Diff of the two buildlogs: -- --- b1/build.log 2021-10-05 16:25:01.699551022 +0000 +++ b2/build.log 2021-10-05 16:36:56.571307821 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Mon Nov 7 10:42:34 -12 2022 -I: pbuilder-time-stamp: 1667860954 +I: Current time: Wed Oct 6 06:25:04 +14 2021 +I: pbuilder-time-stamp: 1633451104 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration @@ -17,8 +17,8 @@ I: copying [./consul_1.8.7+dfsg1-2.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' -gpgv: keyblock resource '/tmp/dpkg-verify-sig.GUPzWpqk/trustedkeys.kbx': General error -gpgv: Signature made Fri Apr 23 23:25:37 2021 -12 +gpgv: keyblock resource '/tmp/dpkg-verify-sig.Ppcw7DXo/trustedkeys.kbx': General error +gpgv: Signature made Sun Apr 25 01:25:37 2021 +14 gpgv: using RSA key C5A5B9DDC33D93FBB63D67C83287D89A97CDA87B gpgv: issuer "vvidic@debian.org" gpgv: Can't check signature: No public key @@ -32,137 +32,171 @@ dpkg-source: info: applying CVE-2020-25864.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/11550/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/D01_modify_environment starting +debug: Running on ionos12-i386. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +Removing 'diversion of /bin/sh to /bin/sh.distrib by dash' +Adding 'diversion of /bin/sh to /bin/sh.distrib by bash' +Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash' +Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash' +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='i386' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=16' - DISTRIBUTION='' - HOME='/root' - HOST_ARCH='i386' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="i686-pc-linux-gnu") + BASH_VERSION='5.1.4(1)-release' + BUILDDIR=/build + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=i386 + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=7' + DIRSTACK=() + DISTRIBUTION= + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=i686 + HOST_ARCH=i386 IFS=' ' - INVOCATION_ID='6f0cc3ddf84147ef8f8875c24a5bbe4c' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - LD_LIBRARY_PATH='/usr/lib/libeatmydata' - LD_PRELOAD='libeatmydata.so' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='11550' - PS1='# ' - PS2='> ' + INVOCATION_ID=874d6c6b50484199b13a1826fff9a81d + LANG=C + LANGUAGE=de_CH:de + LC_ALL=C + LD_LIBRARY_PATH=/usr/lib/libeatmydata + LD_PRELOAD=libeatmydata.so + MACHTYPE=i686-pc-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=12155 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.W8fNMbI5Ts/pbuilderrc_Mfgu --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.W8fNMbI5Ts/b1 --logfile b1/build.log consul_1.8.7+dfsg1-2.dsc' - SUDO_GID='112' - SUDO_UID='107' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://85.184.249.68:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.W8fNMbI5Ts/pbuilderrc_f4oO --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.W8fNMbI5Ts/b2 --logfile b2/build.log consul_1.8.7+dfsg1-2.dsc' + SUDO_GID=112 + SUDO_UID=107 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://78.137.99.97:3128 I: uname -a - Linux ionos16-i386 5.10.0-8-amd64 #1 SMP Debian 5.10.46-5 (2021-09-23) x86_64 GNU/Linux + Linux i-capture-the-hostname 5.10.0-8-686-pae #1 SMP Debian 5.10.46-5 (2021-09-23) i686 GNU/Linux I: ls -l /bin total 5776 - -rwxr-xr-x 1 root root 1367848 Aug 4 2021 bash - -rwxr-xr-x 3 root root 38280 Jul 20 2020 bunzip2 - -rwxr-xr-x 3 root root 38280 Jul 20 2020 bzcat - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff - -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep - -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep - -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep - -rwxr-xr-x 3 root root 38280 Jul 20 2020 bzip2 - -rwxr-xr-x 1 root root 17768 Jul 20 2020 bzip2recover - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore - -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore - -rwxr-xr-x 1 root root 38824 Sep 22 2020 cat - -rwxr-xr-x 1 root root 71624 Sep 22 2020 chgrp - -rwxr-xr-x 1 root root 67528 Sep 22 2020 chmod - -rwxr-xr-x 1 root root 75752 Sep 22 2020 chown - -rwxr-xr-x 1 root root 157960 Sep 22 2020 cp - -rwxr-xr-x 1 root root 128724 Dec 10 2020 dash - -rwxr-xr-x 1 root root 124904 Sep 22 2020 date - -rwxr-xr-x 1 root root 92172 Sep 22 2020 dd - -rwxr-xr-x 1 root root 100752 Sep 22 2020 df - -rwxr-xr-x 1 root root 153964 Sep 22 2020 dir - -rwxr-xr-x 1 root root 83644 Jul 28 2021 dmesg - lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname - lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname - -rwxr-xr-x 1 root root 34664 Sep 22 2020 echo - -rwxr-xr-x 1 root root 28 Nov 9 2020 egrep - -rwxr-xr-x 1 root root 34664 Sep 22 2020 false - -rwxr-xr-x 1 root root 28 Nov 9 2020 fgrep - -rwxr-xr-x 1 root root 71928 Jul 28 2021 findmnt - -rwsr-xr-x 1 root root 30112 Feb 26 2021 fusermount - -rwxr-xr-x 1 root root 210488 Nov 9 2020 grep - -rwxr-xr-x 2 root root 2346 Mar 2 2021 gunzip - -rwxr-xr-x 1 root root 6376 Mar 2 2021 gzexe - -rwxr-xr-x 1 root root 100952 Mar 2 2021 gzip - -rwxr-xr-x 1 root root 21916 Nov 6 2019 hostname - -rwxr-xr-x 1 root root 83980 Sep 22 2020 ln - -rwxr-xr-x 1 root root 55572 Feb 7 2020 login - -rwxr-xr-x 1 root root 153964 Sep 22 2020 ls - -rwxr-xr-x 1 root root 153124 Jul 28 2021 lsblk - -rwxr-xr-x 1 root root 96328 Sep 22 2020 mkdir - -rwxr-xr-x 1 root root 79912 Sep 22 2020 mknod - -rwxr-xr-x 1 root root 47048 Sep 22 2020 mktemp - -rwxr-xr-x 1 root root 58920 Jul 28 2021 more - -rwsr-xr-x 1 root root 50720 Jul 28 2021 mount - -rwxr-xr-x 1 root root 13856 Jul 28 2021 mountpoint - -rwxr-xr-x 1 root root 157996 Sep 22 2020 mv - lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname - lrwxrwxrwx 1 root root 14 Apr 18 2021 pidof -> /sbin/killall5 - -rwxr-xr-x 1 root root 38824 Sep 22 2020 pwd - lrwxrwxrwx 1 root root 4 Aug 4 2021 rbash -> bash - -rwxr-xr-x 1 root root 46984 Sep 22 2020 readlink - -rwxr-xr-x 1 root root 75720 Sep 22 2020 rm - -rwxr-xr-x 1 root root 46984 Sep 22 2020 rmdir - -rwxr-xr-x 1 root root 22292 Sep 27 2020 run-parts - -rwxr-xr-x 1 root root 125036 Dec 22 2018 sed - lrwxrwxrwx 1 root root 4 Oct 24 03:46 sh -> dash - -rwxr-xr-x 1 root root 34696 Sep 22 2020 sleep - -rwxr-xr-x 1 root root 83880 Sep 22 2020 stty - -rwsr-xr-x 1 root root 79396 Jul 28 2021 su - -rwxr-xr-x 1 root root 34696 Sep 22 2020 sync - -rwxr-xr-x 1 root root 602584 Feb 16 2021 tar - -rwxr-xr-x 1 root root 13860 Sep 27 2020 tempfile - -rwxr-xr-x 1 root root 108520 Sep 22 2020 touch - -rwxr-xr-x 1 root root 34664 Sep 22 2020 true - -rwxr-xr-x 1 root root 17768 Feb 26 2021 ulockmgr_server - -rwsr-xr-x 1 root root 30236 Jul 28 2021 umount - -rwxr-xr-x 1 root root 34664 Sep 22 2020 uname - -rwxr-xr-x 2 root root 2346 Mar 2 2021 uncompress - -rwxr-xr-x 1 root root 153964 Sep 22 2020 vdir - -rwxr-xr-x 1 root root 63024 Jul 28 2021 wdctl - lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname - -rwxr-xr-x 1 root root 1984 Mar 2 2021 zcat - -rwxr-xr-x 1 root root 1678 Mar 2 2021 zcmp - -rwxr-xr-x 1 root root 5880 Mar 2 2021 zdiff - -rwxr-xr-x 1 root root 29 Mar 2 2021 zegrep - -rwxr-xr-x 1 root root 29 Mar 2 2021 zfgrep - -rwxr-xr-x 1 root root 2081 Mar 2 2021 zforce - -rwxr-xr-x 1 root root 7585 Mar 2 2021 zgrep - -rwxr-xr-x 1 root root 2206 Mar 2 2021 zless - -rwxr-xr-x 1 root root 1842 Mar 2 2021 zmore - -rwxr-xr-x 1 root root 4553 Mar 2 2021 znew -I: user script /srv/workspace/pbuilder/11550/tmp/hooks/D02_print_environment finished + -rwxr-xr-x 1 root root 1367848 Aug 5 10:25 bash + -rwxr-xr-x 3 root root 38280 Jul 21 2020 bunzip2 + -rwxr-xr-x 3 root root 38280 Jul 21 2020 bzcat + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzcmp -> bzdiff + -rwxr-xr-x 1 root root 2225 Jul 21 2020 bzdiff + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzegrep -> bzgrep + -rwxr-xr-x 1 root root 4877 Sep 5 2019 bzexe + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzfgrep -> bzgrep + -rwxr-xr-x 1 root root 3775 Jul 21 2020 bzgrep + -rwxr-xr-x 3 root root 38280 Jul 21 2020 bzip2 + -rwxr-xr-x 1 root root 17768 Jul 21 2020 bzip2recover + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzless -> bzmore + -rwxr-xr-x 1 root root 1297 Jul 21 2020 bzmore + -rwxr-xr-x 1 root root 38824 Sep 23 2020 cat + -rwxr-xr-x 1 root root 71624 Sep 23 2020 chgrp + -rwxr-xr-x 1 root root 67528 Sep 23 2020 chmod + -rwxr-xr-x 1 root root 75752 Sep 23 2020 chown + -rwxr-xr-x 1 root root 157960 Sep 23 2020 cp + -rwxr-xr-x 1 root root 128724 Dec 11 2020 dash + -rwxr-xr-x 1 root root 124904 Sep 23 2020 date + -rwxr-xr-x 1 root root 92172 Sep 23 2020 dd + -rwxr-xr-x 1 root root 100752 Sep 23 2020 df + -rwxr-xr-x 1 root root 153964 Sep 23 2020 dir + -rwxr-xr-x 1 root root 83644 Jul 29 09:09 dmesg + lrwxrwxrwx 1 root root 8 Nov 8 2019 dnsdomainname -> hostname + lrwxrwxrwx 1 root root 8 Nov 8 2019 domainname -> hostname + -rwxr-xr-x 1 root root 34664 Sep 23 2020 echo + -rwxr-xr-x 1 root root 28 Nov 10 2020 egrep + -rwxr-xr-x 1 root root 34664 Sep 23 2020 false + -rwxr-xr-x 1 root root 28 Nov 10 2020 fgrep + -rwxr-xr-x 1 root root 71928 Jul 29 09:09 findmnt + -rwsr-xr-x 1 root root 30112 Feb 27 2021 fusermount + -rwxr-xr-x 1 root root 210488 Nov 10 2020 grep + -rwxr-xr-x 2 root root 2346 Mar 3 2021 gunzip + -rwxr-xr-x 1 root root 6376 Mar 3 2021 gzexe + -rwxr-xr-x 1 root root 100952 Mar 3 2021 gzip + -rwxr-xr-x 1 root root 21916 Nov 8 2019 hostname + -rwxr-xr-x 1 root root 83980 Sep 23 2020 ln + -rwxr-xr-x 1 root root 55572 Feb 8 2020 login + -rwxr-xr-x 1 root root 153964 Sep 23 2020 ls + -rwxr-xr-x 1 root root 153124 Jul 29 09:09 lsblk + -rwxr-xr-x 1 root root 96328 Sep 23 2020 mkdir + -rwxr-xr-x 1 root root 79912 Sep 23 2020 mknod + -rwxr-xr-x 1 root root 47048 Sep 23 2020 mktemp + -rwxr-xr-x 1 root root 58920 Jul 29 09:09 more + -rwsr-xr-x 1 root root 50720 Jul 29 09:09 mount + -rwxr-xr-x 1 root root 13856 Jul 29 09:09 mountpoint + -rwxr-xr-x 1 root root 157996 Sep 23 2020 mv + lrwxrwxrwx 1 root root 8 Nov 8 2019 nisdomainname -> hostname + lrwxrwxrwx 1 root root 14 Apr 19 05:38 pidof -> /sbin/killall5 + -rwxr-xr-x 1 root root 38824 Sep 23 2020 pwd + lrwxrwxrwx 1 root root 4 Aug 5 10:25 rbash -> bash + -rwxr-xr-x 1 root root 46984 Sep 23 2020 readlink + -rwxr-xr-x 1 root root 75720 Sep 23 2020 rm + -rwxr-xr-x 1 root root 46984 Sep 23 2020 rmdir + -rwxr-xr-x 1 root root 22292 Sep 28 2020 run-parts + -rwxr-xr-x 1 root root 125036 Dec 23 2018 sed + lrwxrwxrwx 1 root root 4 Oct 6 06:25 sh -> bash + lrwxrwxrwx 1 root root 4 Sep 21 23:26 sh.distrib -> dash + -rwxr-xr-x 1 root root 34696 Sep 23 2020 sleep + -rwxr-xr-x 1 root root 83880 Sep 23 2020 stty + -rwsr-xr-x 1 root root 79396 Jul 29 09:09 su + -rwxr-xr-x 1 root root 34696 Sep 23 2020 sync + -rwxr-xr-x 1 root root 602584 Feb 17 2021 tar + -rwxr-xr-x 1 root root 13860 Sep 28 2020 tempfile + -rwxr-xr-x 1 root root 108520 Sep 23 2020 touch + -rwxr-xr-x 1 root root 34664 Sep 23 2020 true + -rwxr-xr-x 1 root root 17768 Feb 27 2021 ulockmgr_server + -rwsr-xr-x 1 root root 30236 Jul 29 09:09 umount + -rwxr-xr-x 1 root root 34664 Sep 23 2020 uname + -rwxr-xr-x 2 root root 2346 Mar 3 2021 uncompress + -rwxr-xr-x 1 root root 153964 Sep 23 2020 vdir + -rwxr-xr-x 1 root root 63024 Jul 29 09:09 wdctl + lrwxrwxrwx 1 root root 8 Nov 8 2019 ypdomainname -> hostname + -rwxr-xr-x 1 root root 1984 Mar 3 2021 zcat + -rwxr-xr-x 1 root root 1678 Mar 3 2021 zcmp + -rwxr-xr-x 1 root root 5880 Mar 3 2021 zdiff + -rwxr-xr-x 1 root root 29 Mar 3 2021 zegrep + -rwxr-xr-x 1 root root 29 Mar 3 2021 zfgrep + -rwxr-xr-x 1 root root 2081 Mar 3 2021 zforce + -rwxr-xr-x 1 root root 7585 Mar 3 2021 zgrep + -rwxr-xr-x 1 root root 2206 Mar 3 2021 zless + -rwxr-xr-x 1 root root 1842 Mar 3 2021 zmore + -rwxr-xr-x 1 root root 4553 Mar 3 2021 znew +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -562,7 +596,7 @@ Get: 210 http://deb.debian.org/debian bullseye/main i386 golang-k8s-sigs-yaml-dev all 1.2.0-3 [15.0 kB] Get: 211 http://deb.debian.org/debian bullseye/main i386 mockery i386 0.0~git20181123.e78b021-3+b6 [1498 kB] Get: 212 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-rootcerts-dev all 1.0.2-2 [8356 B] -Fetched 126 MB in 5s (26.6 MB/s) +Fetched 126 MB in 3s (44.6 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19675 files and directories currently installed.) @@ -1439,7 +1473,8 @@ fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package -I: Running cd /build/consul-1.8.7+dfsg1/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../consul_1.8.7+dfsg1-2_source.changes +hostname: Name or service not known +I: Running cd /build/consul-1.8.7+dfsg1/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../consul_1.8.7+dfsg1-2_source.changes dpkg-buildpackage: info: source package consul dpkg-buildpackage: info: source version 1.8.7+dfsg1-2 dpkg-buildpackage: info: source distribution unstable @@ -1456,65 +1491,65 @@ ## Remove Files-Excluded (when built from checkout or non-DFSG tarball): rm -f -rv `perl -0nE 'say $1 if m{^Files\-Excluded\:\s*(.*?)(?:\n\n|Files:|Comment:)}sm;' debian/copyright` find vendor -type d -empty -delete -print -vendor/gopkg.in -vendor/github.com/beorn7 -vendor/github.com/nicolai86 vendor/github.com/davecgh -vendor/github.com/DataDog -vendor/github.com/denverdino -vendor/github.com/packethost -vendor/github.com/circonus-labs -vendor/github.com/posener +vendor/github.com/digitalocean +vendor/github.com/docker +vendor/github.com/Azure vendor/github.com/joyent -vendor/github.com/googleapis -vendor/github.com/coreos -vendor/github.com/elazarl -vendor/github.com/bgentry -vendor/github.com/shirou -vendor/github.com/boltdb -vendor/github.com/dgrijalva -vendor/github.com/armon -vendor/github.com/matttproud -vendor/github.com/NYTimes +vendor/github.com/spf13 +vendor/github.com/json-iterator vendor/github.com/gophercloud -vendor/github.com/jmespath -vendor/github.com/go-ole vendor/github.com/pierrec -vendor/github.com/tv42 -vendor/github.com/pascaldekloe -vendor/github.com/spf13 -vendor/github.com/cespare -vendor/github.com/fatih -vendor/github.com/sirupsen -vendor/github.com/Azure -vendor/github.com/renier -vendor/github.com/prometheus -vendor/github.com/census-instrumentation -vendor/github.com/stretchr -vendor/github.com/golang +vendor/github.com/coreos +vendor/github.com/dgrijalva vendor/github.com/patrickmn -vendor/github.com/miekg -vendor/github.com/json-iterator -vendor/github.com/vmware +vendor/github.com/shirou vendor/github.com/gogo +vendor/github.com/StackExchange +vendor/github.com/imdario +vendor/github.com/modern-go +vendor/github.com/golang +vendor/github.com/armon vendor/github.com/pquerna +vendor/github.com/mattn +vendor/github.com/Microsoft +vendor/github.com/prometheus +vendor/github.com/boltdb +vendor/github.com/DataDog +vendor/github.com/renier +vendor/github.com/cespare +vendor/github.com/ryanuber +vendor/github.com/stretchr +vendor/github.com/nicolai86 +vendor/github.com/sean- vendor/github.com/konsorten vendor/github.com/pmezard -vendor/github.com/mattn -vendor/github.com/docker +vendor/github.com/beorn7 +vendor/github.com/sirupsen +vendor/github.com/census-instrumentation +vendor/github.com/pascaldekloe +vendor/github.com/miekg vendor/github.com/pkg -vendor/github.com/digitalocean +vendor/github.com/googleapis +vendor/github.com/softlayer +vendor/github.com/denverdino +vendor/github.com/vmware +vendor/github.com/jmespath +vendor/github.com/fatih +vendor/github.com/matttproud +vendor/github.com/packethost vendor/github.com/dimchansky -vendor/github.com/imdario -vendor/github.com/StackExchange -vendor/github.com/sean- +vendor/github.com/circonus-labs +vendor/github.com/bgentry vendor/github.com/aws -vendor/github.com/softlayer -vendor/github.com/modern-go -vendor/github.com/ryanuber vendor/github.com/kr -vendor/github.com/Microsoft +vendor/github.com/posener +vendor/github.com/go-ole +vendor/github.com/tv42 +vendor/github.com/elazarl +vendor/github.com/NYTimes vendor/golang.org +vendor/gopkg.in rm -f -r test/integration make[1]: Leaving directory '/build/consul-1.8.7+dfsg1' debian/rules build @@ -1571,7 +1606,7 @@ CGO_FFLAGS="-g -O2 -fdebug-prefix-map=/build/consul-1.8.7+dfsg1=. -fstack-protector-strong" CGO_LDFLAGS="-Wl,-z,relro" PKG_CONFIG="pkg-config" -GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build555119365=/tmp/go-build -gno-record-gcc-switches" +GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build285295994=/tmp/go-build -gno-record-gcc-switches" dh_auto_build: warning: github.com/hashicorp/consul/agent/routine-leak-checker contains no non-test Go files, removing it from build cd _build && go generate -v github.com/hashicorp/consul github.com/hashicorp/consul/acl github.com/hashicorp/consul/agent github.com/hashicorp/consul/agent/ae github.com/hashicorp/consul/agent/auto-config github.com/hashicorp/consul/agent/cache github.com/hashicorp/consul/agent/cache-types github.com/hashicorp/consul/agent/checks github.com/hashicorp/consul/agent/config github.com/hashicorp/consul/agent/connect github.com/hashicorp/consul/agent/connect/ca github.com/hashicorp/consul/agent/consul github.com/hashicorp/consul/agent/consul/authmethod github.com/hashicorp/consul/agent/consul/authmethod/kubeauth github.com/hashicorp/consul/agent/consul/authmethod/ssoauth github.com/hashicorp/consul/agent/consul/authmethod/testauth github.com/hashicorp/consul/agent/consul/autopilot github.com/hashicorp/consul/agent/consul/discoverychain github.com/hashicorp/consul/agent/consul/fsm github.com/hashicorp/consul/agent/consul/prepared_query github.com/hashicorp/consul/agent/consul/state github.com/hashicorp/consul/agent/consul/wanfed github.com/hashicorp/consul/agent/debug github.com/hashicorp/consul/agent/dns github.com/hashicorp/consul/agent/exec github.com/hashicorp/consul/agent/local github.com/hashicorp/consul/agent/metadata github.com/hashicorp/consul/agent/mock github.com/hashicorp/consul/agent/pool github.com/hashicorp/consul/agent/proxycfg github.com/hashicorp/consul/agent/router github.com/hashicorp/consul/agent/structs github.com/hashicorp/consul/agent/systemd github.com/hashicorp/consul/agent/token github.com/hashicorp/consul/agent/xds github.com/hashicorp/consul/agent/xds/proxysupport github.com/hashicorp/consul/api github.com/hashicorp/consul/api/watch github.com/hashicorp/consul/command github.com/hashicorp/consul/command/acl github.com/hashicorp/consul/command/acl/agenttokens github.com/hashicorp/consul/command/acl/authmethod github.com/hashicorp/consul/command/acl/authmethod/create github.com/hashicorp/consul/command/acl/authmethod/delete github.com/hashicorp/consul/command/acl/authmethod/list github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/update github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/bindingrule/create github.com/hashicorp/consul/command/acl/bindingrule/delete github.com/hashicorp/consul/command/acl/bindingrule/list github.com/hashicorp/consul/command/acl/bindingrule/read github.com/hashicorp/consul/command/acl/bindingrule/update github.com/hashicorp/consul/command/acl/bootstrap github.com/hashicorp/consul/command/acl/policy github.com/hashicorp/consul/command/acl/policy/create github.com/hashicorp/consul/command/acl/policy/delete github.com/hashicorp/consul/command/acl/policy/list github.com/hashicorp/consul/command/acl/policy/read github.com/hashicorp/consul/command/acl/policy/update github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/role/create github.com/hashicorp/consul/command/acl/role/delete github.com/hashicorp/consul/command/acl/role/list github.com/hashicorp/consul/command/acl/role/read github.com/hashicorp/consul/command/acl/role/update github.com/hashicorp/consul/command/acl/rules github.com/hashicorp/consul/command/acl/token github.com/hashicorp/consul/command/acl/token/clone github.com/hashicorp/consul/command/acl/token/create github.com/hashicorp/consul/command/acl/token/delete github.com/hashicorp/consul/command/acl/token/list github.com/hashicorp/consul/command/acl/token/read github.com/hashicorp/consul/command/acl/token/update github.com/hashicorp/consul/command/agent github.com/hashicorp/consul/command/catalog github.com/hashicorp/consul/command/catalog/list/dc github.com/hashicorp/consul/command/catalog/list/nodes github.com/hashicorp/consul/command/catalog/list/services github.com/hashicorp/consul/command/config github.com/hashicorp/consul/command/config/delete github.com/hashicorp/consul/command/config/list github.com/hashicorp/consul/command/config/read github.com/hashicorp/consul/command/config/write github.com/hashicorp/consul/command/connect github.com/hashicorp/consul/command/connect/ca github.com/hashicorp/consul/command/connect/ca/get github.com/hashicorp/consul/command/connect/ca/set github.com/hashicorp/consul/command/connect/envoy github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap github.com/hashicorp/consul/command/connect/expose github.com/hashicorp/consul/command/connect/proxy github.com/hashicorp/consul/command/debug github.com/hashicorp/consul/command/event github.com/hashicorp/consul/command/exec github.com/hashicorp/consul/command/flags github.com/hashicorp/consul/command/forceleave github.com/hashicorp/consul/command/helpers github.com/hashicorp/consul/command/info github.com/hashicorp/consul/command/intention github.com/hashicorp/consul/command/intention/check github.com/hashicorp/consul/command/intention/create github.com/hashicorp/consul/command/intention/delete github.com/hashicorp/consul/command/intention/finder github.com/hashicorp/consul/command/intention/get github.com/hashicorp/consul/command/intention/match github.com/hashicorp/consul/command/join github.com/hashicorp/consul/command/keygen github.com/hashicorp/consul/command/keyring github.com/hashicorp/consul/command/kv github.com/hashicorp/consul/command/kv/del github.com/hashicorp/consul/command/kv/exp github.com/hashicorp/consul/command/kv/get github.com/hashicorp/consul/command/kv/imp github.com/hashicorp/consul/command/kv/impexp github.com/hashicorp/consul/command/kv/put github.com/hashicorp/consul/command/leave github.com/hashicorp/consul/command/lock github.com/hashicorp/consul/command/login github.com/hashicorp/consul/command/logout github.com/hashicorp/consul/command/maint github.com/hashicorp/consul/command/members github.com/hashicorp/consul/command/monitor github.com/hashicorp/consul/command/operator github.com/hashicorp/consul/command/operator/autopilot github.com/hashicorp/consul/command/operator/autopilot/get github.com/hashicorp/consul/command/operator/autopilot/set github.com/hashicorp/consul/command/operator/raft github.com/hashicorp/consul/command/operator/raft/listpeers github.com/hashicorp/consul/command/operator/raft/removepeer github.com/hashicorp/consul/command/reload github.com/hashicorp/consul/command/rtt github.com/hashicorp/consul/command/services github.com/hashicorp/consul/command/services/deregister github.com/hashicorp/consul/command/services/register github.com/hashicorp/consul/command/snapshot github.com/hashicorp/consul/command/snapshot/inspect github.com/hashicorp/consul/command/snapshot/restore github.com/hashicorp/consul/command/snapshot/save github.com/hashicorp/consul/command/tls github.com/hashicorp/consul/command/tls/ca github.com/hashicorp/consul/command/tls/ca/create github.com/hashicorp/consul/command/tls/cert github.com/hashicorp/consul/command/tls/cert/create github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/command/version github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/connect github.com/hashicorp/consul/connect/certgen github.com/hashicorp/consul/connect/proxy github.com/hashicorp/consul/internal/go-sso/oidcauth github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest github.com/hashicorp/consul/ipaddr github.com/hashicorp/consul/lib github.com/hashicorp/consul/lib/decode github.com/hashicorp/consul/lib/file github.com/hashicorp/consul/lib/semaphore github.com/hashicorp/consul/lib/stringslice github.com/hashicorp/consul/lib/template github.com/hashicorp/consul/logging github.com/hashicorp/consul/logging/monitor github.com/hashicorp/consul/proto github.com/hashicorp/consul/proto/pbacl github.com/hashicorp/consul/proto/pbautoconf github.com/hashicorp/consul/proto/pbcommon github.com/hashicorp/consul/proto/pbconfig github.com/hashicorp/consul/proto/pbconnect github.com/hashicorp/consul/sdk/freeport github.com/hashicorp/consul/sdk/testutil github.com/hashicorp/consul/sdk/testutil/retry github.com/hashicorp/consul/sentinel github.com/hashicorp/consul/service_os github.com/hashicorp/consul/snapshot github.com/hashicorp/consul/testrpc github.com/hashicorp/consul/tlsutil github.com/hashicorp/consul/types github.com/hashicorp/consul/version src/github.com/hashicorp/consul/main.go @@ -2572,753 +2607,694 @@ src/github.com/hashicorp/consul/types/checks.go src/github.com/hashicorp/consul/types/node_id.go src/github.com/hashicorp/consul/version/version.go - cd _build && go install -trimpath -v -p 16 github.com/hashicorp/consul github.com/hashicorp/consul/acl github.com/hashicorp/consul/agent github.com/hashicorp/consul/agent/ae github.com/hashicorp/consul/agent/auto-config github.com/hashicorp/consul/agent/cache github.com/hashicorp/consul/agent/cache-types github.com/hashicorp/consul/agent/checks github.com/hashicorp/consul/agent/config github.com/hashicorp/consul/agent/connect github.com/hashicorp/consul/agent/connect/ca github.com/hashicorp/consul/agent/consul github.com/hashicorp/consul/agent/consul/authmethod github.com/hashicorp/consul/agent/consul/authmethod/kubeauth github.com/hashicorp/consul/agent/consul/authmethod/ssoauth github.com/hashicorp/consul/agent/consul/authmethod/testauth github.com/hashicorp/consul/agent/consul/autopilot github.com/hashicorp/consul/agent/consul/discoverychain github.com/hashicorp/consul/agent/consul/fsm github.com/hashicorp/consul/agent/consul/prepared_query github.com/hashicorp/consul/agent/consul/state github.com/hashicorp/consul/agent/consul/wanfed github.com/hashicorp/consul/agent/debug github.com/hashicorp/consul/agent/dns github.com/hashicorp/consul/agent/exec github.com/hashicorp/consul/agent/local github.com/hashicorp/consul/agent/metadata github.com/hashicorp/consul/agent/mock github.com/hashicorp/consul/agent/pool github.com/hashicorp/consul/agent/proxycfg github.com/hashicorp/consul/agent/router github.com/hashicorp/consul/agent/structs github.com/hashicorp/consul/agent/systemd github.com/hashicorp/consul/agent/token github.com/hashicorp/consul/agent/xds github.com/hashicorp/consul/agent/xds/proxysupport github.com/hashicorp/consul/api github.com/hashicorp/consul/api/watch github.com/hashicorp/consul/command github.com/hashicorp/consul/command/acl github.com/hashicorp/consul/command/acl/agenttokens github.com/hashicorp/consul/command/acl/authmethod github.com/hashicorp/consul/command/acl/authmethod/create github.com/hashicorp/consul/command/acl/authmethod/delete github.com/hashicorp/consul/command/acl/authmethod/list github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/update github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/bindingrule/create github.com/hashicorp/consul/command/acl/bindingrule/delete github.com/hashicorp/consul/command/acl/bindingrule/list github.com/hashicorp/consul/command/acl/bindingrule/read github.com/hashicorp/consul/command/acl/bindingrule/update github.com/hashicorp/consul/command/acl/bootstrap github.com/hashicorp/consul/command/acl/policy github.com/hashicorp/consul/command/acl/policy/create github.com/hashicorp/consul/command/acl/policy/delete github.com/hashicorp/consul/command/acl/policy/list github.com/hashicorp/consul/command/acl/policy/read github.com/hashicorp/consul/command/acl/policy/update github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/role/create github.com/hashicorp/consul/command/acl/role/delete github.com/hashicorp/consul/command/acl/role/list github.com/hashicorp/consul/command/acl/role/read github.com/hashicorp/consul/command/acl/role/update github.com/hashicorp/consul/command/acl/rules github.com/hashicorp/consul/command/acl/token github.com/hashicorp/consul/command/acl/token/clone github.com/hashicorp/consul/command/acl/token/create github.com/hashicorp/consul/command/acl/token/delete github.com/hashicorp/consul/command/acl/token/list github.com/hashicorp/consul/command/acl/token/read github.com/hashicorp/consul/command/acl/token/update github.com/hashicorp/consul/command/agent github.com/hashicorp/consul/command/catalog github.com/hashicorp/consul/command/catalog/list/dc github.com/hashicorp/consul/command/catalog/list/nodes github.com/hashicorp/consul/command/catalog/list/services github.com/hashicorp/consul/command/config github.com/hashicorp/consul/command/config/delete github.com/hashicorp/consul/command/config/list github.com/hashicorp/consul/command/config/read github.com/hashicorp/consul/command/config/write github.com/hashicorp/consul/command/connect github.com/hashicorp/consul/command/connect/ca github.com/hashicorp/consul/command/connect/ca/get github.com/hashicorp/consul/command/connect/ca/set github.com/hashicorp/consul/command/connect/envoy github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap github.com/hashicorp/consul/command/connect/expose github.com/hashicorp/consul/command/connect/proxy github.com/hashicorp/consul/command/debug github.com/hashicorp/consul/command/event github.com/hashicorp/consul/command/exec github.com/hashicorp/consul/command/flags github.com/hashicorp/consul/command/forceleave github.com/hashicorp/consul/command/helpers github.com/hashicorp/consul/command/info github.com/hashicorp/consul/command/intention github.com/hashicorp/consul/command/intention/check github.com/hashicorp/consul/command/intention/create github.com/hashicorp/consul/command/intention/delete github.com/hashicorp/consul/command/intention/finder github.com/hashicorp/consul/command/intention/get github.com/hashicorp/consul/command/intention/match github.com/hashicorp/consul/command/join github.com/hashicorp/consul/command/keygen github.com/hashicorp/consul/command/keyring github.com/hashicorp/consul/command/kv github.com/hashicorp/consul/command/kv/del github.com/hashicorp/consul/command/kv/exp github.com/hashicorp/consul/command/kv/get github.com/hashicorp/consul/command/kv/imp github.com/hashicorp/consul/command/kv/impexp github.com/hashicorp/consul/command/kv/put github.com/hashicorp/consul/command/leave github.com/hashicorp/consul/command/lock github.com/hashicorp/consul/command/login github.com/hashicorp/consul/command/logout github.com/hashicorp/consul/command/maint github.com/hashicorp/consul/command/members github.com/hashicorp/consul/command/monitor github.com/hashicorp/consul/command/operator github.com/hashicorp/consul/command/operator/autopilot github.com/hashicorp/consul/command/operator/autopilot/get github.com/hashicorp/consul/command/operator/autopilot/set github.com/hashicorp/consul/command/operator/raft github.com/hashicorp/consul/command/operator/raft/listpeers github.com/hashicorp/consul/command/operator/raft/removepeer github.com/hashicorp/consul/command/reload github.com/hashicorp/consul/command/rtt github.com/hashicorp/consul/command/services github.com/hashicorp/consul/command/services/deregister github.com/hashicorp/consul/command/services/register github.com/hashicorp/consul/command/snapshot github.com/hashicorp/consul/command/snapshot/inspect github.com/hashicorp/consul/command/snapshot/restore github.com/hashicorp/consul/command/snapshot/save github.com/hashicorp/consul/command/tls github.com/hashicorp/consul/command/tls/ca github.com/hashicorp/consul/command/tls/ca/create github.com/hashicorp/consul/command/tls/cert github.com/hashicorp/consul/command/tls/cert/create github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/command/version github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/connect github.com/hashicorp/consul/connect/certgen github.com/hashicorp/consul/connect/proxy github.com/hashicorp/consul/internal/go-sso/oidcauth github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest github.com/hashicorp/consul/ipaddr github.com/hashicorp/consul/lib github.com/hashicorp/consul/lib/decode github.com/hashicorp/consul/lib/file github.com/hashicorp/consul/lib/semaphore github.com/hashicorp/consul/lib/stringslice github.com/hashicorp/consul/lib/template github.com/hashicorp/consul/logging github.com/hashicorp/consul/logging/monitor github.com/hashicorp/consul/proto github.com/hashicorp/consul/proto/pbacl github.com/hashicorp/consul/proto/pbautoconf github.com/hashicorp/consul/proto/pbcommon github.com/hashicorp/consul/proto/pbconfig github.com/hashicorp/consul/proto/pbconnect github.com/hashicorp/consul/sdk/freeport github.com/hashicorp/consul/sdk/testutil github.com/hashicorp/consul/sdk/testutil/retry github.com/hashicorp/consul/sentinel github.com/hashicorp/consul/service_os github.com/hashicorp/consul/snapshot github.com/hashicorp/consul/testrpc github.com/hashicorp/consul/tlsutil github.com/hashicorp/consul/types github.com/hashicorp/consul/version + cd _build && go install -trimpath -v -p 7 github.com/hashicorp/consul github.com/hashicorp/consul/acl github.com/hashicorp/consul/agent github.com/hashicorp/consul/agent/ae github.com/hashicorp/consul/agent/auto-config github.com/hashicorp/consul/agent/cache github.com/hashicorp/consul/agent/cache-types github.com/hashicorp/consul/agent/checks github.com/hashicorp/consul/agent/config github.com/hashicorp/consul/agent/connect github.com/hashicorp/consul/agent/connect/ca github.com/hashicorp/consul/agent/consul github.com/hashicorp/consul/agent/consul/authmethod github.com/hashicorp/consul/agent/consul/authmethod/kubeauth github.com/hashicorp/consul/agent/consul/authmethod/ssoauth github.com/hashicorp/consul/agent/consul/authmethod/testauth github.com/hashicorp/consul/agent/consul/autopilot github.com/hashicorp/consul/agent/consul/discoverychain github.com/hashicorp/consul/agent/consul/fsm github.com/hashicorp/consul/agent/consul/prepared_query github.com/hashicorp/consul/agent/consul/state github.com/hashicorp/consul/agent/consul/wanfed github.com/hashicorp/consul/agent/debug github.com/hashicorp/consul/agent/dns github.com/hashicorp/consul/agent/exec github.com/hashicorp/consul/agent/local github.com/hashicorp/consul/agent/metadata github.com/hashicorp/consul/agent/mock github.com/hashicorp/consul/agent/pool github.com/hashicorp/consul/agent/proxycfg github.com/hashicorp/consul/agent/router github.com/hashicorp/consul/agent/structs github.com/hashicorp/consul/agent/systemd github.com/hashicorp/consul/agent/token github.com/hashicorp/consul/agent/xds github.com/hashicorp/consul/agent/xds/proxysupport github.com/hashicorp/consul/api github.com/hashicorp/consul/api/watch github.com/hashicorp/consul/command github.com/hashicorp/consul/command/acl github.com/hashicorp/consul/command/acl/agenttokens github.com/hashicorp/consul/command/acl/authmethod github.com/hashicorp/consul/command/acl/authmethod/create github.com/hashicorp/consul/command/acl/authmethod/delete github.com/hashicorp/consul/command/acl/authmethod/list github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/update github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/bindingrule/create github.com/hashicorp/consul/command/acl/bindingrule/delete github.com/hashicorp/consul/command/acl/bindingrule/list github.com/hashicorp/consul/command/acl/bindingrule/read github.com/hashicorp/consul/command/acl/bindingrule/update github.com/hashicorp/consul/command/acl/bootstrap github.com/hashicorp/consul/command/acl/policy github.com/hashicorp/consul/command/acl/policy/create github.com/hashicorp/consul/command/acl/policy/delete github.com/hashicorp/consul/command/acl/policy/list github.com/hashicorp/consul/command/acl/policy/read github.com/hashicorp/consul/command/acl/policy/update github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/role/create github.com/hashicorp/consul/command/acl/role/delete github.com/hashicorp/consul/command/acl/role/list github.com/hashicorp/consul/command/acl/role/read github.com/hashicorp/consul/command/acl/role/update github.com/hashicorp/consul/command/acl/rules github.com/hashicorp/consul/command/acl/token github.com/hashicorp/consul/command/acl/token/clone github.com/hashicorp/consul/command/acl/token/create github.com/hashicorp/consul/command/acl/token/delete github.com/hashicorp/consul/command/acl/token/list github.com/hashicorp/consul/command/acl/token/read github.com/hashicorp/consul/command/acl/token/update github.com/hashicorp/consul/command/agent github.com/hashicorp/consul/command/catalog github.com/hashicorp/consul/command/catalog/list/dc github.com/hashicorp/consul/command/catalog/list/nodes github.com/hashicorp/consul/command/catalog/list/services github.com/hashicorp/consul/command/config github.com/hashicorp/consul/command/config/delete github.com/hashicorp/consul/command/config/list github.com/hashicorp/consul/command/config/read github.com/hashicorp/consul/command/config/write github.com/hashicorp/consul/command/connect github.com/hashicorp/consul/command/connect/ca github.com/hashicorp/consul/command/connect/ca/get github.com/hashicorp/consul/command/connect/ca/set github.com/hashicorp/consul/command/connect/envoy github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap github.com/hashicorp/consul/command/connect/expose github.com/hashicorp/consul/command/connect/proxy github.com/hashicorp/consul/command/debug github.com/hashicorp/consul/command/event github.com/hashicorp/consul/command/exec github.com/hashicorp/consul/command/flags github.com/hashicorp/consul/command/forceleave github.com/hashicorp/consul/command/helpers github.com/hashicorp/consul/command/info github.com/hashicorp/consul/command/intention github.com/hashicorp/consul/command/intention/check github.com/hashicorp/consul/command/intention/create github.com/hashicorp/consul/command/intention/delete github.com/hashicorp/consul/command/intention/finder github.com/hashicorp/consul/command/intention/get github.com/hashicorp/consul/command/intention/match github.com/hashicorp/consul/command/join github.com/hashicorp/consul/command/keygen github.com/hashicorp/consul/command/keyring github.com/hashicorp/consul/command/kv github.com/hashicorp/consul/command/kv/del github.com/hashicorp/consul/command/kv/exp github.com/hashicorp/consul/command/kv/get github.com/hashicorp/consul/command/kv/imp github.com/hashicorp/consul/command/kv/impexp github.com/hashicorp/consul/command/kv/put github.com/hashicorp/consul/command/leave github.com/hashicorp/consul/command/lock github.com/hashicorp/consul/command/login github.com/hashicorp/consul/command/logout github.com/hashicorp/consul/command/maint github.com/hashicorp/consul/command/members github.com/hashicorp/consul/command/monitor github.com/hashicorp/consul/command/operator github.com/hashicorp/consul/command/operator/autopilot github.com/hashicorp/consul/command/operator/autopilot/get github.com/hashicorp/consul/command/operator/autopilot/set github.com/hashicorp/consul/command/operator/raft github.com/hashicorp/consul/command/operator/raft/listpeers github.com/hashicorp/consul/command/operator/raft/removepeer github.com/hashicorp/consul/command/reload github.com/hashicorp/consul/command/rtt github.com/hashicorp/consul/command/services github.com/hashicorp/consul/command/services/deregister github.com/hashicorp/consul/command/services/register github.com/hashicorp/consul/command/snapshot github.com/hashicorp/consul/command/snapshot/inspect github.com/hashicorp/consul/command/snapshot/restore github.com/hashicorp/consul/command/snapshot/save github.com/hashicorp/consul/command/tls github.com/hashicorp/consul/command/tls/ca github.com/hashicorp/consul/command/tls/ca/create github.com/hashicorp/consul/command/tls/cert github.com/hashicorp/consul/command/tls/cert/create github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/command/version github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/connect github.com/hashicorp/consul/connect/certgen github.com/hashicorp/consul/connect/proxy github.com/hashicorp/consul/internal/go-sso/oidcauth github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest github.com/hashicorp/consul/ipaddr github.com/hashicorp/consul/lib github.com/hashicorp/consul/lib/decode github.com/hashicorp/consul/lib/file github.com/hashicorp/consul/lib/semaphore github.com/hashicorp/consul/lib/stringslice github.com/hashicorp/consul/lib/template github.com/hashicorp/consul/logging github.com/hashicorp/consul/logging/monitor github.com/hashicorp/consul/proto github.com/hashicorp/consul/proto/pbacl github.com/hashicorp/consul/proto/pbautoconf github.com/hashicorp/consul/proto/pbcommon github.com/hashicorp/consul/proto/pbconfig github.com/hashicorp/consul/proto/pbconnect github.com/hashicorp/consul/sdk/freeport github.com/hashicorp/consul/sdk/testutil github.com/hashicorp/consul/sdk/testutil/retry github.com/hashicorp/consul/sentinel github.com/hashicorp/consul/service_os github.com/hashicorp/consul/snapshot github.com/hashicorp/consul/testrpc github.com/hashicorp/consul/tlsutil github.com/hashicorp/consul/types github.com/hashicorp/consul/version internal/unsafeheader -unicode/utf16 unicode/utf8 -container/list -crypto/subtle -internal/nettrace math/bits -sync/atomic -encoding -crypto/internal/subtle -runtime/internal/sys internal/race -internal/cpu runtime/internal/atomic +internal/cpu +runtime/internal/sys +sync/atomic unicode +encoding +unicode/utf16 +container/list +runtime/internal/math +internal/testlog +math +internal/nettrace +internal/bytealg +crypto/internal/subtle +runtime/cgo +crypto/subtle vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle github.com/circonus-labs/circonus-gometrics/api/config -runtime/cgo golang.org/x/sys/internal/unsafeheader -github.com/hashicorp/consul/lib/stringslice golang.org/x/net/internal/iana +github.com/hashicorp/consul/lib/stringslice +github.com/hashicorp/consul/types google.golang.org/grpc/serviceconfig github.com/aws/aws-sdk-go/aws/client/metadata github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/selection github.com/hashicorp/consul/vendor/k8s.io/utils/integer -github.com/hashicorp/consul/types github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil -go.opencensus.io github.com/golang/groupcache/lru +go.opencensus.io go.opencensus.io/trace/internal -math -runtime/internal/math -internal/testlog go.opencensus.io/internal/tagencoding -github.com/hashicorp/consul/agent/xds/proxysupport -github.com/hashicorp/consul/service_os -internal/bytealg github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/pkg/wellknown runtime +github.com/hashicorp/consul/agent/xds/proxysupport +github.com/hashicorp/consul/service_os github.com/aws/aws-sdk-go/internal/sdkmath internal/reflectlite sync internal/singleflight google.golang.org/grpc/internal/buffer -github.com/aws/aws-sdk-go/internal/sync/singleflight +math/rand google.golang.org/grpc/internal/grpcsync +github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/hashicorp/consul/agent/token -math/rand errors sort +io +strconv internal/oserror -github.com/hashicorp/hcl/hcl/strconv github.com/hashicorp/golang-lru/simplelru -io +github.com/hashicorp/hcl/hcl/strconv vendor/golang.org/x/net/dns/dnsmessage -strconv syscall container/heap -github.com/beorn7/perks/quantile github.com/gogo/protobuf/sortkeys +github.com/beorn7/perks/quantile hash bytes -crypto/internal/randutil -github.com/aws/aws-sdk-go/internal/sdkio -text/tabwriter strings +text/tabwriter hash/crc32 +crypto/internal/randutil crypto/hmac hash/crc64 -hash/fnv -vendor/golang.org/x/crypto/hkdf -golang.org/x/crypto/pbkdf2 -crypto/rc4 -crypto -encoding/base32 reflect -vendor/golang.org/x/text/transform -github.com/kr/text -golang.org/x/text/transform +crypto +crypto/rc4 +vendor/golang.org/x/crypto/hkdf github.com/armon/go-radix -github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg -path -github.com/hashicorp/go-immutable-radix +regexp/syntax bufio -github.com/aws/aws-sdk-go/internal/strings -google.golang.org/grpc/encoding -github.com/ryanuber/go-glob +github.com/hashicorp/go-immutable-radix +vendor/golang.org/x/text/transform +path +github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg html -regexp/syntax -github.com/aws/aws-sdk-go/internal/sdkuri +encoding/base32 +hash/fnv internal/syscall/unix -internal/syscall/execenv time +internal/syscall/execenv +github.com/kr/text +google.golang.org/grpc/encoding +golang.org/x/text/transform +github.com/aws/aws-sdk-go/internal/sdkio +github.com/aws/aws-sdk-go/internal/strings +github.com/aws/aws-sdk-go/internal/sdkuri +github.com/ryanuber/go-glob +golang.org/x/crypto/pbkdf2 regexp internal/poll -google.golang.org/grpc/internal/grpcrand +google.golang.org/grpc/backoff context +google.golang.org/grpc/internal/grpcrand google.golang.org/grpc/keepalive -google.golang.org/grpc/backoff -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/clock github.com/aws/aws-sdk-go/internal/sdkrand +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/clock google.golang.org/grpc/internal/backoff +internal/fmtsort +encoding/binary +github.com/hashicorp/errwrap +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/sets +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/version google.golang.org/grpc/tap github.com/hashicorp/consul/lib/semaphore -golang.org/x/net/context github.com/hashicorp/consul/agent/dns -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/version google.golang.org/api/internal/third_party/uritemplates +golang.org/x/net/context os -github.com/hashicorp/errwrap -internal/fmtsort -encoding/binary -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/sets crypto/md5 encoding/base64 golang.org/x/crypto/blake2b crypto/cipher +crypto/sha512 crypto/ed25519/internal/edwards25519 crypto/sha1 -crypto/sha512 -github.com/cespare/xxhash +crypto/aes +crypto/des crypto/sha256 +encoding/pem +vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/poly1305 -github.com/pierrec/lz4/internal/xxh32 -github.com/google/gofuzz/bytesource -github.com/golang/snappy fmt -golang.org/x/sys/unix net path/filepath os/signal -encoding/pem +vendor/golang.org/x/crypto/chacha20poly1305 +github.com/cespare/xxhash runtime/debug google.golang.org/grpc/internal/envconfig -internal/lazyregexp -crypto/aes -crypto/des -vendor/golang.org/x/crypto/chacha20 +golang.org/x/sys/unix io/ioutil -github.com/aws/aws-sdk-go/internal/shareddefaults os/exec -vendor/golang.org/x/crypto/chacha20poly1305 +github.com/aws/aws-sdk-go/internal/shareddefaults github.com/prometheus/procfs/internal/util -github.com/hashicorp/consul/vendor/github.com/rboyer/safeio +github.com/golang/snappy encoding/json github.com/hashicorp/hcl/hcl/token log compress/flate -mime -vendor/golang.org/x/net/http2/hpack +github.com/hashicorp/hcl/hcl/ast +github.com/hashicorp/hcl/hcl/scanner +github.com/hashicorp/hcl/json/token +github.com/hashicorp/hcl/json/scanner math/big -mime/quotedprintable -net/http/internal +github.com/hashicorp/hcl/hcl/parser +github.com/hashicorp/hcl/json/parser +compress/gzip +github.com/hashicorp/hcl/hcl/printer +github.com/hashicorp/hcl encoding/hex net/url -vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/crypto/curve25519 -github.com/pkg/errors vendor/golang.org/x/text/unicode/bidi -github.com/hashicorp/hcl/hcl/ast -github.com/hashicorp/hcl/hcl/scanner -github.com/hashicorp/hcl/json/token +github.com/hashicorp/consul/acl +vendor/golang.org/x/text/unicode/norm +vendor/golang.org/x/text/secure/bidirule +vendor/golang.org/x/net/http2/hpack +mime +mime/quotedprintable +github.com/pkg/errors +net/http/internal +github.com/circonus-labs/circonusllhist +crypto/rand +github.com/golang/protobuf/proto +crypto/elliptic +encoding/asn1 +crypto/dsa +crypto/ed25519 +crypto/rsa +vendor/golang.org/x/net/idna +github.com/prometheus/common/model github.com/prometheus/procfs/internal/fs +vendor/golang.org/x/crypto/cryptobyte +crypto/x509/pkix github.com/mitchellh/go-testing-interface github.com/hashicorp/consul/version +github.com/hashicorp/go-uuid encoding/gob -github.com/hashicorp/hcl/json/scanner -github.com/hashicorp/hcl/hcl/parser go/token +crypto/ecdsa text/template/parse compress/lzw github.com/google/btree github.com/hashicorp/go-multierror -github.com/hashicorp/hcl/json/parser +golang.org/x/crypto/ed25519 golang.org/x/net/bpf +github.com/hashicorp/memberlist/vendor/github.com/sean-/seed github.com/mitchellh/reflectwalk github.com/davecgh/go-spew/spew -vendor/golang.org/x/text/secure/bidirule -github.com/hashicorp/hcl/hcl/printer -compress/gzip github.com/mitchellh/copystructure github.com/pmezard/go-difflib/difflib -gopkg.in/yaml.v3 -github.com/hashicorp/hcl +text/template +github.com/stretchr/objx flag -github.com/hashicorp/go-version +gopkg.in/yaml.v3 golang.org/x/time/rate +github.com/mattn/go-isatty +github.com/golang/protobuf/ptypes/any +crypto/x509 +net/textproto +vendor/golang.org/x/net/http/httpproxy +github.com/DataDog/datadog-go/statsd +github.com/golang/protobuf/ptypes/duration +vendor/golang.org/x/net/http/httpguts +mime/multipart +github.com/golang/protobuf/ptypes/timestamp +github.com/matttproud/golang_protobuf_extensions/pbutil +github.com/golang/protobuf/ptypes +github.com/prometheus/client_model/go +github.com/prometheus/procfs +github.com/hashicorp/consul/ipaddr +github.com/hashicorp/consul/sdk/freeport +html/template +github.com/hashicorp/go-sockaddr +github.com/prometheus/client_golang/prometheus/internal +golang.org/x/net/internal/socket +github.com/hashicorp/yamux +crypto/tls +github.com/mitchellh/mapstructure +golang.org/x/net/ipv4 +golang.org/x/net/ipv6 +github.com/mattn/go-colorable +github.com/fatih/color +github.com/hashicorp/go-hclog +log/syslog +github.com/hashicorp/go-version runtime/trace +github.com/hashicorp/go-syslog github.com/hashicorp/consul/lib/decode +testing github.com/hashicorp/golang-lru github.com/mitchellh/hashstructure github.com/bgentry/speakeasy -github.com/circonus-labs/circonusllhist -github.com/hashicorp/consul/acl -github.com/prometheus/common/model -github.com/golang/protobuf/proto -github.com/stretchr/objx -github.com/mattn/go-isatty -text/template -vendor/golang.org/x/net/idna -testing os/user github.com/hashicorp/consul/command/helpers -github.com/mattn/go-colorable github.com/armon/circbuf +github.com/golang/protobuf/ptypes/struct +github.com/hashicorp/consul/logging +golang.org/x/net/internal/socks github.com/hashicorp/consul/agent/exec +github.com/golang/protobuf/jsonpb golang.org/x/net/internal/timeseries google.golang.org/grpc/grpclog +golang.org/x/net/proxy +google.golang.org/grpc/credentials/internal google.golang.org/grpc/metadata -github.com/fatih/color google.golang.org/grpc/attributes -crypto/rand -crypto/elliptic -encoding/asn1 -crypto/dsa google.golang.org/grpc/connectivity google.golang.org/grpc/codes -github.com/hashicorp/go-hclog +google.golang.org/grpc/encoding/proto google.golang.org/grpc/internal google.golang.org/grpc/internal/balancerload +google.golang.org/grpc/binarylog/grpc_binarylog_v1 +google.golang.org/genproto/googleapis/rpc/status +golang.org/x/text/unicode/norm golang.org/x/text/unicode/bidi golang.org/x/net/http2/hpack -crypto/ed25519 -golang.org/x/text/unicode/norm -crypto/rsa -github.com/hashicorp/go-uuid -github.com/hashicorp/memberlist/vendor/github.com/sean-/seed +google.golang.org/grpc/status +net/http/httptrace +github.com/hashicorp/go-rootcerts +github.com/miekg/dns +google.golang.org/grpc/credentials +net/http +google.golang.org/grpc/internal/binarylog +google.golang.org/grpc/resolver +google.golang.org/grpc/internal/channelz +google.golang.org/grpc/balancer +google.golang.org/grpc/internal/resolver/dns +google.golang.org/grpc/internal/resolver/passthrough +google.golang.org/grpc/balancer/base +golang.org/x/text/secure/bidirule +google.golang.org/grpc/balancer/roundrobin +google.golang.org/grpc/internal/syscall +google.golang.org/grpc/peer +google.golang.org/grpc/stats +golang.org/x/net/idna +google.golang.org/grpc/naming github.com/aws/aws-sdk-go/aws/awserr github.com/jmespath/go-jmespath -golang.org/x/crypto/ed25519 -html/template encoding/xml -github.com/hashicorp/hil/ast github.com/aws/aws-sdk-go/internal/ini +golang.org/x/net/http/httpguts +github.com/posener/complete/cmd/install +github.com/aws/aws-sdk-go/aws/endpoints +github.com/aws/aws-sdk-go/aws/credentials +github.com/aws/aws-sdk-go/aws/awsutil +github.com/posener/complete/cmd +github.com/hashicorp/hil/ast +github.com/aws/aws-sdk-go/aws/credentials/processcreds +github.com/posener/complete +github.com/hashicorp/hil github.com/gogo/protobuf/proto -vendor/golang.org/x/crypto/cryptobyte -crypto/x509/pkix -golang.org/x/text/secure/bidirule github.com/hashicorp/go-memdb -github.com/aws/aws-sdk-go/aws/endpoints +github.com/mitchellh/cli github.com/hashicorp/consul/sdk/testutil/retry github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/consts github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/hclutil -github.com/pierrec/lz4 +github.com/pierrec/lz4/internal/xxh32 github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/strutil +gopkg.in/square/go-jose.v2/cipher +github.com/pierrec/lz4 +github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/parseutil gopkg.in/square/go-jose.v2/json github.com/hashicorp/go-bexpr -crypto/ecdsa +github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/compressutil +github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/jsonutil +github.com/google/gofuzz/bytesource github.com/google/gofuzz -github.com/aws/aws-sdk-go/aws/awsutil -github.com/aws/aws-sdk-go/aws/credentials gopkg.in/inf.v0 -github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/compressutil -gopkg.in/square/go-jose.v2/cipher +gopkg.in/square/go-jose.v2 github.com/hashicorp/consul/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect -golang.org/x/net/idna -github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/jsonutil +github.com/hashicorp/consul/vendor/github.com/coredns/coredns/plugin/pkg/dnsutil github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/fields -github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/errors -go/scanner -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/conversion/queryparams -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/json -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/naming -github.com/golang/protobuf/ptypes/any -github.com/golang/protobuf/ptypes/duration -github.com/golang/protobuf/ptypes/timestamp -github.com/matttproud/golang_protobuf_extensions/pbutil -github.com/golang/protobuf/ptypes/struct -google.golang.org/grpc/encoding/proto -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/conversion github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/validation/field -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/types -gopkg.in/yaml.v2 -google.golang.org/genproto/googleapis/rpc/status -github.com/golang/protobuf/ptypes -github.com/prometheus/client_model/go -google.golang.org/grpc/binarylog/grpc_binarylog_v1 -github.com/modern-go/concurrent -go/ast -github.com/golang/protobuf/jsonpb -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/framer -github.com/hashicorp/consul/vendor/k8s.io/client-go/pkg/version -golang.org/x/term -github.com/hashicorp/consul/vendor/k8s.io/client-go/tools/metrics -github.com/hashicorp/consul/vendor/k8s.io/client-go/util/flowcontrol +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/conversion github.com/hashicorp/consul/vendor/k8s.io/klog -github.com/posener/complete/cmd/install -github.com/patrickmn/go-cache -google.golang.org/grpc/status -github.com/modern-go/reflect2 -github.com/prometheus/client_golang/prometheus/internal -github.com/googleapis/gnostic/extensions -github.com/hashicorp/go-raftchunking/types -golang.org/x/crypto/ssh/terminal -github.com/hashicorp/consul/proto/pbconfig -github.com/boltdb/bolt -archive/tar -golang.org/x/sync/singleflight -golang.org/x/oauth2/jws -go.opencensus.io/internal -crypto/x509 -net/textproto -vendor/golang.org/x/net/http/httpproxy -github.com/DataDog/datadog-go/statsd -github.com/prometheus/procfs -github.com/hashicorp/consul/ipaddr -github.com/hashicorp/consul/sdk/freeport -golang.org/x/net/internal/socket -github.com/hashicorp/yamux -vendor/golang.org/x/net/http/httpguts -github.com/hashicorp/go-sockaddr -mime/multipart -github.com/mitchellh/mapstructure -github.com/posener/complete/cmd -log/syslog -golang.org/x/net/internal/socks -github.com/posener/complete -github.com/hashicorp/go-syslog -google.golang.org/grpc/credentials/internal -google.golang.org/grpc/internal/binarylog -golang.org/x/net/http/httpguts -google.golang.org/grpc/internal/syscall -google.golang.org/grpc/stats -github.com/hashicorp/consul/logging -google.golang.org/grpc/naming -golang.org/x/net/ipv4 -golang.org/x/net/ipv6 -golang.org/x/net/proxy -github.com/mitchellh/cli -crypto/tls -gopkg.in/square/go-jose.v2 -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/validation -go/doc -go/parser -sigs.k8s.io/yaml -github.com/hashicorp/consul/vendor/k8s.io/client-go/util/connrotation -github.com/hashicorp/hil -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/labels -github.com/hashicorp/consul/vendor/k8s.io/client-go/util/keyutil -github.com/hashicorp/consul/vendor/github.com/mitchellh/pointerstructure -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/yaml -github.com/hashicorp/consul/vendor/k8s.io/client-go/util/cert -go.opencensus.io/trace/tracestate -github.com/json-iterator/go -go.opencensus.io/resource -go.opencensus.io/trace -runtime/pprof -github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/sdk/helper/parseutil -github.com/shirou/gopsutil/internal/common -github.com/hashicorp/go-sockaddr/template -github.com/hashicorp/consul/agent/systemd -go.opencensus.io/metric/metricdata -github.com/golang/protobuf/protoc-gen-go/descriptor -net/mail -github.com/golang/protobuf/ptypes/wrappers -github.com/golang/protobuf/ptypes/empty -github.com/census-instrumentation/opencensus-proto/gen-go/resource/v1 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/pkg/conversion -go.opencensus.io/metric/metricproducer -github.com/gogo/protobuf/protoc-gen-gogo/descriptor -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/api/resource -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/schema -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/intstr -github.com/hashicorp/consul/lib/template -github.com/hashicorp/consul/lib/file -github.com/gogo/protobuf/types -github.com/census-instrumentation/opencensus-proto/gen-go/trace/v1 -github.com/hashicorp/consul/logging/monitor -github.com/imdario/mergo -gopkg.in/square/go-jose.v2/jwt -internal/profile -github.com/ryanuber/columnize -go.opencensus.io/tag -github.com/shirou/gopsutil/disk -github.com/shirou/gopsutil/cpu -github.com/shirou/gopsutil/host -github.com/shirou/gopsutil/mem -go.opencensus.io/stats/internal -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/annotations -github.com/hashicorp/consul/vendor/github.com/cncf/udpa/go/udpa/annotations -github.com/hashicorp/consul/vendor/github.com/envoyproxy/protoc-gen-validate/validate -google.golang.org/genproto/googleapis/api/annotations -go.opencensus.io/stats -github.com/hashicorp/consul/agent/debug -go.opencensus.io/stats/view -github.com/gogo/protobuf/gogoproto -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/listener/v2 -github.com/hashicorp/consul/proto/pbacl -net/http/httptrace -github.com/hashicorp/go-rootcerts -google.golang.org/grpc/credentials -google.golang.org/api/transport/cert -github.com/miekg/dns -net/http -google.golang.org/grpc/resolver -google.golang.org/grpc/internal/channelz -google.golang.org/grpc/peer -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/metadata/v2 -google.golang.org/grpc/balancer -google.golang.org/grpc/internal/resolver/dns -google.golang.org/grpc/internal/resolver/passthrough -google.golang.org/grpc/balancer/base -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/tracing/v2 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/matcher -google.golang.org/grpc/balancer/roundrobin -github.com/hashicorp/consul/proto -github.com/hashicorp/consul/proto/pbcommon -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/core -github.com/hashicorp/consul/proto/pbconnect -github.com/hashicorp/consul/proto/pbautoconf -github.com/hashicorp/mdns -github.com/hashicorp/consul/vendor/github.com/coredns/coredns/plugin/pkg/dnsutil +github.com/armon/go-metrics github.com/hashicorp/go-cleanhttp +github.com/hashicorp/go-retryablehttp github.com/tv42/httpunix expvar +github.com/prometheus/common/expfmt +github.com/armon/go-metrics/datadog +github.com/hashicorp/serf/coordinate +net/rpc +github.com/circonus-labs/circonus-gometrics/api net/http/httptest -github.com/elazarl/go-bindata-assetfs github.com/NYTimes/gziphandler -net/rpc -github.com/docker/go-connections/sockets -golang.org/x/net/trace -net/http/httputil -github.com/armon/go-metrics -github.com/prometheus/common/expfmt -github.com/aws/aws-sdk-go/aws -golang.org/x/net/http2 -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/runtime -github.com/googleapis/gnostic/compiler -github.com/hashicorp/go-retryablehttp -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime -golang.org/x/net/context/ctxhttp -github.com/pquerna/cachecontrol/cacheobject -github.com/hashicorp/consul/vendor/github.com/hashicorp/go-connlimit -github.com/denverdino/aliyungo/util +github.com/hashicorp/consul/api github.com/stretchr/testify/assert -golang.org/x/oauth2/internal -cloud.google.com/go/compute/metadata -google.golang.org/api/googleapi -go.opencensus.io/trace/propagation -github.com/googleapis/gnostic/OpenAPIv2 +github.com/elazarl/go-bindata-assetfs github.com/hashicorp/go-msgpack/codec -github.com/pquerna/cachecontrol -github.com/aws/aws-sdk-go/aws/request -golang.org/x/oauth2 -go.opencensus.io/plugin/ochttp/propagation/b3 -github.com/circonus-labs/circonus-gometrics/api -google.golang.org/api/googleapi/transport -github.com/armon/go-metrics/datadog -github.com/hashicorp/serf/coordinate -google.golang.org/api/transport/http/internal/propagation -github.com/hashicorp/go-discover/provider/mdns -github.com/denverdino/aliyungo/common -go.opencensus.io/plugin/ochttp -github.com/gophercloud/gophercloud github.com/prometheus/client_golang/prometheus -github.com/coreos/go-oidc -github.com/packethost/packngo -golang.org/x/oauth2/jwt -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/auth -github.com/hashicorp/consul/api -golang.org/x/oauth2/google -github.com/denverdino/aliyungo/ecs -github.com/hashicorp/consul/internal/go-sso/oidcauth -github.com/aws/aws-sdk-go/aws/client +github.com/docker/go-connections/sockets +golang.org/x/net/trace +github.com/circonus-labs/circonus-gometrics/checkmgr +github.com/circonus-labs/circonus-gometrics github.com/stretchr/testify/mock github.com/stretchr/testify/require -github.com/aws/aws-sdk-go/aws/corehandlers -github.com/aws/aws-sdk-go/private/protocol -github.com/circonus-labs/circonus-gometrics/checkmgr -github.com/aws/aws-sdk-go/aws/csm -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/recognizer -github.com/aws/aws-sdk-go/aws/ec2metadata -github.com/hashicorp/memberlist -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming -github.com/hashicorp/raft -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json -github.com/hashicorp/consul/vendor/k8s.io/client-go/tools/clientcmd/api +golang.org/x/net/http2 +net/http/httputil github.com/prometheus/client_golang/prometheus/push -github.com/hashicorp/net-rpc-msgpackrpc -github.com/aws/aws-sdk-go/private/protocol/rest -github.com/aws/aws-sdk-go/private/protocol/query/queryutil -github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil -github.com/aws/aws-sdk-go/private/protocol/json/jsonutil -google.golang.org/grpc/internal/transport -github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds -github.com/circonus-labs/circonus-gometrics -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/net -github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/api -github.com/gophercloud/gophercloud/openstack/identity/v3/tokens -github.com/gophercloud/gophercloud/pagination -github.com/armon/go-metrics/prometheus -github.com/hashicorp/go-discover/provider/aliyun -github.com/aws/aws-sdk-go/aws/credentials/endpointcreds -github.com/gophercloud/gophercloud/openstack/utils -github.com/aws/aws-sdk-go/aws/signer/v4 -github.com/aws/aws-sdk-go/private/protocol/jsonrpc -github.com/hashicorp/go-discover/provider/packet -github.com/aws/aws-sdk-go/private/protocol/query -github.com/aws/aws-sdk-go/private/protocol/ec2query -github.com/aws/aws-sdk-go/aws/defaults -github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/ec2tokens -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/watch -github.com/hashicorp/consul/vendor/k8s.io/client-go/transport -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/cluster -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/endpoint +github.com/hashicorp/raft +github.com/hashicorp/memberlist github.com/armon/go-metrics/circonus -github.com/gophercloud/gophercloud/openstack/identity/v2/tenants -github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/oauth1 -github.com/gophercloud/gophercloud/openstack/compute/v2/servers -github.com/aws/aws-sdk-go/service/sts github.com/hashicorp/consul/command/flags -github.com/aws/aws-sdk-go/service/acmpca -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/listener -github.com/gophercloud/gophercloud/openstack/identity/v2/tokens -github.com/hashicorp/serf/serf -github.com/aws/aws-sdk-go/service/ec2 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/route +github.com/armon/go-metrics/prometheus +github.com/aws/aws-sdk-go/aws +github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/hashicorp/consul/command/acl/agenttokens github.com/hashicorp/consul/command/acl/authmethod github.com/hashicorp/consul/command/acl/authmethod/delete -github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/authmethod/create -github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/list +github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/update +github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/token github.com/hashicorp/consul/command/acl/policy -github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/bindingrule/create github.com/hashicorp/consul/command/acl/bindingrule/list +github.com/hashicorp/serf/serf github.com/hashicorp/consul/command/acl/bootstrap -google.golang.org/grpc +github.com/hashicorp/consul/command/acl/policy/list +github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/token/list -github.com/aws/aws-sdk-go/service/sts/stsiface +google.golang.org/grpc/internal/transport github.com/hashicorp/consul/command/acl/role/list -github.com/hashicorp/consul/command/acl/policy/list -github.com/hashicorp/go-raftchunking -github.com/hashicorp/consul/snapshot -github.com/hashicorp/raft-boltdb -github.com/gophercloud/gophercloud/openstack -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/ext_authz/v2 -github.com/aws/aws-sdk-go/aws/credentials/stscreds -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/trace/v2 -github.com/hashicorp/consul/api/watch -github.com/hashicorp/consul/sdk/testutil -github.com/prometheus/client_golang/prometheus/promhttp -net/http/pprof -github.com/hashicorp/go-checkpoint -github.com/hashicorp/consul/command/catalog -github.com/hashicorp/consul/command/catalog/list/dc +github.com/aws/aws-sdk-go/aws/request +github.com/gogo/protobuf/gogoproto +gopkg.in/square/go-jose.v2/jwt +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/api/resource +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/validation +github.com/hashicorp/consul/proto/pbacl +go/scanner +github.com/hashicorp/consul/vendor/github.com/hashicorp/vault/api +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/labels +github.com/aws/aws-sdk-go/aws/client github.com/hashicorp/consul/lib github.com/hashicorp/consul/agent/consul/autopilot -github.com/aws/aws-sdk-go/aws/session -github.com/hashicorp/go-discover/provider/os -github.com/hashicorp/consul/command/catalog/list/nodes -github.com/hashicorp/consul/command/catalog/list/services -github.com/hashicorp/consul/command/config -github.com/hashicorp/consul/command/config/delete -github.com/hashicorp/consul/command/config/list -github.com/hashicorp/consul/command/config/write -github.com/hashicorp/consul/command/config/read -github.com/hashicorp/consul/command/connect -github.com/hashicorp/consul/command/connect/ca -github.com/hashicorp/consul/command/connect/ca/get -github.com/hashicorp/consul/command/connect/ca/set -github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap -github.com/hashicorp/consul/command/intention/finder -github.com/hashicorp/consul/command/debug -github.com/hashicorp/consul/command/event -github.com/hashicorp/consul/agent/cache -github.com/hashicorp/consul/command/exec +github.com/aws/aws-sdk-go/aws/corehandlers +github.com/aws/aws-sdk-go/private/protocol +github.com/aws/aws-sdk-go/aws/csm github.com/hashicorp/consul/agent/ae -github.com/hashicorp/consul/command/forceleave -github.com/hashicorp/consul/command/info -github.com/hashicorp/consul/command/intention -github.com/hashicorp/consul/command/intention/check -github.com/hashicorp/consul/command/intention/create -github.com/hashicorp/consul/command/intention/delete -github.com/hashicorp/consul/command/intention/get -github.com/hashicorp/consul/command/intention/match -github.com/hashicorp/consul/command/join -github.com/hashicorp/consul/command/keygen -github.com/hashicorp/consul/command/kv -github.com/hashicorp/consul/command/kv/del -github.com/hashicorp/consul/command/kv/impexp -github.com/hashicorp/consul/command/kv/get -github.com/hashicorp/consul/command/kv/put -github.com/hashicorp/consul/command/leave -github.com/hashicorp/consul/command/login -github.com/hashicorp/consul/command/kv/exp -github.com/hashicorp/consul/command/kv/imp -github.com/hashicorp/consul/command/logout -github.com/hashicorp/consul/command/maint +github.com/hashicorp/consul/agent/cache +github.com/aws/aws-sdk-go/aws/ec2metadata +go/ast +google.golang.org/grpc +internal/lazyregexp +github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds +github.com/aws/aws-sdk-go/private/protocol/rest +github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/hashicorp/consul/agent/structs -github.com/hashicorp/consul/command/members -github.com/hashicorp/consul/command/monitor -github.com/hashicorp/consul/command/operator -github.com/hashicorp/consul/command/operator/autopilot -github.com/hashicorp/consul/command/operator/autopilot/get -github.com/hashicorp/consul/command/operator/autopilot/set -github.com/hashicorp/consul/command/operator/raft -github.com/hashicorp/consul/command/operator/raft/listpeers -github.com/hashicorp/consul/command/operator/raft/removepeer -github.com/hashicorp/consul/command/reload -github.com/hashicorp/consul/command/rtt -github.com/hashicorp/consul/command/snapshot -github.com/hashicorp/consul/command/snapshot/inspect -github.com/hashicorp/consul/command/snapshot/restore -github.com/hashicorp/consul/command/snapshot/save -github.com/hashicorp/consul/command/tls -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/accesslog/v2 -github.com/hashicorp/consul/command/tls/ca -github.com/hashicorp/consul/command/tls/cert -github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest -github.com/hashicorp/consul/sentinel +github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil +github.com/aws/aws-sdk-go/private/protocol/json/jsonutil +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/conversion/queryparams +github.com/aws/aws-sdk-go/aws/signer/v4 +github.com/aws/aws-sdk-go/aws/credentials/endpointcreds +github.com/aws/aws-sdk-go/private/protocol/jsonrpc +github.com/aws/aws-sdk-go/private/protocol/query +go/doc +github.com/aws/aws-sdk-go/aws/defaults +go/parser +github.com/aws/aws-sdk-go/service/sts +github.com/aws/aws-sdk-go/service/acmpca +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/schema +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/json +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/naming +github.com/aws/aws-sdk-go/service/sts/stsiface +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/runtime +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/types google.golang.org/grpc/health/grpc_health_v1 -github.com/googleapis/gax-go/v2 -google.golang.org/api/internal -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/auth/v2 -google.golang.org/api/internal/gensupport -google.golang.org/api/option +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/intstr +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/net +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime +github.com/aws/aws-sdk-go/aws/credentials/stscreds +github.com/googleapis/gnostic/extensions +gopkg.in/yaml.v2 +github.com/modern-go/concurrent +github.com/aws/aws-sdk-go/aws/session +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/framer +github.com/modern-go/reflect2 +github.com/hashicorp/consul/vendor/k8s.io/client-go/pkg/version +golang.org/x/term +golang.org/x/net/context/ctxhttp +golang.org/x/oauth2/internal +golang.org/x/crypto/ssh/terminal +github.com/hashicorp/consul/vendor/k8s.io/client-go/util/connrotation +github.com/hashicorp/consul/vendor/k8s.io/client-go/tools/metrics +github.com/hashicorp/consul/vendor/k8s.io/client-go/util/keyutil +github.com/hashicorp/consul/vendor/k8s.io/client-go/util/flowcontrol +golang.org/x/oauth2 +github.com/hashicorp/consul/vendor/k8s.io/client-go/util/cert +github.com/pquerna/cachecontrol/cacheobject +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/watch +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/recognizer +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming +github.com/hashicorp/consul/vendor/k8s.io/client-go/tools/clientcmd/api +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/transport +github.com/pquerna/cachecontrol +github.com/googleapis/gnostic/compiler +sigs.k8s.io/yaml +github.com/coreos/go-oidc +github.com/googleapis/gnostic/OpenAPIv2 +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/util/yaml +github.com/hashicorp/consul/vendor/github.com/mitchellh/pointerstructure +github.com/json-iterator/go +github.com/patrickmn/go-cache +github.com/hashicorp/go-raftchunking/types +github.com/hashicorp/net-rpc-msgpackrpc +github.com/hashicorp/go-raftchunking +github.com/hashicorp/consul/lib/template +github.com/gogo/protobuf/types +github.com/hashicorp/consul/proto/pbconfig +github.com/hashicorp/consul/internal/go-sso/oidcauth +github.com/hashicorp/consul/command/acl +github.com/hashicorp/consul/agent/connect +github.com/hashicorp/consul/command/acl/bindingrule/delete +github.com/hashicorp/consul/command/acl/bindingrule/read +github.com/hashicorp/consul/command/acl/bindingrule/update +github.com/hashicorp/consul/command/acl/policy/create +github.com/hashicorp/consul/command/acl/policy/delete +github.com/hashicorp/consul/command/acl/policy/read +github.com/hashicorp/consul/command/acl/policy/update +github.com/hashicorp/consul/command/acl/role/create +github.com/hashicorp/consul/command/acl/role/delete +github.com/hashicorp/consul/command/acl/role/read +github.com/hashicorp/consul/command/acl/role/update +github.com/hashicorp/consul/command/acl/rules +github.com/hashicorp/consul/command/acl/token/clone +github.com/hashicorp/consul/command/acl/token/create +github.com/hashicorp/consul/command/acl/token/delete +github.com/hashicorp/consul/command/acl/token/read +github.com/hashicorp/consul/command/acl/token/update +github.com/hashicorp/consul/agent/checks +github.com/hashicorp/consul/agent/consul/discoverychain +github.com/hashicorp/consul/agent/consul/prepared_query +github.com/hashicorp/consul/agent/consul/authmethod github.com/hashicorp/consul/vendor/k8s.io/api/authentication/v1 -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/protobuf +github.com/hashicorp/consul/vendor/k8s.io/api/core/v1 github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/api/errors +github.com/hashicorp/consul/agent/consul/state +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/protobuf +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured github.com/hashicorp/consul/vendor/k8s.io/api/admissionregistration/v1 github.com/hashicorp/consul/vendor/k8s.io/api/admissionregistration/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/api/auditregistration/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/authentication/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning github.com/hashicorp/consul/vendor/k8s.io/api/authorization/v1 -github.com/hashicorp/consul/vendor/k8s.io/api/auditregistration/v1alpha1 -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured -github.com/hashicorp/consul/vendor/k8s.io/api/core/v1 +github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer github.com/hashicorp/consul/vendor/k8s.io/api/authorization/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/certificates/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/coordination/v1 github.com/hashicorp/consul/vendor/k8s.io/api/coordination/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/api/rbac/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/rbac/v1 -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning +github.com/hashicorp/consul/vendor/k8s.io/api/rbac/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/rbac/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/pkg/apis/clientauthentication github.com/hashicorp/consul/vendor/k8s.io/client-go/rest/watch github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/api/meta -github.com/hashicorp/consul/vendor/k8s.io/apimachinery/pkg/runtime/serializer -google.golang.org/api/option/internaloption -google.golang.org/api/transport/http github.com/hashicorp/consul/vendor/k8s.io/client-go/pkg/apis/clientauthentication/v1alpha1 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2 +github.com/hashicorp/consul/agent/connect/ca github.com/hashicorp/consul/vendor/k8s.io/client-go/pkg/apis/clientauthentication/v1beta1 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/tcp_proxy/v2 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/auth/v2alpha +github.com/hashicorp/consul/agent/consul/authmethod/ssoauth github.com/hashicorp/consul/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec -google.golang.org/api/compute/v1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/rest -github.com/hashicorp/consul/command/acl -github.com/hashicorp/consul/agent/connect -github.com/hashicorp/consul/agent/consul/authmethod -github.com/hashicorp/consul/agent/consul/prepared_query -github.com/hashicorp/consul/agent/checks +github.com/hashicorp/consul/agent/consul/fsm +github.com/hashicorp/consul/tlsutil github.com/hashicorp/consul/agent/metadata -github.com/hashicorp/consul/agent/local -github.com/hashicorp/consul/testrpc -github.com/hashicorp/consul/agent/mock github.com/hashicorp/consul/agent/router -github.com/hashicorp/consul/agent/consul/authmethod/ssoauth -github.com/hashicorp/consul/agent/consul/authmethod/testauth -github.com/hashicorp/consul/command/acl/bindingrule/read -github.com/hashicorp/consul/command/acl/bindingrule/update -github.com/hashicorp/consul/command/acl/bindingrule/delete -github.com/hashicorp/consul/command/acl/policy/delete -github.com/hashicorp/consul/command/acl/policy/read -github.com/hashicorp/consul/command/acl/policy/create -github.com/hashicorp/consul/command/acl/policy/update -github.com/hashicorp/consul/command/acl/role/create -github.com/hashicorp/consul/command/acl/role/delete -github.com/hashicorp/consul/command/acl/role/read -github.com/hashicorp/consul/command/acl/role/update -github.com/hashicorp/consul/command/acl/rules -github.com/hashicorp/consul/command/acl/token/clone -github.com/hashicorp/consul/command/acl/token/create -github.com/hashicorp/consul/command/acl/token/delete -github.com/hashicorp/consul/agent/consul/discoverychain -github.com/hashicorp/consul/command/acl/token/read -github.com/hashicorp/consul/command/acl/token/update -github.com/hashicorp/consul/tlsutil -github.com/hashicorp/consul/connect/certgen -github.com/hashicorp/consul/connect -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/http_connection_manager/v2 -github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2 -github.com/hashicorp/consul/agent/consul/state +github.com/hashicorp/consul/vendor/k8s.io/client-go/rest +github.com/hashicorp/consul/proto github.com/hashicorp/consul/agent/pool -github.com/hashicorp/consul/command/tls/ca/create -github.com/hashicorp/consul/command/tls/cert/create -github.com/hashicorp/consul/connect/proxy +github.com/hashicorp/consul/proto/pbcommon +github.com/hashicorp/consul/vendor/github.com/hashicorp/go-connlimit +archive/tar +github.com/boltdb/bolt github.com/hashicorp/consul/agent/consul/wanfed -github.com/hashicorp/consul/command/connect/proxy -github.com/hashicorp/consul/agent/connect/ca -github.com/hashicorp/consul/agent/consul/fsm +golang.org/x/sync/singleflight +github.com/hashicorp/consul/agent/local +github.com/hashicorp/go-sockaddr/template +github.com/hashicorp/consul/proto/pbconnect +github.com/denverdino/aliyungo/util +github.com/aws/aws-sdk-go/private/protocol/ec2query +github.com/hashicorp/consul/snapshot +cloud.google.com/go/compute/metadata +github.com/aws/aws-sdk-go/service/ec2 +github.com/denverdino/aliyungo/common +golang.org/x/oauth2/jws +github.com/hashicorp/consul/proto/pbautoconf +github.com/hashicorp/raft-boltdb +google.golang.org/api/googleapi +golang.org/x/oauth2/jwt +github.com/googleapis/gax-go/v2 +golang.org/x/oauth2/google +go.opencensus.io/internal +go.opencensus.io/trace/tracestate +github.com/denverdino/aliyungo/ecs +google.golang.org/api/internal/gensupport +go.opencensus.io/resource +go.opencensus.io/trace +go.opencensus.io/metric/metricdata +runtime/pprof +google.golang.org/api/internal +go.opencensus.io/trace/propagation +google.golang.org/api/option +go.opencensus.io/metric/metricproducer +go.opencensus.io/plugin/ochttp/propagation/b3 +google.golang.org/api/googleapi/transport +google.golang.org/api/option/internaloption +google.golang.org/api/transport/cert +google.golang.org/api/transport/http/internal/propagation +github.com/hashicorp/mdns +go.opencensus.io/tag +github.com/gophercloud/gophercloud +github.com/packethost/packngo +github.com/hashicorp/go-discover/provider/aliyun +go.opencensus.io/stats/internal +github.com/hashicorp/go-discover/provider/mdns +go.opencensus.io/stats +github.com/shirou/gopsutil/internal/common +github.com/hashicorp/consul/agent/systemd +go.opencensus.io/stats/view +github.com/golang/protobuf/protoc-gen-go/descriptor +go.opencensus.io/plugin/ochttp +github.com/shirou/gopsutil/cpu +github.com/gophercloud/gophercloud/pagination +github.com/gophercloud/gophercloud/openstack/identity/v3/tokens +github.com/gophercloud/gophercloud/openstack/utils +github.com/hashicorp/go-discover/provider/packet +github.com/gophercloud/gophercloud/openstack/identity/v2/tenants +github.com/gophercloud/gophercloud/openstack/compute/v2/servers +github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/ec2tokens +github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/oauth1 +github.com/shirou/gopsutil/disk +github.com/gophercloud/gophercloud/openstack/identity/v2/tokens +google.golang.org/api/transport/http +github.com/shirou/gopsutil/host +github.com/shirou/gopsutil/mem +github.com/gophercloud/gophercloud/openstack +net/mail +github.com/hashicorp/consul/vendor/github.com/envoyproxy/protoc-gen-validate/validate +github.com/hashicorp/consul/agent/debug +google.golang.org/api/compute/v1 +github.com/golang/protobuf/ptypes/wrappers +github.com/hashicorp/go-discover/provider/os +github.com/hashicorp/consul/vendor/github.com/cncf/udpa/go/udpa/annotations +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/annotations +github.com/golang/protobuf/ptypes/empty +google.golang.org/genproto/googleapis/api/annotations +github.com/census-instrumentation/opencensus-proto/gen-go/resource/v1 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/pkg/conversion +github.com/census-instrumentation/opencensus-proto/gen-go/trace/v1 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/metadata/v2 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/listener/v2 +github.com/hashicorp/consul/api/watch +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/tracing/v2 +github.com/hashicorp/consul/lib/file +github.com/hashicorp/consul/logging/monitor +github.com/hashicorp/consul/sdk/testutil +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/type/matcher +github.com/imdario/mergo +github.com/prometheus/client_golang/prometheus/promhttp +internal/profile +github.com/hashicorp/go-checkpoint +github.com/hashicorp/consul/command/catalog +github.com/hashicorp/consul/command/catalog/list/dc +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/core +github.com/ryanuber/columnize +github.com/hashicorp/consul/command/catalog/list/services +github.com/hashicorp/consul/command/catalog/list/nodes +net/http/pprof +github.com/hashicorp/consul/command/config +github.com/hashicorp/consul/command/config/delete +github.com/hashicorp/consul/command/config/list +github.com/hashicorp/consul/command/config/read +github.com/hashicorp/consul/command/config/write +github.com/hashicorp/consul/command/connect +github.com/hashicorp/consul/command/connect/ca +github.com/hashicorp/consul/command/connect/ca/get +github.com/hashicorp/consul/command/connect/ca/set +github.com/hashicorp/consul/connect +github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap +github.com/hashicorp/consul/command/intention/finder github.com/hashicorp/consul/vendor/k8s.io/api/apps/v1 -github.com/hashicorp/consul/vendor/k8s.io/api/autoscaling/v2beta1 github.com/hashicorp/consul/vendor/k8s.io/api/apps/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/apps/v1beta2 +github.com/hashicorp/consul/vendor/k8s.io/api/autoscaling/v1 +github.com/hashicorp/consul/vendor/k8s.io/api/autoscaling/v2beta1 github.com/hashicorp/consul/vendor/k8s.io/api/autoscaling/v2beta2 -github.com/hashicorp/consul/vendor/k8s.io/api/networking/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/api/events/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/batch/v1 -github.com/hashicorp/consul/vendor/k8s.io/api/node/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/api/policy/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/api/node/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/discovery/v1alpha1 -github.com/hashicorp/consul/vendor/k8s.io/api/autoscaling/v1 +github.com/hashicorp/consul/vendor/k8s.io/api/events/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/api/batch/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/api/batch/v2alpha1 +github.com/hashicorp/consul/vendor/k8s.io/api/extensions/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/networking/v1 +github.com/hashicorp/consul/vendor/k8s.io/api/networking/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/api/node/v1alpha1 +github.com/hashicorp/consul/vendor/k8s.io/api/node/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/api/policy/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/scheduling/v1 github.com/hashicorp/consul/vendor/k8s.io/api/scheduling/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/scheduling/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/api/settings/v1alpha1 -github.com/hashicorp/consul/vendor/k8s.io/api/batch/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/api/batch/v2alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/storage/v1 github.com/hashicorp/consul/vendor/k8s.io/api/storage/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/api/storage/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/tools/reference -github.com/hashicorp/consul/vendor/k8s.io/api/extensions/v1beta1 -github.com/hashicorp/go-discover/provider/aws -github.com/hashicorp/go-discover/provider/gce -github.com/hashicorp/go-discover +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/auth +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/cluster +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/endpoint +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/route +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/ext_authz/v2 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2/listener +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/auth/v2 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/trace/v2 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/scheme -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/batch/v1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/batch/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authentication/v1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v1 +github.com/hashicorp/consul/connect/proxy +github.com/hashicorp/consul/command/intention/create +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/auth/v2alpha github.com/hashicorp/consul/vendor/k8s.io/client-go/discovery github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/auditregistration/v1alpha1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/apps/v1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta2 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta2 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/auditregistration/v1alpha1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authentication/v1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authentication/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/apps/v1beta1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authorization/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authorization/v1 -github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/apps/v1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/authorization/v1beta1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/autoscaling/v2beta2 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/batch/v1 +github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/batch/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/batch/v2alpha1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/certificates/v1beta1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/coordination/v1 @@ -3342,26 +3318,85 @@ github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/storage/v1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/storage/v1alpha1 github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes/typed/storage/v1beta1 +github.com/hashicorp/go-discover/provider/gce +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/accesslog/v2 +github.com/hashicorp/consul/command/connect/proxy +github.com/hashicorp/consul/command/debug +github.com/hashicorp/consul/command/event +github.com/hashicorp/consul/command/exec +github.com/hashicorp/go-discover/provider/aws +github.com/hashicorp/consul/command/forceleave github.com/hashicorp/consul/vendor/k8s.io/client-go/kubernetes +github.com/hashicorp/consul/command/info +github.com/hashicorp/consul/command/intention +github.com/hashicorp/consul/command/intention/check +github.com/hashicorp/consul/command/intention/delete +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/tcp_proxy/v2 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/api/v2 +github.com/hashicorp/consul/command/intention/get +github.com/hashicorp/consul/command/intention/match +github.com/hashicorp/go-discover +github.com/hashicorp/consul/command/join +github.com/hashicorp/consul/command/keygen +github.com/hashicorp/consul/command/kv +github.com/hashicorp/consul/command/kv/del +github.com/hashicorp/consul/command/kv/impexp +github.com/hashicorp/consul/command/kv/get +github.com/hashicorp/consul/command/kv/exp +github.com/hashicorp/consul/command/kv/put +github.com/hashicorp/consul/command/kv/imp +github.com/hashicorp/consul/command/leave +github.com/hashicorp/consul/command/logout +github.com/hashicorp/consul/command/login +github.com/hashicorp/consul/command/maint +github.com/hashicorp/consul/command/members +github.com/hashicorp/consul/command/monitor +github.com/hashicorp/consul/command/operator +github.com/hashicorp/consul/command/operator/autopilot +github.com/hashicorp/consul/command/operator/autopilot/get +github.com/hashicorp/consul/command/operator/autopilot/set +github.com/hashicorp/consul/command/operator/raft +github.com/hashicorp/consul/command/operator/raft/listpeers +github.com/hashicorp/consul/command/operator/raft/removepeer +github.com/hashicorp/consul/command/reload +github.com/hashicorp/consul/command/rtt +github.com/hashicorp/consul/command/snapshot +github.com/hashicorp/consul/command/snapshot/inspect +github.com/hashicorp/consul/command/snapshot/restore +github.com/hashicorp/consul/vendor/github.com/rboyer/safeio +github.com/hashicorp/consul/command/tls +github.com/hashicorp/consul/command/tls/ca +github.com/hashicorp/consul/command/tls/ca/create +github.com/hashicorp/consul/command/snapshot/save +github.com/hashicorp/consul/command/tls/cert +github.com/hashicorp/consul/command/tls/cert/create +github.com/hashicorp/consul/agent/consul/authmethod/testauth +github.com/hashicorp/consul/agent/mock +github.com/hashicorp/consul/connect/certgen github.com/hashicorp/consul/agent/consul/authmethod/kubeauth +github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest +github.com/hashicorp/consul/sentinel +github.com/hashicorp/consul/testrpc github.com/hashicorp/consul/agent/consul +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/config/filter/network/http_connection_manager/v2 +github.com/hashicorp/consul/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2 github.com/hashicorp/consul/command/version github.com/hashicorp/consul/agent/cache-types github.com/hashicorp/consul/agent/config github.com/hashicorp/consul/agent/proxycfg github.com/hashicorp/consul/agent/xds -github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/command/services +github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/agent/auto-config github.com/hashicorp/consul/command/services/deregister github.com/hashicorp/consul/command/services/register github.com/hashicorp/consul/command/connect/envoy github.com/hashicorp/consul/agent github.com/hashicorp/consul/command/agent -github.com/hashicorp/consul/command/keyring github.com/hashicorp/consul/command/connect/expose -github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/command/lock +github.com/hashicorp/consul/command/keyring +github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/command github.com/hashicorp/consul make[1]: Leaving directory '/build/consul-1.8.7+dfsg1' @@ -3373,7 +3408,7 @@ cd _build && go test -vet=off -v -p 4 -short -failfast -timeout 8m github.com/hashicorp/consul github.com/hashicorp/consul/acl github.com/hashicorp/consul/agent github.com/hashicorp/consul/agent/ae github.com/hashicorp/consul/agent/auto-config github.com/hashicorp/consul/agent/config github.com/hashicorp/consul/agent/debug github.com/hashicorp/consul/agent/dns github.com/hashicorp/consul/agent/exec github.com/hashicorp/consul/agent/local github.com/hashicorp/consul/agent/metadata github.com/hashicorp/consul/agent/mock github.com/hashicorp/consul/agent/pool github.com/hashicorp/consul/agent/router github.com/hashicorp/consul/agent/structs github.com/hashicorp/consul/agent/systemd github.com/hashicorp/consul/agent/token github.com/hashicorp/consul/agent/xds github.com/hashicorp/consul/agent/xds/proxysupport github.com/hashicorp/consul/command github.com/hashicorp/consul/command/acl github.com/hashicorp/consul/command/acl/agenttokens github.com/hashicorp/consul/command/acl/authmethod github.com/hashicorp/consul/command/acl/authmethod/create github.com/hashicorp/consul/command/acl/authmethod/delete github.com/hashicorp/consul/command/acl/authmethod/list github.com/hashicorp/consul/command/acl/authmethod/read github.com/hashicorp/consul/command/acl/authmethod/update github.com/hashicorp/consul/command/acl/bindingrule github.com/hashicorp/consul/command/acl/bindingrule/create github.com/hashicorp/consul/command/acl/bindingrule/delete github.com/hashicorp/consul/command/acl/bindingrule/list github.com/hashicorp/consul/command/acl/bindingrule/read github.com/hashicorp/consul/command/acl/bindingrule/update github.com/hashicorp/consul/command/acl/bootstrap github.com/hashicorp/consul/command/acl/policy github.com/hashicorp/consul/command/acl/policy/create github.com/hashicorp/consul/command/acl/policy/delete github.com/hashicorp/consul/command/acl/policy/list github.com/hashicorp/consul/command/acl/policy/read github.com/hashicorp/consul/command/acl/policy/update github.com/hashicorp/consul/command/acl/role github.com/hashicorp/consul/command/acl/role/create github.com/hashicorp/consul/command/acl/role/delete github.com/hashicorp/consul/command/acl/role/list github.com/hashicorp/consul/command/acl/role/read github.com/hashicorp/consul/command/acl/role/update github.com/hashicorp/consul/command/acl/rules github.com/hashicorp/consul/command/acl/token github.com/hashicorp/consul/command/acl/token/clone github.com/hashicorp/consul/command/acl/token/create github.com/hashicorp/consul/command/acl/token/delete github.com/hashicorp/consul/command/acl/token/list github.com/hashicorp/consul/command/acl/token/read github.com/hashicorp/consul/command/acl/token/update github.com/hashicorp/consul/command/agent github.com/hashicorp/consul/command/catalog github.com/hashicorp/consul/command/catalog/list/dc github.com/hashicorp/consul/command/catalog/list/nodes github.com/hashicorp/consul/command/catalog/list/services github.com/hashicorp/consul/command/config github.com/hashicorp/consul/command/config/delete github.com/hashicorp/consul/command/config/list github.com/hashicorp/consul/command/config/read github.com/hashicorp/consul/command/config/write github.com/hashicorp/consul/command/connect github.com/hashicorp/consul/command/connect/ca github.com/hashicorp/consul/command/connect/ca/get github.com/hashicorp/consul/command/connect/ca/set github.com/hashicorp/consul/command/connect/envoy github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap github.com/hashicorp/consul/command/connect/expose github.com/hashicorp/consul/command/connect/proxy github.com/hashicorp/consul/command/event github.com/hashicorp/consul/command/exec github.com/hashicorp/consul/command/forceleave github.com/hashicorp/consul/command/helpers github.com/hashicorp/consul/command/info github.com/hashicorp/consul/command/intention github.com/hashicorp/consul/command/intention/check github.com/hashicorp/consul/command/intention/create github.com/hashicorp/consul/command/intention/delete github.com/hashicorp/consul/command/intention/finder github.com/hashicorp/consul/command/intention/get github.com/hashicorp/consul/command/intention/match github.com/hashicorp/consul/command/join github.com/hashicorp/consul/command/keygen github.com/hashicorp/consul/command/keyring github.com/hashicorp/consul/command/kv github.com/hashicorp/consul/command/kv/del github.com/hashicorp/consul/command/kv/exp github.com/hashicorp/consul/command/kv/get github.com/hashicorp/consul/command/kv/imp github.com/hashicorp/consul/command/kv/impexp github.com/hashicorp/consul/command/kv/put github.com/hashicorp/consul/command/leave github.com/hashicorp/consul/command/lock github.com/hashicorp/consul/command/login github.com/hashicorp/consul/command/logout github.com/hashicorp/consul/command/maint github.com/hashicorp/consul/command/members github.com/hashicorp/consul/command/operator github.com/hashicorp/consul/command/operator/autopilot github.com/hashicorp/consul/command/operator/autopilot/get github.com/hashicorp/consul/command/operator/autopilot/set github.com/hashicorp/consul/command/operator/raft github.com/hashicorp/consul/command/operator/raft/listpeers github.com/hashicorp/consul/command/operator/raft/removepeer github.com/hashicorp/consul/command/reload github.com/hashicorp/consul/command/rtt github.com/hashicorp/consul/command/services github.com/hashicorp/consul/command/services/deregister github.com/hashicorp/consul/command/services/register github.com/hashicorp/consul/command/snapshot github.com/hashicorp/consul/command/snapshot/inspect github.com/hashicorp/consul/command/snapshot/restore github.com/hashicorp/consul/command/snapshot/save github.com/hashicorp/consul/command/validate github.com/hashicorp/consul/command/version github.com/hashicorp/consul/command/watch github.com/hashicorp/consul/connect github.com/hashicorp/consul/connect/certgen github.com/hashicorp/consul/connect/proxy github.com/hashicorp/consul/internal/go-sso/oidcauth github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest github.com/hashicorp/consul/ipaddr github.com/hashicorp/consul/lib github.com/hashicorp/consul/lib/decode github.com/hashicorp/consul/lib/file github.com/hashicorp/consul/lib/semaphore github.com/hashicorp/consul/lib/stringslice github.com/hashicorp/consul/lib/template github.com/hashicorp/consul/logging github.com/hashicorp/consul/logging/monitor github.com/hashicorp/consul/proto github.com/hashicorp/consul/proto/pbacl github.com/hashicorp/consul/proto/pbautoconf github.com/hashicorp/consul/proto/pbcommon github.com/hashicorp/consul/proto/pbconfig github.com/hashicorp/consul/proto/pbconnect github.com/hashicorp/consul/sdk/freeport github.com/hashicorp/consul/sdk/testutil github.com/hashicorp/consul/sdk/testutil/retry github.com/hashicorp/consul/sentinel github.com/hashicorp/consul/service_os github.com/hashicorp/consul/snapshot github.com/hashicorp/consul/testrpc github.com/hashicorp/consul/tlsutil github.com/hashicorp/consul/types github.com/hashicorp/consul/version testing: warning: no tests to run PASS -ok github.com/hashicorp/consul 0.029s [no tests to run] +ok github.com/hashicorp/consul 0.060s [no tests to run] === RUN TestACL === RUN TestACL/DenyAll === RUN TestACL/DenyAll/DenyACLRead @@ -3979,7 +4014,7 @@ === RUN TestACL/KeyWritePrefixDefaultAllow === RUN TestACL/KeyWritePrefixDefaultAllow/KeyWritePrefixDenied.Prefix(foo) === RUN TestACL/KeyWritePrefixDefaultAllow/KeyWritePrefixAllowed.Prefix(bar) ---- PASS: TestACL (0.03s) +--- PASS: TestACL (0.04s) --- PASS: TestACL/DenyAll (0.00s) --- PASS: TestACL/DenyAll/DenyACLRead (0.00s) --- PASS: TestACL/DenyAll/DenyACLWrite (0.00s) @@ -4254,7 +4289,7 @@ --- PASS: TestACL/NodeDefaultAllow/ChildRWSuffixWriteAllowed.Prefix(child-rw-prefix) (0.00s) --- PASS: TestACL/NodeDefaultAllow/ChildOverrideReadAllowed.Prefix(override) (0.00s) --- PASS: TestACL/NodeDefaultAllow/ChildOverrideWriteAllowed.Prefix(override) (0.00s) - --- PASS: TestACL/SessionDefaultDeny (0.00s) + --- PASS: TestACL/SessionDefaultDeny (0.01s) --- PASS: TestACL/SessionDefaultDeny/DefaultReadDenied.Prefix(nope) (0.00s) --- PASS: TestACL/SessionDefaultDeny/DefaultWriteDenied.Prefix(nope) (0.00s) --- PASS: TestACL/SessionDefaultDeny/DenyReadDenied.Prefix(root-nope) (0.00s) @@ -4352,7 +4387,7 @@ --- PASS: TestACL/Parent/ACLWriteDenied (0.00s) --- PASS: TestACL/Parent/SnapshotDenied (0.00s) --- PASS: TestACL/Parent/IntentionDefaultAllowDenied (0.00s) - --- PASS: TestACL/ComplexDefaultAllow (0.00s) + --- PASS: TestACL/ComplexDefaultAllow (0.01s) --- PASS: TestACL/ComplexDefaultAllow/KeyReadAllowed.Prefix(other) (0.00s) --- PASS: TestACL/ComplexDefaultAllow/KeyWriteAllowed.Prefix(other) (0.00s) --- PASS: TestACL/ComplexDefaultAllow/KeyWritePrefixAllowed.Prefix(other) (0.00s) @@ -4602,7 +4637,7 @@ === RUN TestACLEnforce/RuleDenyRequireRead === RUN TestACLEnforce/RuleDenyRequireWrite === RUN TestACLEnforce/RuleDenyRequireList ---- PASS: TestACLEnforce (0.00s) +--- PASS: TestACLEnforce (0.01s) --- PASS: TestACLEnforce/RuleNoneRequireRead (0.00s) --- PASS: TestACLEnforce/RuleNoneRequireWrite (0.00s) --- PASS: TestACLEnforce/RuleNoneRequireList (0.00s) @@ -4653,7 +4688,7 @@ === RUN TestPolicySourceParse/Bad_Policy_-_Operator === RUN TestPolicySourceParse/Keyring_Empty === RUN TestPolicySourceParse/Operator_Empty ---- PASS: TestPolicySourceParse (0.00s) +--- PASS: TestPolicySourceParse (0.01s) --- PASS: TestPolicySourceParse/Legacy_Basic (0.00s) --- PASS: TestPolicySourceParse/Legacy_(JSON) (0.00s) --- PASS: TestPolicySourceParse/Service_No_Intentions_(Legacy) (0.00s) @@ -4689,7 +4724,7 @@ === RUN TestMergePolicies/Write_Precedence === RUN TestMergePolicies/Deny_Precedence === RUN TestMergePolicies/Read_Precedence ---- PASS: TestMergePolicies (0.00s) +--- PASS: TestMergePolicies (0.01s) --- PASS: TestMergePolicies/Agents (0.00s) --- PASS: TestMergePolicies/Events (0.00s) --- PASS: TestMergePolicies/Node (0.00s) @@ -4751,48 +4786,42 @@ === CONT TestACL_Enforce === RUN TestACL_Enforce/acl/read/Deny === CONT TestAllAllowed -=== RUN TestAllAllowed/prefix-allow-other-read-prefix -=== RUN TestAllAllowed/prefix-allow-other-read-exact === CONT TestACL_Enforce/acl/read/Deny authorizer_test.go:618: PASS: ACLRead(*acl.AuthorizerContext) -=== RUN TestAllAllowed/prefix-allow-other-deny-prefix -=== RUN TestACL_Enforce/acl/read/Allow -=== RUN TestAllAllowed/prefix-list-allowed -=== RUN TestAllAllowed/prefix-read-allowed -=== RUN TestAllAllowed/prefix-allow-other-list-prefix -=== CONT TestACL_Enforce/acl/read/Allow - authorizer_test.go:618: PASS: ACLRead(*acl.AuthorizerContext) -=== RUN TestACL_Enforce/acl/write/Deny -=== RUN TestAllAllowed/prefix-allow-other-list-exact -=== RUN TestAllAllowed/prefix-allow-other-write-exact -=== RUN TestAllAllowed/prefix-allow-other-deny-exact -=== CONT TestACL_Enforce/acl/write/Deny - authorizer_test.go:618: PASS: ACLWrite(*acl.AuthorizerContext) -=== CONT TestStaticAuthorizer -=== RUN TestStaticAuthorizer/AllowAll -=== RUN TestACL_Enforce/acl/write/Allow === CONT TestAnyAllowed -=== RUN TestAnyAllowed/no-rules-default -=== RUN TestAnyAllowed/prefix-list-allowed -=== CONT TestACL_Enforce/acl/write/Allow - authorizer_test.go:618: PASS: ACLWrite(*acl.AuthorizerContext) -=== RUN TestAnyAllowed/prefix-deny -=== RUN TestACL_Enforce/acl/list/Deny -=== RUN TestAnyAllowed/prefix-deny-other-write-prefix -=== RUN TestACL_Enforce/operator/read/Deny -=== RUN TestAnyAllowed/prefix-deny-other-write-exact -=== RUN TestAnyAllowed/prefix-deny-other-deny-prefix -=== RUN TestAnyAllowed/prefix-deny-other-deny-exact -=== CONT TestACL_Enforce/operator/read/Deny - authorizer_test.go:618: PASS: OperatorRead(*acl.AuthorizerContext) === CONT TestPolicyAuthorizer +=== RUN TestAllAllowed/prefix-read-allowed +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default +=== RUN TestPolicyAuthorizer/Defaults +=== PAUSE TestPolicyAuthorizer/Defaults +=== RUN TestPolicyAuthorizer/Prefer_Exact_Matches +=== RUN TestAllAllowed/prefix-deny +=== RUN TestAllAllowed/prefix-allow-other-write-exact +=== PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied +=== RUN TestAllAllowed/prefix-allow-other-list-exact === RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed === PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed === RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed === PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed +=== CONT TestStaticAuthorizer +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default +=== RUN TestStaticAuthorizer/AllowAll +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default === PAUSE TestStaticAuthorizer/AllowAll -=== RUN TestACL_Enforce/operator/read/Allow +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default === RUN TestStaticAuthorizer/DenyAll +=== PAUSE TestStaticAuthorizer/DenyAll +=== RUN TestStaticAuthorizer/ManageAll +=== PAUSE TestStaticAuthorizer/ManageAll +=== CONT TestStaticAuthorizer/AllowAll +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) === CONT TestChainedAuthorizer === RUN TestChainedAuthorizer/No_Authorizers === PAUSE TestChainedAuthorizer/No_Authorizers @@ -4801,164 +4830,125 @@ === RUN TestChainedAuthorizer/Authorizer_No_Defaults === PAUSE TestChainedAuthorizer/Authorizer_No_Defaults === RUN TestChainedAuthorizer/First_Found +=== RUN TestACL_Enforce/acl/read/Allow === PAUSE TestChainedAuthorizer/First_Found === CONT TestChainedAuthorizer/No_Authorizers -=== CONT TestChainedAuthorizer/First_Found -=== CONT TestACL_Enforce/operator/read/Allow +=== CONT TestACL_Enforce/acl/read/Allow + authorizer_test.go:618: PASS: ACLRead(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/acl/write/Deny +=== RUN TestAllAllowed/prefix-allow-other-read-prefix +=== RUN TestAllAllowed/prefix-allow-other-read-exact +=== CONT TestACL_Enforce/acl/write/Deny + authorizer_test.go:618: PASS: ACLWrite(*acl.AuthorizerContext) +=== RUN TestAllAllowed/prefix-allow-other-deny-exact +=== RUN TestACL_Enforce/acl/write/Allow +=== RUN TestAllAllowed/no-rules-default +=== RUN TestAllAllowed/prefix-write-allowed +=== CONT TestACL_Enforce/acl/write/Allow + authorizer_test.go:618: PASS: ACLWrite(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/acl/list/Deny +=== RUN TestAllAllowed/prefix-list-allowed +=== RUN TestAllAllowed/prefix-allow-other-write-prefix +=== RUN TestACL_Enforce/operator/read/Deny +=== RUN TestAllAllowed/prefix-allow-other-list-prefix +=== RUN TestAllAllowed/prefix-allow-other-deny-prefix +=== CONT TestACL_Enforce/operator/read/Deny + authorizer_test.go:618: PASS: OperatorRead(*acl.AuthorizerContext) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed +--- PASS: TestAllAllowed (0.00s) + --- PASS: TestAllAllowed/prefix-read-allowed (0.00s) + --- PASS: TestAllAllowed/prefix-deny (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-write-exact (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-list-exact (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-read-prefix (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-read-exact (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-deny-exact (0.00s) + --- PASS: TestAllAllowed/no-rules-default (0.00s) + --- PASS: TestAllAllowed/prefix-write-allowed (0.00s) + --- PASS: TestAllAllowed/prefix-list-allowed (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-write-prefix (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-list-prefix (0.00s) + --- PASS: TestAllAllowed/prefix-allow-other-deny-prefix (0.00s) +=== RUN TestACL_Enforce/operator/read/Allow authorizer_test.go:618: PASS: OperatorRead(*acl.AuthorizerContext) === RUN TestACL_Enforce/operator/write/Deny -=== CONT TestChainedAuthorizer/Authorizer_No_Defaults -=== CONT TestChainedAuthorizer/Authorizer_Defaults -=== CONT TestACL_Enforce/operator/write/Deny authorizer_test.go:618: PASS: OperatorWrite(*acl.AuthorizerContext) === RUN TestACL_Enforce/operator/write/Allow -=== RUN TestAnyAllowed/prefix-write-allowed ---- PASS: TestChainedAuthorizer (0.00s) - --- PASS: TestChainedAuthorizer/No_Authorizers (0.00s) - --- PASS: TestChainedAuthorizer/Authorizer_No_Defaults (0.00s) - --- PASS: TestChainedAuthorizer/Authorizer_Defaults (0.00s) - --- PASS: TestChainedAuthorizer/First_Found (0.00s) -=== RUN TestAllAllowed/no-rules-default -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default -=== PAUSE TestStaticAuthorizer/DenyAll -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default -=== RUN TestStaticAuthorizer/ManageAll -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default -=== PAUSE TestStaticAuthorizer/ManageAll -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default -=== CONT TestStaticAuthorizer/DenyAll -=== CONT TestACL_Enforce/operator/write/Allow authorizer_test.go:618: PASS: OperatorWrite(*acl.AuthorizerContext) -=== RUN TestPolicyAuthorizer/Defaults -=== PAUSE TestPolicyAuthorizer/Defaults === RUN TestACL_Enforce/operator/list/Deny -=== RUN TestPolicyAuthorizer/Prefer_Exact_Matches -=== CONT TestStaticAuthorizer/ManageAll === RUN TestACL_Enforce/keyring/read/Deny -=== CONT TestStaticAuthorizer/AllowAll -=== PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches -=== RUN TestAnyAllowed/prefix-read-allowed -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed -=== RUN TestAnyAllowed/prefix-deny-other-list-prefix -=== CONT TestPolicyAuthorizer/Defaults -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied -=== RUN TestAnyAllowed/prefix-deny-other-list-exact -=== RUN TestAllAllowed/prefix-write-allowed -=== RUN TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) -=== RUN TestAnyAllowed/prefix-deny-other-read-prefix -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default -=== RUN TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed -=== RUN TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) + authorizer_test.go:618: PASS: KeyringRead(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/keyring/read/Allow + authorizer_test.go:618: PASS: KeyringRead(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/keyring/write/Deny + authorizer_test.go:618: PASS: KeyringWrite(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/keyring/write/Allow + authorizer_test.go:618: PASS: KeyringWrite(*acl.AuthorizerContext) +=== RUN TestACL_Enforce/keyring/list/Deny +=== RUN TestACL_Enforce/agent/foo/read/Deny + authorizer_test.go:618: PASS: AgentRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/agent/foo/read/Allow + authorizer_test.go:618: PASS: AgentRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/agent/foo/write/Deny === RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AnyAllowed.Prefix(*) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) === PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches -=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) === RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) -=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default === PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) +=== CONT TestACL_Enforce/agent/foo/write/Deny + authorizer_test.go:618: PASS: AgentWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/agent/foo/write/Allow +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) +=== CONT TestStaticAuthorizer/DenyAll +=== CONT TestACL_Enforce/agent/foo/write/Allow + authorizer_test.go:618: PASS: AgentWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/agent/foo/list/Deny +=== RUN TestACL_Enforce/event/foo/read/Deny +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) +=== CONT TestPolicyAuthorizer/Defaults +=== RUN TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) +=== CONT TestACL_Enforce/event/foo/read/Deny + authorizer_test.go:618: PASS: EventRead(string,*acl.AuthorizerContext) +=== CONT TestChainedAuthorizer/Authorizer_No_Defaults +=== RUN TestACL_Enforce/event/foo/read/Allow === CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AnyAllowed.Prefix(*) -=== RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(fo) ---- PASS: TestStaticAuthorizer (0.00s) - --- PASS: TestStaticAuthorizer/DenyAll (0.00s) - --- PASS: TestStaticAuthorizer/ManageAll (0.00s) - --- PASS: TestStaticAuthorizer/AllowAll (0.00s) -=== RUN TestAllAllowed/prefix-deny -=== CONT TestACL_Enforce/keyring/read/Deny - authorizer_test.go:618: PASS: KeyringRead(*acl.AuthorizerContext) -=== RUN TestAnyAllowed/prefix-deny-other-read-exact -=== RUN TestACL_Enforce/keyring/read/Allow -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) ---- PASS: TestAnyAllowed (0.00s) - --- PASS: TestAnyAllowed/no-rules-default (0.00s) - --- PASS: TestAnyAllowed/prefix-list-allowed (0.00s) - --- PASS: TestAnyAllowed/prefix-deny (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-write-prefix (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-write-exact (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-deny-prefix (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-deny-exact (0.00s) - --- PASS: TestAnyAllowed/prefix-write-allowed (0.00s) - --- PASS: TestAnyAllowed/prefix-read-allowed (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-list-prefix (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-list-exact (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-read-prefix (0.00s) - --- PASS: TestAnyAllowed/prefix-deny-other-read-exact (0.00s) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default +=== CONT TestACL_Enforce/event/foo/read/Allow + authorizer_test.go:618: PASS: EventRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/event/foo/write/Deny +=== CONT TestChainedAuthorizer/Authorizer_Defaults +=== CONT TestACL_Enforce/event/foo/write/Deny + authorizer_test.go:618: PASS: EventWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/event/foo/write/Allow +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) === RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) === PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) === CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) -=== CONT TestACL_Enforce/keyring/read/Allow - authorizer_test.go:618: PASS: KeyringRead(*acl.AuthorizerContext) -=== RUN TestAllAllowed/prefix-allow-other-write-prefix +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) +=== CONT TestACL_Enforce/event/foo/write/Allow + authorizer_test.go:618: PASS: EventWrite(string,*acl.AuthorizerContext) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) +=== RUN TestACL_Enforce/event/foo/list/Deny +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) +=== RUN TestACL_Enforce/intention/foo/read/Deny +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches +=== RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(fo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(fo) -=== RUN TestACL_Enforce/keyring/write/Deny === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(fo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(fo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) ---- PASS: TestAllAllowed (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-read-prefix (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-read-exact (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-deny-prefix (0.00s) - --- PASS: TestAllAllowed/prefix-list-allowed (0.00s) - --- PASS: TestAllAllowed/prefix-read-allowed (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-list-prefix (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-list-exact (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-write-exact (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-deny-exact (0.00s) - --- PASS: TestAllAllowed/no-rules-default (0.00s) - --- PASS: TestAllAllowed/prefix-write-allowed (0.00s) - --- PASS: TestAllAllowed/prefix-deny (0.00s) - --- PASS: TestAllAllowed/prefix-allow-other-write-prefix (0.00s) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(for) -=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) -=== CONT TestACL_Enforce/keyring/write/Deny - authorizer_test.go:618: PASS: KeyringWrite(*acl.AuthorizerContext) -=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) -=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) -=== RUN TestACL_Enforce/keyring/write/Allow === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(for) +=== CONT TestACL_Enforce/intention/foo/read/Deny + authorizer_test.go:618: PASS: IntentionRead(string,*acl.AuthorizerContext) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(for) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(for) -=== CONT TestACL_Enforce/keyring/write/Allow - authorizer_test.go:618: PASS: KeyringWrite(*acl.AuthorizerContext) -=== RUN TestACL_Enforce/keyring/list/Deny -=== RUN TestACL_Enforce/agent/foo/read/Deny +=== RUN TestACL_Enforce/intention/foo/read/Allow === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteAllowed.Prefix(foo) @@ -4966,83 +4956,20 @@ === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot) -=== CONT TestACL_Enforce/agent/foo/read/Deny - authorizer_test.go:618: PASS: AgentRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/agent/foo/read/Allow -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) -=== RUN TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot) -=== RUN TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot2) -=== RUN TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot2) -=== RUN TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) -=== RUN TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) +=== CONT TestACL_Enforce/intention/foo/read/Allow + authorizer_test.go:618: PASS: IntentionRead(string,*acl.AuthorizerContext) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(food) -=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(food) -=== CONT TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(food) +=== RUN TestACL_Enforce/intention/foo/write/Deny === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(food) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) -=== CONT TestACL_Enforce/agent/foo/read/Allow - authorizer_test.go:618: PASS: AgentRead(string,*acl.AuthorizerContext) -=== CONT TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) -=== RUN TestACL_Enforce/agent/foo/write/Deny -=== CONT TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) -=== CONT TestACL_Enforce/agent/foo/write/Deny - authorizer_test.go:618: PASS: AgentWrite(string,*acl.AuthorizerContext) -=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) -=== RUN TestACL_Enforce/agent/foo/write/Allow -=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) -=== CONT TestACL_Enforce/agent/foo/write/Allow - authorizer_test.go:618: PASS: AgentWrite(string,*acl.AuthorizerContext) -=== CONT TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) -=== RUN TestACL_Enforce/agent/foo/list/Deny -=== CONT TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) -=== RUN TestACL_Enforce/event/foo/read/Deny -=== CONT TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) -=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteDenied.Prefix(football) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteDenied.Prefix(football) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(fo) @@ -5052,7 +4979,10 @@ === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(for) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(for) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) +=== CONT TestACL_Enforce/intention/foo/write/Deny + authorizer_test.go:618: PASS: IntentionWrite(string,*acl.AuthorizerContext) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) +=== RUN TestACL_Enforce/intention/foo/write/Allow === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteAllowed.Prefix(foo) @@ -5064,6 +4994,10 @@ === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) +=== CONT TestACL_Enforce/intention/foo/write/Allow + authorizer_test.go:618: PASS: IntentionWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/intention/foo/list/Deny +=== RUN TestACL_Enforce/node/foo/read/Deny === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) @@ -5075,8 +5009,11 @@ === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteDenied.Prefix(football) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) +=== CONT TestACL_Enforce/node/foo/read/Deny + authorizer_test.go:618: PASS: NodeRead(string,*acl.AuthorizerContext) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) +=== RUN TestACL_Enforce/node/foo/read/Allow === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(for) @@ -5090,7 +5027,10 @@ === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) +=== CONT TestACL_Enforce/node/foo/read/Allow + authorizer_test.go:618: PASS: NodeRead(string,*acl.AuthorizerContext) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) +=== CONT TestChainedAuthorizer/First_Found === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food) @@ -5115,20 +5055,109 @@ === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteAllowed.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot) +--- PASS: TestChainedAuthorizer (0.00s) + --- PASS: TestChainedAuthorizer/No_Authorizers (0.00s) + --- PASS: TestChainedAuthorizer/Authorizer_No_Defaults (0.00s) + --- PASS: TestChainedAuthorizer/Authorizer_Defaults (0.00s) + --- PASS: TestChainedAuthorizer/First_Found (0.00s) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) +=== CONT TestStaticAuthorizer/ManageAll +=== RUN TestAnyAllowed/prefix-deny-other-read-exact +=== RUN TestAnyAllowed/prefix-deny-other-deny-prefix +=== RUN TestAnyAllowed/prefix-deny-other-deny-exact +=== RUN TestACL_Enforce/node/foo/write/Deny + authorizer_test.go:618: PASS: NodeWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/node/foo/write/Allow + authorizer_test.go:618: PASS: NodeWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/node/foo/list/Deny +=== RUN TestACL_Enforce/query/foo/read/Deny + authorizer_test.go:618: PASS: PreparedQueryRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/query/foo/read/Allow + authorizer_test.go:618: PASS: PreparedQueryRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/query/foo/write/Deny + authorizer_test.go:618: PASS: PreparedQueryWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/query/foo/write/Allow + authorizer_test.go:618: PASS: PreparedQueryWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/query/foo/list/Deny +=== RUN TestACL_Enforce/service/foo/read/Deny + authorizer_test.go:618: PASS: ServiceRead(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/service/foo/read/Allow +--- PASS: TestStaticAuthorizer (0.00s) + --- PASS: TestStaticAuthorizer/AllowAll (0.00s) + --- PASS: TestStaticAuthorizer/DenyAll (0.00s) + --- PASS: TestStaticAuthorizer/ManageAll (0.01s) +=== RUN TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) +=== CONT TestACL_Enforce/service/foo/read/Allow + authorizer_test.go:618: PASS: ServiceRead(string,*acl.AuthorizerContext) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot2) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot2) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) +=== RUN TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) +=== RUN TestAnyAllowed/no-rules-default === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(food) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(food) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadDenied.Prefix(football) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadDenied.Prefix(football) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteDenied.Prefix(football) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteDenied.Prefix(football) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) +=== RUN TestAnyAllowed/prefix-read-allowed +=== RUN TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) +=== RUN TestAnyAllowed/prefix-deny +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) +=== RUN TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) +=== PAUSE TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo)#01 === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo)#01 === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo)#01 @@ -5270,23 +5299,79 @@ === RUN TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteDenied.Prefix(football) === PAUSE TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(fo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultSessionRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultServiceWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) === CONT TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) === CONT TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) === CONT TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food)#01 +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteAllowed.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) +=== CONT TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadAllowed.Prefix(foo) +=== RUN TestAnyAllowed/prefix-deny-other-write-prefix +=== CONT TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(for) +=== RUN TestAnyAllowed/prefix-deny-other-write-exact +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) +=== RUN TestAnyAllowed/prefix-deny-other-list-prefix +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) +=== RUN TestAnyAllowed/prefix-deny-other-list-exact +=== RUN TestAnyAllowed/prefix-deny-other-read-prefix +=== RUN TestAnyAllowed/prefix-write-allowed +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) +=== RUN TestAnyAllowed/prefix-list-allowed +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteDenied.Prefix(football) +--- PASS: TestAnyAllowed (0.07s) + --- PASS: TestAnyAllowed/prefix-deny-other-read-exact (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-deny-prefix (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-deny-exact (0.01s) + --- PASS: TestAnyAllowed/no-rules-default (0.00s) + --- PASS: TestAnyAllowed/prefix-read-allowed (0.00s) + --- PASS: TestAnyAllowed/prefix-deny (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-write-prefix (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-write-exact (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-list-prefix (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-list-exact (0.00s) + --- PASS: TestAnyAllowed/prefix-deny-other-read-prefix (0.00s) + --- PASS: TestAnyAllowed/prefix-write-allowed (0.00s) + --- PASS: TestAnyAllowed/prefix-list-allowed (0.00s) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadDenied.Prefix(football) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(food) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) +=== RUN TestACL_Enforce/service/foo/write/Deny === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(for) +=== CONT TestACL_Enforce/service/foo/write/Deny + authorizer_test.go:618: PASS: ServiceWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/service/foo/write/Allow === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(for) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(fo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(fo) +=== CONT TestACL_Enforce/service/foo/write/Allow + authorizer_test.go:618: PASS: ServiceWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/session/foo/list/Deny +=== RUN TestACL_Enforce/session/foo/read/Deny +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/EventWriteDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/EventReadDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/EventWritePrefixDenied.Prefix(food) @@ -5308,7 +5393,9 @@ === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(fo) +=== CONT TestACL_Enforce/session/foo/read/Deny + authorizer_test.go:618: PASS: SessionRead(string,*acl.AuthorizerContext) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWriteAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(for) @@ -5332,7 +5419,13 @@ === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football)#01 === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football)#01 === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food)#01 -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food)#01 +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteAllowed.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2)#01 === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2)#01 === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot)#01 @@ -5345,76 +5438,77 @@ === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo)#01 === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadDenied.Prefix(football) +=== RUN TestACL_Enforce/session/foo/read/Allow +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot) +=== RUN TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(food) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(food) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(fo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteAllowed.Prefix(foo) +=== CONT TestACL_Enforce/session/foo/read/Allow + authorizer_test.go:618: PASS: SessionRead(string,*acl.AuthorizerContext) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(for) +=== RUN TestACL_Enforce/session/foo/write/Deny === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(for) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(fo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(fo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteAllowed.Prefix(foo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadAllowed.Prefix(foo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(for) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteAllowed.Prefix(foo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(for) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(fo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(fo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteDenied.Prefix(football) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(food) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(food) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteAllowed.Prefix(foo) -=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(for) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(for) === CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(fo) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) +=== PAUSE TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) +=== CONT TestACL_Enforce/session/foo/write/Deny + authorizer_test.go:618: PASS: SessionWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/session/foo/write/Allow +=== CONT TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) +=== CONT TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) +=== CONT TestACL_Enforce/session/foo/write/Allow + authorizer_test.go:618: PASS: SessionWrite(string,*acl.AuthorizerContext) +=== RUN TestACL_Enforce/session/foo/list/Deny#01 +=== RUN TestACL_Enforce/key/foo/read/Deny --- PASS: TestPolicyAuthorizer (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AnyAllowed.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) (0.00s) --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_any_default (0.00s) --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AnyDefault.Prefix(*) (0.00s) --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_any_default/AllDenied.Prefix(*) (0.00s) --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_default (0.00s) - --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) (0.00s) --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AnyAllowed.Prefix(*) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_default/AllDefault.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AnyAllowed.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_all_allowed/AllAllowed.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AnyAllowed.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_allowed/AllDenied.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults (0.06s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultACLRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultSnapshot.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultSessionWrite.Prefix(foo) (0.00s) @@ -5423,40 +5517,47 @@ --- PASS: TestPolicyAuthorizer/Defaults/DefaultServiceRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultPreparedQueryWrite.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultPreparedQueryRead.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyringWrite.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultIntentionWrite.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultIntentionRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultIntentionDefaultAllow.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultEventRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultAgentWrite.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultAgentRead.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Defaults/DefaultACLWrite.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultNodeRead.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultOperatorWrite.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultOperatorRead.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultNodeWrite.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyWrite.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyWritePrefix.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyringRead.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyList.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Defaults/DefaultKeyRead.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches (0.00s) + --- PASS: TestPolicyAuthorizer/Defaults/DefaultEventWrite.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied (0.07s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AnyAllowed.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Intention_Wildcards_-_prefix_denied/AllDenied.Prefix(*) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches (0.06s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(fo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food)#01 (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadAllowed.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(for) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadDenied.Prefix(football) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(food) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWriteAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(fo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(fo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/EventWriteDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/EventReadDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/EventWritePrefixDenied.Prefix(food) (0.00s) @@ -5478,7 +5579,8 @@ --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteAllowed.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(fo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWriteAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionWritePrefixDenied.Prefix(for) (0.00s) @@ -5502,7 +5604,13 @@ --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football)#01 (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football)#01 (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food)#01 (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food)#01 (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteAllowed.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2)#01 (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2)#01 (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot)#01 (0.00s) @@ -5515,10 +5623,10 @@ --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo)#01 (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadDenied.Prefix(football) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(food) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(fo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(foot) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(foot) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWriteAllowed.Prefix(foo) (0.00s) @@ -5526,103 +5634,38 @@ --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceWritePrefixDenied.Prefix(fo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(fo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteAllowed.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadAllowed.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(for) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(for) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(fo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(fo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot2) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWriteAllowed.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadAllowed.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(for) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyWritePrefixDenied.Prefix(fo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/KeyReadPrefixAllowed.Prefix(fo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteDenied.Prefix(football) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(food) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(food) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(food) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWriteDenied.Prefix(football) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadDenied.Prefix(football) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(food) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(foot) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWriteAllowed.Prefix(foo) (0.00s) - --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot2) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot2) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(for) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadAllowed.Prefix(foo) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(for) (0.00s) --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentWritePrefixDenied.Prefix(fo) (0.00s) -=== CONT TestACL_Enforce/event/foo/read/Deny - authorizer_test.go:618: PASS: EventRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/event/foo/read/Allow - authorizer_test.go:618: PASS: EventRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/event/foo/write/Deny - authorizer_test.go:618: PASS: EventWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/event/foo/write/Allow - authorizer_test.go:618: PASS: EventWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/event/foo/list/Deny -=== RUN TestACL_Enforce/intention/foo/read/Deny - authorizer_test.go:618: PASS: IntentionRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/intention/foo/read/Allow - authorizer_test.go:618: PASS: IntentionRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/intention/foo/write/Deny - authorizer_test.go:618: PASS: IntentionWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/intention/foo/write/Allow - authorizer_test.go:618: PASS: IntentionWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/intention/foo/list/Deny -=== RUN TestACL_Enforce/node/foo/read/Deny - authorizer_test.go:618: PASS: NodeRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/node/foo/read/Allow - authorizer_test.go:618: PASS: NodeRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/node/foo/write/Deny - authorizer_test.go:618: PASS: NodeWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/node/foo/write/Allow - authorizer_test.go:618: PASS: NodeWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/node/foo/list/Deny -=== RUN TestACL_Enforce/query/foo/read/Deny - authorizer_test.go:618: PASS: PreparedQueryRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/query/foo/read/Allow - authorizer_test.go:618: PASS: PreparedQueryRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/query/foo/write/Deny - authorizer_test.go:618: PASS: PreparedQueryWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/query/foo/write/Allow - authorizer_test.go:618: PASS: PreparedQueryWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/query/foo/list/Deny -=== RUN TestACL_Enforce/service/foo/read/Deny - authorizer_test.go:618: PASS: ServiceRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/service/foo/read/Allow - authorizer_test.go:618: PASS: ServiceRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/service/foo/write/Deny - authorizer_test.go:618: PASS: ServiceWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/service/foo/write/Allow - authorizer_test.go:618: PASS: ServiceWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/session/foo/list/Deny -=== RUN TestACL_Enforce/session/foo/read/Deny - authorizer_test.go:618: PASS: SessionRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/session/foo/read/Allow - authorizer_test.go:618: PASS: SessionRead(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/session/foo/write/Deny - authorizer_test.go:618: PASS: SessionWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/session/foo/write/Allow - authorizer_test.go:618: PASS: SessionWrite(string,*acl.AuthorizerContext) -=== RUN TestACL_Enforce/session/foo/list/Deny#01 -=== RUN TestACL_Enforce/key/foo/read/Deny + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadDenied.Prefix(football) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/PreparedQueryWritePrefixDenied.Prefix(food) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/SessionReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/AgentReadPrefixAllowed.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/NodeWritePrefixDenied.Prefix(foot) (0.00s) + --- PASS: TestPolicyAuthorizer/Prefer_Exact_Matches/ServiceReadPrefixAllowed.Prefix(food) (0.00s) +=== CONT TestACL_Enforce/key/foo/read/Deny authorizer_test.go:618: PASS: KeyRead(string,*acl.AuthorizerContext) === RUN TestACL_Enforce/key/foo/read/Allow authorizer_test.go:618: PASS: KeyRead(string,*acl.AuthorizerContext) @@ -5636,7 +5679,7 @@ authorizer_test.go:618: PASS: KeyList(string,*acl.AuthorizerContext) === RUN TestACL_Enforce/key/foo/deny/Deny === RUN TestACL_Enforce/not-a-real-resource/read/Deny ---- PASS: TestACL_Enforce (0.01s) +--- PASS: TestACL_Enforce (0.08s) --- PASS: TestACL_Enforce/acl/read/Deny (0.00s) --- PASS: TestACL_Enforce/acl/read/Allow (0.00s) --- PASS: TestACL_Enforce/acl/write/Deny (0.00s) @@ -5678,11 +5721,11 @@ --- PASS: TestACL_Enforce/query/foo/write/Allow (0.00s) --- PASS: TestACL_Enforce/query/foo/list/Deny (0.00s) --- PASS: TestACL_Enforce/service/foo/read/Deny (0.00s) - --- PASS: TestACL_Enforce/service/foo/read/Allow (0.00s) + --- PASS: TestACL_Enforce/service/foo/read/Allow (0.04s) --- PASS: TestACL_Enforce/service/foo/write/Deny (0.00s) --- PASS: TestACL_Enforce/service/foo/write/Allow (0.00s) --- PASS: TestACL_Enforce/session/foo/list/Deny (0.00s) - --- PASS: TestACL_Enforce/session/foo/read/Deny (0.00s) + --- PASS: TestACL_Enforce/session/foo/read/Deny (0.01s) --- PASS: TestACL_Enforce/session/foo/read/Allow (0.00s) --- PASS: TestACL_Enforce/session/foo/write/Deny (0.00s) --- PASS: TestACL_Enforce/session/foo/write/Allow (0.00s) @@ -5696,7 +5739,7 @@ --- PASS: TestACL_Enforce/key/foo/deny/Deny (0.00s) --- PASS: TestACL_Enforce/not-a-real-resource/read/Deny (0.00s) PASS -ok github.com/hashicorp/consul/acl 0.050s +ok github.com/hashicorp/consul/acl 0.200s === RUN TestACL_Legacy_Disabled_Response === PAUSE TestACL_Legacy_Disabled_Response === RUN TestACL_Legacy_Update @@ -5788,69 +5831,72 @@ [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:44:36.299Z [WARN] TestAgent_ReloadDoesNotTriggerWatch: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:36.273Z [INFO] TestAgent_ReloadDoesNotTriggerWatch: Watch not yet populated, retrying -2022-11-07T22:44:35.272Z [INFO] TestAgent_ReloadDoesNotTriggerWatch: Consul is now ready -2022-11-07T22:44:35.103Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:35.116Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:35.139 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:601a4c36-cb90-3866-721d-0c43c76e32a0 Address:127.0.0.1:11027}]" -44:35.139 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11027 [Follower]" leader= -44:35.140 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-601a4c36-cb90-3866-721d-0c43c76e32a0.dc1 127.0.0.1 -44:35.140 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-601a4c36-cb90-3866-721d-0c43c76e32a0 127.0.0.1 -2022-11-07T22:44:35.140Z [INFO] agent.router: Initializing LAN area manager -44:35.141 [INFO] TestAgent.server: Adding LAN server: server="Node-601a4c36-cb90-3866-721d-0c43c76e32a0 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" -44:35.141 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-601a4c36-cb90-3866-721d-0c43c76e32a0.dc1 area=wan -44:35.141 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=tcp -44:35.141 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=udp -44:35.141 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11023 network=tcp -44:35.141 [INFO] TestAgent: started state syncer -44:35.141 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11028 network=tcp -44:35.206 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:35.206 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11027 [Candidate]" term=2 -44:35.215 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:35.215 [DEBUG] TestAgent.server.raft: vote granted: from=601a4c36-cb90-3866-721d-0c43c76e32a0 term=2 tally=1 -44:35.215 [INFO] TestAgent.server.raft: election won: tally=1 -44:35.215 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11027 [Leader]" -44:35.215 [INFO] TestAgent.server: cluster leadership acquired -44:35.215 [INFO] TestAgent.server: New leader elected: payload=Node-601a4c36-cb90-3866-721d-0c43c76e32a0 -44:35.219 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11027 -44:35.231 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:35.251 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:35.251 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:35.251 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:35.251 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:35.251 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:35.251 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-601a4c36-cb90-3866-721d-0c43c76e32a0 -44:35.251 [INFO] TestAgent.server: member joined, marking health alive: member=Node-601a4c36-cb90-3866-721d-0c43c76e32a0 -44:35.264 [INFO] TestAgent.server: federation state anti-entropy synced -44:35.272 [DEBUG] TestAgent: Check status updated: check=red-is-dead status=passing -44:35.324 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/redis?dc=dc1 from=127.0.0.1:42522 latency=182.235107ms -44:35.333 [DEBUG] TestAgent: watch handler output: watch_handler=[bash, -c, "(cat ; echo CONSUL_INDEX $CONSUL_INDEX) | tee '/tmp/rexec029173692.atomic' ; mv '/tmp/rexec029173692.atomic' '/tmp/rexec029173692'"] output="[] -CONSUL_INDEX 10 +2021-10-05T16:29:35.624Z [WARN] TestAgent_ReloadDoesNotTriggerWatch: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:35.565Z [INFO] TestAgent_ReloadDoesNotTriggerWatch: Watch not yet populated, retrying +2021-10-05T16:29:34.564Z [INFO] TestAgent_ReloadDoesNotTriggerWatch: Consul is now ready +2021-10-05T16:29:34.122Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:34.150Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:34.347 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eda9219b-0d2f-b31a-275b-066696eb17b0 Address:127.0.0.1:29405}]" +29:34.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eda9219b-0d2f-b31a-275b-066696eb17b0.dc1 127.0.0.1 +29:34.350 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eda9219b-0d2f-b31a-275b-066696eb17b0 127.0.0.1 +2021-10-05T16:29:34.350Z [INFO] agent.router: Initializing LAN area manager +29:34.356 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= +29:34.357 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eda9219b-0d2f-b31a-275b-066696eb17b0.dc1 area=wan +29:34.358 [INFO] TestAgent.server: Adding LAN server: server="Node-eda9219b-0d2f-b31a-275b-066696eb17b0 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" +29:34.358 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp +29:34.358 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp +29:34.364 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp +29:34.364 [INFO] TestAgent: started state syncer +29:34.364 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp +29:34.408 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:34.408 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 +29:34.436 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:34.436 [DEBUG] TestAgent.server.raft: vote granted: from=eda9219b-0d2f-b31a-275b-066696eb17b0 term=2 tally=1 +29:34.436 [INFO] TestAgent.server.raft: election won: tally=1 +29:34.436 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" +29:34.436 [INFO] TestAgent.server: cluster leadership acquired +29:34.436 [INFO] TestAgent.server: New leader elected: payload=Node-eda9219b-0d2f-b31a-275b-066696eb17b0 +29:34.446 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 +29:34.471 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:34.536 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:34.536 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:34.536 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:34.536 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:34.536 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:34.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eda9219b-0d2f-b31a-275b-066696eb17b0 +29:34.536 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eda9219b-0d2f-b31a-275b-066696eb17b0 +29:34.556 [INFO] TestAgent.server: federation state anti-entropy synced +29:34.564 [DEBUG] TestAgent: Check status updated: check=red-is-dead status=passing +29:34.598 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:34.606 [INFO] TestAgent: Synced node info +29:34.608 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/redis?dc=dc1 from=127.0.0.1:60580 latency=242.265873ms +29:34.614 [INFO] TestAgent: Synced service: service=redis +29:34.614 [DEBUG] TestAgent: Check in sync: check=red-is-dead +29:34.614 [DEBUG] TestAgent: Node info in sync +29:34.614 [DEBUG] TestAgent: Service in sync: service=redis +29:34.614 [DEBUG] TestAgent: Check in sync: check=red-is-dead +29:34.639 [DEBUG] TestAgent: watch handler output: watch_handler=[bash, -c, "(cat ; echo CONSUL_INDEX $CONSUL_INDEX) | tee '/tmp/rexec738890493.atomic' ; mv '/tmp/rexec738890493.atomic' '/tmp/rexec738890493'"] output="[] +CONSUL_INDEX 12 " -44:35.451 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:35.455 [INFO] TestAgent: Synced node info -44:35.460 [INFO] TestAgent: Synced service: service=redis -44:35.460 [DEBUG] TestAgent: Check in sync: check=red-is-dead -44:35.460 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/redis?dc=dc1&index=10 from=127.0.0.1:42522 latency=126.486115ms -44:35.466 [DEBUG] TestAgent: watch handler output: watch_handler=[bash, -c, "(cat ; echo CONSUL_INDEX $CONSUL_INDEX) | tee '/tmp/rexec029173692.atomic' ; mv '/tmp/rexec029173692.atomic' '/tmp/rexec029173692'"] output="[{"Node":{"ID":"601a4c36-cb90-3866-721d-0c43c76e32a0","Node":"Node-601a4c36-cb90-3866-721d-0c43c76e32a0","Address":"127.0.0.1","Datacenter":"dc1","TaggedAddresses":{"lan":"127.0.0.1","lan_ipv4":"127.0.0.1","wan":"127.0.0.1","wan_ipv4":"127.0.0.1"},"Meta":{"consul-network-segment":""},"CreateIndex":10,"ModifyIndex":12},"Service":{"ID":"redis","Service":"redis","Tags":[],"Meta":null,"Port":0,"Address":"","Weights":{"Passing":1,"Warning":1},"EnableTagOverride":false,"CreateIndex":13,"ModifyIndex":13,"Proxy":{"MeshGateway":{},"Expose":{}},"Connect":{}},"Checks":[{"Node":"Node-601a4c36-cb90-3866-721d-0c43c76e32a0","CheckID":"serfHealth","Name":"Serf Health Status","Status":"passing","Notes":"","Output":"Agent alive and reachable","ServiceID":"","ServiceName":"","ServiceTags":[],"Type":"","Definition":{"Interval":"0s","Timeout":"0s","DeregisterCriticalServiceAfter":"0s","HTTP":"","Header":null,"Method":"","Body":"","TLSSkipVerify":false,"TCP":""},"CreateIndex":10,"ModifyIndex":10},{"Node":"Node-601a4c36-cb90-3866-721d-0c43c76e32a0","CheckID":"red-is-dead","Name":"Service 'redis' check","Status":"passing","Notes":"initial check","Output":"testing-agent-reload-001","ServiceID":"redis","ServiceName":"redis","ServiceTags":[],"Type":"ttl","Definition":{"Interval":"0s","Timeout":"0s","DeregisterCriticalServiceAfter":"0s","HTTP":"","Header":null,"Method":"","Body":"","TLSSkipVerify":false,"TCP":""},"CreateIndex":13,"ModifyIndex":13}]}] +29:34.640 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/redis?dc=dc1&index=12 from=127.0.0.1:60580 latency=1.183746ms +29:34.664 [DEBUG] TestAgent: watch handler output: watch_handler=[bash, -c, "(cat ; echo CONSUL_INDEX $CONSUL_INDEX) | tee '/tmp/rexec738890493.atomic' ; mv '/tmp/rexec738890493.atomic' '/tmp/rexec738890493'"] output="[{"Node":{"ID":"eda9219b-0d2f-b31a-275b-066696eb17b0","Node":"Node-eda9219b-0d2f-b31a-275b-066696eb17b0","Address":"127.0.0.1","Datacenter":"dc1","TaggedAddresses":{"lan":"127.0.0.1","lan_ipv4":"127.0.0.1","wan":"127.0.0.1","wan_ipv4":"127.0.0.1"},"Meta":{"consul-network-segment":""},"CreateIndex":10,"ModifyIndex":12},"Service":{"ID":"redis","Service":"redis","Tags":[],"Meta":null,"Port":0,"Address":"","Weights":{"Passing":1,"Warning":1},"EnableTagOverride":false,"CreateIndex":13,"ModifyIndex":13,"Proxy":{"MeshGateway":{},"Expose":{}},"Connect":{}},"Checks":[{"Node":"Node-eda9219b-0d2f-b31a-275b-066696eb17b0","CheckID":"serfHealth","Name":"Serf Health Status","Status":"passing","Notes":"","Output":"Agent alive and reachable","ServiceID":"","ServiceName":"","ServiceTags":[],"Type":"","Definition":{"Interval":"0s","Timeout":"0s","DeregisterCriticalServiceAfter":"0s","HTTP":"","Header":null,"Method":"","Body":"","TLSSkipVerify":false,"TCP":""},"CreateIndex":10,"ModifyIndex":10},{"Node":"Node-eda9219b-0d2f-b31a-275b-066696eb17b0","CheckID":"red-is-dead","Name":"Service 'redis' check","Status":"passing","Notes":"initial check","Output":"testing-agent-reload-001","ServiceID":"redis","ServiceName":"redis","ServiceTags":[],"Type":"ttl","Definition":{"Interval":"0s","Timeout":"0s","DeregisterCriticalServiceAfter":"0s","HTTP":"","Header":null,"Method":"","Body":"","TLSSkipVerify":false,"TCP":""},"CreateIndex":13,"ModifyIndex":13}]}] CONSUL_INDEX 13 " -44:36.300 [INFO] TestAgent: Requesting shutdown -44:36.300 [INFO] TestAgent.server: shutting down server -44:36.300 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:36.301 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:36.307 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:36.312Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:36.312Z [INFO] agent.router.manager: shutting down -44:36.312 [INFO] TestAgent: consul server down -44:36.312 [INFO] TestAgent: shutdown complete -44:36.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=tcp -44:36.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=udp -44:36.312 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11023 network=tcp -44:36.812 [INFO] TestAgent: Waiting for endpoints to shut down -44:36.813 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReloadDoesNotTriggerWatch (1.75s) +29:35.625 [INFO] TestAgent: Requesting shutdown +29:35.625 [INFO] TestAgent.server: shutting down server +29:35.625 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:35.626 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:35.634 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:35.639Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:35.639Z [INFO] agent.router.manager: shutting down +29:35.639 [INFO] TestAgent: consul server down +29:35.639 [INFO] TestAgent: shutdown complete +29:35.639 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp +29:35.639 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp +29:35.639 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp +29:36.140 [INFO] TestAgent: Waiting for endpoints to shut down +29:36.140 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReloadDoesNotTriggerWatch (2.12s) === RUN TestAgent_Reload_ACLDeny === PAUSE TestAgent_Reload_ACLDeny === RUN TestAgent_Members @@ -5921,131 +5967,131 @@ === PAUSE TestAgent_RegisterService/service_manager === CONT TestAgent_RegisterService/normal === CONT TestAgent_RegisterService/service_manager -2022-11-07T22:44:36.827Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:36.842Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:36.875 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cdc6aba5-c04c-cca2-6dd4-66198c253281 Address:127.0.0.1:11034}]" -44:36.876 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11034 [Follower]" leader= -44:36.876 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cdc6aba5-c04c-cca2-6dd4-66198c253281.dc1 127.0.0.1 -44:36.877 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cdc6aba5-c04c-cca2-6dd4-66198c253281 127.0.0.1 -2022-11-07T22:44:36.877Z [INFO] agent.router: Initializing LAN area manager -44:36.877 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=udp -44:36.883 [INFO] TestAgent.server: Adding LAN server: server="Node-cdc6aba5-c04c-cca2-6dd4-66198c253281 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" -44:36.883 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cdc6aba5-c04c-cca2-6dd4-66198c253281.dc1 area=wan -44:36.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=tcp -44:36.884 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11030 network=tcp -44:36.884 [INFO] TestAgent: started state syncer -44:36.884 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11035 network=tcp -44:36.929 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:36.929 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11034 [Candidate]" term=2 -44:36.937 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:36.937 [DEBUG] TestAgent.server.raft: vote granted: from=cdc6aba5-c04c-cca2-6dd4-66198c253281 term=2 tally=1 -44:36.937 [INFO] TestAgent.server.raft: election won: tally=1 -44:36.937 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11034 [Leader]" -44:36.937 [INFO] TestAgent.server: cluster leadership acquired -44:36.937 [INFO] TestAgent.server: New leader elected: payload=Node-cdc6aba5-c04c-cca2-6dd4-66198c253281 -44:36.941 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11034 -44:36.954 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:36.977 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:36.977 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:36.977 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:36.977 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:36.977 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:36.977 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cdc6aba5-c04c-cca2-6dd4-66198c253281 -44:36.977 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cdc6aba5-c04c-cca2-6dd4-66198c253281 -44:36.983 [INFO] TestAgent.server: federation state anti-entropy synced -44:37.246 [INFO] TestAgent: Synced node info -44:37.255 [INFO] TestAgent: Synced service: service=test -44:37.255 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:37.255 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:37.255 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:37.255 [INFO] TestAgent: Requesting shutdown -44:37.255 [INFO] TestAgent.server: shutting down server -44:37.255 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:37.255 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:37.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:37.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:37.255 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:37.255 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:37.255 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:37.255 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:37.255 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:37.255 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:37.255 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:37.258 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:37.261Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:37.261Z [INFO] agent.router.manager: shutting down -44:37.261 [INFO] TestAgent: consul server down -44:37.261 [INFO] TestAgent: shutdown complete -44:37.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=tcp -44:37.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=udp -44:37.261 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11030 network=tcp -44:37.762 [INFO] TestAgent: Waiting for endpoints to shut down -44:37.762 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:36.832Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:36.871Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:36.917 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:34aba9a1-cc65-f1c7-e076-d8847df46e23 Address:127.0.0.1:11041}]" -44:36.918 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11041 [Follower]" leader= -44:36.918 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-34aba9a1-cc65-f1c7-e076-d8847df46e23.dc1 127.0.0.1 -44:36.918 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-34aba9a1-cc65-f1c7-e076-d8847df46e23 127.0.0.1 -2022-11-07T22:44:36.919Z [INFO] agent.router: Initializing LAN area manager -44:36.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=udp -44:36.919 [INFO] TestAgent.server: Adding LAN server: server="Node-34aba9a1-cc65-f1c7-e076-d8847df46e23 (Addr: tcp/127.0.0.1:11041) (DC: dc1)" -44:36.919 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-34aba9a1-cc65-f1c7-e076-d8847df46e23.dc1 area=wan -44:36.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=tcp -44:36.920 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11037 network=tcp -44:36.920 [INFO] TestAgent: started state syncer -44:36.920 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11042 network=tcp -44:36.953 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:36.953 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11041 [Candidate]" term=2 -44:36.963 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:36.963 [DEBUG] TestAgent.server.raft: vote granted: from=34aba9a1-cc65-f1c7-e076-d8847df46e23 term=2 tally=1 -44:36.963 [INFO] TestAgent.server.raft: election won: tally=1 -44:36.963 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11041 [Leader]" -44:36.963 [INFO] TestAgent.server: cluster leadership acquired -44:36.964 [INFO] TestAgent.server: New leader elected: payload=Node-34aba9a1-cc65-f1c7-e076-d8847df46e23 -44:36.969 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11041 -44:36.978 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:37.001 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:37.001 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:37.001 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:37.001 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:37.001 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:37.001 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-34aba9a1-cc65-f1c7-e076-d8847df46e23 -44:37.001 [INFO] TestAgent.server: member joined, marking health alive: member=Node-34aba9a1-cc65-f1c7-e076-d8847df46e23 -44:37.013 [INFO] TestAgent.server: federation state anti-entropy synced -44:37.027 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:37.030 [INFO] TestAgent: Synced node info -44:37.030 [DEBUG] TestAgent: Node info in sync -44:37.345 [DEBUG] TestAgent: Node info in sync -44:37.357 [INFO] TestAgent: Synced service: service=test -44:37.357 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:37.357 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:37.357 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:37.357 [INFO] TestAgent: Requesting shutdown -44:37.357 [INFO] TestAgent.server: shutting down server -44:37.357 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:37.357 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:37.357 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:37.357 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:37.357 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:37.357 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:37.357 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:37.357 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:37.357 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:37.358 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:37.366 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:37.372Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:37.372Z [INFO] agent.router.manager: shutting down -44:37.372 [INFO] TestAgent: consul server down -44:37.372 [INFO] TestAgent: shutdown complete -44:37.372 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=tcp -44:37.372 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=udp -44:37.372 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11037 network=tcp -44:37.873 [INFO] TestAgent: Waiting for endpoints to shut down -44:37.873 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:36.160Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:36.188Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:36.259 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c3d324f5-53c8-8241-6121-58f404b971e4 Address:127.0.0.1:29412}]" +29:36.259 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= +29:36.260 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c3d324f5-53c8-8241-6121-58f404b971e4.dc1 127.0.0.1 +29:36.265 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c3d324f5-53c8-8241-6121-58f404b971e4 127.0.0.1 +2021-10-05T16:29:36.265Z [INFO] agent.router: Initializing LAN area manager +29:36.265 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp +29:36.265 [INFO] TestAgent.server: Adding LAN server: server="Node-c3d324f5-53c8-8241-6121-58f404b971e4 (Addr: tcp/127.0.0.1:29412) (DC: dc1)" +29:36.265 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c3d324f5-53c8-8241-6121-58f404b971e4.dc1 area=wan +29:36.265 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp +29:36.266 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp +29:36.266 [INFO] TestAgent: started state syncer +29:36.266 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp +29:36.316 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:36.316 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 +29:36.325 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:36.325 [DEBUG] TestAgent.server.raft: vote granted: from=c3d324f5-53c8-8241-6121-58f404b971e4 term=2 tally=1 +29:36.325 [INFO] TestAgent.server.raft: election won: tally=1 +29:36.325 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" +29:36.326 [INFO] TestAgent.server: cluster leadership acquired +29:36.326 [INFO] TestAgent.server: New leader elected: payload=Node-c3d324f5-53c8-8241-6121-58f404b971e4 +29:36.332 [INFO] TestAgent: Synced node info +29:36.332 [DEBUG] TestAgent: Node info in sync +29:36.332 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 +29:36.372 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:36.395 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:36.395 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:36.395 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:36.395 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:36.395 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:36.395 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c3d324f5-53c8-8241-6121-58f404b971e4 +29:36.396 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c3d324f5-53c8-8241-6121-58f404b971e4 +29:36.436 [INFO] TestAgent.server: federation state anti-entropy synced +29:36.504 [DEBUG] TestAgent: Node info in sync +29:36.517 [INFO] TestAgent: Synced service: service=test +29:36.517 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:36.517 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:36.517 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:36.517 [INFO] TestAgent: Requesting shutdown +29:36.517 [INFO] TestAgent.server: shutting down server +29:36.517 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:36.517 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:36.517 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:36.517 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:36.517 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:36.517 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:36.517 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:36.517 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:36.518 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:36.518 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:36.528 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:36.536Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:36.536Z [INFO] agent.router.manager: shutting down +29:36.536 [INFO] TestAgent: consul server down +29:36.536 [INFO] TestAgent: shutdown complete +29:36.536 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp +29:36.536 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp +29:36.536 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp +29:37.037 [INFO] TestAgent: Waiting for endpoints to shut down +29:37.037 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:36.192Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:36.229Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:36.316 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:90462246-0a5c-a9fb-96b4-ebb04ecdf915 Address:127.0.0.1:29419}]" +29:36.316 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29419 [Follower]" leader= +29:36.317 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915.dc1 127.0.0.1 +29:36.318 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915 127.0.0.1 +2021-10-05T16:29:36.318Z [INFO] agent.router: Initializing LAN area manager +29:36.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=udp +29:36.318 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915.dc1 area=wan +29:36.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=tcp +29:36.318 [INFO] TestAgent.server: Adding LAN server: server="Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915 (Addr: tcp/127.0.0.1:29419) (DC: dc1)" +29:36.319 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29415 network=tcp +29:36.319 [INFO] TestAgent: started state syncer +29:36.319 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29420 network=tcp +29:36.382 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:36.382 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29419 [Candidate]" term=2 +29:36.420 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:36.420 [DEBUG] TestAgent.server.raft: vote granted: from=90462246-0a5c-a9fb-96b4-ebb04ecdf915 term=2 tally=1 +29:36.420 [INFO] TestAgent.server.raft: election won: tally=1 +29:36.420 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29419 [Leader]" +29:36.420 [INFO] TestAgent.server: cluster leadership acquired +29:36.420 [INFO] TestAgent.server: New leader elected: payload=Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915 +29:36.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29419 +29:36.491 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:36.540 [INFO] TestAgent: Synced node info +29:36.540 [DEBUG] TestAgent: Node info in sync +29:36.564 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:36.564 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:36.564 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:36.564 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:36.564 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:36.564 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915 +29:36.564 [INFO] TestAgent.server: member joined, marking health alive: member=Node-90462246-0a5c-a9fb-96b4-ebb04ecdf915 +29:36.607 [INFO] TestAgent.server: federation state anti-entropy synced +29:36.708 [DEBUG] TestAgent: Node info in sync +29:36.729 [INFO] TestAgent: Synced service: service=test +29:36.729 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:36.729 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:36.729 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:36.729 [INFO] TestAgent: Requesting shutdown +29:36.729 [INFO] TestAgent.server: shutting down server +29:36.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:36.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:36.729 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:36.729 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:36.729 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:36.729 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:36.729 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:36.729 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:36.729 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:36.729 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:36.736 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:36.736Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:36.736Z [INFO] agent.router.manager: shutting down +29:36.736 [INFO] TestAgent: consul server down +29:36.736 [INFO] TestAgent: shutdown complete +29:36.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=tcp +29:36.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=udp +29:36.736 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29415 network=tcp +29:37.240 [INFO] TestAgent: Waiting for endpoints to shut down +29:37.240 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService (0.00s) - --- PASS: TestAgent_RegisterService/normal (0.95s) - --- PASS: TestAgent_RegisterService/service_manager (1.06s) + --- PASS: TestAgent_RegisterService/normal (0.90s) + --- PASS: TestAgent_RegisterService/service_manager (1.10s) === RUN TestAgent_RegisterService_ReRegister === RUN TestAgent_RegisterService_ReRegister/normal === PAUSE TestAgent_RegisterService_ReRegister/normal @@ -6053,137 +6099,139 @@ === PAUSE TestAgent_RegisterService_ReRegister/service_manager === CONT TestAgent_RegisterService_ReRegister/normal === CONT TestAgent_RegisterService_ReRegister/service_manager -2022-11-07T22:44:37.890Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:37.903Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:37.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b33881c8-c651-e514-b36c-bf4c8678f29a Address:127.0.0.1:11048}]" -44:37.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11048 [Follower]" leader= -44:37.931 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b33881c8-c651-e514-b36c-bf4c8678f29a.dc1 127.0.0.1 -44:37.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b33881c8-c651-e514-b36c-bf4c8678f29a 127.0.0.1 -2022-11-07T22:44:37.931Z [INFO] agent.router: Initializing LAN area manager -44:37.932 [INFO] TestAgent.server: Adding LAN server: server="Node-b33881c8-c651-e514-b36c-bf4c8678f29a (Addr: tcp/127.0.0.1:11048) (DC: dc1)" -44:37.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=udp -44:37.932 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b33881c8-c651-e514-b36c-bf4c8678f29a.dc1 area=wan -44:37.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=tcp -44:37.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11044 network=tcp -44:37.932 [INFO] TestAgent: started state syncer -44:37.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11049 network=tcp -44:37.979 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:37.979 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11048 [Candidate]" term=2 -44:37.989 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:37.989 [DEBUG] TestAgent.server.raft: vote granted: from=b33881c8-c651-e514-b36c-bf4c8678f29a term=2 tally=1 -44:37.989 [INFO] TestAgent.server.raft: election won: tally=1 -44:37.989 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11048 [Leader]" -44:37.989 [INFO] TestAgent.server: cluster leadership acquired -44:37.989 [INFO] TestAgent.server: New leader elected: payload=Node-b33881c8-c651-e514-b36c-bf4c8678f29a -44:37.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11048 -44:38.011 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b33881c8-c651-e514-b36c-bf4c8678f29a -44:38.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b33881c8-c651-e514-b36c-bf4c8678f29a -44:38.060 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.180 [INFO] TestAgent: Synced node info -44:38.186 [INFO] TestAgent: Synced service: service=test -44:38.186 [DEBUG] TestAgent: Check in sync: check=check_1 -44:38.186 [DEBUG] TestAgent: Check in sync: check=check_2 -44:38.197 [DEBUG] TestAgent: Node info in sync -44:38.200 [INFO] TestAgent: Synced service: service=test -44:38.200 [DEBUG] TestAgent: Check in sync: check=check_1 -44:38.200 [DEBUG] TestAgent: Check in sync: check=check_2 -44:38.200 [DEBUG] TestAgent: Check in sync: check=check_3 -44:38.201 [INFO] TestAgent: Requesting shutdown -44:38.201 [INFO] TestAgent.server: shutting down server -44:38.201 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.201 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.201 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.201 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.201 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.201 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.201 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.201 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.201 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.201 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.201 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.203 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.207Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.207Z [INFO] agent.router.manager: shutting down -44:38.207 [INFO] TestAgent: consul server down -44:38.207 [INFO] TestAgent: shutdown complete -44:38.207 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=tcp -44:38.207 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=udp -44:38.207 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11044 network=tcp -44:38.707 [INFO] TestAgent: Waiting for endpoints to shut down -44:38.707 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:37.893Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:37.903Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:37.932 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf331507-27e0-e515-1e28-2aa28ddb008c Address:127.0.0.1:11055}]" -44:37.932 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11055 [Follower]" leader= -44:37.933 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bf331507-27e0-e515-1e28-2aa28ddb008c.dc1 127.0.0.1 -44:37.933 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bf331507-27e0-e515-1e28-2aa28ddb008c 127.0.0.1 -2022-11-07T22:44:37.933Z [INFO] agent.router: Initializing LAN area manager -44:37.933 [INFO] TestAgent.server: Adding LAN server: server="Node-bf331507-27e0-e515-1e28-2aa28ddb008c (Addr: tcp/127.0.0.1:11055) (DC: dc1)" -44:37.934 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bf331507-27e0-e515-1e28-2aa28ddb008c.dc1 area=wan -44:37.934 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=tcp -44:37.934 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=udp -44:37.937 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11051 network=tcp -44:37.937 [INFO] TestAgent: started state syncer -44:37.937 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11056 network=tcp -44:37.984 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:37.984 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11055 [Candidate]" term=2 -44:37.995 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:37.995 [DEBUG] TestAgent.server.raft: vote granted: from=bf331507-27e0-e515-1e28-2aa28ddb008c term=2 tally=1 -44:37.995 [INFO] TestAgent.server.raft: election won: tally=1 -44:37.995 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11055 [Leader]" -44:37.996 [INFO] TestAgent.server: cluster leadership acquired -44:37.996 [INFO] TestAgent.server: New leader elected: payload=Node-bf331507-27e0-e515-1e28-2aa28ddb008c -44:38.002 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11055 -44:38.023 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.046 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.046 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.046 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.046 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.046 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.047 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf331507-27e0-e515-1e28-2aa28ddb008c -44:38.047 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bf331507-27e0-e515-1e28-2aa28ddb008c -44:38.059 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.313 [INFO] TestAgent: Synced node info -44:38.319 [INFO] TestAgent: Synced service: service=test -44:38.319 [DEBUG] TestAgent: Check in sync: check=check_1 -44:38.319 [DEBUG] TestAgent: Check in sync: check=check_2 -44:38.329 [DEBUG] TestAgent: Node info in sync -44:38.332 [INFO] TestAgent: Synced service: service=test -44:38.332 [DEBUG] TestAgent: Check in sync: check=check_3 -44:38.332 [DEBUG] TestAgent: Check in sync: check=check_1 -44:38.332 [DEBUG] TestAgent: Check in sync: check=check_2 -44:38.332 [INFO] TestAgent: Requesting shutdown -44:38.332 [INFO] TestAgent.server: shutting down server -44:38.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.332 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.332 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.332 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.332 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.332 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.332 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.332 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.335 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.339Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.339Z [INFO] agent.router.manager: shutting down -44:38.339 [INFO] TestAgent: consul server down -44:38.339 [INFO] TestAgent: shutdown complete -44:38.339 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=tcp -44:38.339 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=udp -44:38.339 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11051 network=tcp -44:38.840 [INFO] TestAgent: Waiting for endpoints to shut down -44:38.840 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:37.269Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:37.308Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:37.548 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8d5ff3fe-50d1-0df3-ad32-134ac16a6085 Address:127.0.0.1:29433}]" +29:37.549 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29433 [Follower]" leader= +29:37.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085.dc1 127.0.0.1 +29:37.550 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085 127.0.0.1 +2021-10-05T16:29:37.550Z [INFO] agent.router: Initializing LAN area manager +29:37.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=udp +29:37.551 [INFO] TestAgent.server: Adding LAN server: server="Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085 (Addr: tcp/127.0.0.1:29433) (DC: dc1)" +29:37.551 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085.dc1 area=wan +29:37.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=tcp +29:37.552 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29429 network=tcp +29:37.552 [INFO] TestAgent: started state syncer +29:37.552 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29434 network=tcp +29:37.606 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:37.606 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29433 [Candidate]" term=2 +29:37.625 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:37.625 [DEBUG] TestAgent.server.raft: vote granted: from=8d5ff3fe-50d1-0df3-ad32-134ac16a6085 term=2 tally=1 +29:37.625 [INFO] TestAgent.server.raft: election won: tally=1 +29:37.625 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29433 [Leader]" +29:37.628 [INFO] TestAgent.server: cluster leadership acquired +29:37.628 [INFO] TestAgent.server: New leader elected: payload=Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085 +29:37.648 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29433 +29:37.678 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:37.728 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:37.728 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:37.728 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:37.728 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:37.728 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:37.728 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085 +29:37.728 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8d5ff3fe-50d1-0df3-ad32-134ac16a6085 +29:37.758 [INFO] TestAgent.server: federation state anti-entropy synced +29:37.783 [INFO] TestAgent: Synced node info +29:37.789 [INFO] TestAgent: Synced service: service=test +29:37.789 [DEBUG] TestAgent: Check in sync: check=check_1 +29:37.789 [DEBUG] TestAgent: Check in sync: check=check_2 +29:37.801 [DEBUG] TestAgent: Node info in sync +29:37.818 [INFO] TestAgent: Synced service: service=test +29:37.819 [DEBUG] TestAgent: Check in sync: check=check_1 +29:37.819 [DEBUG] TestAgent: Check in sync: check=check_2 +29:37.819 [DEBUG] TestAgent: Check in sync: check=check_3 +29:37.819 [INFO] TestAgent: Requesting shutdown +29:37.819 [INFO] TestAgent.server: shutting down server +29:37.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:37.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:37.819 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:37.819 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:37.819 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:37.819 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:37.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:37.819 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:37.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:37.819 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:37.819 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:37.828 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:37.840Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:37.840Z [INFO] agent.router.manager: shutting down +29:37.840 [INFO] TestAgent: consul server down +29:37.840 [INFO] TestAgent: shutdown complete +29:37.840 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=tcp +29:37.840 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=udp +29:37.840 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29429 network=tcp +29:38.340 [INFO] TestAgent: Waiting for endpoints to shut down +29:38.340 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:37.268Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:37.336Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:37.551 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 Address:127.0.0.1:29426}]" +29:37.552 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2.dc1 127.0.0.1 +29:37.552 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 127.0.0.1 +2021-10-05T16:29:37.552Z [INFO] agent.router: Initializing LAN area manager +29:37.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=udp +29:37.553 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29426 [Follower]" leader= +29:37.553 [INFO] TestAgent.server: Adding LAN server: server="Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 (Addr: tcp/127.0.0.1:29426) (DC: dc1)" +29:37.553 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2.dc1 area=wan +29:37.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=tcp +29:37.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29422 network=tcp +29:37.554 [INFO] TestAgent: started state syncer +29:37.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29427 network=tcp +29:37.616 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:37.616 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29426 [Candidate]" term=2 +29:37.652 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:37.652 [DEBUG] TestAgent.server.raft: vote granted: from=1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 term=2 tally=1 +29:37.652 [INFO] TestAgent.server.raft: election won: tally=1 +29:37.652 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29426 [Leader]" +29:37.656 [INFO] TestAgent.server: cluster leadership acquired +29:37.656 [INFO] TestAgent.server: New leader elected: payload=Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 +29:37.671 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29426 +29:37.690 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:37.741 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:37.741 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:37.741 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:37.741 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:37.741 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:37.741 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 +29:37.741 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1cee6c4e-431e-84c3-2b5e-2df2e74fadf2 +29:37.772 [INFO] TestAgent.server: federation state anti-entropy synced +29:37.892 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:37.916 [INFO] TestAgent: Synced node info +29:37.916 [DEBUG] TestAgent: Node info in sync +29:37.936 [DEBUG] TestAgent: Node info in sync +29:37.952 [INFO] TestAgent: Synced service: service=test +29:37.952 [DEBUG] TestAgent: Check in sync: check=check_1 +29:37.952 [DEBUG] TestAgent: Check in sync: check=check_2 +29:37.980 [DEBUG] TestAgent: Node info in sync +29:37.996 [INFO] TestAgent: Synced service: service=test +29:37.996 [DEBUG] TestAgent: Check in sync: check=check_1 +29:37.996 [DEBUG] TestAgent: Check in sync: check=check_2 +29:37.996 [DEBUG] TestAgent: Check in sync: check=check_3 +29:37.996 [INFO] TestAgent: Requesting shutdown +29:37.996 [INFO] TestAgent.server: shutting down server +29:37.996 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:37.996 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:37.996 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:37.996 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:37.996 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:37.996 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:37.996 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:37.997 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:37.997 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:37.997 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:38.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:38.024Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:38.024Z [INFO] agent.router.manager: shutting down +29:38.024 [INFO] TestAgent: consul server down +29:38.024 [INFO] TestAgent: shutdown complete +29:38.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=tcp +29:38.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=udp +29:38.024 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29422 network=tcp +29:38.526 [INFO] TestAgent: Waiting for endpoints to shut down +29:38.526 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ReRegister (0.00s) - --- PASS: TestAgent_RegisterService_ReRegister/normal (0.83s) - --- PASS: TestAgent_RegisterService_ReRegister/service_manager (0.97s) + --- PASS: TestAgent_RegisterService_ReRegister/service_manager (1.10s) + --- PASS: TestAgent_RegisterService_ReRegister/normal (1.29s) === RUN TestAgent_RegisterService_ReRegister_ReplaceExistingChecks === RUN TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/normal === PAUSE TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/normal @@ -6191,141 +6239,156 @@ === PAUSE TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/service_manager === CONT TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/normal === CONT TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/service_manager -2022-11-07T22:44:38.857Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.866Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.915 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a473203-1b67-80ab-004b-30d8d6951964 Address:127.0.0.1:11069}]" -44:38.915 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11069 [Follower]" leader= -44:38.920 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a473203-1b67-80ab-004b-30d8d6951964.dc1 127.0.0.1 -44:38.920 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a473203-1b67-80ab-004b-30d8d6951964 127.0.0.1 -2022-11-07T22:44:38.920Z [INFO] agent.router: Initializing LAN area manager -44:38.921 [INFO] TestAgent.server: Adding LAN server: server="Node-1a473203-1b67-80ab-004b-30d8d6951964 (Addr: tcp/127.0.0.1:11069) (DC: dc1)" -44:38.921 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a473203-1b67-80ab-004b-30d8d6951964.dc1 area=wan -44:38.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11064 network=udp -44:38.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11064 network=tcp -44:38.921 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11065 network=tcp -44:38.921 [INFO] TestAgent: started state syncer -44:38.921 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11070 network=tcp -44:38.963 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.963 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11069 [Candidate]" term=2 -44:38.977 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.977 [DEBUG] TestAgent.server.raft: vote granted: from=1a473203-1b67-80ab-004b-30d8d6951964 term=2 tally=1 -44:38.977 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.977 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11069 [Leader]" -44:38.977 [INFO] TestAgent.server: cluster leadership acquired -44:38.977 [INFO] TestAgent.server: New leader elected: payload=Node-1a473203-1b67-80ab-004b-30d8d6951964 -44:38.985 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11069 -44:39.002 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:39.029 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:39.029 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:39.029 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:39.029 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:39.029 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:39.029 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1a473203-1b67-80ab-004b-30d8d6951964 -44:39.029 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a473203-1b67-80ab-004b-30d8d6951964 -44:39.037 [INFO] TestAgent.server: federation state anti-entropy synced -44:39.191 [INFO] TestAgent: Synced node info -44:39.196 [INFO] TestAgent: Synced service: service=test -44:39.196 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:39.196 [DEBUG] TestAgent: Check in sync: check=check_2 -44:39.206 [DEBUG] TestAgent: removed check: check=check_2 -44:39.206 [DEBUG] TestAgent: Node info in sync -44:39.212 [INFO] TestAgent: Synced service: service=test -44:39.212 [DEBUG] TestAgent: Check in sync: check=check_3 -44:39.212 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:39.217 [INFO] TestAgent: Deregistered check: check=check_2 -44:39.217 [INFO] TestAgent: Requesting shutdown -44:39.217 [INFO] TestAgent.server: shutting down server -44:39.217 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.217 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.217 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.217 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.217 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.217 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.217 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:39.217 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.217 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.217 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.217 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.220 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.223Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.224Z [INFO] agent.router.manager: shutting down -44:39.224 [INFO] TestAgent: consul server down -44:39.224 [INFO] TestAgent: shutdown complete -44:39.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11064 network=tcp -44:39.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11064 network=udp -44:39.224 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11065 network=tcp -44:39.724 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.724 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:38.852Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.864Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.914 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e61b490d-085f-762c-cfc5-b650c200a0b1 Address:127.0.0.1:11062}]" -44:38.914 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11062 [Follower]" leader= -44:38.915 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e61b490d-085f-762c-cfc5-b650c200a0b1.dc1 127.0.0.1 -44:38.915 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e61b490d-085f-762c-cfc5-b650c200a0b1 127.0.0.1 -2022-11-07T22:44:38.915Z [INFO] agent.router: Initializing LAN area manager -44:38.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=udp -44:38.919 [INFO] TestAgent.server: Adding LAN server: server="Node-e61b490d-085f-762c-cfc5-b650c200a0b1 (Addr: tcp/127.0.0.1:11062) (DC: dc1)" -44:38.919 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e61b490d-085f-762c-cfc5-b650c200a0b1.dc1 area=wan -44:38.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=tcp -44:38.920 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11058 network=tcp -44:38.920 [INFO] TestAgent: started state syncer -44:38.920 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11063 network=tcp -44:38.956 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.956 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11062 [Candidate]" term=2 -44:38.969 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.969 [DEBUG] TestAgent.server.raft: vote granted: from=e61b490d-085f-762c-cfc5-b650c200a0b1 term=2 tally=1 -44:38.969 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.969 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11062 [Leader]" -44:38.969 [INFO] TestAgent.server: cluster leadership acquired -44:38.969 [INFO] TestAgent.server: New leader elected: payload=Node-e61b490d-085f-762c-cfc5-b650c200a0b1 -44:38.976 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11062 -44:38.998 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:39.026 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:39.026 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:39.026 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:39.026 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:39.026 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:39.026 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e61b490d-085f-762c-cfc5-b650c200a0b1 -44:39.026 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e61b490d-085f-762c-cfc5-b650c200a0b1 -44:39.040 [INFO] TestAgent.server: federation state anti-entropy synced -44:39.205 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.210 [INFO] TestAgent: Synced node info -44:39.210 [DEBUG] TestAgent: Node info in sync -44:39.345 [DEBUG] TestAgent: Node info in sync -44:39.352 [INFO] TestAgent: Synced service: service=test -44:39.352 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:39.352 [DEBUG] TestAgent: Check in sync: check=check_2 -44:39.364 [DEBUG] TestAgent: removed check: check=check_2 -44:39.364 [DEBUG] TestAgent: Node info in sync -44:39.367 [INFO] TestAgent: Synced service: service=test -44:39.367 [DEBUG] TestAgent: Check in sync: check=check_3 -44:39.367 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:39.370 [INFO] TestAgent: Deregistered check: check=check_2 -44:39.370 [INFO] TestAgent: Requesting shutdown -44:39.370 [INFO] TestAgent.server: shutting down server -44:39.370 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.370 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.370 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.370 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.370 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.370 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.370 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.370 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.370 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.370 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.373 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.377Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.377Z [INFO] agent.router.manager: shutting down -44:39.377 [INFO] TestAgent: consul server down -44:39.377 [INFO] TestAgent: shutdown complete -44:39.377 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=tcp -44:39.377 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=udp -44:39.377 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11058 network=tcp -44:39.877 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.877 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:38.558Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:38.601Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:38.668 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7bb8f77-8a31-72bd-4c31-66ed7c431b60 Address:127.0.0.1:29440}]" +29:38.669 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60.dc1 127.0.0.1 +29:38.670 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60 127.0.0.1 +2021-10-05T16:29:38.670Z [INFO] agent.router: Initializing LAN area manager +29:38.670 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=udp +29:38.670 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29440 [Follower]" leader= +29:38.671 [INFO] TestAgent.server: Adding LAN server: server="Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60 (Addr: tcp/127.0.0.1:29440) (DC: dc1)" +29:38.671 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60.dc1 area=wan +29:38.671 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=tcp +29:38.671 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29436 network=tcp +29:38.671 [INFO] TestAgent: started state syncer +29:38.672 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29441 network=tcp +29:38.706 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:38.706 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29440 [Candidate]" term=2 +29:38.752 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:38.752 [DEBUG] TestAgent.server.raft: vote granted: from=c7bb8f77-8a31-72bd-4c31-66ed7c431b60 term=2 tally=1 +29:38.752 [INFO] TestAgent.server.raft: election won: tally=1 +29:38.752 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29440 [Leader]" +29:38.753 [INFO] TestAgent.server: cluster leadership acquired +29:38.753 [INFO] TestAgent.server: New leader elected: payload=Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60 +29:38.779 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29440 +29:38.801 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:38.844 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:38.844 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:38.844 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:38.844 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:38.844 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:38.844 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60 +29:38.844 [INFO] TestAgent: Synced node info +29:38.844 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7bb8f77-8a31-72bd-4c31-66ed7c431b60 +29:38.849 [INFO] TestAgent.server: federation state anti-entropy synced +29:38.940 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:38.940 [DEBUG] TestAgent: Node info in sync +29:38.940 [DEBUG] TestAgent: Node info in sync +29:38.964 [DEBUG] TestAgent: Node info in sync +29:38.980 [INFO] TestAgent: Synced service: service=test +29:38.980 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:38.980 [DEBUG] TestAgent: Check in sync: check=check_2 +29:38.984 [DEBUG] TestAgent: Node info in sync +29:38.997 [INFO] TestAgent: Synced service: service=test +29:38.997 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:38.997 [DEBUG] TestAgent: Check in sync: check=check_2 +29:38.997 [DEBUG] TestAgent: Check in sync: check=check_3 +29:39.016 [DEBUG] TestAgent: removed check: check=check_2 +29:39.016 [DEBUG] TestAgent: Node info in sync +29:39.016 [DEBUG] TestAgent: Service in sync: service=test +29:39.016 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:39.036 [INFO] TestAgent: Deregistered check: check=check_2 +29:39.036 [DEBUG] TestAgent: Check in sync: check=check_3 +29:39.036 [INFO] TestAgent: Requesting shutdown +29:39.036 [INFO] TestAgent.server: shutting down server +29:39.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.036 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.036 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.036 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.036 [DEBUG] TestAgent: Node info in sync +29:39.036 [DEBUG] TestAgent: Service in sync: service=test +29:39.036 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:39.036 [DEBUG] TestAgent: Check in sync: check=check_3 +29:39.036 [DEBUG] TestAgent: Node info in sync +29:39.036 [DEBUG] TestAgent: Service in sync: service=test +29:39.036 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:39.036 [DEBUG] TestAgent: Check in sync: check=check_3 +29:39.036 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.036 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.036 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.043 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.060Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.060Z [INFO] agent.router.manager: shutting down +29:39.060 [INFO] TestAgent: consul server down +29:39.060 [INFO] TestAgent: shutdown complete +29:39.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=tcp +29:39.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=udp +29:39.060 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29436 network=tcp +29:39.564 [INFO] TestAgent: Waiting for endpoints to shut down +29:39.564 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:38.570Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:38.629Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:38.660 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bbe865c9-6ecd-1280-a8c9-65623840b710 Address:127.0.0.1:29447}]" +29:38.660 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29447 [Follower]" leader= +29:38.661 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bbe865c9-6ecd-1280-a8c9-65623840b710.dc1 127.0.0.1 +29:38.661 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bbe865c9-6ecd-1280-a8c9-65623840b710 127.0.0.1 +2021-10-05T16:29:38.661Z [INFO] agent.router: Initializing LAN area manager +29:38.662 [INFO] TestAgent.server: Adding LAN server: server="Node-bbe865c9-6ecd-1280-a8c9-65623840b710 (Addr: tcp/127.0.0.1:29447) (DC: dc1)" +29:38.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=udp +29:38.662 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bbe865c9-6ecd-1280-a8c9-65623840b710.dc1 area=wan +29:38.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=tcp +29:38.662 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29443 network=tcp +29:38.662 [INFO] TestAgent: started state syncer +29:38.663 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29448 network=tcp +29:38.720 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:38.720 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29447 [Candidate]" term=2 +29:38.735 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:38.735 [DEBUG] TestAgent.server.raft: vote granted: from=bbe865c9-6ecd-1280-a8c9-65623840b710 term=2 tally=1 +29:38.735 [INFO] TestAgent.server.raft: election won: tally=1 +29:38.735 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29447 [Leader]" +29:38.736 [INFO] TestAgent.server: cluster leadership acquired +29:38.736 [INFO] TestAgent.server: New leader elected: payload=Node-bbe865c9-6ecd-1280-a8c9-65623840b710 +29:38.744 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29447 +29:38.758 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:38.811 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:38.811 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:38.811 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:38.811 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:38.811 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:38.811 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bbe865c9-6ecd-1280-a8c9-65623840b710 +29:38.811 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bbe865c9-6ecd-1280-a8c9-65623840b710 +29:38.840 [INFO] TestAgent.server: federation state anti-entropy synced +29:38.867 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:38.872 [INFO] TestAgent: Synced node info +29:39.118 [DEBUG] TestAgent: Node info in sync +29:39.143 [INFO] TestAgent: Synced service: service=test +29:39.143 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:39.143 [DEBUG] TestAgent: Check in sync: check=check_2 +29:39.172 [DEBUG] TestAgent: removed check: check=check_2 +29:39.172 [DEBUG] TestAgent: Node info in sync +29:39.188 [INFO] TestAgent: Synced service: service=test +29:39.188 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:39.224 [INFO] TestAgent: Deregistered check: check=check_2 +29:39.224 [DEBUG] TestAgent: Check in sync: check=check_3 +29:39.224 [INFO] TestAgent: Requesting shutdown +29:39.224 [INFO] TestAgent.server: shutting down server +29:39.224 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.224 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.224 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.224 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.224 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.224 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.224 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.225 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.225 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.225 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.238 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.252Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.252Z [INFO] agent.router.manager: shutting down +29:39.252 [INFO] TestAgent: consul server down +29:39.252 [INFO] TestAgent: shutdown complete +29:39.252 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=tcp +29:39.252 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=udp +29:39.252 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29443 network=tcp +29:39.753 [INFO] TestAgent: Waiting for endpoints to shut down +29:39.753 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ReRegister_ReplaceExistingChecks (0.00s) - --- PASS: TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/service_manager (0.88s) --- PASS: TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/normal (1.04s) + --- PASS: TestAgent_RegisterService_ReRegister_ReplaceExistingChecks/service_manager (1.23s) === RUN TestAgent_RegisterService_TranslateKeys === RUN TestAgent_RegisterService_TranslateKeys/normal === PAUSE TestAgent_RegisterService_TranslateKeys/normal @@ -6337,184 +6400,172 @@ === RUN TestAgent_RegisterService_TranslateKeys/normal/root_token === RUN TestAgent_RegisterService_TranslateKeys/service_manager/no_token === RUN TestAgent_RegisterService_TranslateKeys/service_manager/root_token -2022-11-07T22:44:39.894Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:39.894Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:39.903Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:39.903Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:39.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:74be4055-1e17-fb62-b14b-e6b0ed7e16bd Address:127.0.0.1:11076}]" -44:39.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11076 [Follower]" leader= -44:39.930 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd.dc1 127.0.0.1 -44:39.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd 127.0.0.1 -2022-11-07T22:44:39.931Z [INFO] agent.router: Initializing LAN area manager -44:39.931 [INFO] TestAgent.server: Adding LAN server: server="Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd (Addr: tcp/127.0.0.1:11076) (DC: dc1)" -44:39.931 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd.dc1 area=wan -44:39.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11071 network=udp -44:39.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11071 network=tcp -44:39.931 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11072 network=tcp -44:39.932 [INFO] TestAgent: started state syncer -44:39.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11077 network=tcp -44:39.981 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:39.989 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:39.989 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11076 [Candidate]" term=2 -44:39.999 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:39.999 [DEBUG] TestAgent.server.raft: vote granted: from=74be4055-1e17-fb62-b14b-e6b0ed7e16bd term=2 tally=1 -44:39.999 [INFO] TestAgent.server.raft: election won: tally=1 -44:39.999 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11076 [Leader]" -44:39.999 [INFO] TestAgent.server: cluster leadership acquired -44:39.999 [INFO] TestAgent.server: New leader elected: payload=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.005 [INFO] TestAgent.server: initializing acls -44:40.008 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:40.008 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:40.013 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:40.022 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:40.022 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:40.022 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:40.022 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.022 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd.dc1 -44:40.022 [INFO] TestAgent.server: Updating LAN server: server="Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd (Addr: tcp/127.0.0.1:11076) (DC: dc1)" -44:40.022 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd.dc1 area=wan -44:40.031 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:40.049 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:40.049 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:40.049 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:40.049 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:40.049 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:40.049 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.049 [INFO] TestAgent.server: member joined, marking health alive: member=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.062 [INFO] TestAgent.server: federation state anti-entropy synced -44:40.062 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.136 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.136 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-74be4055-1e17-fb62-b14b-e6b0ed7e16bd -44:40.154 [INFO] TestAgent: Synced node info -44:40.157 [INFO] TestAgent: Synced service: service=test -44:40.157 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:40.157 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:40.158 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:40.158 [INFO] TestAgent: Requesting shutdown -44:40.158 [INFO] TestAgent.server: shutting down server -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:40.158 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:40.158 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:40.158 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:40.158 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:40.158 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:40.161 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:40.164Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:40.164Z [INFO] agent.router.manager: shutting down -44:40.164 [INFO] TestAgent: consul server down -44:40.164 [INFO] TestAgent: shutdown complete -44:40.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11071 network=tcp -44:40.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11071 network=udp -44:40.164 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11072 network=tcp -44:40.664 [INFO] TestAgent: Waiting for endpoints to shut down -44:40.664 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:39.897Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:39.897Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:39.912Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:39.912Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:39.936 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2f4ef0be-c2d4-3aaf-dc68-256beff05f41 Address:127.0.0.1:11083}]" -44:39.937 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11083 [Follower]" leader= -44:39.937 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 127.0.0.1 -44:39.937 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 127.0.0.1 -2022-11-07T22:44:39.938Z [INFO] agent.router: Initializing LAN area manager -44:39.938 [INFO] TestAgent.server: Adding LAN server: server="Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 (Addr: tcp/127.0.0.1:11083) (DC: dc1)" -44:39.938 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 area=wan -44:39.938 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11078 network=tcp -44:39.938 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11078 network=udp -44:39.938 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11079 network=tcp -44:39.938 [INFO] TestAgent: started state syncer -44:39.938 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11084 network=tcp -44:39.973 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:39.973 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11083 [Candidate]" term=2 -44:39.981 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:39.981 [DEBUG] TestAgent.server.raft: vote granted: from=2f4ef0be-c2d4-3aaf-dc68-256beff05f41 term=2 tally=1 -44:39.981 [INFO] TestAgent.server.raft: election won: tally=1 -44:39.981 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11083 [Leader]" -44:39.981 [INFO] TestAgent.server: cluster leadership acquired -44:39.981 [INFO] TestAgent.server: New leader elected: payload=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:39.986 [INFO] TestAgent.server: initializing acls -44:39.988 [INFO] TestAgent.server: initializing acls -44:39.994 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:39.994 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:39.994 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:39.994 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:40.006 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:40.006 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:40.009 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:40.009 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:40.009 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:40.009 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.009 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 -44:40.009 [INFO] TestAgent.server: Updating LAN server: server="Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 (Addr: tcp/127.0.0.1:11083) (DC: dc1)" -44:40.009 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 area=wan -44:40.014 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:40.014 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:40.014 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.014 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 -44:40.014 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41.dc1 area=wan -44:40.014 [INFO] TestAgent.server: Updating LAN server: server="Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 (Addr: tcp/127.0.0.1:11083) (DC: dc1)" -44:40.028 [INFO] TestAgent: Synced node info -44:40.028 [DEBUG] TestAgent: Node info in sync -44:40.028 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:40.062 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:40.062 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:40.062 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:40.062 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:40.062 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:40.062 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.062 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.065 [INFO] TestAgent.server: federation state anti-entropy synced -44:40.070 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.070 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.178 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.178 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2f4ef0be-c2d4-3aaf-dc68-256beff05f41 -44:40.192 [DEBUG] TestAgent: Node info in sync -44:40.196 [INFO] TestAgent: Synced service: service=test -44:40.196 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:40.196 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:40.196 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:40.196 [INFO] TestAgent: Requesting shutdown -44:40.196 [INFO] TestAgent.server: shutting down server -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:40.196 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:40.196 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:40.196 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:40.196 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:40.196 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:40.196 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:40.196 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:40.197 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:40.197 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:40.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:40.203Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:40.203Z [INFO] agent.router.manager: shutting down -44:40.203 [INFO] TestAgent: consul server down -44:40.203 [INFO] TestAgent: shutdown complete -44:40.203 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11078 network=tcp -44:40.203 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11078 network=udp -44:40.203 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11079 network=tcp -44:40.703 [INFO] TestAgent: Waiting for endpoints to shut down -44:40.703 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:39.775Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:39.775Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.810Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:39.810Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.878 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3084a5d-2522-102f-fd67-09d3a28708e8 Address:127.0.0.1:29454}]" +29:39.878 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29454 [Follower]" leader= +29:39.879 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3084a5d-2522-102f-fd67-09d3a28708e8.dc1 127.0.0.1 +29:39.880 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3084a5d-2522-102f-fd67-09d3a28708e8 127.0.0.1 +2021-10-05T16:29:39.880Z [INFO] agent.router: Initializing LAN area manager +29:39.880 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=udp +29:39.880 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3084a5d-2522-102f-fd67-09d3a28708e8.dc1 area=wan +29:39.881 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=tcp +29:39.881 [INFO] TestAgent.server: Adding LAN server: server="Node-e3084a5d-2522-102f-fd67-09d3a28708e8 (Addr: tcp/127.0.0.1:29454) (DC: dc1)" +29:39.881 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29450 network=tcp +29:39.881 [INFO] TestAgent: started state syncer +29:39.881 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29455 network=tcp +29:39.929 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.929 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29454 [Candidate]" term=2 +29:39.931 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:39.948 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.948 [DEBUG] TestAgent.server.raft: vote granted: from=e3084a5d-2522-102f-fd67-09d3a28708e8 term=2 tally=1 +29:39.948 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.948 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29454 [Leader]" +29:39.951 [INFO] TestAgent.server: cluster leadership acquired +29:39.951 [INFO] TestAgent.server: New leader elected: payload=Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:39.969 [INFO] TestAgent.server: initializing acls +29:39.973 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:39.973 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:39.981 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:39.996 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:39.996 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:39.996 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:39.996 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:39.996 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e3084a5d-2522-102f-fd67-09d3a28708e8.dc1 +29:39.998 [INFO] TestAgent.server: Updating LAN server: server="Node-e3084a5d-2522-102f-fd67-09d3a28708e8 (Addr: tcp/127.0.0.1:29454) (DC: dc1)" +29:39.998 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e3084a5d-2522-102f-fd67-09d3a28708e8.dc1 area=wan +29:40.013 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.053 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.053 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.053 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.053 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.053 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.053 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:40.053 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:40.060 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:40.068 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e3084a5d-2522-102f-fd67-09d3a28708e8 +29:40.077 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.142 [INFO] TestAgent: Synced node info +29:40.159 [INFO] TestAgent: Synced service: service=test +29:40.159 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:40.159 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:40.159 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:40.160 [INFO] TestAgent: Requesting shutdown +29:40.160 [INFO] TestAgent.server: shutting down server +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.160 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.160 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.160 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.160 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:40.160 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:40.165 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.167Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.167Z [INFO] agent.router.manager: shutting down +29:40.167 [INFO] TestAgent: consul server down +29:40.167 [INFO] TestAgent: shutdown complete +29:40.167 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=tcp +29:40.167 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=udp +29:40.167 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29450 network=tcp +29:40.673 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.673 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:39.770Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:39.770Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.804Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:39.804Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.840 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f4fe148e-d4c7-990e-ce62-a1c96e643066 Address:127.0.0.1:29461}]" +29:39.841 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4fe148e-d4c7-990e-ce62-a1c96e643066.dc1 127.0.0.1 +29:39.842 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 127.0.0.1 +2021-10-05T16:29:39.842Z [INFO] agent.router: Initializing LAN area manager +29:39.842 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29456 network=udp +29:39.842 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29461 [Follower]" leader= +29:39.842 [INFO] TestAgent.server: Adding LAN server: server="Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 (Addr: tcp/127.0.0.1:29461) (DC: dc1)" +29:39.842 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066.dc1 area=wan +29:39.842 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29456 network=tcp +29:39.844 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29457 network=tcp +29:39.844 [INFO] TestAgent: started state syncer +29:39.844 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29462 network=tcp +29:39.884 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.884 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29461 [Candidate]" term=2 +29:39.893 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:39.914 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.914 [DEBUG] TestAgent.server.raft: vote granted: from=f4fe148e-d4c7-990e-ce62-a1c96e643066 term=2 tally=1 +29:39.914 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.914 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29461 [Leader]" +29:39.914 [INFO] TestAgent.server: cluster leadership acquired +29:39.914 [INFO] TestAgent.server: New leader elected: payload=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:39.916 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +29:39.921 [INFO] TestAgent.server: initializing acls +29:39.925 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:39.925 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:39.932 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:39.947 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:39.947 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:39.947 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:39.947 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:39.948 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f4fe148e-d4c7-990e-ce62-a1c96e643066.dc1 +29:39.951 [INFO] TestAgent.server: Updating LAN server: server="Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 (Addr: tcp/127.0.0.1:29461) (DC: dc1)" +29:39.951 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066.dc1 area=wan +29:39.967 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.002 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.002 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.002 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.002 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.002 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.002 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:40.002 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:40.012 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:40.036 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.247 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:40.247 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f4fe148e-d4c7-990e-ce62-a1c96e643066 +29:40.300 [INFO] TestAgent: Synced node info +29:40.325 [INFO] TestAgent: Synced service: service=test +29:40.325 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:40.325 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:40.325 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:40.325 [INFO] TestAgent: Requesting shutdown +29:40.325 [INFO] TestAgent.server: shutting down server +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:40.325 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.325 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.325 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:40.325 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:40.340 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.349Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.349Z [INFO] agent.router.manager: shutting down +29:40.349 [INFO] TestAgent: consul server down +29:40.349 [INFO] TestAgent: shutdown complete +29:40.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29456 network=tcp +29:40.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29456 network=udp +29:40.349 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29457 network=tcp +29:40.849 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.849 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_TranslateKeys (0.00s) - --- PASS: TestAgent_RegisterService_TranslateKeys/normal (0.79s) + --- PASS: TestAgent_RegisterService_TranslateKeys/normal (0.92s) --- PASS: TestAgent_RegisterService_TranslateKeys/normal/no_token (0.00s) - --- PASS: TestAgent_RegisterService_TranslateKeys/normal/root_token (0.02s) - --- PASS: TestAgent_RegisterService_TranslateKeys/service_manager (0.83s) + --- PASS: TestAgent_RegisterService_TranslateKeys/normal/root_token (0.08s) + --- PASS: TestAgent_RegisterService_TranslateKeys/service_manager (1.10s) --- PASS: TestAgent_RegisterService_TranslateKeys/service_manager/no_token (0.00s) - --- PASS: TestAgent_RegisterService_TranslateKeys/service_manager/root_token (0.02s) + --- PASS: TestAgent_RegisterService_TranslateKeys/service_manager/root_token (0.08s) === RUN TestAgent_RegisterService_ACLDeny === RUN TestAgent_RegisterService_ACLDeny/normal === PAUSE TestAgent_RegisterService_ACLDeny/normal @@ -6526,175 +6577,182 @@ === RUN TestAgent_RegisterService_ACLDeny/normal/root_token === RUN TestAgent_RegisterService_ACLDeny/service_manager/no_token === RUN TestAgent_RegisterService_ACLDeny/service_manager/root_token -2022-11-07T22:44:40.719Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:40.719Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:40.732Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:40.732Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:40.755 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3539b934-3c9f-ce7c-58d1-e2823fe83b66 Address:127.0.0.1:11090}]" -44:40.755 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11090 [Follower]" leader= -44:40.756 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66.dc1 127.0.0.1 -44:40.757 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 127.0.0.1 -2022-11-07T22:44:40.757Z [INFO] agent.router: Initializing LAN area manager -44:40.758 [INFO] TestAgent.server: Adding LAN server: server="Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 (Addr: tcp/127.0.0.1:11090) (DC: dc1)" -44:40.758 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66.dc1 area=wan -44:40.758 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11085 network=tcp -44:40.759 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11085 network=udp -44:40.760 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11086 network=tcp -44:40.760 [INFO] TestAgent: started state syncer -44:40.760 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11091 network=tcp -44:40.808 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:40.811 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:40.811 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11090 [Candidate]" term=2 -44:40.818 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:40.818 [DEBUG] TestAgent.server.raft: vote granted: from=3539b934-3c9f-ce7c-58d1-e2823fe83b66 term=2 tally=1 -44:40.818 [INFO] TestAgent.server.raft: election won: tally=1 -44:40.818 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11090 [Leader]" -44:40.818 [INFO] TestAgent.server: cluster leadership acquired -44:40.819 [INFO] TestAgent.server: New leader elected: payload=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:40.825 [INFO] TestAgent.server: initializing acls -44:40.828 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:40.828 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:40.834 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:40.845 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:40.845 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:40.845 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:40.845 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:40.845 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66.dc1 -44:40.845 [INFO] TestAgent.server: Updating LAN server: server="Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 (Addr: tcp/127.0.0.1:11090) (DC: dc1)" -44:40.845 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66.dc1 area=wan -44:40.857 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:40.873 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:40.873 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:40.873 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:40.873 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:40.873 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:40.873 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:40.873 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:40.879 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:40.884 [INFO] TestAgent.server: federation state anti-entropy synced -44:41.077 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:41.081 [INFO] TestAgent: Synced node info -44:41.124 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:41.124 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3539b934-3c9f-ce7c-58d1-e2823fe83b66 -44:41.141 [DEBUG] TestAgent: Node info in sync -44:41.145 [INFO] TestAgent: Synced service: service=test -44:41.145 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:41.145 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:41.145 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:41.145 [INFO] TestAgent: Requesting shutdown -44:41.145 [INFO] TestAgent.server: shutting down server -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:41.145 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:41.145 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:41.145 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:41.145 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:41.149 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:41.153Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:41.153Z [INFO] agent.router.manager: shutting down -44:41.153 [INFO] TestAgent: consul server down -44:41.153 [INFO] TestAgent: shutdown complete -44:41.153 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11085 network=tcp -44:41.153 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11085 network=udp -44:41.153 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11086 network=tcp -44:41.654 [INFO] TestAgent: Waiting for endpoints to shut down -44:41.654 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:40.719Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:40.719Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:40.732Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:40.732Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:40.753 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:81c360bf-39f7-5522-d41a-3f5857ed4bb7 Address:127.0.0.1:11097}]" -44:40.753 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11097 [Follower]" leader= -44:40.754 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7.dc1 127.0.0.1 -44:40.754 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 127.0.0.1 -2022-11-07T22:44:40.754Z [INFO] agent.router: Initializing LAN area manager -44:40.754 [INFO] TestAgent.server: Adding LAN server: server="Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 (Addr: tcp/127.0.0.1:11097) (DC: dc1)" -44:40.754 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7.dc1 area=wan -44:40.754 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11092 network=udp -44:40.754 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11092 network=tcp -44:40.755 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11093 network=tcp -44:40.755 [INFO] TestAgent: started state syncer -44:40.755 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11098 network=tcp -44:40.804 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:40.811 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:40.811 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11097 [Candidate]" term=2 -44:40.819 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:40.819 [DEBUG] TestAgent.server.raft: vote granted: from=81c360bf-39f7-5522-d41a-3f5857ed4bb7 term=2 tally=1 -44:40.819 [INFO] TestAgent.server.raft: election won: tally=1 -44:40.819 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11097 [Leader]" -44:40.819 [INFO] TestAgent.server: cluster leadership acquired -44:40.819 [INFO] TestAgent.server: New leader elected: payload=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:40.825 [INFO] TestAgent.server: initializing acls -44:40.829 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:40.829 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:40.836 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:40.847 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:40.847 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:40.847 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:40.847 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:40.847 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7.dc1 -44:40.847 [INFO] TestAgent.server: Updating LAN server: server="Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 (Addr: tcp/127.0.0.1:11097) (DC: dc1)" -44:40.847 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7.dc1 area=wan -44:40.857 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:40.874 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:40.874 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:40.874 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:40.874 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:40.874 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:40.874 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:40.874 [INFO] TestAgent.server: member joined, marking health alive: member=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:40.880 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:40.886 [INFO] TestAgent.server: federation state anti-entropy synced -44:41.082 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:41.087 [INFO] TestAgent: Synced node info -44:41.141 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:41.141 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-81c360bf-39f7-5522-d41a-3f5857ed4bb7 -44:41.159 [DEBUG] TestAgent: Node info in sync -44:41.162 [INFO] TestAgent: Synced service: service=test -44:41.162 [DEBUG] TestAgent: Check in sync: check=service:test:1 -44:41.162 [DEBUG] TestAgent: Check in sync: check=service:test:2 -44:41.162 [DEBUG] TestAgent: Check in sync: check=service:test:3 -44:41.162 [INFO] TestAgent: Requesting shutdown -44:41.162 [INFO] TestAgent.server: shutting down server -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:41.162 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:41.162 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:41.162 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:41.162 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:41.165 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:41.168Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:41.168Z [INFO] agent.router.manager: shutting down -44:41.168 [INFO] TestAgent: consul server down -44:41.168 [INFO] TestAgent: shutdown complete -44:41.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11092 network=tcp -44:41.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11092 network=udp -44:41.168 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11093 network=tcp -44:41.668 [INFO] TestAgent: Waiting for endpoints to shut down -44:41.668 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:40.862Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.862Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.926Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.926Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.979 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eddde1dc-2628-cafe-6c2c-6e1e8013ee55 Address:127.0.0.1:29468}]" +29:40.979 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55.dc1 127.0.0.1 +29:40.980 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29468 [Follower]" leader= +29:40.980 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 127.0.0.1 +2021-10-05T16:29:40.980Z [INFO] agent.router: Initializing LAN area manager +29:40.981 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29463 network=udp +29:40.981 [INFO] TestAgent.server: Adding LAN server: server="Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 (Addr: tcp/127.0.0.1:29468) (DC: dc1)" +29:40.981 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55.dc1 area=wan +29:40.981 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29463 network=tcp +29:40.981 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29464 network=tcp +29:40.981 [INFO] TestAgent: started state syncer +29:40.982 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29469 network=tcp +29:41.031 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:41.040 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:41.040 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29468 [Candidate]" term=2 +29:41.066 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:41.066 [DEBUG] TestAgent.server.raft: vote granted: from=eddde1dc-2628-cafe-6c2c-6e1e8013ee55 term=2 tally=1 +29:41.066 [INFO] TestAgent.server.raft: election won: tally=1 +29:41.066 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29468 [Leader]" +29:41.068 [INFO] TestAgent.server: cluster leadership acquired +29:41.068 [INFO] TestAgent.server: New leader elected: payload=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.078 [INFO] TestAgent.server: initializing acls +29:41.083 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:41.083 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:41.095 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:41.115 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:41.115 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:41.115 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:41.115 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.115 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55.dc1 +29:41.115 [INFO] TestAgent.server: Updating LAN server: server="Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 (Addr: tcp/127.0.0.1:29468) (DC: dc1)" +29:41.115 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55.dc1 area=wan +29:41.130 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:41.171 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:41.171 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:41.171 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:41.171 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:41.171 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:41.171 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.171 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.197 [INFO] TestAgent: Synced node info +29:41.197 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.206 [INFO] TestAgent.server: federation state anti-entropy synced +29:41.211 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-eddde1dc-2628-cafe-6c2c-6e1e8013ee55 +29:41.237 [DEBUG] TestAgent: Node info in sync +29:41.242 [INFO] TestAgent: Synced service: service=test +29:41.242 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:41.242 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:41.242 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:41.242 [INFO] TestAgent: Requesting shutdown +29:41.242 [INFO] TestAgent.server: shutting down server +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:41.242 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:41.242 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:41.242 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:41.242 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:41.246 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:41.259Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:41.259Z [INFO] agent.router.manager: shutting down +29:41.259 [INFO] TestAgent: consul server down +29:41.259 [INFO] TestAgent: shutdown complete +29:41.259 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29463 network=tcp +29:41.259 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29463 network=udp +29:41.259 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29464 network=tcp +29:41.760 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.760 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:40.924Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.924Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.945Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.945Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.980 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dbd29818-72b6-1529-2a0f-448c670d20f5 Address:127.0.0.1:29475}]" +29:40.980 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29475 [Follower]" leader= +29:40.981 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dbd29818-72b6-1529-2a0f-448c670d20f5.dc1 127.0.0.1 +29:40.981 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dbd29818-72b6-1529-2a0f-448c670d20f5 127.0.0.1 +2021-10-05T16:29:40.981Z [INFO] agent.router: Initializing LAN area manager +29:40.982 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29470 network=udp +29:40.982 [INFO] TestAgent.server: Adding LAN server: server="Node-dbd29818-72b6-1529-2a0f-448c670d20f5 (Addr: tcp/127.0.0.1:29475) (DC: dc1)" +29:40.982 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dbd29818-72b6-1529-2a0f-448c670d20f5.dc1 area=wan +29:40.983 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29470 network=tcp +29:40.983 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29471 network=tcp +29:40.983 [INFO] TestAgent: started state syncer +29:40.983 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29476 network=tcp +29:41.026 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:41.026 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29475 [Candidate]" term=2 +29:41.034 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:41.054 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:41.054 [DEBUG] TestAgent.server.raft: vote granted: from=dbd29818-72b6-1529-2a0f-448c670d20f5 term=2 tally=1 +29:41.054 [INFO] TestAgent.server.raft: election won: tally=1 +29:41.054 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29475 [Leader]" +29:41.055 [INFO] TestAgent.server: cluster leadership acquired +29:41.055 [INFO] TestAgent.server: New leader elected: payload=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.065 [INFO] TestAgent.server: initializing acls +29:41.070 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:41.070 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:41.082 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:41.096 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:41.096 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:41.096 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:41.097 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.097 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dbd29818-72b6-1529-2a0f-448c670d20f5.dc1 +29:41.097 [INFO] TestAgent.server: Updating LAN server: server="Node-dbd29818-72b6-1529-2a0f-448c670d20f5 (Addr: tcp/127.0.0.1:29475) (DC: dc1)" +29:41.097 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dbd29818-72b6-1529-2a0f-448c670d20f5.dc1 area=wan +29:41.122 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:41.187 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:41.187 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:41.187 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:41.187 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:41.187 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:41.187 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.187 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.199 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.207 [INFO] TestAgent.server: federation state anti-entropy synced +29:41.290 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.290 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-dbd29818-72b6-1529-2a0f-448c670d20f5 +29:41.324 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:41.337 [INFO] TestAgent: Synced node info +29:41.341 [INFO] TestAgent: Synced service: service=test +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:41.341 [DEBUG] TestAgent: Node info in sync +29:41.341 [DEBUG] TestAgent: Service in sync: service=test +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:41.341 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:41.342 [DEBUG] TestAgent: Node info in sync +29:41.342 [DEBUG] TestAgent: Service in sync: service=test +29:41.342 [DEBUG] TestAgent: Check in sync: check=service:test:2 +29:41.342 [DEBUG] TestAgent: Check in sync: check=service:test:3 +29:41.342 [DEBUG] TestAgent: Check in sync: check=service:test:1 +29:41.342 [INFO] TestAgent: Requesting shutdown +29:41.342 [INFO] TestAgent.server: shutting down server +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:41.342 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:41.342 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:41.342 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:41.342 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:41.346 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:41.350Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:41.350Z [INFO] agent.router.manager: shutting down +29:41.351 [INFO] TestAgent: consul server down +29:41.351 [INFO] TestAgent: shutdown complete +29:41.351 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29470 network=tcp +29:41.351 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29470 network=udp +29:41.351 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29471 network=tcp +29:41.852 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.852 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ACLDeny (0.00s) - --- PASS: TestAgent_RegisterService_ACLDeny/normal (0.95s) + --- PASS: TestAgent_RegisterService_ACLDeny/normal (0.91s) --- PASS: TestAgent_RegisterService_ACLDeny/normal/no_token (0.00s) - --- PASS: TestAgent_RegisterService_ACLDeny/normal/root_token (0.02s) - --- PASS: TestAgent_RegisterService_ACLDeny/service_manager (0.97s) + --- PASS: TestAgent_RegisterService_ACLDeny/normal/root_token (0.03s) + --- PASS: TestAgent_RegisterService_ACLDeny/service_manager (1.00s) --- PASS: TestAgent_RegisterService_ACLDeny/service_manager/no_token (0.00s) - --- PASS: TestAgent_RegisterService_ACLDeny/service_manager/root_token (0.02s) + --- PASS: TestAgent_RegisterService_ACLDeny/service_manager/root_token (0.05s) === RUN TestAgent_RegisterService_InvalidAddress === RUN TestAgent_RegisterService_InvalidAddress/normal === PAUSE TestAgent_RegisterService_InvalidAddress/normal @@ -6705,129 +6763,130 @@ === RUN TestAgent_RegisterService_InvalidAddress/service_manager/addr_0.0.0.0 === RUN TestAgent_RegisterService_InvalidAddress/service_manager/addr_:: === RUN TestAgent_RegisterService_InvalidAddress/service_manager/addr_[::] -2022-11-07T22:44:41.686Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:41.700Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:41.727 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cc338cf8-5e59-be28-9965-3cd93b0345e1 Address:127.0.0.1:11104}]" -44:41.727 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11104 [Follower]" leader= -44:41.727 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cc338cf8-5e59-be28-9965-3cd93b0345e1.dc1 127.0.0.1 -44:41.728 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cc338cf8-5e59-be28-9965-3cd93b0345e1 127.0.0.1 -2022-11-07T22:44:41.728Z [INFO] agent.router: Initializing LAN area manager -44:41.730 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cc338cf8-5e59-be28-9965-3cd93b0345e1.dc1 area=wan -44:41.730 [INFO] TestAgent.server: Adding LAN server: server="Node-cc338cf8-5e59-be28-9965-3cd93b0345e1 (Addr: tcp/127.0.0.1:11104) (DC: dc1)" -44:41.730 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11099 network=udp -44:41.730 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11099 network=tcp -44:41.731 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11100 network=tcp -44:41.731 [INFO] TestAgent: started state syncer -44:41.731 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11105 network=tcp -44:41.786 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:41.786 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11104 [Candidate]" term=2 -44:41.796 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:41.796 [DEBUG] TestAgent.server.raft: vote granted: from=cc338cf8-5e59-be28-9965-3cd93b0345e1 term=2 tally=1 -44:41.796 [INFO] TestAgent.server.raft: election won: tally=1 -44:41.796 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11104 [Leader]" -44:41.796 [INFO] TestAgent.server: cluster leadership acquired -44:41.796 [INFO] TestAgent.server: New leader elected: payload=Node-cc338cf8-5e59-be28-9965-3cd93b0345e1 -44:41.801 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11104 -44:41.822 [INFO] TestAgent: Synced node info -44:41.822 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:41.822 [DEBUG] TestAgent: Node info in sync -44:41.835 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:41.835 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:41.835 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:41.835 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:41.835 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:41.835 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cc338cf8-5e59-be28-9965-3cd93b0345e1 -44:41.835 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cc338cf8-5e59-be28-9965-3cd93b0345e1 -44:41.844 [INFO] TestAgent.server: federation state anti-entropy synced -44:42.050 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.050 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.050 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.050 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.053 [DEBUG] TestAgent: Node info in sync -44:42.059 [INFO] TestAgent: Synced service: service=connect-proxy -44:42.059 [INFO] TestAgent: Requesting shutdown -44:42.059 [INFO] TestAgent.server: shutting down server -44:42.059 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:42.060 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:42.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:42.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:42.060 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:42.060 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:42.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:42.060 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:42.060 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:42.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:42.063 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:42.067Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:42.067Z [INFO] agent.router.manager: shutting down -44:42.067 [INFO] TestAgent: consul server down -44:42.067 [INFO] TestAgent: shutdown complete -44:42.067 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11099 network=tcp -44:42.067 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11099 network=udp -44:42.067 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11100 network=tcp -44:42.567 [INFO] TestAgent: Waiting for endpoints to shut down -44:42.567 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:41.682Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:41.698Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:41.725 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab Address:127.0.0.1:11111}]" -44:41.725 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11111 [Follower]" leader= -44:41.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab.dc1 127.0.0.1 -44:41.726 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab 127.0.0.1 -2022-11-07T22:44:41.726Z [INFO] agent.router: Initializing LAN area manager -44:41.726 [INFO] TestAgent.server: Adding LAN server: server="Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab (Addr: tcp/127.0.0.1:11111) (DC: dc1)" -44:41.726 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab.dc1 area=wan -44:41.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11106 network=udp -44:41.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11106 network=tcp -44:41.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11107 network=tcp -44:41.727 [INFO] TestAgent: started state syncer -44:41.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11112 network=tcp -44:41.774 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:41.774 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11111 [Candidate]" term=2 -44:41.784 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:41.784 [DEBUG] TestAgent.server.raft: vote granted: from=07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab term=2 tally=1 -44:41.784 [INFO] TestAgent.server.raft: election won: tally=1 -44:41.784 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11111 [Leader]" -44:41.784 [INFO] TestAgent.server: cluster leadership acquired -44:41.785 [INFO] TestAgent.server: New leader elected: payload=Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab -44:41.790 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11111 -44:41.802 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:41.820 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:41.820 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:41.820 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:41.820 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:41.820 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:41.820 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab -44:41.820 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07eb1d9f-24a1-e86d-ea8d-f7b2e50ef2ab -44:41.826 [INFO] TestAgent.server: federation state anti-entropy synced -44:42.074 [INFO] TestAgent: Requesting shutdown -44:42.074 [INFO] TestAgent.server: shutting down server -44:42.075 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:42.075 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:42.075 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:42.075 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:42.075 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:42.075 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:42.075 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:42.075 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:42.075 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:42.075 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:42.075 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:42.078 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:42.081Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:42.081Z [INFO] agent.router.manager: shutting down -44:42.081 [INFO] TestAgent: consul server down -44:42.081 [INFO] TestAgent: shutdown complete -44:42.081 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11106 network=tcp -44:42.081 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11106 network=udp -44:42.081 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11107 network=tcp -44:42.582 [INFO] TestAgent: Waiting for endpoints to shut down -44:42.582 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:41.883Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:41.904Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:41.945 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a9a19f34-fda2-008b-fc1b-b6e5871e3805 Address:127.0.0.1:29489}]" +29:41.946 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805.dc1 127.0.0.1 +29:41.946 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805 127.0.0.1 +2021-10-05T16:29:41.947Z [INFO] agent.router: Initializing LAN area manager +29:41.947 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29484 network=udp +29:41.947 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29489 [Follower]" leader= +29:41.947 [INFO] TestAgent.server: Adding LAN server: server="Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805 (Addr: tcp/127.0.0.1:29489) (DC: dc1)" +29:41.947 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805.dc1 area=wan +29:41.947 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29484 network=tcp +29:41.948 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29485 network=tcp +29:41.948 [INFO] TestAgent: started state syncer +29:41.948 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29490 network=tcp +29:42.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:42.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29489 [Candidate]" term=2 +29:42.024 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:42.024 [DEBUG] TestAgent.server.raft: vote granted: from=a9a19f34-fda2-008b-fc1b-b6e5871e3805 term=2 tally=1 +29:42.024 [INFO] TestAgent.server.raft: election won: tally=1 +29:42.024 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29489 [Leader]" +29:42.024 [INFO] TestAgent.server: cluster leadership acquired +29:42.025 [INFO] TestAgent.server: New leader elected: payload=Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805 +29:42.032 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29489 +29:42.072 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:42.110 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:42.110 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:42.110 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:42.110 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:42.110 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:42.110 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805 +29:42.110 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a9a19f34-fda2-008b-fc1b-b6e5871e3805 +29:42.118 [INFO] TestAgent.server: federation state anti-entropy synced +29:42.126 [INFO] TestAgent: Requesting shutdown +29:42.126 [INFO] TestAgent.server: shutting down server +29:42.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:42.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:42.126 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:42.126 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:42.126 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:42.126 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:42.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:42.126 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:42.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:42.126 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:42.126 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:42.130 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:42.134Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:42.134Z [INFO] agent.router.manager: shutting down +29:42.135 [INFO] TestAgent: consul server down +29:42.135 [INFO] TestAgent: shutdown complete +29:42.135 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29484 network=tcp +29:42.135 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29484 network=udp +29:42.135 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29485 network=tcp +29:42.636 [INFO] TestAgent: Waiting for endpoints to shut down +29:42.636 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:41.881Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:41.901Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:41.941 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dada5e18-0f62-adb5-3050-138ed4909205 Address:127.0.0.1:29482}]" +29:41.941 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dada5e18-0f62-adb5-3050-138ed4909205.dc1 127.0.0.1 +29:41.942 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dada5e18-0f62-adb5-3050-138ed4909205 127.0.0.1 +2021-10-05T16:29:41.942Z [INFO] agent.router: Initializing LAN area manager +29:41.942 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29477 network=udp +29:41.942 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29482 [Follower]" leader= +29:41.942 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dada5e18-0f62-adb5-3050-138ed4909205.dc1 area=wan +29:41.942 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29477 network=tcp +29:41.943 [INFO] TestAgent.server: Adding LAN server: server="Node-dada5e18-0f62-adb5-3050-138ed4909205 (Addr: tcp/127.0.0.1:29482) (DC: dc1)" +29:41.943 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29478 network=tcp +29:41.943 [INFO] TestAgent: started state syncer +29:41.943 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29483 network=tcp +29:41.988 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:41.988 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29482 [Candidate]" term=2 +29:41.998 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:41.998 [DEBUG] TestAgent.server.raft: vote granted: from=dada5e18-0f62-adb5-3050-138ed4909205 term=2 tally=1 +29:41.998 [INFO] TestAgent.server.raft: election won: tally=1 +29:41.998 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29482 [Leader]" +29:41.998 [INFO] TestAgent.server: cluster leadership acquired +29:41.999 [INFO] TestAgent.server: New leader elected: payload=Node-dada5e18-0f62-adb5-3050-138ed4909205 +29:42.005 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29482 +29:42.033 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:42.058 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:42.058 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:42.058 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:42.058 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:42.058 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:42.058 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dada5e18-0f62-adb5-3050-138ed4909205 +29:42.058 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dada5e18-0f62-adb5-3050-138ed4909205 +29:42.080 [INFO] TestAgent.server: federation state anti-entropy synced +29:42.196 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:42.212 [INFO] TestAgent: Synced node info +29:42.212 [DEBUG] TestAgent: Node info in sync +29:42.348 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:42.348 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:42.348 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:42.348 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:42.356 [DEBUG] TestAgent: Node info in sync +29:42.388 [INFO] TestAgent: Synced service: service=connect-proxy +29:42.388 [INFO] TestAgent: Requesting shutdown +29:42.388 [INFO] TestAgent.server: shutting down server +29:42.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:42.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:42.388 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:42.388 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:42.388 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:42.388 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:42.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:42.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:42.388 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:42.388 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:42.396 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:42.401Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:42.401Z [INFO] agent.router.manager: shutting down +29:42.402 [INFO] TestAgent: consul server down +29:42.402 [INFO] TestAgent: shutdown complete +29:42.402 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29477 network=tcp +29:42.402 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29477 network=udp +29:42.402 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29478 network=tcp +29:42.904 [INFO] TestAgent: Waiting for endpoints to shut down +29:42.904 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_InvalidAddress (0.00s) - --- PASS: TestAgent_RegisterService_InvalidAddress/normal (0.90s) - --- PASS: TestAgent_RegisterService_InvalidAddress/service_manager (0.91s) + --- PASS: TestAgent_RegisterService_InvalidAddress/service_manager (0.78s) --- PASS: TestAgent_RegisterService_InvalidAddress/service_manager/addr_0.0.0.0 (0.00s) --- PASS: TestAgent_RegisterService_InvalidAddress/service_manager/addr_:: (0.00s) --- PASS: TestAgent_RegisterService_InvalidAddress/service_manager/addr_[::] (0.00s) + --- PASS: TestAgent_RegisterService_InvalidAddress/normal (1.05s) === RUN TestAgent_RegisterService_UnmanagedConnectProxy === RUN TestAgent_RegisterService_UnmanagedConnectProxy/normal === PAUSE TestAgent_RegisterService_UnmanagedConnectProxy/normal @@ -6835,2290 +6894,2289 @@ === PAUSE TestAgent_RegisterService_UnmanagedConnectProxy/service_manager === CONT TestAgent_RegisterService_UnmanagedConnectProxy/normal === CONT TestAgent_RegisterService_UnmanagedConnectProxy/service_manager -2022-11-07T22:44:42.598Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:42.611Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:42.644 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b7f824b-5181-b9c9-e69d-9475960e428c Address:127.0.0.1:11125}]" -44:42.644 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11125 [Follower]" leader= -44:42.645 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b7f824b-5181-b9c9-e69d-9475960e428c.dc1 127.0.0.1 -44:42.646 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b7f824b-5181-b9c9-e69d-9475960e428c 127.0.0.1 -2022-11-07T22:44:42.646Z [INFO] agent.router: Initializing LAN area manager -44:42.648 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b7f824b-5181-b9c9-e69d-9475960e428c.dc1 area=wan -44:42.648 [INFO] TestAgent.server: Adding LAN server: server="Node-8b7f824b-5181-b9c9-e69d-9475960e428c (Addr: tcp/127.0.0.1:11125) (DC: dc1)" -44:42.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11120 network=udp -44:42.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11120 network=tcp -44:42.649 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11121 network=tcp -44:42.649 [INFO] TestAgent: started state syncer -44:42.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11126 network=tcp -44:42.696 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:42.696 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11125 [Candidate]" term=2 -44:42.704 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:42.704 [DEBUG] TestAgent.server.raft: vote granted: from=8b7f824b-5181-b9c9-e69d-9475960e428c term=2 tally=1 -44:42.704 [INFO] TestAgent.server.raft: election won: tally=1 -44:42.704 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11125 [Leader]" -44:42.704 [INFO] TestAgent.server: cluster leadership acquired -44:42.704 [INFO] TestAgent.server: New leader elected: payload=Node-8b7f824b-5181-b9c9-e69d-9475960e428c -44:42.709 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11125 -44:42.721 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:42.741 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:42.741 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:42.741 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:42.741 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:42.741 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:42.741 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b7f824b-5181-b9c9-e69d-9475960e428c -44:42.741 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b7f824b-5181-b9c9-e69d-9475960e428c -44:42.752 [INFO] TestAgent.server: federation state anti-entropy synced -44:42.842 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.842 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.842 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.842 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.858 [INFO] TestAgent: Synced node info -44:42.863 [INFO] TestAgent: Synced service: service=connect-proxy -44:42.863 [INFO] TestAgent: Requesting shutdown -44:42.863 [INFO] TestAgent.server: shutting down server -44:42.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:42.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:42.863 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:42.863 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:42.863 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:42.863 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:42.863 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:42.863 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:42.863 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:42.867 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -44:42.867 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:42.869 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:42.875Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:42.875Z [INFO] agent.router.manager: shutting down -44:42.875 [INFO] TestAgent: consul server down -44:42.875 [INFO] TestAgent: shutdown complete -44:42.875 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11120 network=tcp -44:42.875 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11120 network=udp -44:42.875 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11121 network=tcp -44:43.375 [INFO] TestAgent: Waiting for endpoints to shut down -44:43.375 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:42.593Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:42.608Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:42.646 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:59c4f4ae-21df-1a7d-c8c3-e884ad088b2e Address:127.0.0.1:11118}]" -44:42.648 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11118 [Follower]" leader= -44:42.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e.dc1 127.0.0.1 -44:42.650 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e 127.0.0.1 -2022-11-07T22:44:42.650Z [INFO] agent.router: Initializing LAN area manager -44:42.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11113 network=udp -44:42.650 [INFO] TestAgent.server: Adding LAN server: server="Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e (Addr: tcp/127.0.0.1:11118) (DC: dc1)" -44:42.650 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e.dc1 area=wan -44:42.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11113 network=tcp -44:42.651 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11114 network=tcp -44:42.651 [INFO] TestAgent: started state syncer -44:42.651 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11119 network=tcp -44:42.718 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:42.718 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11118 [Candidate]" term=2 -44:42.730 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:42.730 [DEBUG] TestAgent.server.raft: vote granted: from=59c4f4ae-21df-1a7d-c8c3-e884ad088b2e term=2 tally=1 -44:42.730 [INFO] TestAgent.server.raft: election won: tally=1 -44:42.730 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11118 [Leader]" -44:42.730 [INFO] TestAgent.server: cluster leadership acquired -44:42.730 [INFO] TestAgent.server: New leader elected: payload=Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e -44:42.739 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11118 -44:42.739 [INFO] TestAgent: Synced node info -44:42.739 [DEBUG] TestAgent: Node info in sync -44:42.758 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:42.771 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:42.771 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:42.771 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:42.771 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:42.771 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:42.772 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e -44:42.772 [INFO] TestAgent.server: member joined, marking health alive: member=Node-59c4f4ae-21df-1a7d-c8c3-e884ad088b2e -44:42.780 [INFO] TestAgent.server: federation state anti-entropy synced -44:42.861 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.861 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.861 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.861 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:42.865 [DEBUG] TestAgent: Node info in sync -44:42.875 [INFO] TestAgent: Synced service: service=connect-proxy -44:42.875 [INFO] TestAgent: Requesting shutdown -44:42.875 [INFO] TestAgent.server: shutting down server -44:42.875 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:42.875 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:42.875 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:42.875 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:42.875 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:42.875 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:42.875 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:42.875 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:42.877 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -44:42.877 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:42.879 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:42.882Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:42.882Z [INFO] agent.router.manager: shutting down -44:42.882 [INFO] TestAgent: consul server down -44:42.882 [INFO] TestAgent: shutdown complete -44:42.882 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11113 network=tcp -44:42.882 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11113 network=udp -44:42.882 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11114 network=tcp -44:43.382 [INFO] TestAgent: Waiting for endpoints to shut down -44:43.382 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:42.966Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:43.001Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:43.134 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:aeecbdb6-6875-6ec6-e514-6fa25162a905 Address:127.0.0.1:29503}]" +29:43.135 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-aeecbdb6-6875-6ec6-e514-6fa25162a905.dc1 127.0.0.1 +29:43.135 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-aeecbdb6-6875-6ec6-e514-6fa25162a905 127.0.0.1 +2021-10-05T16:29:43.135Z [INFO] agent.router: Initializing LAN area manager +29:43.135 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29498 network=udp +29:43.135 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29503 [Follower]" leader= +29:43.136 [INFO] TestAgent.server: Adding LAN server: server="Node-aeecbdb6-6875-6ec6-e514-6fa25162a905 (Addr: tcp/127.0.0.1:29503) (DC: dc1)" +29:43.136 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-aeecbdb6-6875-6ec6-e514-6fa25162a905.dc1 area=wan +29:43.136 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29498 network=tcp +29:43.144 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29499 network=tcp +29:43.144 [INFO] TestAgent: started state syncer +29:43.144 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29504 network=tcp +29:43.173 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:43.173 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29503 [Candidate]" term=2 +29:43.186 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:43.186 [DEBUG] TestAgent.server.raft: vote granted: from=aeecbdb6-6875-6ec6-e514-6fa25162a905 term=2 tally=1 +29:43.186 [INFO] TestAgent.server.raft: election won: tally=1 +29:43.186 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29503 [Leader]" +29:43.188 [INFO] TestAgent.server: cluster leadership acquired +29:43.188 [INFO] TestAgent.server: New leader elected: payload=Node-aeecbdb6-6875-6ec6-e514-6fa25162a905 +29:43.197 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29503 +29:43.214 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:43.243 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:43.243 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:43.243 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:43.243 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:43.243 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:43.243 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-aeecbdb6-6875-6ec6-e514-6fa25162a905 +29:43.243 [INFO] TestAgent.server: member joined, marking health alive: member=Node-aeecbdb6-6875-6ec6-e514-6fa25162a905 +29:43.262 [INFO] TestAgent.server: federation state anti-entropy synced +29:43.264 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.265 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.265 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.265 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.273 [INFO] TestAgent: Synced node info +29:43.288 [INFO] TestAgent: Synced service: service=connect-proxy +29:43.288 [INFO] TestAgent: Requesting shutdown +29:43.288 [INFO] TestAgent.server: shutting down server +29:43.288 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:43.288 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:43.288 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:43.288 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:43.288 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:43.288 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:43.288 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:43.289 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:43.289 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:43.289 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:43.289 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:43.296 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:43.304Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:43.304Z [INFO] agent.router.manager: shutting down +29:43.304 [INFO] TestAgent: consul server down +29:43.304 [INFO] TestAgent: shutdown complete +29:43.304 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29498 network=tcp +29:43.304 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29498 network=udp +29:43.304 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29499 network=tcp +29:43.808 [INFO] TestAgent: Waiting for endpoints to shut down +29:43.808 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:42.972Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:43.005Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:43.121 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7e35369e-7b77-0caa-25ac-31a25a08498e Address:127.0.0.1:29496}]" +29:43.121 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29496 [Follower]" leader= +29:43.122 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7e35369e-7b77-0caa-25ac-31a25a08498e.dc1 127.0.0.1 +29:43.122 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7e35369e-7b77-0caa-25ac-31a25a08498e 127.0.0.1 +2021-10-05T16:29:43.122Z [INFO] agent.router: Initializing LAN area manager +29:43.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29491 network=udp +29:43.123 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7e35369e-7b77-0caa-25ac-31a25a08498e.dc1 area=wan +29:43.123 [INFO] TestAgent.server: Adding LAN server: server="Node-7e35369e-7b77-0caa-25ac-31a25a08498e (Addr: tcp/127.0.0.1:29496) (DC: dc1)" +29:43.123 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29491 network=tcp +29:43.123 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29492 network=tcp +29:43.123 [INFO] TestAgent: started state syncer +29:43.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29497 network=tcp +29:43.178 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:43.178 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29496 [Candidate]" term=2 +29:43.194 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:43.194 [DEBUG] TestAgent.server.raft: vote granted: from=7e35369e-7b77-0caa-25ac-31a25a08498e term=2 tally=1 +29:43.194 [INFO] TestAgent.server.raft: election won: tally=1 +29:43.194 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29496 [Leader]" +29:43.196 [INFO] TestAgent.server: cluster leadership acquired +29:43.196 [INFO] TestAgent.server: New leader elected: payload=Node-7e35369e-7b77-0caa-25ac-31a25a08498e +29:43.203 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29496 +29:43.223 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:43.259 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:43.259 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:43.259 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:43.259 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:43.259 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:43.260 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7e35369e-7b77-0caa-25ac-31a25a08498e +29:43.260 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7e35369e-7b77-0caa-25ac-31a25a08498e +29:43.291 [INFO] TestAgent.server: federation state anti-entropy synced +29:43.301 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.302 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.302 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.302 [ERROR] TestAgent.proxycfg: watch error: service_id=connect-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:43.315 [INFO] TestAgent: Synced node info +29:43.321 [INFO] TestAgent: Synced service: service=connect-proxy +29:43.321 [INFO] TestAgent: Requesting shutdown +29:43.321 [INFO] TestAgent.server: shutting down server +29:43.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:43.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:43.321 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:43.321 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:43.321 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:43.321 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:43.321 [DEBUG] TestAgent: Node info in sync +29:43.321 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:43.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:43.322 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:43.327 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +29:43.329 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +29:43.329 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:43.329 [INFO] TestAgent: Synced service: service=connect-proxy +2021-10-05T16:29:43.332Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:43.332Z [INFO] agent.router.manager: shutting down +29:43.332 [INFO] TestAgent: consul server down +29:43.332 [INFO] TestAgent: shutdown complete +29:43.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29491 network=tcp +29:43.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29491 network=udp +29:43.332 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29492 network=tcp +29:43.836 [INFO] TestAgent: Waiting for endpoints to shut down +29:43.836 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_UnmanagedConnectProxy (0.00s) - --- PASS: TestAgent_RegisterService_UnmanagedConnectProxy/service_manager (0.79s) - --- PASS: TestAgent_RegisterService_UnmanagedConnectProxy/normal (0.80s) + --- PASS: TestAgent_RegisterService_UnmanagedConnectProxy/service_manager (0.90s) + --- PASS: TestAgent_RegisterService_UnmanagedConnectProxy/normal (0.93s) === RUN TestAgent_RegisterServiceDeregisterService_Sidecar === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal === PAUSE TestAgent_RegisterServiceDeregisterService_Sidecar/normal === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager === PAUSE TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager === CONT TestAgent_RegisterServiceDeregisterService_Sidecar/normal -=== CONT TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/sanity_check_no_sidecar_case +=== CONT TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/sanity_check_no_sidecar_case -2022-11-07T22:44:43.401Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:43.416Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:43.442 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c Address:127.0.0.1:11132}]" -44:43.442 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11132 [Follower]" leader= -44:43.442 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c.dc1 127.0.0.1 -44:43.443 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c 127.0.0.1 -2022-11-07T22:44:43.443Z [INFO] agent.router: Initializing LAN area manager -44:43.443 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c.dc1 area=wan -44:43.443 [INFO] TestAgent.server: Adding LAN server: server="Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c (Addr: tcp/127.0.0.1:11132) (DC: dc1)" -44:43.443 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11127 network=udp -44:43.443 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11127 network=tcp -44:43.444 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11128 network=tcp -44:43.444 [INFO] TestAgent: started state syncer -44:43.444 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11133 network=tcp -44:43.511 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:43.511 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11132 [Candidate]" term=2 -44:43.522 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:43.522 [DEBUG] TestAgent.server.raft: vote granted: from=a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c term=2 tally=1 -44:43.522 [INFO] TestAgent.server.raft: election won: tally=1 -44:43.522 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11132 [Leader]" -44:43.522 [INFO] TestAgent.server: cluster leadership acquired -44:43.522 [INFO] TestAgent.server: New leader elected: payload=Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c -44:43.529 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11132 -44:43.542 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:43.560 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:43.560 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:43.560 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:43.560 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:43.560 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:43.560 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c -44:43.560 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a7197a5b-bb6e-7b6e-61e0-5ce51d3a887c -44:43.569 [INFO] TestAgent.server: federation state anti-entropy synced -44:43.706 [INFO] TestAgent: Synced node info -44:43.711 [INFO] TestAgent: Synced service: service=web -44:43.711 [INFO] TestAgent: Requesting shutdown -44:43.711 [INFO] TestAgent.server: shutting down server -44:43.711 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:43.711 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:43.711 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:43.711 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:43.711 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:43.711 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:43.711 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:43.711 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:43.711 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:43.711 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:43.711 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:43.715 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:43.720Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:43.720Z [INFO] agent.router.manager: shutting down -44:43.720 [INFO] TestAgent: consul server down -44:43.720 [INFO] TestAgent: shutdown complete -44:43.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11127 network=tcp -44:43.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11127 network=udp -44:43.720 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11128 network=tcp -44:44.220 [INFO] TestAgent: Waiting for endpoints to shut down -44:44.220 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/default_sidecar -2022-11-07T22:44:43.398Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:43.406Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:43.430 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5bea7a41-9801-ed02-1233-c52afea9bc54 Address:127.0.0.1:11139}]" -44:43.430 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11139 [Follower]" leader= -44:43.430 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5bea7a41-9801-ed02-1233-c52afea9bc54.dc1 127.0.0.1 -44:43.432 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5bea7a41-9801-ed02-1233-c52afea9bc54 127.0.0.1 -2022-11-07T22:44:43.432Z [INFO] agent.router: Initializing LAN area manager -44:43.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11134 network=udp -44:43.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11134 network=tcp -44:43.435 [INFO] TestAgent.server: Adding LAN server: server="Node-5bea7a41-9801-ed02-1233-c52afea9bc54 (Addr: tcp/127.0.0.1:11139) (DC: dc1)" -44:43.435 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5bea7a41-9801-ed02-1233-c52afea9bc54.dc1 area=wan -44:43.435 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11135 network=tcp -44:43.435 [INFO] TestAgent: started state syncer -44:43.435 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11140 network=tcp -44:43.493 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:43.493 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11139 [Candidate]" term=2 -44:43.501 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:43.501 [DEBUG] TestAgent.server.raft: vote granted: from=5bea7a41-9801-ed02-1233-c52afea9bc54 term=2 tally=1 -44:43.501 [INFO] TestAgent.server.raft: election won: tally=1 -44:43.501 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11139 [Leader]" -44:43.501 [INFO] TestAgent.server: cluster leadership acquired -44:43.501 [INFO] TestAgent.server: New leader elected: payload=Node-5bea7a41-9801-ed02-1233-c52afea9bc54 -44:43.506 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11139 -44:43.521 [INFO] TestAgent: Synced node info -44:43.529 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:43.543 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:43.543 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:43.543 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:43.543 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:43.543 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:43.543 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5bea7a41-9801-ed02-1233-c52afea9bc54 -44:43.543 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5bea7a41-9801-ed02-1233-c52afea9bc54 -44:43.550 [INFO] TestAgent.server: federation state anti-entropy synced -44:43.717 [DEBUG] TestAgent: Node info in sync -44:43.720 [INFO] TestAgent: Synced service: service=web -44:43.720 [INFO] TestAgent: Requesting shutdown -44:43.720 [INFO] TestAgent.server: shutting down server -44:43.720 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:43.720 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:43.720 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:43.720 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:43.720 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:43.720 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:43.720 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:43.720 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:43.720 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:43.720 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:43.724 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:43.727Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:43.727Z [INFO] agent.router.manager: shutting down -44:43.727 [INFO] TestAgent: consul server down -44:43.727 [INFO] TestAgent: shutdown complete -44:43.727 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11134 network=tcp -44:43.727 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11134 network=udp -44:43.727 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11135 network=tcp -44:44.227 [INFO] TestAgent: Waiting for endpoints to shut down -44:44.227 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:43.869Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:43.912Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:43.969 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c23d5368-46a2-bbf8-5130-fa52ae4529e4 Address:127.0.0.1:29517}]" +29:43.969 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29517 [Follower]" leader= +29:43.970 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4.dc1 127.0.0.1 +29:43.971 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4 127.0.0.1 +2021-10-05T16:29:43.971Z [INFO] agent.router: Initializing LAN area manager +29:43.971 [INFO] TestAgent.server: Adding LAN server: server="Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4 (Addr: tcp/127.0.0.1:29517) (DC: dc1)" +29:43.971 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4.dc1 area=wan +29:43.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29512 network=udp +29:43.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29512 network=tcp +29:43.972 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29513 network=tcp +29:43.972 [INFO] TestAgent: started state syncer +29:43.972 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29518 network=tcp +29:44.014 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:44.015 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29517 [Candidate]" term=2 +29:44.043 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:44.043 [DEBUG] TestAgent.server.raft: vote granted: from=c23d5368-46a2-bbf8-5130-fa52ae4529e4 term=2 tally=1 +29:44.043 [INFO] TestAgent.server.raft: election won: tally=1 +29:44.043 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29517 [Leader]" +29:44.043 [INFO] TestAgent.server: cluster leadership acquired +29:44.044 [INFO] TestAgent.server: New leader elected: payload=Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4 +29:44.054 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29517 +29:44.073 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:44.132 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:44.132 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:44.132 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:44.132 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:44.132 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:44.132 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4 +29:44.132 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c23d5368-46a2-bbf8-5130-fa52ae4529e4 +29:44.163 [INFO] TestAgent.server: federation state anti-entropy synced +29:44.220 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:44.236 [INFO] TestAgent: Synced node info +29:44.236 [DEBUG] TestAgent: Node info in sync +29:44.402 [DEBUG] TestAgent: Node info in sync +29:44.412 [INFO] TestAgent: Synced service: service=web +29:44.412 [INFO] TestAgent: Requesting shutdown +29:44.412 [INFO] TestAgent.server: shutting down server +29:44.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:44.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:44.412 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:44.412 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:44.412 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:44.412 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:44.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:44.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:44.412 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:44.412 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:44.536 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:44.662Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:44.662Z [INFO] agent.router.manager: shutting down +29:44.662 [INFO] TestAgent: consul server down +29:44.662 [INFO] TestAgent: shutdown complete +29:44.662 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29512 network=tcp +29:44.662 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29512 network=udp +29:44.662 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29513 network=tcp +29:45.164 [INFO] TestAgent: Waiting for endpoints to shut down +29:45.164 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/default_sidecar -2022-11-07T22:44:44.231Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:44.242Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:44.331 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:307a1abf-2dbe-8971-a775-21d5f4f5a791 Address:127.0.0.1:11146}]" -44:44.331 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11146 [Follower]" leader= -44:44.331 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-307a1abf-2dbe-8971-a775-21d5f4f5a791.dc1 127.0.0.1 -44:44.332 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-307a1abf-2dbe-8971-a775-21d5f4f5a791 127.0.0.1 -2022-11-07T22:44:44.332Z [INFO] agent.router: Initializing LAN area manager -44:44.332 [INFO] TestAgent.server: Adding LAN server: server="Node-307a1abf-2dbe-8971-a775-21d5f4f5a791 (Addr: tcp/127.0.0.1:11146) (DC: dc1)" -44:44.332 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-307a1abf-2dbe-8971-a775-21d5f4f5a791.dc1 area=wan -44:44.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11141 network=udp -44:44.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11141 network=tcp -44:44.332 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11142 network=tcp -44:44.333 [INFO] TestAgent: started state syncer -44:44.333 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11147 network=tcp -44:44.369 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:44.369 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11146 [Candidate]" term=2 -44:44.376 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:44.376 [DEBUG] TestAgent.server.raft: vote granted: from=307a1abf-2dbe-8971-a775-21d5f4f5a791 term=2 tally=1 -44:44.376 [INFO] TestAgent.server.raft: election won: tally=1 -44:44.376 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11146 [Leader]" -44:44.376 [INFO] TestAgent.server: cluster leadership acquired -44:44.377 [INFO] TestAgent.server: New leader elected: payload=Node-307a1abf-2dbe-8971-a775-21d5f4f5a791 -44:44.381 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11146 -44:44.392 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:44.411 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:44.411 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:44.411 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:44.411 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:44.411 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:44.411 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-307a1abf-2dbe-8971-a775-21d5f4f5a791 -44:44.411 [INFO] TestAgent.server: member joined, marking health alive: member=Node-307a1abf-2dbe-8971-a775-21d5f4f5a791 -44:44.423 [INFO] TestAgent.server: federation state anti-entropy synced -44:44.478 [INFO] TestAgent: Synced node info -44:44.483 [INFO] TestAgent: Synced service: service=web -44:44.487 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:44.487 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:44.487 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:44.487 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:44.487 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:44.487 [DEBUG] TestAgent: removed service: service=web -44:44.487 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:44.487 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:44.487 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:44.487 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:44.487 [DEBUG] TestAgent: Node info in sync -44:44.491 [INFO] TestAgent: Deregistered service: service=web -44:44.493 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:44.493 [INFO] TestAgent: Requesting shutdown -44:44.494 [INFO] TestAgent.server: shutting down server -44:44.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:44.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:44.494 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:44.494 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:44.494 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:44.494 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:44.494 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:44.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:44.494 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:44.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:44.494 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:44.496 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:44.499Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:44.499Z [INFO] agent.router.manager: shutting down -44:44.499 [INFO] TestAgent: consul server down -44:44.499 [INFO] TestAgent: shutdown complete -44:44.499 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11141 network=tcp -44:44.499 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11141 network=udp -44:44.499 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11142 network=tcp -44:44.999 [INFO] TestAgent: Waiting for endpoints to shut down -44:45.000 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_defaults -2022-11-07T22:44:44.239Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:44.248Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:44.331 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a154a40-9b2d-d620-3f34-489dedf4462a Address:127.0.0.1:11153}]" -44:44.332 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11153 [Follower]" leader= -44:44.332 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8a154a40-9b2d-d620-3f34-489dedf4462a.dc1 127.0.0.1 -44:44.333 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8a154a40-9b2d-d620-3f34-489dedf4462a 127.0.0.1 -2022-11-07T22:44:44.333Z [INFO] agent.router: Initializing LAN area manager -44:44.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11148 network=udp -44:44.333 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8a154a40-9b2d-d620-3f34-489dedf4462a.dc1 area=wan -44:44.333 [INFO] TestAgent.server: Adding LAN server: server="Node-8a154a40-9b2d-d620-3f34-489dedf4462a (Addr: tcp/127.0.0.1:11153) (DC: dc1)" -44:44.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11148 network=tcp -44:44.333 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11149 network=tcp -44:44.333 [INFO] TestAgent: started state syncer -44:44.333 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11154 network=tcp -44:44.388 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:44.388 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11153 [Candidate]" term=2 -44:44.397 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:44.397 [DEBUG] TestAgent.server.raft: vote granted: from=8a154a40-9b2d-d620-3f34-489dedf4462a term=2 tally=1 -44:44.397 [INFO] TestAgent.server.raft: election won: tally=1 -44:44.397 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11153 [Leader]" -44:44.397 [INFO] TestAgent.server: cluster leadership acquired -44:44.397 [INFO] TestAgent.server: New leader elected: payload=Node-8a154a40-9b2d-d620-3f34-489dedf4462a -44:44.403 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11153 -44:44.415 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:44.434 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:44.434 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:44.434 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:44.434 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:44.434 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:44.434 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8a154a40-9b2d-d620-3f34-489dedf4462a -44:44.434 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8a154a40-9b2d-d620-3f34-489dedf4462a -44:44.443 [INFO] TestAgent.server: federation state anti-entropy synced -44:44.661 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -44:44.664 [INFO] TestAgent: Synced node info -44:44.668 [INFO] TestAgent: Synced service: service=web -44:44.672 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:44.672 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:44.672 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:44.684 [DEBUG] TestAgent: removed service: service=web -44:44.684 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:44.684 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:44.684 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:44.684 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:44.684 [DEBUG] TestAgent: Node info in sync -44:44.687 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:44.689 [INFO] TestAgent: Deregistered service: service=web -44:44.689 [INFO] TestAgent: Requesting shutdown -44:44.690 [INFO] TestAgent.server: shutting down server -44:44.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:44.690 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:44.690 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:44.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:44.690 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:44.690 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:44.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:44.690 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:44.690 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:44.690 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:44.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:44.693 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:44.696Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:44.696Z [INFO] agent.router.manager: shutting down -44:44.696 [INFO] TestAgent: consul server down -44:44.696 [INFO] TestAgent: shutdown complete -44:44.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11148 network=tcp -44:44.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11148 network=udp -44:44.696 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11149 network=tcp -44:45.196 [INFO] TestAgent: Waiting for endpoints to shut down -44:45.196 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:43.862Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:43.891Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:43.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a2e05680-7c1f-8eb2-adb7-317ab22384d1 Address:127.0.0.1:29510}]" +29:43.931 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1.dc1 127.0.0.1 +29:43.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1 127.0.0.1 +2021-10-05T16:29:43.931Z [INFO] agent.router: Initializing LAN area manager +29:43.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29505 network=udp +29:43.931 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29510 [Follower]" leader= +29:43.932 [INFO] TestAgent.server: Adding LAN server: server="Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1 (Addr: tcp/127.0.0.1:29510) (DC: dc1)" +29:43.932 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1.dc1 area=wan +29:43.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29505 network=tcp +29:43.933 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29506 network=tcp +29:43.933 [INFO] TestAgent: started state syncer +29:43.933 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29511 network=tcp +29:43.998 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:43.998 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29510 [Candidate]" term=2 +29:44.022 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:44.022 [DEBUG] TestAgent.server.raft: vote granted: from=a2e05680-7c1f-8eb2-adb7-317ab22384d1 term=2 tally=1 +29:44.022 [INFO] TestAgent.server.raft: election won: tally=1 +29:44.022 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29510 [Leader]" +29:44.024 [INFO] TestAgent.server: cluster leadership acquired +29:44.024 [INFO] TestAgent.server: New leader elected: payload=Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1 +29:44.036 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29510 +29:44.060 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:44.116 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:44.116 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:44.116 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:44.116 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:44.116 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:44.116 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1 +29:44.116 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a2e05680-7c1f-8eb2-adb7-317ab22384d1 +29:44.156 [INFO] TestAgent.server: federation state anti-entropy synced +29:44.348 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:44.532 [INFO] TestAgent: Synced node info +29:44.532 [DEBUG] TestAgent: Node info in sync +29:44.662 [DEBUG] TestAgent: Node info in sync +29:44.674 [INFO] TestAgent: Synced service: service=web +29:44.675 [INFO] TestAgent: Requesting shutdown +29:44.675 [INFO] TestAgent.server: shutting down server +29:44.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:44.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:44.675 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:44.675 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:44.675 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:44.675 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:44.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:44.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:44.675 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:44.675 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:44.700 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:44.720Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:44.720Z [INFO] agent.router.manager: shutting down +29:44.720 [INFO] TestAgent: consul server down +29:44.720 [INFO] TestAgent: shutdown complete +29:44.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29505 network=tcp +29:44.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29505 network=udp +29:44.720 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29506 network=tcp +29:45.221 [INFO] TestAgent: Waiting for endpoints to shut down +29:45.221 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/default_sidecar +2021-10-05T16:29:45.193Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:45.209Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:45.552 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83fe9710-05ae-4ef8-acfb-1c90e4af383c Address:127.0.0.1:29524}]" +29:45.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c.dc1 127.0.0.1 +29:45.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c 127.0.0.1 +2021-10-05T16:29:45.554Z [INFO] agent.router: Initializing LAN area manager +29:45.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29519 network=udp +29:45.554 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29524 [Follower]" leader= +29:45.554 [INFO] TestAgent.server: Adding LAN server: server="Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c (Addr: tcp/127.0.0.1:29524) (DC: dc1)" +29:45.554 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c.dc1 area=wan +29:45.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29519 network=tcp +29:45.555 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29520 network=tcp +29:45.555 [INFO] TestAgent: started state syncer +29:45.555 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29525 network=tcp +29:45.621 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:45.621 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29524 [Candidate]" term=2 +29:45.634 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:45.634 [DEBUG] TestAgent.server.raft: vote granted: from=83fe9710-05ae-4ef8-acfb-1c90e4af383c term=2 tally=1 +29:45.634 [INFO] TestAgent.server.raft: election won: tally=1 +29:45.634 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29524 [Leader]" +29:45.634 [INFO] TestAgent.server: cluster leadership acquired +29:45.635 [INFO] TestAgent.server: New leader elected: payload=Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c +29:45.639 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29524 +29:45.663 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:45.699 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:45.699 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:45.699 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:45.699 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:45.699 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:45.699 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c +29:45.699 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83fe9710-05ae-4ef8-acfb-1c90e4af383c +29:45.713 [INFO] TestAgent.server: federation state anti-entropy synced +29:45.725 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:45.731 [INFO] TestAgent: Synced node info +29:45.758 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +29:45.758 [DEBUG] TestAgent: Node info in sync +29:45.769 [INFO] TestAgent: Synced service: service=web +29:45.772 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:45.772 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:45.772 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:45.790 [DEBUG] TestAgent: removed service: service=web +29:45.790 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:45.790 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:45.790 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:45.790 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:45.790 [DEBUG] TestAgent: Node info in sync +29:45.790 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:45.793 [INFO] TestAgent: Deregistered service: service=web +29:45.796 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:45.796 [INFO] TestAgent: Requesting shutdown +29:45.797 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:45.797 [INFO] TestAgent.server: shutting down server +29:45.797 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:45.797 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:45.797 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:45.797 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:45.797 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:45.797 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:45.797 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:45.797 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:45.797 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:45.797 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:45.801 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:45.805Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:45.805Z [INFO] agent.router.manager: shutting down +29:45.805 [INFO] TestAgent: consul server down +29:45.806 [INFO] TestAgent: shutdown complete +29:45.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29519 network=tcp +29:45.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29519 network=udp +29:45.806 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29520 network=tcp +29:46.307 [INFO] TestAgent: Waiting for endpoints to shut down +29:46.307 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_defaults -2022-11-07T22:44:45.013Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:45.013Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:45.022Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:45.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:45.180 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:abfd0af7-83fb-ed81-4c59-5ad17f52b9af Address:127.0.0.1:11160}]" -44:45.180 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11160 [Follower]" leader= -44:45.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 127.0.0.1 -44:45.181 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af 127.0.0.1 -2022-11-07T22:44:45.181Z [INFO] agent.router: Initializing LAN area manager -44:45.181 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 area=wan -44:45.181 [INFO] TestAgent.server: Adding LAN server: server="Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af (Addr: tcp/127.0.0.1:11160) (DC: dc1)" -44:45.181 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11155 network=udp -44:45.181 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11155 network=tcp -44:45.182 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11156 network=tcp -44:45.182 [INFO] TestAgent: started state syncer -44:45.182 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11161 network=tcp -44:45.219 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:45.219 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11160 [Candidate]" term=2 -44:45.231 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:45.231 [DEBUG] TestAgent.server.raft: vote granted: from=abfd0af7-83fb-ed81-4c59-5ad17f52b9af term=2 tally=1 -44:45.231 [INFO] TestAgent.server.raft: election won: tally=1 -44:45.231 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11160 [Leader]" -44:45.231 [INFO] TestAgent.server: cluster leadership acquired -44:45.231 [INFO] TestAgent.server: New leader elected: payload=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.232 [INFO] TestAgent.server: initializing acls -44:45.237 [INFO] TestAgent.server: initializing acls -44:45.237 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:45.237 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:45.243 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:45.243 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:45.250 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:45.253 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:45.255 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:45.255 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:45.255 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:45.255 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:45.255 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.255 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 -44:45.255 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 area=wan -44:45.255 [INFO] TestAgent.server: Updating LAN server: server="Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af (Addr: tcp/127.0.0.1:11160) (DC: dc1)" -44:45.258 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:45.258 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.258 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 -44:45.258 [INFO] TestAgent.server: Updating LAN server: server="Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af (Addr: tcp/127.0.0.1:11160) (DC: dc1)" -44:45.258 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af.dc1 area=wan -44:45.268 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:45.284 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:45.284 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:45.284 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:45.284 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:45.284 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:45.284 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.284 [INFO] TestAgent.server: member joined, marking health alive: member=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.290 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.290 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.296 [INFO] TestAgent.server: federation state anti-entropy synced -44:45.436 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:45.441 [INFO] TestAgent: Synced node info -44:45.441 [DEBUG] TestAgent: Node info in sync -44:45.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-abfd0af7-83fb-ed81-4c59-5ad17f52b9af -44:45.613 [DEBUG] TestAgent: Node info in sync -44:45.616 [INFO] TestAgent: Synced service: service=web -44:45.619 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:45.619 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:45.619 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:45.619 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.619 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.619 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.619 [DEBUG] TestAgent: removed service: service=web -44:45.619 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.619 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:45.619 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:45.619 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:45.619 [DEBUG] TestAgent: Node info in sync -44:45.623 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:45.631 [INFO] TestAgent: Deregistered service: service=web -44:45.631 [INFO] TestAgent: Requesting shutdown -44:45.631 [INFO] TestAgent.server: shutting down server -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:45.631 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:45.631 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:45.631 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:45.631 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:45.635 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:45.638Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:45.638Z [INFO] agent.router.manager: shutting down -44:45.638 [INFO] TestAgent: consul server down -44:45.638 [INFO] TestAgent: shutdown complete -44:45.638 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11155 network=tcp -44:45.638 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11155 network=udp -44:45.638 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11156 network=tcp -44:46.138 [INFO] TestAgent: Waiting for endpoints to shut down -44:46.138 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_denied -2022-11-07T22:44:45.210Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:45.210Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:45.222Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:45.222Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:45.245 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2878e179-c335-254f-baa3-7349bc67b8e0 Address:127.0.0.1:11167}]" -44:45.245 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11167 [Follower]" leader= -44:45.246 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 127.0.0.1 -44:45.246 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2878e179-c335-254f-baa3-7349bc67b8e0 127.0.0.1 -2022-11-07T22:44:45.246Z [INFO] agent.router: Initializing LAN area manager -44:45.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11162 network=udp -44:45.247 [INFO] TestAgent.server: Adding LAN server: server="Node-2878e179-c335-254f-baa3-7349bc67b8e0 (Addr: tcp/127.0.0.1:11167) (DC: dc1)" -44:45.247 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 area=wan -44:45.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11162 network=tcp -44:45.247 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11163 network=tcp -44:45.247 [INFO] TestAgent: started state syncer -44:45.247 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11168 network=tcp -44:45.284 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:45.284 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11167 [Candidate]" term=2 -44:45.295 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:45.295 [DEBUG] TestAgent.server.raft: vote granted: from=2878e179-c335-254f-baa3-7349bc67b8e0 term=2 tally=1 -44:45.295 [INFO] TestAgent.server.raft: election won: tally=1 -44:45.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11167 [Leader]" -44:45.295 [INFO] TestAgent.server: cluster leadership acquired -44:45.295 [INFO] TestAgent.server: New leader elected: payload=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.297 [INFO] TestAgent.server: initializing acls -44:45.301 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:45.301 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:45.301 [INFO] TestAgent.server: initializing acls -44:45.301 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:45.307 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:45.313 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:45.318 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:45.318 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:45.318 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:45.318 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:45.318 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:45.318 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.318 [INFO] TestAgent.server: Updating LAN server: server="Node-2878e179-c335-254f-baa3-7349bc67b8e0 (Addr: tcp/127.0.0.1:11167) (DC: dc1)" -44:45.318 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 -44:45.318 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 area=wan -44:45.318 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.318 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 -44:45.318 [INFO] TestAgent.server: Updating LAN server: server="Node-2878e179-c335-254f-baa3-7349bc67b8e0 (Addr: tcp/127.0.0.1:11167) (DC: dc1)" -44:45.318 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2878e179-c335-254f-baa3-7349bc67b8e0.dc1 area=wan -44:45.326 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:45.345 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:45.345 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:45.345 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:45.345 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:45.345 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:45.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.345 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.351 [INFO] TestAgent.server: federation state anti-entropy synced -44:45.352 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.352 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.583 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:45.586 [INFO] TestAgent: Synced node info -44:45.586 [DEBUG] TestAgent: Node info in sync -44:45.678 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.678 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2878e179-c335-254f-baa3-7349bc67b8e0 -44:45.718 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -44:45.718 [DEBUG] TestAgent: Node info in sync -44:45.720 [INFO] TestAgent: Synced service: service=web -44:45.725 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:45.725 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:45.725 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:45.739 [DEBUG] TestAgent: removed service: service=web -44:45.740 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.740 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:45.740 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:45.740 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:45.740 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:45.740 [DEBUG] TestAgent: Node info in sync -44:45.743 [INFO] TestAgent: Deregistered service: service=web -44:45.745 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:45.746 [INFO] TestAgent: Requesting shutdown -44:45.746 [INFO] TestAgent.server: shutting down server -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:45.746 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:45.746 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:45.746 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:45.746 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:45.749 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:45.752Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:45.752Z [INFO] agent.router.manager: shutting down -44:45.753 [INFO] TestAgent: consul server down -44:45.753 [INFO] TestAgent: shutdown complete -44:45.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11162 network=tcp -44:45.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11162 network=udp -44:45.753 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11163 network=tcp -44:46.253 [INFO] TestAgent: Waiting for endpoints to shut down -44:46.253 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:45.246Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:45.278Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:45.544 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0e824cd7-bdcf-8bde-47df-cbb432bbf526 Address:127.0.0.1:29531}]" +29:45.545 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526.dc1 127.0.0.1 +29:45.545 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526 127.0.0.1 +2021-10-05T16:29:45.545Z [INFO] agent.router: Initializing LAN area manager +29:45.546 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29526 network=udp +29:45.546 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29531 [Follower]" leader= +29:45.546 [INFO] TestAgent.server: Adding LAN server: server="Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526 (Addr: tcp/127.0.0.1:29531) (DC: dc1)" +29:45.546 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526.dc1 area=wan +29:45.546 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29526 network=tcp +29:45.547 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29527 network=tcp +29:45.547 [INFO] TestAgent: started state syncer +29:45.547 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29532 network=tcp +29:45.612 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:45.612 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29531 [Candidate]" term=2 +29:45.627 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:45.627 [DEBUG] TestAgent.server.raft: vote granted: from=0e824cd7-bdcf-8bde-47df-cbb432bbf526 term=2 tally=1 +29:45.627 [INFO] TestAgent.server.raft: election won: tally=1 +29:45.627 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29531 [Leader]" +29:45.627 [INFO] TestAgent.server: cluster leadership acquired +29:45.628 [INFO] TestAgent.server: New leader elected: payload=Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526 +29:45.640 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29531 +29:45.658 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:45.691 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:45.691 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:45.691 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:45.691 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:45.691 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:45.691 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526 +29:45.691 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0e824cd7-bdcf-8bde-47df-cbb432bbf526 +29:45.706 [INFO] TestAgent.server: federation state anti-entropy synced +29:45.720 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:45.728 [INFO] TestAgent: Synced node info +29:45.746 [INFO] TestAgent: Synced service: service=web +29:45.755 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:45.755 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:45.755 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:45.968 [DEBUG] TestAgent: Node info in sync +29:45.968 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy +29:45.968 [DEBUG] TestAgent: Service in sync: service=web +29:45.988 [INFO] TestAgent: Synced check: check=service:web-sidecar-proxy:2 +29:45.988 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:45.989 [DEBUG] TestAgent: removed service: service=web +29:45.989 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:45.989 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:45.989 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:45.989 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:45.989 [DEBUG] TestAgent: Node info in sync +29:45.989 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:45.993 [INFO] TestAgent: Deregistered service: service=web +29:45.997 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:45.997 [INFO] TestAgent: Requesting shutdown +29:45.997 [INFO] TestAgent.server: shutting down server +29:45.997 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:45.997 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:45.997 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:45.997 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:45.997 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:45.997 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:45.997 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:45.997 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:45.997 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:45.997 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:46.003 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:46.009Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:46.009Z [INFO] agent.router.manager: shutting down +29:46.009 [INFO] TestAgent: consul server down +29:46.009 [INFO] TestAgent: shutdown complete +29:46.009 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29526 network=tcp +29:46.009 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29526 network=udp +29:46.009 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29527 network=tcp +29:46.510 [INFO] TestAgent: Waiting for endpoints to shut down +29:46.510 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_defaults +2021-10-05T16:29:46.330Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:46.330Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:46.386Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:46.386Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:46.461 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15cd3c90-10f4-bd57-249c-88744a42b30d Address:127.0.0.1:29538}]" +29:46.462 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15cd3c90-10f4-bd57-249c-88744a42b30d.dc1 127.0.0.1 +29:46.463 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15cd3c90-10f4-bd57-249c-88744a42b30d 127.0.0.1 +2021-10-05T16:29:46.463Z [INFO] agent.router: Initializing LAN area manager +29:46.463 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29533 network=udp +29:46.463 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29538 [Follower]" leader= +29:46.463 [INFO] TestAgent.server: Adding LAN server: server="Node-15cd3c90-10f4-bd57-249c-88744a42b30d (Addr: tcp/127.0.0.1:29538) (DC: dc1)" +29:46.463 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15cd3c90-10f4-bd57-249c-88744a42b30d.dc1 area=wan +29:46.463 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29533 network=tcp +29:46.464 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29534 network=tcp +29:46.464 [INFO] TestAgent: started state syncer +29:46.464 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29539 network=tcp +29:46.512 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:46.512 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29538 [Candidate]" term=2 +29:46.516 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:46.550 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:46.550 [DEBUG] TestAgent.server.raft: vote granted: from=15cd3c90-10f4-bd57-249c-88744a42b30d term=2 tally=1 +29:46.550 [INFO] TestAgent.server.raft: election won: tally=1 +29:46.550 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29538 [Leader]" +29:46.550 [INFO] TestAgent.server: cluster leadership acquired +29:46.550 [INFO] TestAgent.server: New leader elected: payload=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.558 [INFO] TestAgent.server: initializing acls +29:46.563 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:46.563 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:46.580 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:46.589 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:46.589 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:46.589 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:46.589 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-15cd3c90-10f4-bd57-249c-88744a42b30d.dc1 +29:46.592 [INFO] TestAgent.server: Updating LAN server: server="Node-15cd3c90-10f4-bd57-249c-88744a42b30d (Addr: tcp/127.0.0.1:29538) (DC: dc1)" +29:46.595 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-15cd3c90-10f4-bd57-249c-88744a42b30d.dc1 area=wan +29:46.618 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:46.648 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:46.648 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:46.648 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:46.648 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:46.648 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:46.648 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.648 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.679 [INFO] TestAgent.server: federation state anti-entropy synced +29:46.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.684 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.684 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-15cd3c90-10f4-bd57-249c-88744a42b30d +29:46.742 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +29:46.760 [INFO] TestAgent: Synced node info +29:46.775 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:46.784 [INFO] TestAgent: Synced service: service=web +29:46.784 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:46.784 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:46.823 [DEBUG] TestAgent: removed service: service=web +29:46.823 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:46.823 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:46.823 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:46.823 [DEBUG] TestAgent: Node info in sync +29:46.824 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:46.832 [INFO] TestAgent: Deregistered service: service=web +29:46.848 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:46.848 [INFO] TestAgent: Requesting shutdown +29:46.848 [INFO] TestAgent.server: shutting down server +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:46.848 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:46.848 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:46.848 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:46.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:46.849 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:46.849 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:46.849 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:46.849 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:46.849 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:46.849 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:46.858 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:46.864Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:46.864Z [INFO] agent.router.manager: shutting down +29:46.864 [INFO] TestAgent: consul server down +29:46.864 [INFO] TestAgent: shutdown complete +29:46.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29533 network=tcp +29:46.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29533 network=udp +29:46.864 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29534 network=tcp +29:47.368 [INFO] TestAgent: Waiting for endpoints to shut down +29:47.368 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_denied -2022-11-07T22:44:46.269Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:46.269Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.285Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:46.285Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.310 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:00138c2d-50a5-7488-2413-ad45f4d8bb63 Address:127.0.0.1:11181}]" -44:46.310 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11181 [Follower]" leader= -44:46.311 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-00138c2d-50a5-7488-2413-ad45f4d8bb63.dc1 127.0.0.1 -44:46.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 127.0.0.1 -2022-11-07T22:44:46.312Z [INFO] agent.router: Initializing LAN area manager -44:46.312 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63.dc1 area=wan -44:46.312 [INFO] TestAgent.server: Adding LAN server: server="Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 (Addr: tcp/127.0.0.1:11181) (DC: dc1)" -44:46.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11176 network=udp -44:46.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11176 network=tcp -44:46.312 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11177 network=tcp -44:46.313 [INFO] TestAgent: started state syncer -44:46.313 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11182 network=tcp -44:46.359 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.359 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11181 [Candidate]" term=2 -44:46.362 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:46.367 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.367 [DEBUG] TestAgent.server.raft: vote granted: from=00138c2d-50a5-7488-2413-ad45f4d8bb63 term=2 tally=1 -44:46.367 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.367 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11181 [Leader]" -44:46.367 [INFO] TestAgent.server: cluster leadership acquired -44:46.367 [INFO] TestAgent.server: New leader elected: payload=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.372 [INFO] TestAgent.server: initializing acls -44:46.375 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.375 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.379 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.388 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.388 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.388 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.388 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.388 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-00138c2d-50a5-7488-2413-ad45f4d8bb63.dc1 -44:46.388 [INFO] TestAgent.server: Updating LAN server: server="Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 (Addr: tcp/127.0.0.1:11181) (DC: dc1)" -44:46.388 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63.dc1 area=wan -44:46.396 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.410 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.410 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.410 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.410 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.410 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.410 [INFO] TestAgent.server: member joined, marking health alive: member=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.413 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.417 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.473 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.473 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-00138c2d-50a5-7488-2413-ad45f4d8bb63 -44:46.473 [INFO] TestAgent: Requesting shutdown -44:46.473 [INFO] TestAgent.server: shutting down server -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.473 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.473 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.473 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.477 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.480Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.480Z [INFO] agent.router.manager: shutting down -44:46.480 [INFO] TestAgent: consul server down -44:46.480 [INFO] TestAgent: shutdown complete -44:46.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11176 network=tcp -44:46.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11176 network=udp -44:46.480 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11177 network=tcp -44:46.980 [INFO] TestAgent: Waiting for endpoints to shut down -44:46.980 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:46.531Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:46.531Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:46.547Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:46.547Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:46.581 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:92e3d203-7060-5344-eab2-ce86c26d50b4 Address:127.0.0.1:29545}]" +29:46.582 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-92e3d203-7060-5344-eab2-ce86c26d50b4.dc1 127.0.0.1 +29:46.582 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29545 [Follower]" leader= +29:46.582 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-92e3d203-7060-5344-eab2-ce86c26d50b4 127.0.0.1 +2021-10-05T16:29:46.582Z [INFO] agent.router: Initializing LAN area manager +29:46.582 [INFO] TestAgent.server: Adding LAN server: server="Node-92e3d203-7060-5344-eab2-ce86c26d50b4 (Addr: tcp/127.0.0.1:29545) (DC: dc1)" +29:46.582 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-92e3d203-7060-5344-eab2-ce86c26d50b4.dc1 area=wan +29:46.582 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29540 network=udp +29:46.582 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29540 network=tcp +29:46.583 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29541 network=tcp +29:46.583 [INFO] TestAgent: started state syncer +29:46.583 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29546 network=tcp +29:46.633 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:46.638 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:46.638 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29545 [Candidate]" term=2 +29:46.656 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:46.656 [DEBUG] TestAgent.server.raft: vote granted: from=92e3d203-7060-5344-eab2-ce86c26d50b4 term=2 tally=1 +29:46.656 [INFO] TestAgent.server.raft: election won: tally=1 +29:46.656 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29545 [Leader]" +29:46.656 [INFO] TestAgent.server: cluster leadership acquired +29:46.657 [INFO] TestAgent.server: New leader elected: payload=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.667 [INFO] TestAgent.server: initializing acls +29:46.671 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:46.671 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:46.681 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:46.696 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:46.696 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:46.696 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:46.696 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-92e3d203-7060-5344-eab2-ce86c26d50b4.dc1 +29:46.696 [INFO] TestAgent.server: Updating LAN server: server="Node-92e3d203-7060-5344-eab2-ce86c26d50b4 (Addr: tcp/127.0.0.1:29545) (DC: dc1)" +29:46.697 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-92e3d203-7060-5344-eab2-ce86c26d50b4.dc1 area=wan +29:46.720 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:46.720 [INFO] TestAgent: Synced node info +29:46.720 [DEBUG] TestAgent: Node info in sync +29:46.747 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:46.747 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:46.747 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:46.747 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:46.747 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:46.747 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.747 [INFO] TestAgent.server: member joined, marking health alive: member=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.767 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.779 [INFO] TestAgent.server: federation state anti-entropy synced +29:46.789 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.789 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-92e3d203-7060-5344-eab2-ce86c26d50b4 +29:46.833 [DEBUG] TestAgent: Node info in sync +29:46.855 [INFO] TestAgent: Synced service: service=web +29:46.864 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:46.864 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:46.864 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:46.865 [DEBUG] TestAgent: removed service: service=web +29:46.865 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:46.865 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:46.865 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:46.865 [DEBUG] TestAgent: Node info in sync +29:46.884 [INFO] TestAgent: Deregistered service: service=web +29:46.900 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:46.900 [INFO] TestAgent: Requesting shutdown +29:46.900 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:46.900 [INFO] TestAgent.server: shutting down server +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:46.900 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:46.900 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:46.900 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:46.900 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:46.907 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:46.912Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:46.912Z [INFO] agent.router.manager: shutting down +29:46.912 [INFO] TestAgent: consul server down +29:46.912 [INFO] TestAgent: shutdown complete +29:46.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29540 network=tcp +29:46.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29540 network=udp +29:46.912 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29541 network=tcp +29:47.414 [INFO] TestAgent: Waiting for endpoints to shut down +29:47.414 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_denied +2021-10-05T16:29:47.404Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:47.404Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:47.443Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:47.443Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:47.494 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:538a415e-6eb8-1708-ec89-5302609386fa Address:127.0.0.1:29552}]" +29:47.495 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-538a415e-6eb8-1708-ec89-5302609386fa.dc1 127.0.0.1 +29:47.495 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-538a415e-6eb8-1708-ec89-5302609386fa 127.0.0.1 +2021-10-05T16:29:47.495Z [INFO] agent.router: Initializing LAN area manager +29:47.496 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29547 network=udp +29:47.496 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29552 [Follower]" leader= +29:47.496 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29547 network=tcp +29:47.496 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-538a415e-6eb8-1708-ec89-5302609386fa.dc1 area=wan +29:47.496 [INFO] TestAgent.server: Adding LAN server: server="Node-538a415e-6eb8-1708-ec89-5302609386fa (Addr: tcp/127.0.0.1:29552) (DC: dc1)" +29:47.496 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29548 network=tcp +29:47.496 [INFO] TestAgent: started state syncer +29:47.497 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29553 network=tcp +29:47.547 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:47.549 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:47.549 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29552 [Candidate]" term=2 +29:47.569 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:47.569 [DEBUG] TestAgent.server.raft: vote granted: from=538a415e-6eb8-1708-ec89-5302609386fa term=2 tally=1 +29:47.569 [INFO] TestAgent.server.raft: election won: tally=1 +29:47.569 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29552 [Leader]" +29:47.569 [INFO] TestAgent.server: cluster leadership acquired +29:47.569 [INFO] TestAgent.server: New leader elected: payload=Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.576 [INFO] TestAgent.server: initializing acls +29:47.580 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:47.580 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:47.587 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:47.612 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:47.612 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:47.612 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:47.612 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.612 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-538a415e-6eb8-1708-ec89-5302609386fa.dc1 +29:47.614 [INFO] TestAgent.server: Updating LAN server: server="Node-538a415e-6eb8-1708-ec89-5302609386fa (Addr: tcp/127.0.0.1:29552) (DC: dc1)" +29:47.614 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-538a415e-6eb8-1708-ec89-5302609386fa.dc1 area=wan +29:47.626 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:47.660 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:47.660 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:47.660 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:47.660 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:47.660 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:47.660 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.660 [INFO] TestAgent.server: member joined, marking health alive: member=Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.671 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.677 [INFO] TestAgent.server: federation state anti-entropy synced +29:47.686 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-538a415e-6eb8-1708-ec89-5302609386fa +29:47.687 [INFO] TestAgent: Requesting shutdown +29:47.687 [INFO] TestAgent.server: shutting down server +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:47.687 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:47.687 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:47.687 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:47.687 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:47.687 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:47.690 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:47.691Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:47.691Z [INFO] agent.router.manager: shutting down +29:47.691 [INFO] TestAgent: consul server down +29:47.691 [INFO] TestAgent: shutdown complete +29:47.691 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29547 network=tcp +29:47.692 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29547 network=udp +29:47.692 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29548 network=tcp +29:48.193 [INFO] TestAgent: Waiting for endpoints to shut down +29:48.193 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_sidecar -2022-11-07T22:44:46.157Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:46.157Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.173Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:46.173Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.210 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ad194b3c-1d21-0fe9-ebbc-212a047d2376 Address:127.0.0.1:11174}]" -44:46.211 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11174 [Follower]" leader= -44:46.211 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376.dc1 127.0.0.1 -44:46.212 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 127.0.0.1 -2022-11-07T22:44:46.212Z [INFO] agent.router: Initializing LAN area manager -44:46.212 [INFO] TestAgent.server: Adding LAN server: server="Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 (Addr: tcp/127.0.0.1:11174) (DC: dc1)" -44:46.212 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376.dc1 area=wan -44:46.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11169 network=tcp -44:46.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11169 network=udp -44:46.212 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11170 network=tcp -44:46.212 [INFO] TestAgent: started state syncer -44:46.212 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11175 network=tcp -44:46.263 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:46.268 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.268 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11174 [Candidate]" term=2 -44:46.276 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.276 [DEBUG] TestAgent.server.raft: vote granted: from=ad194b3c-1d21-0fe9-ebbc-212a047d2376 term=2 tally=1 -44:46.277 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.277 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11174 [Leader]" -44:46.277 [INFO] TestAgent.server: cluster leadership acquired -44:46.277 [INFO] TestAgent.server: New leader elected: payload=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.282 [INFO] TestAgent.server: initializing acls -44:46.285 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.285 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.291 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.303 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.303 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.303 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376.dc1 -44:46.303 [INFO] TestAgent.server: Updating LAN server: server="Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 (Addr: tcp/127.0.0.1:11174) (DC: dc1)" -44:46.304 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376.dc1 area=wan -44:46.312 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.328 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.328 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.328 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.328 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.328 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.328 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.328 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.331 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.336 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.369 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:46.373 [INFO] TestAgent: Synced node info -44:46.373 [DEBUG] TestAgent: Node info in sync -44:46.601 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.601 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ad194b3c-1d21-0fe9-ebbc-212a047d2376 -44:46.602 [INFO] TestAgent: Requesting shutdown -44:46.602 [INFO] TestAgent.server: shutting down server -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.602 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.602 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.602 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.602 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.606 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.613Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.613Z [INFO] agent.router.manager: shutting down -44:46.613 [INFO] TestAgent: consul server down -44:46.613 [INFO] TestAgent: shutdown complete -44:46.613 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11169 network=tcp -44:46.613 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11169 network=udp -44:46.613 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11170 network=tcp -44:47.113 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.113 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:47.446Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:47.446Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:47.475Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:47.475Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:47.504 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:60eda825-34e1-9550-c719-61050aa4fdd6 Address:127.0.0.1:29559}]" +29:47.504 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-60eda825-34e1-9550-c719-61050aa4fdd6.dc1 127.0.0.1 +29:47.505 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-60eda825-34e1-9550-c719-61050aa4fdd6 127.0.0.1 +2021-10-05T16:29:47.505Z [INFO] agent.router: Initializing LAN area manager +29:47.505 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29554 network=udp +29:47.505 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29559 [Follower]" leader= +29:47.506 [INFO] TestAgent.server: Adding LAN server: server="Node-60eda825-34e1-9550-c719-61050aa4fdd6 (Addr: tcp/127.0.0.1:29559) (DC: dc1)" +29:47.506 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-60eda825-34e1-9550-c719-61050aa4fdd6.dc1 area=wan +29:47.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29554 network=tcp +29:47.506 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29555 network=tcp +29:47.506 [INFO] TestAgent: started state syncer +29:47.506 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29560 network=tcp +29:47.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:47.557 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29559 [Candidate]" term=2 +29:47.557 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:47.576 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:47.576 [DEBUG] TestAgent.server.raft: vote granted: from=60eda825-34e1-9550-c719-61050aa4fdd6 term=2 tally=1 +29:47.576 [INFO] TestAgent.server.raft: election won: tally=1 +29:47.576 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29559 [Leader]" +29:47.576 [INFO] TestAgent.server: cluster leadership acquired +29:47.576 [INFO] TestAgent.server: New leader elected: payload=Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.582 [INFO] TestAgent.server: initializing acls +29:47.585 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:47.585 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:47.593 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:47.621 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:47.621 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:47.621 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:47.621 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-60eda825-34e1-9550-c719-61050aa4fdd6.dc1 +29:47.622 [INFO] TestAgent.server: Updating LAN server: server="Node-60eda825-34e1-9550-c719-61050aa4fdd6 (Addr: tcp/127.0.0.1:29559) (DC: dc1)" +29:47.622 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-60eda825-34e1-9550-c719-61050aa4fdd6.dc1 area=wan +29:47.640 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:47.683 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:47.683 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:47.683 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:47.683 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:47.683 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:47.683 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.683 [INFO] TestAgent.server: member joined, marking health alive: member=Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.689 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.693 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-60eda825-34e1-9550-c719-61050aa4fdd6 +29:47.693 [INFO] TestAgent: Requesting shutdown +29:47.693 [INFO] TestAgent.server: shutting down server +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:47.693 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:47.693 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:47.693 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:47.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:47.693 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:47.693 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:47.693 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:47.693 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:47.696 [INFO] TestAgent.server: federation state anti-entropy synced +29:47.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:47.697 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:47.698Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:47.698Z [INFO] agent.router.manager: shutting down +29:47.698 [INFO] TestAgent: consul server down +29:47.698 [INFO] TestAgent: shutdown complete +29:47.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29554 network=tcp +29:47.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29554 network=udp +29:47.698 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29555 network=tcp +29:48.199 [INFO] TestAgent: Waiting for endpoints to shut down +29:48.199 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_sidecar -2022-11-07T22:44:46.992Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:46.992Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:47.002Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:47.002Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:47.022 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:16d977f2-9ae7-6830-1840-3192255bad25 Address:127.0.0.1:11188}]" -44:47.022 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11188 [Follower]" leader= -44:47.022 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-16d977f2-9ae7-6830-1840-3192255bad25.dc1 127.0.0.1 -44:47.023 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-16d977f2-9ae7-6830-1840-3192255bad25 127.0.0.1 -2022-11-07T22:44:47.023Z [INFO] agent.router: Initializing LAN area manager -44:47.023 [INFO] TestAgent.server: Adding LAN server: server="Node-16d977f2-9ae7-6830-1840-3192255bad25 (Addr: tcp/127.0.0.1:11188) (DC: dc1)" -44:47.023 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-16d977f2-9ae7-6830-1840-3192255bad25.dc1 area=wan -44:47.023 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11183 network=udp -44:47.023 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11183 network=tcp -44:47.023 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11184 network=tcp -44:47.023 [INFO] TestAgent: started state syncer -44:47.024 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11189 network=tcp -44:47.072 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:47.072 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11188 [Candidate]" term=2 -44:47.073 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:47.080 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:47.080 [DEBUG] TestAgent.server.raft: vote granted: from=16d977f2-9ae7-6830-1840-3192255bad25 term=2 tally=1 -44:47.080 [INFO] TestAgent.server.raft: election won: tally=1 -44:47.080 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11188 [Leader]" -44:47.080 [INFO] TestAgent.server: cluster leadership acquired -44:47.080 [INFO] TestAgent.server: New leader elected: payload=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.085 [INFO] TestAgent.server: initializing acls -44:47.091 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:47.091 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:47.100 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:47.108 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:47.108 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:47.108 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:47.108 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.108 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-16d977f2-9ae7-6830-1840-3192255bad25.dc1 -44:47.108 [INFO] TestAgent.server: Updating LAN server: server="Node-16d977f2-9ae7-6830-1840-3192255bad25 (Addr: tcp/127.0.0.1:11188) (DC: dc1)" -44:47.108 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-16d977f2-9ae7-6830-1840-3192255bad25.dc1 area=wan -44:47.116 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:47.131 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:47.131 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:47.131 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:47.131 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:47.131 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:47.131 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.131 [INFO] TestAgent.server: member joined, marking health alive: member=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.137 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.143 [INFO] TestAgent.server: federation state anti-entropy synced -44:47.165 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:47.172 [INFO] TestAgent: Synced node info -44:47.440 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.440 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-16d977f2-9ae7-6830-1840-3192255bad25 -44:47.446 [INFO] TestAgent: Requesting shutdown -44:47.446 [INFO] TestAgent.server: shutting down server -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:47.446 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:47.446 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:47.447 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:47.447 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:47.456 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:47.459Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:47.459Z [INFO] agent.router.manager: shutting down -44:47.459 [INFO] TestAgent: consul server down -44:47.459 [INFO] TestAgent: shutdown complete -44:47.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11183 network=tcp -44:47.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11183 network=udp -44:47.459 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11184 network=tcp -44:47.960 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.960 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:48.206Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:48.206Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:48.258Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:48.258Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:48.352 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fdc6412-0a85-be41-a872-f85fb88884ca Address:127.0.0.1:29566}]" +29:48.352 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29566 [Follower]" leader= +29:48.353 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 127.0.0.1 +29:48.353 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fdc6412-0a85-be41-a872-f85fb88884ca 127.0.0.1 +2021-10-05T16:29:48.354Z [INFO] agent.router: Initializing LAN area manager +29:48.354 [INFO] TestAgent.server: Adding LAN server: server="Node-2fdc6412-0a85-be41-a872-f85fb88884ca (Addr: tcp/127.0.0.1:29566) (DC: dc1)" +29:48.354 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 area=wan +29:48.354 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29561 network=udp +29:48.354 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29561 network=tcp +29:48.355 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29562 network=tcp +29:48.355 [INFO] TestAgent: started state syncer +29:48.355 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29567 network=tcp +29:48.396 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:48.396 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29566 [Candidate]" term=2 +29:48.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:48.456 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:48.456 [DEBUG] TestAgent.server.raft: vote granted: from=2fdc6412-0a85-be41-a872-f85fb88884ca term=2 tally=1 +29:48.456 [INFO] TestAgent.server.raft: election won: tally=1 +29:48.456 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29566 [Leader]" +29:48.456 [INFO] TestAgent.server: cluster leadership acquired +29:48.456 [INFO] TestAgent.server: New leader elected: payload=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.487 [INFO] TestAgent.server: initializing acls +29:48.500 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:48.500 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:48.509 [INFO] TestAgent.server: initializing acls +29:48.509 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:48.514 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:48.564 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:48.572 [INFO] TestAgent: Synced node info +29:48.572 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:48.572 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:48.572 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:48.572 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.572 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 +29:48.572 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 area=wan +29:48.573 [INFO] TestAgent.server: Updating LAN server: server="Node-2fdc6412-0a85-be41-a872-f85fb88884ca (Addr: tcp/127.0.0.1:29566) (DC: dc1)" +29:48.580 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:48.580 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:48.580 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.580 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 +29:48.580 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2fdc6412-0a85-be41-a872-f85fb88884ca.dc1 area=wan +29:48.580 [INFO] TestAgent.server: Updating LAN server: server="Node-2fdc6412-0a85-be41-a872-f85fb88884ca (Addr: tcp/127.0.0.1:29566) (DC: dc1)" +29:48.587 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:48.640 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:48.640 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:48.640 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:48.640 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:48.640 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:48.640 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.640 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.651 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.651 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.669 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.669 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fdc6412-0a85-be41-a872-f85fb88884ca +29:48.709 [INFO] TestAgent.server: federation state anti-entropy synced +29:48.715 [INFO] TestAgent: Requesting shutdown +29:48.715 [INFO] TestAgent.server: shutting down server +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:48.715 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:48.715 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:48.715 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:48.715 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:48.719 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:48.719Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:48.719Z [INFO] agent.router.manager: shutting down +29:48.719 [INFO] TestAgent: consul server down +29:48.719 [INFO] TestAgent: shutdown complete +29:48.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29561 network=tcp +29:48.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29561 network=udp +29:48.719 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29562 network=tcp +29:49.219 [INFO] TestAgent: Waiting for endpoints to shut down +29:49.219 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination -2022-11-07T22:44:47.127Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:47.127Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:47.136Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:47.136Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:47.159 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5317da0a-c741-af3c-7c86-196ce72f68ef Address:127.0.0.1:11195}]" -44:47.159 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11195 [Follower]" leader= -44:47.159 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5317da0a-c741-af3c-7c86-196ce72f68ef.dc1 127.0.0.1 -44:47.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5317da0a-c741-af3c-7c86-196ce72f68ef 127.0.0.1 -2022-11-07T22:44:47.160Z [INFO] agent.router: Initializing LAN area manager -44:47.160 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5317da0a-c741-af3c-7c86-196ce72f68ef.dc1 area=wan -44:47.160 [INFO] TestAgent.server: Adding LAN server: server="Node-5317da0a-c741-af3c-7c86-196ce72f68ef (Addr: tcp/127.0.0.1:11195) (DC: dc1)" -44:47.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11190 network=udp -44:47.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11190 network=tcp -44:47.160 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11191 network=tcp -44:47.160 [INFO] TestAgent: started state syncer -44:47.160 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11196 network=tcp -44:47.210 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:47.228 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:47.228 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11195 [Candidate]" term=2 -44:47.236 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:47.236 [DEBUG] TestAgent.server.raft: vote granted: from=5317da0a-c741-af3c-7c86-196ce72f68ef term=2 tally=1 -44:47.236 [INFO] TestAgent.server.raft: election won: tally=1 -44:47.236 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11195 [Leader]" -44:47.236 [INFO] TestAgent.server: cluster leadership acquired -44:47.236 [INFO] TestAgent.server: New leader elected: payload=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.242 [INFO] TestAgent.server: initializing acls -44:47.245 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:47.245 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:47.252 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:47.261 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:47.261 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:47.261 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:47.262 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.262 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5317da0a-c741-af3c-7c86-196ce72f68ef.dc1 -44:47.262 [INFO] TestAgent.server: Updating LAN server: server="Node-5317da0a-c741-af3c-7c86-196ce72f68ef (Addr: tcp/127.0.0.1:11195) (DC: dc1)" -44:47.262 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5317da0a-c741-af3c-7c86-196ce72f68ef.dc1 area=wan -44:47.270 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:47.288 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:47.288 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:47.288 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:47.288 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:47.288 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:47.288 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.288 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.299 [INFO] TestAgent.server: federation state anti-entropy synced -44:47.299 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.304 [INFO] TestAgent: Synced node info -44:47.554 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.554 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5317da0a-c741-af3c-7c86-196ce72f68ef -44:47.561 [INFO] TestAgent: Requesting shutdown -44:47.561 [INFO] TestAgent.server: shutting down server -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:47.561 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:47.561 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:47.561 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:47.561 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:47.569 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:47.573Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:47.573Z [INFO] agent.router.manager: shutting down -44:47.573 [INFO] TestAgent: consul server down -44:47.573 [INFO] TestAgent: shutdown complete -44:47.573 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11190 network=tcp -44:47.573 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11190 network=udp -44:47.573 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11191 network=tcp -44:48.074 [INFO] TestAgent: Waiting for endpoints to shut down -44:48.074 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:48.214Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:48.214Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:48.269Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:48.269Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:48.364 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18d09ab7-400d-1efd-f361-697ac232fb4b Address:127.0.0.1:29573}]" +29:48.365 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 127.0.0.1 +29:48.365 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29573 [Follower]" leader= +29:48.365 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18d09ab7-400d-1efd-f361-697ac232fb4b 127.0.0.1 +2021-10-05T16:29:48.365Z [INFO] agent.router: Initializing LAN area manager +29:48.366 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29568 network=udp +29:48.366 [INFO] TestAgent.server: Adding LAN server: server="Node-18d09ab7-400d-1efd-f361-697ac232fb4b (Addr: tcp/127.0.0.1:29573) (DC: dc1)" +29:48.366 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29568 network=tcp +29:48.366 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 area=wan +29:48.366 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29569 network=tcp +29:48.366 [INFO] TestAgent: started state syncer +29:48.366 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29574 network=tcp +29:48.420 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:48.436 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:48.436 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29573 [Candidate]" term=2 +29:48.460 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:48.460 [DEBUG] TestAgent.server.raft: vote granted: from=18d09ab7-400d-1efd-f361-697ac232fb4b term=2 tally=1 +29:48.460 [INFO] TestAgent.server.raft: election won: tally=1 +29:48.460 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29573 [Leader]" +29:48.460 [INFO] TestAgent.server: cluster leadership acquired +29:48.460 [INFO] TestAgent.server: New leader elected: payload=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.469 [INFO] TestAgent.server: initializing acls +29:48.484 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:48.484 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:48.509 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:48.524 [INFO] TestAgent.server: initializing acls +29:48.524 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:48.556 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:48.556 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:48.556 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:48.556 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.556 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 +29:48.556 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 area=wan +29:48.557 [INFO] TestAgent.server: Updating LAN server: server="Node-18d09ab7-400d-1efd-f361-697ac232fb4b (Addr: tcp/127.0.0.1:29573) (DC: dc1)" +29:48.560 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:48.560 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:48.560 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.560 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 +29:48.560 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-18d09ab7-400d-1efd-f361-697ac232fb4b.dc1 area=wan +29:48.560 [INFO] TestAgent: Synced node info +29:48.560 [INFO] TestAgent.server: Updating LAN server: server="Node-18d09ab7-400d-1efd-f361-697ac232fb4b (Addr: tcp/127.0.0.1:29573) (DC: dc1)" +29:48.572 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:48.599 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:48.599 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:48.599 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:48.599 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:48.599 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:48.599 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.599 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.637 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.637 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.660 [INFO] TestAgent.server: federation state anti-entropy synced +29:48.758 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.758 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-18d09ab7-400d-1efd-f361-697ac232fb4b +29:48.767 [INFO] TestAgent: Requesting shutdown +29:48.767 [INFO] TestAgent.server: shutting down server +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:48.767 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:48.767 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:48.767 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:48.767 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:48.780 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:48.784Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:48.784Z [INFO] agent.router.manager: shutting down +29:48.784 [INFO] TestAgent: consul server down +29:48.784 [INFO] TestAgent: shutdown complete +29:48.784 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29568 network=tcp +29:48.784 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29568 network=udp +29:48.784 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29569 network=tcp +29:49.286 [INFO] TestAgent: Waiting for endpoints to shut down +29:49.286 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination -2022-11-07T22:44:47.975Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:47.975Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:47.984Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:47.984Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:48.658 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:28d2c14d-3088-428f-21ed-db7bfc915156 Address:127.0.0.1:11202}]" -44:48.658 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11202 [Follower]" leader= -44:48.659 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-28d2c14d-3088-428f-21ed-db7bfc915156.dc1 127.0.0.1 -44:48.659 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-28d2c14d-3088-428f-21ed-db7bfc915156 127.0.0.1 -2022-11-07T22:44:48.659Z [INFO] agent.router: Initializing LAN area manager -44:48.659 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11197 network=udp -44:48.660 [INFO] TestAgent.server: Adding LAN server: server="Node-28d2c14d-3088-428f-21ed-db7bfc915156 (Addr: tcp/127.0.0.1:11202) (DC: dc1)" -44:48.660 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-28d2c14d-3088-428f-21ed-db7bfc915156.dc1 area=wan -44:48.660 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11197 network=tcp -44:48.660 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11198 network=tcp -44:48.660 [INFO] TestAgent: started state syncer -44:48.660 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11203 network=tcp -44:48.710 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:48.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:48.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11202 [Candidate]" term=2 -44:48.721 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:48.721 [DEBUG] TestAgent.server.raft: vote granted: from=28d2c14d-3088-428f-21ed-db7bfc915156 term=2 tally=1 -44:48.721 [INFO] TestAgent.server.raft: election won: tally=1 -44:48.721 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11202 [Leader]" -44:48.721 [INFO] TestAgent.server: cluster leadership acquired -44:48.721 [INFO] TestAgent.server: New leader elected: payload=Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.728 [INFO] TestAgent.server: initializing acls -44:48.732 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:48.732 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:48.739 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:48.751 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:48.751 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:48.751 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:48.751 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.751 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-28d2c14d-3088-428f-21ed-db7bfc915156.dc1 -44:48.751 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-28d2c14d-3088-428f-21ed-db7bfc915156.dc1 area=wan -44:48.751 [INFO] TestAgent.server: Updating LAN server: server="Node-28d2c14d-3088-428f-21ed-db7bfc915156 (Addr: tcp/127.0.0.1:11202) (DC: dc1)" -44:48.761 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:48.779 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:48.779 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:48.779 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:48.779 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:48.779 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:48.779 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.779 [INFO] TestAgent.server: member joined, marking health alive: member=Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.785 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.790 [INFO] TestAgent.server: federation state anti-entropy synced -44:48.810 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-28d2c14d-3088-428f-21ed-db7bfc915156 -44:48.819 [INFO] TestAgent: Requesting shutdown -44:48.819 [INFO] TestAgent.server: shutting down server -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:48.819 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:48.819 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:48.819 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:48.819 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:48.819 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:48.822 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:48.826Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:48.826Z [INFO] agent.router.manager: shutting down -44:48.826 [INFO] TestAgent: consul server down -44:48.826 [INFO] TestAgent: shutdown complete -44:48.826 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11197 network=tcp -44:48.826 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11197 network=udp -44:48.826 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11198 network=tcp -44:49.326 [INFO] TestAgent: Waiting for endpoints to shut down -44:49.326 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:49.232Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:49.232Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:49.243Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:49.243Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:49.302 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3fd1976a-f63f-513a-45bb-59c7eef2e263 Address:127.0.0.1:29580}]" +29:49.303 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29580 [Follower]" leader= +29:49.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3fd1976a-f63f-513a-45bb-59c7eef2e263.dc1 127.0.0.1 +29:49.304 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 127.0.0.1 +2021-10-05T16:29:49.304Z [INFO] agent.router: Initializing LAN area manager +29:49.304 [INFO] TestAgent.server: Adding LAN server: server="Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 (Addr: tcp/127.0.0.1:29580) (DC: dc1)" +29:49.304 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263.dc1 area=wan +29:49.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29575 network=udp +29:49.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29575 network=tcp +29:49.321 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29576 network=tcp +29:49.321 [INFO] TestAgent: started state syncer +29:49.321 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29581 network=tcp +29:49.349 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:49.349 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29580 [Candidate]" term=2 +29:49.355 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:49.363 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:49.363 [DEBUG] TestAgent.server.raft: vote granted: from=3fd1976a-f63f-513a-45bb-59c7eef2e263 term=2 tally=1 +29:49.363 [INFO] TestAgent.server.raft: election won: tally=1 +29:49.363 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29580 [Leader]" +29:49.363 [INFO] TestAgent.server: cluster leadership acquired +29:49.364 [INFO] TestAgent.server: New leader elected: payload=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.381 [INFO] TestAgent.server: initializing acls +29:49.386 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:49.386 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:49.392 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:49.406 [INFO] TestAgent: Synced node info +29:49.406 [DEBUG] TestAgent: Node info in sync +29:49.406 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:49.406 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:49.406 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:49.406 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.406 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3fd1976a-f63f-513a-45bb-59c7eef2e263.dc1 +29:49.407 [INFO] TestAgent.server: Updating LAN server: server="Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 (Addr: tcp/127.0.0.1:29580) (DC: dc1)" +29:49.407 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263.dc1 area=wan +29:49.439 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:49.459 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:49.459 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:49.459 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:49.459 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:49.459 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:49.459 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.459 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.464 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.464 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.466 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3fd1976a-f63f-513a-45bb-59c7eef2e263 +29:49.477 [INFO] TestAgent.server: federation state anti-entropy synced +29:49.478 [INFO] TestAgent: Requesting shutdown +29:49.478 [INFO] TestAgent.server: shutting down server +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:49.478 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:49.478 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:49.478 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:49.478 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:49.482 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:49.487Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:49.487Z [INFO] agent.router.manager: shutting down +29:49.487 [INFO] TestAgent: consul server down +29:49.487 [INFO] TestAgent: shutdown complete +29:49.487 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29575 network=tcp +29:49.487 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29575 network=udp +29:49.487 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29576 network=tcp +29:49.988 [INFO] TestAgent: Waiting for endpoints to shut down +29:49.988 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_overridden_sidecar -2022-11-07T22:44:48.182Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:48.182Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:48.191Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:48.191Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:48.661 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:78c37c03-677e-8dbb-c9df-7beda74a4b3e Address:127.0.0.1:11209}]" -44:48.661 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11209 [Follower]" leader= -44:48.662 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e.dc1 127.0.0.1 -44:48.662 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e 127.0.0.1 -2022-11-07T22:44:48.662Z [INFO] agent.router: Initializing LAN area manager -44:48.662 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e.dc1 area=wan -44:48.662 [INFO] TestAgent.server: Adding LAN server: server="Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e (Addr: tcp/127.0.0.1:11209) (DC: dc1)" -44:48.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11204 network=udp -44:48.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11204 network=tcp -44:48.663 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11205 network=tcp -44:48.663 [INFO] TestAgent: started state syncer -44:48.663 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11210 network=tcp -44:48.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:48.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11209 [Candidate]" term=2 -44:48.712 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:48.720 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:48.720 [DEBUG] TestAgent.server.raft: vote granted: from=78c37c03-677e-8dbb-c9df-7beda74a4b3e term=2 tally=1 -44:48.720 [INFO] TestAgent.server.raft: election won: tally=1 -44:48.720 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11209 [Leader]" -44:48.720 [INFO] TestAgent.server: cluster leadership acquired -44:48.721 [INFO] TestAgent.server: New leader elected: payload=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:48.727 [INFO] TestAgent.server: initializing acls -44:48.731 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:48.731 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:48.736 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:48.746 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:48.746 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:48.746 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:48.746 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:48.747 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e.dc1 -44:48.747 [INFO] TestAgent.server: Updating LAN server: server="Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e (Addr: tcp/127.0.0.1:11209) (DC: dc1)" -44:48.747 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e.dc1 area=wan -44:48.757 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:48.773 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:48.773 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:48.773 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:48.773 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:48.773 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:48.773 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:48.773 [INFO] TestAgent.server: member joined, marking health alive: member=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:48.782 [INFO] TestAgent.server: federation state anti-entropy synced -44:48.782 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:49.012 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:49.012 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-78c37c03-677e-8dbb-c9df-7beda74a4b3e -44:49.020 [INFO] TestAgent: Requesting shutdown -44:49.020 [INFO] TestAgent.server: shutting down server -44:49.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:49.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:49.020 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:49.020 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:49.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:49.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:49.021 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:49.021 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:49.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:49.021 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:49.024 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:49.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:49.027Z [INFO] agent.router.manager: shutting down -44:49.027 [INFO] TestAgent: consul server down -44:49.027 [INFO] TestAgent: shutdown complete -44:49.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11204 network=tcp -44:49.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11204 network=udp -44:49.027 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11205 network=tcp -44:49.527 [INFO] TestAgent: Waiting for endpoints to shut down -44:49.527 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:49.323Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:49.323Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:49.343Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:49.343Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:49.382 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a75768be-6dea-aaeb-3034-79cd0b407199 Address:127.0.0.1:29587}]" +29:49.383 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a75768be-6dea-aaeb-3034-79cd0b407199.dc1 127.0.0.1 +29:49.383 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a75768be-6dea-aaeb-3034-79cd0b407199 127.0.0.1 +2021-10-05T16:29:49.383Z [INFO] agent.router: Initializing LAN area manager +29:49.384 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29587 [Follower]" leader= +29:49.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29582 network=udp +29:49.384 [INFO] TestAgent.server: Adding LAN server: server="Node-a75768be-6dea-aaeb-3034-79cd0b407199 (Addr: tcp/127.0.0.1:29587) (DC: dc1)" +29:49.384 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a75768be-6dea-aaeb-3034-79cd0b407199.dc1 area=wan +29:49.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29582 network=tcp +29:49.385 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29583 network=tcp +29:49.385 [INFO] TestAgent: started state syncer +29:49.385 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29588 network=tcp +29:49.434 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:49.444 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:49.444 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29587 [Candidate]" term=2 +29:49.460 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:49.460 [DEBUG] TestAgent.server.raft: vote granted: from=a75768be-6dea-aaeb-3034-79cd0b407199 term=2 tally=1 +29:49.460 [INFO] TestAgent.server.raft: election won: tally=1 +29:49.460 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29587 [Leader]" +29:49.460 [INFO] TestAgent.server: cluster leadership acquired +29:49.460 [INFO] TestAgent.server: New leader elected: payload=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.479 [INFO] TestAgent.server: initializing acls +29:49.484 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:49.484 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:49.496 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:49.511 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:49.511 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:49.511 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:49.511 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.511 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a75768be-6dea-aaeb-3034-79cd0b407199.dc1 +29:49.512 [INFO] TestAgent.server: Updating LAN server: server="Node-a75768be-6dea-aaeb-3034-79cd0b407199 (Addr: tcp/127.0.0.1:29587) (DC: dc1)" +29:49.512 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a75768be-6dea-aaeb-3034-79cd0b407199.dc1 area=wan +29:49.530 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:49.548 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:49.548 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:49.548 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:49.548 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:49.548 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:49.548 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.548 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.556 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.563 [INFO] TestAgent.server: federation state anti-entropy synced +29:49.594 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.594 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a75768be-6dea-aaeb-3034-79cd0b407199 +29:49.604 [INFO] TestAgent: Requesting shutdown +29:49.604 [INFO] TestAgent.server: shutting down server +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:49.604 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:49.604 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:49.605 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:49.605 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:49.605 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:49.610 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:49.614Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:49.614Z [INFO] agent.router.manager: shutting down +29:49.614 [INFO] TestAgent: consul server down +29:49.614 [INFO] TestAgent: shutdown complete +29:49.614 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29582 network=tcp +29:49.614 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29582 network=udp +29:49.614 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29583 network=tcp +29:50.116 [INFO] TestAgent: Waiting for endpoints to shut down +29:50.116 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_overridden_sidecar -2022-11-07T22:44:49.337Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:49.337Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:49.346Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:49.346Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:49.368 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:13ceb9aa-1ea5-546d-d1d2-539835489574 Address:127.0.0.1:11216}]" -44:49.368 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11216 [Follower]" leader= -44:49.371 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-13ceb9aa-1ea5-546d-d1d2-539835489574.dc1 127.0.0.1 -44:49.371 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-13ceb9aa-1ea5-546d-d1d2-539835489574 127.0.0.1 -2022-11-07T22:44:49.371Z [INFO] agent.router: Initializing LAN area manager -44:49.374 [INFO] TestAgent.server: Adding LAN server: server="Node-13ceb9aa-1ea5-546d-d1d2-539835489574 (Addr: tcp/127.0.0.1:11216) (DC: dc1)" -44:49.374 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-13ceb9aa-1ea5-546d-d1d2-539835489574.dc1 area=wan -44:49.374 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11211 network=udp -44:49.374 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11211 network=tcp -44:49.375 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11212 network=tcp -44:49.375 [INFO] TestAgent: started state syncer -44:49.375 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11217 network=tcp -44:49.419 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:49.419 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11216 [Candidate]" term=2 -44:49.424 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:49.428 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:49.428 [DEBUG] TestAgent.server.raft: vote granted: from=13ceb9aa-1ea5-546d-d1d2-539835489574 term=2 tally=1 -44:49.428 [INFO] TestAgent.server.raft: election won: tally=1 -44:49.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11216 [Leader]" -44:49.428 [INFO] TestAgent.server: cluster leadership acquired -44:49.428 [INFO] TestAgent.server: New leader elected: payload=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.433 [INFO] TestAgent.server: initializing acls -44:49.436 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:49.436 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:49.442 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:49.450 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:49.450 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:49.450 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:49.450 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.450 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-13ceb9aa-1ea5-546d-d1d2-539835489574.dc1 -44:49.450 [INFO] TestAgent.server: Updating LAN server: server="Node-13ceb9aa-1ea5-546d-d1d2-539835489574 (Addr: tcp/127.0.0.1:11216) (DC: dc1)" -44:49.450 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-13ceb9aa-1ea5-546d-d1d2-539835489574.dc1 area=wan -44:49.460 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:49.474 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:49.475 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:49.475 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:49.475 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:49.475 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:49.475 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.475 [INFO] TestAgent.server: member joined, marking health alive: member=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.480 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.484 [INFO] TestAgent.server: federation state anti-entropy synced -44:49.794 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:49.801 [INFO] TestAgent: Synced node info -44:49.801 [DEBUG] TestAgent: Node info in sync -44:49.803 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.804 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-13ceb9aa-1ea5-546d-d1d2-539835489574 -44:49.810 [INFO] TestAgent: Requesting shutdown -44:49.810 [INFO] TestAgent.server: shutting down server -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:49.810 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:49.810 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:49.810 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:49.811 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:49.811 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:49.811 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:49.811 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:49.811 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:49.811 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:49.814 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:49.817Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:49.817Z [INFO] agent.router.manager: shutting down -44:49.817 [INFO] TestAgent: consul server down -44:49.817 [INFO] TestAgent: shutdown complete -44:49.817 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11211 network=tcp -44:49.818 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11211 network=udp -44:49.818 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11212 network=tcp -44:50.318 [INFO] TestAgent: Waiting for endpoints to shut down -44:50.318 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:50.003Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.004Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:50.028Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.028Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:50.063 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6fca75e1-699d-616e-9856-d99e52fa5c6b Address:127.0.0.1:29594}]" +29:50.063 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29594 [Follower]" leader= +29:50.064 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6fca75e1-699d-616e-9856-d99e52fa5c6b.dc1 127.0.0.1 +29:50.065 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6fca75e1-699d-616e-9856-d99e52fa5c6b 127.0.0.1 +2021-10-05T16:29:50.065Z [INFO] agent.router: Initializing LAN area manager +29:50.065 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b.dc1 area=wan +29:50.073 [INFO] TestAgent.server: Adding LAN server: server="Node-6fca75e1-699d-616e-9856-d99e52fa5c6b (Addr: tcp/127.0.0.1:29594) (DC: dc1)" +29:50.076 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29589 network=udp +29:50.076 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29589 network=tcp +29:50.077 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29590 network=tcp +29:50.077 [INFO] TestAgent: started state syncer +29:50.077 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29595 network=tcp +29:50.112 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:50.112 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29594 [Candidate]" term=2 +29:50.116 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:50.160 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:50.160 [DEBUG] TestAgent.server.raft: vote granted: from=6fca75e1-699d-616e-9856-d99e52fa5c6b term=2 tally=1 +29:50.160 [INFO] TestAgent.server.raft: election won: tally=1 +29:50.160 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29594 [Leader]" +29:50.160 [INFO] TestAgent.server: cluster leadership acquired +29:50.160 [INFO] TestAgent.server: New leader elected: payload=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.168 [INFO] TestAgent.server: initializing acls +29:50.171 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:50.171 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:50.177 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:50.187 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:50.187 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:50.187 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:50.187 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.187 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6fca75e1-699d-616e-9856-d99e52fa5c6b.dc1 +29:50.187 [INFO] TestAgent.server: Updating LAN server: server="Node-6fca75e1-699d-616e-9856-d99e52fa5c6b (Addr: tcp/127.0.0.1:29594) (DC: dc1)" +29:50.188 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b.dc1 area=wan +29:50.197 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:50.234 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:50.234 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:50.234 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:50.234 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:50.234 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:50.234 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.234 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.263 [INFO] TestAgent.server: federation state anti-entropy synced +29:50.309 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.309 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6fca75e1-699d-616e-9856-d99e52fa5c6b +29:50.322 [INFO] TestAgent: Requesting shutdown +29:50.323 [INFO] TestAgent.server: shutting down server +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:50.323 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:50.323 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:50.323 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:50.323 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:50.323 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:50.326 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:50.333Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:50.333Z [INFO] agent.router.manager: shutting down +29:50.334 [INFO] TestAgent: consul server down +29:50.334 [INFO] TestAgent: shutdown complete +29:50.334 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29589 network=tcp +29:50.334 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29589 network=udp +29:50.334 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29590 network=tcp +29:50.840 [INFO] TestAgent: Waiting for endpoints to shut down +29:50.840 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_and_overridden_for_sidecar -2022-11-07T22:44:49.539Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:49.539Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:49.548Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:49.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:49.568 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:df480eab-2b3e-e072-d337-94d26ca642bc Address:127.0.0.1:11223}]" -44:49.568 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11223 [Follower]" leader= -44:49.568 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-df480eab-2b3e-e072-d337-94d26ca642bc.dc1 127.0.0.1 -44:49.569 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-df480eab-2b3e-e072-d337-94d26ca642bc 127.0.0.1 -2022-11-07T22:44:49.569Z [INFO] agent.router: Initializing LAN area manager -44:49.569 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-df480eab-2b3e-e072-d337-94d26ca642bc.dc1 area=wan -44:49.569 [INFO] TestAgent.server: Adding LAN server: server="Node-df480eab-2b3e-e072-d337-94d26ca642bc (Addr: tcp/127.0.0.1:11223) (DC: dc1)" -44:49.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11218 network=udp -44:49.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11218 network=tcp -44:49.570 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11219 network=tcp -44:49.570 [INFO] TestAgent: started state syncer -44:49.570 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11224 network=tcp -44:49.619 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:49.621 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:49.621 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11223 [Candidate]" term=2 -44:49.629 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:49.629 [DEBUG] TestAgent.server.raft: vote granted: from=df480eab-2b3e-e072-d337-94d26ca642bc term=2 tally=1 -44:49.629 [INFO] TestAgent.server.raft: election won: tally=1 -44:49.629 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11223 [Leader]" -44:49.629 [INFO] TestAgent.server: cluster leadership acquired -44:49.630 [INFO] TestAgent.server: New leader elected: payload=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.636 [INFO] TestAgent.server: initializing acls -44:49.641 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:49.641 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:49.648 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:49.656 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:49.656 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:49.656 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:49.656 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.656 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-df480eab-2b3e-e072-d337-94d26ca642bc.dc1 -44:49.656 [INFO] TestAgent.server: Updating LAN server: server="Node-df480eab-2b3e-e072-d337-94d26ca642bc (Addr: tcp/127.0.0.1:11223) (DC: dc1)" -44:49.656 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-df480eab-2b3e-e072-d337-94d26ca642bc.dc1 area=wan -44:49.666 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:49.681 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:49.681 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:49.681 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:49.681 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:49.681 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:49.681 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.681 [INFO] TestAgent.server: member joined, marking health alive: member=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.690 [INFO] TestAgent.server: federation state anti-entropy synced -44:49.817 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.817 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-df480eab-2b3e-e072-d337-94d26ca642bc -44:49.826 [INFO] TestAgent: Requesting shutdown -44:49.826 [INFO] TestAgent.server: shutting down server -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:49.826 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:49.826 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:49.826 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:49.826 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:49.826 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:49.829 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:49.833Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:49.833Z [INFO] agent.router.manager: shutting down -44:49.833 [INFO] TestAgent: consul server down -44:49.833 [INFO] TestAgent: shutdown complete -44:49.833 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11218 network=tcp -44:49.833 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11218 network=udp -44:49.833 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11219 network=tcp -44:50.334 [INFO] TestAgent: Waiting for endpoints to shut down -44:50.334 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:50.157Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.157Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:50.187Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.187Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:50.280 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f58ae808-c3f5-c748-47d3-1ef4656dadc2 Address:127.0.0.1:29601}]" +29:50.281 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2.dc1 127.0.0.1 +29:50.281 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 127.0.0.1 +2021-10-05T16:29:50.281Z [INFO] agent.router: Initializing LAN area manager +29:50.281 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=udp +29:50.281 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29601 [Follower]" leader= +29:50.282 [INFO] TestAgent.server: Adding LAN server: server="Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 (Addr: tcp/127.0.0.1:29601) (DC: dc1)" +29:50.282 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2.dc1 area=wan +29:50.282 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=tcp +29:50.282 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29597 network=tcp +29:50.282 [INFO] TestAgent: started state syncer +29:50.282 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29602 network=tcp +29:50.327 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:50.327 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29601 [Candidate]" term=2 +29:50.332 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:50.357 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:50.358 [DEBUG] TestAgent.server.raft: vote granted: from=f58ae808-c3f5-c748-47d3-1ef4656dadc2 term=2 tally=1 +29:50.358 [INFO] TestAgent.server.raft: election won: tally=1 +29:50.358 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29601 [Leader]" +29:50.358 [INFO] TestAgent.server: cluster leadership acquired +29:50.358 [INFO] TestAgent.server: New leader elected: payload=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.368 [INFO] TestAgent.server: initializing acls +29:50.384 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:50.384 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:50.400 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:50.424 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:50.424 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:50.424 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:50.424 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.424 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2.dc1 +29:50.425 [INFO] TestAgent.server: Updating LAN server: server="Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 (Addr: tcp/127.0.0.1:29601) (DC: dc1)" +29:50.425 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2.dc1 area=wan +29:50.442 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:50.462 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:50.462 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:50.462 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:50.462 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:50.462 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:50.462 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.462 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.469 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.474 [INFO] TestAgent.server: federation state anti-entropy synced +29:50.486 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f58ae808-c3f5-c748-47d3-1ef4656dadc2 +29:50.499 [INFO] TestAgent: Requesting shutdown +29:50.499 [INFO] TestAgent.server: shutting down server +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:50.499 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:50.499 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:50.499 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:50.499 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:50.499 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:50.504 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:50.508Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:50.508Z [INFO] agent.router.manager: shutting down +29:50.508 [INFO] TestAgent: consul server down +29:50.508 [INFO] TestAgent: shutdown complete +29:50.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=tcp +29:50.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=udp +29:50.508 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29597 network=tcp +29:51.009 [INFO] TestAgent: Waiting for endpoints to shut down +29:51.009 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_and_overridden_for_sidecar -2022-11-07T22:44:50.328Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:50.328Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:50.342Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:50.342Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:50.419 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9c32e4e4-1577-0acc-e6db-1be49cf4c91d Address:127.0.0.1:11230}]" -44:50.419 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11230 [Follower]" leader= -44:50.419 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d.dc1 127.0.0.1 -44:50.420 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d 127.0.0.1 -2022-11-07T22:44:50.420Z [INFO] agent.router: Initializing LAN area manager -44:50.420 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11225 network=udp -44:50.420 [INFO] TestAgent.server: Adding LAN server: server="Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d (Addr: tcp/127.0.0.1:11230) (DC: dc1)" -44:50.420 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d.dc1 area=wan -44:50.420 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11225 network=tcp -44:50.420 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11226 network=tcp -44:50.421 [INFO] TestAgent: started state syncer -44:50.421 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11231 network=tcp -44:50.470 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:50.483 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:50.483 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11230 [Candidate]" term=2 -44:50.491 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:50.491 [DEBUG] TestAgent.server.raft: vote granted: from=9c32e4e4-1577-0acc-e6db-1be49cf4c91d term=2 tally=1 -44:50.491 [INFO] TestAgent.server.raft: election won: tally=1 -44:50.491 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11230 [Leader]" -44:50.491 [INFO] TestAgent.server: cluster leadership acquired -44:50.491 [INFO] TestAgent.server: New leader elected: payload=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.497 [INFO] TestAgent.server: initializing acls -44:50.500 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:50.500 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:50.505 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:50.515 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:50.515 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:50.515 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:50.515 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.515 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d.dc1 -44:50.515 [INFO] TestAgent.server: Updating LAN server: server="Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d (Addr: tcp/127.0.0.1:11230) (DC: dc1)" -44:50.515 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d.dc1 area=wan -44:50.525 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:50.541 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:50.541 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:50.541 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:50.541 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:50.541 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:50.541 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.541 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.549 [INFO] TestAgent.server: federation state anti-entropy synced -44:50.549 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.623 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.623 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-9c32e4e4-1577-0acc-e6db-1be49cf4c91d -44:50.652 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -44:50.656 [INFO] TestAgent: Synced node info -44:50.659 [INFO] TestAgent: Synced service: service=web -44:50.662 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:50.662 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:50.662 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:50.676 [DEBUG] TestAgent: removed service: service=web -44:50.676 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:50.676 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:50.676 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:50.676 [DEBUG] TestAgent: Node info in sync -44:50.679 [INFO] TestAgent: Deregistered service: service=web -44:50.682 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:50.682 [INFO] TestAgent: Requesting shutdown -44:50.682 [INFO] TestAgent.server: shutting down server -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:50.682 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:50.682 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:50.682 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:50.682 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:50.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:50.686 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:50.689Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:50.689Z [INFO] agent.router.manager: shutting down -44:50.689 [INFO] TestAgent: consul server down -44:50.689 [INFO] TestAgent: shutdown complete -44:50.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11225 network=tcp -44:50.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11225 network=udp -44:50.689 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11226 network=tcp -44:51.190 [INFO] TestAgent: Waiting for endpoints to shut down -44:51.190 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:50.859Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.859Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:50.886Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:50.886Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:50.944 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3dbb1ce0-57f6-2e87-339a-64648aa2ace1 Address:127.0.0.1:29608}]" +29:50.945 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1.dc1 127.0.0.1 +29:50.945 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 127.0.0.1 +2021-10-05T16:29:50.945Z [INFO] agent.router: Initializing LAN area manager +29:50.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=udp +29:50.946 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29608 [Follower]" leader= +29:50.946 [INFO] TestAgent.server: Adding LAN server: server="Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 (Addr: tcp/127.0.0.1:29608) (DC: dc1)" +29:50.946 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1.dc1 area=wan +29:50.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=tcp +29:50.947 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29604 network=tcp +29:50.947 [INFO] TestAgent: started state syncer +29:50.947 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29609 network=tcp +29:50.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:51.009 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.009 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29608 [Candidate]" term=2 +29:51.025 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.025 [DEBUG] TestAgent.server.raft: vote granted: from=3dbb1ce0-57f6-2e87-339a-64648aa2ace1 term=2 tally=1 +29:51.025 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.025 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29608 [Leader]" +29:51.028 [INFO] TestAgent.server: cluster leadership acquired +29:51.028 [INFO] TestAgent.server: New leader elected: payload=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.040 [INFO] TestAgent.server: initializing acls +29:51.044 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.044 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.059 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.086 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.086 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.086 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.087 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.087 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1.dc1 +29:51.087 [INFO] TestAgent.server: Updating LAN server: server="Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 (Addr: tcp/127.0.0.1:29608) (DC: dc1)" +29:51.088 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1.dc1 area=wan +29:51.104 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.119 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.119 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.119 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.119 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.119 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.119 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.119 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.142 [INFO] TestAgent.server: federation state anti-entropy synced +29:51.143 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.154 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3dbb1ce0-57f6-2e87-339a-64648aa2ace1 +29:51.234 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +29:51.240 [INFO] TestAgent: Synced node info +29:51.252 [INFO] TestAgent: Synced service: service=web +29:51.257 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:51.257 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:51.257 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:51.292 [DEBUG] TestAgent: removed service: service=web +29:51.293 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:51.293 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:51.293 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:51.293 [DEBUG] TestAgent: Node info in sync +29:51.293 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:51.302 [INFO] TestAgent: Deregistered service: service=web +29:51.307 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:51.307 [INFO] TestAgent: Requesting shutdown +29:51.307 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:51.307 [INFO] TestAgent.server: shutting down server +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.307 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.307 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.307 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.307 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.307 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.316 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.324Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.324Z [INFO] agent.router.manager: shutting down +29:51.324 [INFO] TestAgent: consul server down +29:51.324 [INFO] TestAgent: shutdown complete +29:51.324 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=tcp +29:51.324 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=udp +29:51.324 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29604 network=tcp +29:51.828 [INFO] TestAgent: Waiting for endpoints to shut down +29:51.828 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_definition_in_sidecar -2022-11-07T22:44:50.347Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:50.347Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:50.359Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:50.359Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:50.419 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0795ac45-5c23-8cae-243e-4885dd36531b Address:127.0.0.1:11237}]" -44:50.419 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11237 [Follower]" leader= -44:50.420 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0795ac45-5c23-8cae-243e-4885dd36531b.dc1 127.0.0.1 -44:50.420 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0795ac45-5c23-8cae-243e-4885dd36531b 127.0.0.1 -2022-11-07T22:44:50.420Z [INFO] agent.router: Initializing LAN area manager -44:50.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11232 network=udp -44:50.421 [INFO] TestAgent.server: Adding LAN server: server="Node-0795ac45-5c23-8cae-243e-4885dd36531b (Addr: tcp/127.0.0.1:11237) (DC: dc1)" -44:50.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11232 network=tcp -44:50.421 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0795ac45-5c23-8cae-243e-4885dd36531b.dc1 area=wan -44:50.421 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11233 network=tcp -44:50.421 [INFO] TestAgent: started state syncer -44:50.421 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11238 network=tcp -44:50.470 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:50.470 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11237 [Candidate]" term=2 -44:50.471 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:50.477 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:50.477 [DEBUG] TestAgent.server.raft: vote granted: from=0795ac45-5c23-8cae-243e-4885dd36531b term=2 tally=1 -44:50.477 [INFO] TestAgent.server.raft: election won: tally=1 -44:50.477 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11237 [Leader]" -44:50.478 [INFO] TestAgent.server: cluster leadership acquired -44:50.478 [INFO] TestAgent.server: New leader elected: payload=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.483 [INFO] TestAgent.server: initializing acls -44:50.486 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:50.486 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:50.492 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:50.499 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:50.499 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:50.499 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:50.500 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.500 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0795ac45-5c23-8cae-243e-4885dd36531b.dc1 -44:50.500 [INFO] TestAgent.server: Updating LAN server: server="Node-0795ac45-5c23-8cae-243e-4885dd36531b (Addr: tcp/127.0.0.1:11237) (DC: dc1)" -44:50.500 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0795ac45-5c23-8cae-243e-4885dd36531b.dc1 area=wan -44:50.510 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:50.525 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:50.525 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:50.525 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:50.525 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:50.525 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:50.525 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.525 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.529 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.533 [INFO] TestAgent.server: federation state anti-entropy synced -44:50.579 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:50.584 [INFO] TestAgent: Synced node info -44:50.584 [DEBUG] TestAgent: Node info in sync -44:50.779 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.779 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0795ac45-5c23-8cae-243e-4885dd36531b -44:50.801 [DEBUG] TestAgent: Node info in sync -44:50.804 [INFO] TestAgent: Synced service: service=web -44:50.807 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:50.807 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:50.807 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:50.807 [DEBUG] TestAgent: removed service: service=web -44:50.807 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:50.807 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:50.807 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:50.807 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:50.807 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:50.807 [DEBUG] TestAgent: Node info in sync -44:50.810 [INFO] TestAgent: Deregistered service: service=web -44:50.815 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:50.815 [INFO] TestAgent: Requesting shutdown -44:50.815 [INFO] TestAgent.server: shutting down server -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:50.815 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:50.815 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:50.815 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:50.815 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:50.819 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:50.822Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:50.822Z [INFO] agent.router.manager: shutting down -44:50.822 [INFO] TestAgent: consul server down -44:50.822 [INFO] TestAgent: shutdown complete -44:50.822 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11232 network=tcp -44:50.822 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11232 network=udp -44:50.822 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11233 network=tcp -44:51.322 [INFO] TestAgent: Waiting for endpoints to shut down -44:51.322 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:51.026Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:51.026Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:51.048Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:51.048Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.102 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76d577c9-9558-42da-b8f2-fe3cb97afa4a Address:127.0.0.1:29615}]" +29:51.102 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29615 [Follower]" leader= +29:51.102 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 127.0.0.1 +29:51.103 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a 127.0.0.1 +2021-10-05T16:29:51.103Z [INFO] agent.router: Initializing LAN area manager +29:51.103 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29610 network=udp +29:51.103 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 area=wan +29:51.103 [INFO] TestAgent.server: Adding LAN server: server="Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a (Addr: tcp/127.0.0.1:29615) (DC: dc1)" +29:51.103 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29610 network=tcp +29:51.104 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29611 network=tcp +29:51.104 [INFO] TestAgent: started state syncer +29:51.104 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29616 network=tcp +29:51.141 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.141 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29615 [Candidate]" term=2 +29:51.154 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.154 [DEBUG] TestAgent.server.raft: vote granted: from=76d577c9-9558-42da-b8f2-fe3cb97afa4a term=2 tally=1 +29:51.154 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.154 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29615 [Leader]" +29:51.156 [INFO] TestAgent.server: cluster leadership acquired +29:51.156 [INFO] TestAgent.server: initializing acls +29:51.156 [INFO] TestAgent.server: New leader elected: payload=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.176 [INFO] TestAgent.server: initializing acls +29:51.176 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.177 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.177 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.191 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.217 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.236 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.236 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.236 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.236 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:51.236 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.236 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 +29:51.236 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 area=wan +29:51.236 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.237 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.237 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 +29:51.237 [INFO] TestAgent.server: Updating LAN server: server="Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a (Addr: tcp/127.0.0.1:29615) (DC: dc1)" +29:51.237 [INFO] TestAgent.server: Updating LAN server: server="Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a (Addr: tcp/127.0.0.1:29615) (DC: dc1)" +29:51.237 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a.dc1 area=wan +29:51.266 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.302 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.302 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.302 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.302 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.302 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.303 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.303 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.316 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.316 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.331 [INFO] TestAgent.server: federation state anti-entropy synced +29:51.349 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:51.359 [INFO] TestAgent: Synced node info +29:51.359 [DEBUG] TestAgent: Node info in sync +29:51.411 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.411 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-76d577c9-9558-42da-b8f2-fe3cb97afa4a +29:51.464 [DEBUG] TestAgent: Node info in sync +29:51.469 [INFO] TestAgent: Synced service: service=web +29:51.477 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:51.477 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:51.477 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:51.477 [DEBUG] TestAgent: removed service: service=web +29:51.478 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:51.478 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:51.478 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:51.478 [DEBUG] TestAgent: Node info in sync +29:51.482 [INFO] TestAgent: Deregistered service: service=web +29:51.489 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:51.489 [INFO] TestAgent: Requesting shutdown +29:51.489 [INFO] TestAgent.server: shutting down server +29:51.489 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.489 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.490 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.490 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.490 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.490 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.490 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.496 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.503Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.503Z [INFO] agent.router.manager: shutting down +29:51.503 [INFO] TestAgent: consul server down +29:51.503 [INFO] TestAgent: shutdown complete +29:51.503 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29610 network=tcp +29:51.503 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29610 network=udp +29:51.503 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29611 network=tcp +29:52.003 [INFO] TestAgent: Waiting for endpoints to shut down +29:52.003 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_definition_in_sidecar -2022-11-07T22:44:51.200Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:51.209Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:51.239 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f Address:127.0.0.1:11244}]" -44:51.239 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11244 [Follower]" leader= -44:51.239 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f.dc1 127.0.0.1 -44:51.240 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f 127.0.0.1 -2022-11-07T22:44:51.240Z [INFO] agent.router: Initializing LAN area manager -44:51.240 [INFO] TestAgent.server: Adding LAN server: server="Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f (Addr: tcp/127.0.0.1:11244) (DC: dc1)" -44:51.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11239 network=udp -44:51.240 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f.dc1 area=wan -44:51.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11239 network=tcp -44:51.241 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11240 network=tcp -44:51.241 [INFO] TestAgent: started state syncer -44:51.241 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11245 network=tcp -44:51.295 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:51.295 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11244 [Candidate]" term=2 -44:51.303 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:51.303 [DEBUG] TestAgent.server.raft: vote granted: from=14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f term=2 tally=1 -44:51.303 [INFO] TestAgent.server.raft: election won: tally=1 -44:51.303 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11244 [Leader]" -44:51.303 [INFO] TestAgent.server: cluster leadership acquired -44:51.303 [INFO] TestAgent.server: New leader elected: payload=Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f -44:51.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11244 -44:51.320 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:51.340 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:51.340 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:51.340 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:51.340 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:51.340 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:51.340 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f -44:51.340 [INFO] TestAgent.server: member joined, marking health alive: member=Node-14420a3e-f4e4-8d56-83ea-df1cdf6c1e3f -44:51.352 [INFO] TestAgent.server: federation state anti-entropy synced -44:51.378 [INFO] TestAgent: Requesting shutdown -44:51.378 [INFO] TestAgent.server: shutting down server -44:51.378 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:51.378 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:51.378 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:51.378 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:51.378 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:51.379 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:51.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:51.379 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:51.379 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:51.379 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:51.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:51.382 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:51.386Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:51.386Z [INFO] agent.router.manager: shutting down -44:51.386 [INFO] TestAgent: consul server down -44:51.386 [INFO] TestAgent: shutdown complete -44:51.386 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11239 network=tcp -44:51.386 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11239 network=udp -44:51.386 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11240 network=tcp -44:51.886 [INFO] TestAgent: Waiting for endpoints to shut down -44:51.886 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:51.880Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:51.916Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.969 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f709500d-5a12-8f63-c481-443953c68c54 Address:127.0.0.1:29622}]" +29:51.970 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f709500d-5a12-8f63-c481-443953c68c54.dc1 127.0.0.1 +29:51.970 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f709500d-5a12-8f63-c481-443953c68c54 127.0.0.1 +2021-10-05T16:29:51.970Z [INFO] agent.router: Initializing LAN area manager +29:51.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29617 network=udp +29:51.971 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29622 [Follower]" leader= +29:51.971 [INFO] TestAgent.server: Adding LAN server: server="Node-f709500d-5a12-8f63-c481-443953c68c54 (Addr: tcp/127.0.0.1:29622) (DC: dc1)" +29:51.971 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f709500d-5a12-8f63-c481-443953c68c54.dc1 area=wan +29:51.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29617 network=tcp +29:51.972 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29618 network=tcp +29:51.972 [INFO] TestAgent: started state syncer +29:51.972 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29623 network=tcp +29:52.040 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:52.040 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29622 [Candidate]" term=2 +29:52.057 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:52.057 [DEBUG] TestAgent.server.raft: vote granted: from=f709500d-5a12-8f63-c481-443953c68c54 term=2 tally=1 +29:52.057 [INFO] TestAgent.server.raft: election won: tally=1 +29:52.057 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29622 [Leader]" +29:52.058 [INFO] TestAgent.server: cluster leadership acquired +29:52.058 [INFO] TestAgent.server: New leader elected: payload=Node-f709500d-5a12-8f63-c481-443953c68c54 +29:52.065 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29622 +29:52.078 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:52.096 [INFO] TestAgent: Synced node info +29:52.096 [DEBUG] TestAgent: Node info in sync +29:52.112 [DEBUG] TestAgent: Node info in sync +29:52.123 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:52.123 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:52.123 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:52.123 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:52.123 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:52.123 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f709500d-5a12-8f63-c481-443953c68c54 +29:52.123 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f709500d-5a12-8f63-c481-443953c68c54 +29:52.152 [INFO] TestAgent.server: federation state anti-entropy synced +29:52.287 [INFO] TestAgent: Requesting shutdown +29:52.287 [INFO] TestAgent.server: shutting down server +29:52.287 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:52.287 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:52.287 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:52.287 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:52.287 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:52.287 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:52.287 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:52.287 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:52.287 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:52.287 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:52.301 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:52.307Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:52.307Z [INFO] agent.router.manager: shutting down +29:52.307 [INFO] TestAgent: consul server down +29:52.307 [INFO] TestAgent: shutdown complete +29:52.307 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29617 network=tcp +29:52.307 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29617 network=udp +29:52.307 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29618 network=tcp +29:52.808 [INFO] TestAgent: Waiting for endpoints to shut down +29:52.808 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_definitions_in_sidecar -2022-11-07T22:44:51.334Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:51.344Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:51.367 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:68090945-bee7-3619-3fcd-32071850ff59 Address:127.0.0.1:11251}]" -44:51.367 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11251 [Follower]" leader= -44:51.368 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-68090945-bee7-3619-3fcd-32071850ff59.dc1 127.0.0.1 -44:51.375 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-68090945-bee7-3619-3fcd-32071850ff59 127.0.0.1 -2022-11-07T22:44:51.375Z [INFO] agent.router: Initializing LAN area manager -44:51.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11246 network=udp -44:51.375 [INFO] TestAgent.server: Adding LAN server: server="Node-68090945-bee7-3619-3fcd-32071850ff59 (Addr: tcp/127.0.0.1:11251) (DC: dc1)" -44:51.375 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-68090945-bee7-3619-3fcd-32071850ff59.dc1 area=wan -44:51.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11246 network=tcp -44:51.376 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11247 network=tcp -44:51.376 [INFO] TestAgent: started state syncer -44:51.376 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11252 network=tcp -44:51.403 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:51.403 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11251 [Candidate]" term=2 -44:51.410 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:51.410 [DEBUG] TestAgent.server.raft: vote granted: from=68090945-bee7-3619-3fcd-32071850ff59 term=2 tally=1 -44:51.410 [INFO] TestAgent.server.raft: election won: tally=1 -44:51.410 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11251 [Leader]" -44:51.410 [INFO] TestAgent.server: cluster leadership acquired -44:51.411 [INFO] TestAgent.server: New leader elected: payload=Node-68090945-bee7-3619-3fcd-32071850ff59 -44:51.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11251 -44:51.426 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:51.444 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:51.444 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:51.444 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:51.444 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:51.444 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:51.444 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-68090945-bee7-3619-3fcd-32071850ff59 -44:51.444 [INFO] TestAgent.server: member joined, marking health alive: member=Node-68090945-bee7-3619-3fcd-32071850ff59 -44:51.453 [INFO] TestAgent.server: federation state anti-entropy synced -44:51.687 [INFO] TestAgent: Requesting shutdown -44:51.687 [INFO] TestAgent.server: shutting down server -44:51.687 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:51.687 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:51.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:51.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:51.687 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:51.687 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:51.687 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:51.687 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:51.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:51.687 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:51.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:51.691 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:51.698Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:51.698Z [INFO] agent.router.manager: shutting down -44:51.698 [INFO] TestAgent: consul server down -44:51.698 [INFO] TestAgent: shutdown complete -44:51.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11246 network=tcp -44:51.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11246 network=udp -44:51.698 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11247 network=tcp -44:52.199 [INFO] TestAgent: Waiting for endpoints to shut down -44:52.199 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:52.019Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:52.034Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:52.120 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:046def86-ccf2-1ecd-af27-611867d0d636 Address:127.0.0.1:29629}]" +29:52.120 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-046def86-ccf2-1ecd-af27-611867d0d636.dc1 127.0.0.1 +29:52.121 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-046def86-ccf2-1ecd-af27-611867d0d636 127.0.0.1 +2021-10-05T16:29:52.121Z [INFO] agent.router: Initializing LAN area manager +29:52.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29624 network=udp +29:52.121 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29629 [Follower]" leader= +29:52.122 [INFO] TestAgent.server: Adding LAN server: server="Node-046def86-ccf2-1ecd-af27-611867d0d636 (Addr: tcp/127.0.0.1:29629) (DC: dc1)" +29:52.122 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-046def86-ccf2-1ecd-af27-611867d0d636.dc1 area=wan +29:52.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29624 network=tcp +29:52.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29625 network=tcp +29:52.122 [INFO] TestAgent: started state syncer +29:52.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29630 network=tcp +29:52.166 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:52.166 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29629 [Candidate]" term=2 +29:52.175 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:52.175 [DEBUG] TestAgent.server.raft: vote granted: from=046def86-ccf2-1ecd-af27-611867d0d636 term=2 tally=1 +29:52.175 [INFO] TestAgent.server.raft: election won: tally=1 +29:52.175 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29629 [Leader]" +29:52.176 [INFO] TestAgent.server: cluster leadership acquired +29:52.176 [INFO] TestAgent.server: New leader elected: payload=Node-046def86-ccf2-1ecd-af27-611867d0d636 +29:52.180 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29629 +29:52.225 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:52.250 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:52.250 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:52.250 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:52.250 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:52.250 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:52.250 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-046def86-ccf2-1ecd-af27-611867d0d636 +29:52.250 [INFO] TestAgent.server: member joined, marking health alive: member=Node-046def86-ccf2-1ecd-af27-611867d0d636 +29:52.263 [INFO] TestAgent.server: federation state anti-entropy synced +29:52.301 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:52.311 [INFO] TestAgent: Synced node info +29:52.334 [INFO] TestAgent: Requesting shutdown +29:52.334 [INFO] TestAgent.server: shutting down server +29:52.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:52.334 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:52.334 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:52.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:52.334 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:52.334 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:52.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:52.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:52.334 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:52.334 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:52.340 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:52.347Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:52.347Z [INFO] agent.router.manager: shutting down +29:52.347 [INFO] TestAgent: consul server down +29:52.347 [INFO] TestAgent: shutdown complete +29:52.347 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29624 network=tcp +29:52.347 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29624 network=udp +29:52.347 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29625 network=tcp +29:52.848 [INFO] TestAgent: Waiting for endpoints to shut down +29:52.848 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_definitions_in_sidecar -2022-11-07T22:44:51.898Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:51.907Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:51.927 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3120c71-dad6-b1d5-17a6-c5ee8024f43b Address:127.0.0.1:11258}]" -44:51.927 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11258 [Follower]" leader= -44:51.928 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b.dc1 127.0.0.1 -44:51.928 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b 127.0.0.1 -2022-11-07T22:44:51.928Z [INFO] agent.router: Initializing LAN area manager -44:51.928 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b.dc1 area=wan -44:51.928 [INFO] TestAgent.server: Adding LAN server: server="Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b (Addr: tcp/127.0.0.1:11258) (DC: dc1)" -44:51.929 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11253 network=tcp -44:51.929 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11253 network=udp -44:51.929 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11254 network=tcp -44:51.929 [INFO] TestAgent: started state syncer -44:51.929 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11259 network=tcp -44:51.976 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:51.976 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11258 [Candidate]" term=2 -44:51.985 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:51.985 [DEBUG] TestAgent.server.raft: vote granted: from=f3120c71-dad6-b1d5-17a6-c5ee8024f43b term=2 tally=1 -44:51.985 [INFO] TestAgent.server.raft: election won: tally=1 -44:51.985 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11258 [Leader]" -44:51.986 [INFO] TestAgent.server: cluster leadership acquired -44:51.986 [INFO] TestAgent.server: New leader elected: payload=Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b -44:51.990 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11258 -44:52.002 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:52.025 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:52.025 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:52.025 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:52.025 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:52.025 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:52.025 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b -44:52.025 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f3120c71-dad6-b1d5-17a6-c5ee8024f43b -44:52.036 [INFO] TestAgent.server: federation state anti-entropy synced -44:52.145 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:52.148 [INFO] TestAgent: Synced node info -44:52.148 [DEBUG] TestAgent: Node info in sync -44:52.297 [INFO] TestAgent: Requesting shutdown -44:52.297 [INFO] TestAgent.server: shutting down server -44:52.297 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:52.297 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:52.297 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:52.297 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:52.297 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:52.297 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:52.297 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:52.297 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:52.297 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:52.297 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:52.301 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:52.304Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:52.304Z [INFO] agent.router.manager: shutting down -44:52.305 [INFO] TestAgent: consul server down -44:52.305 [INFO] TestAgent: shutdown complete -44:52.305 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11253 network=tcp -44:52.305 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11253 network=udp -44:52.305 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11254 network=tcp -44:52.805 [INFO] TestAgent: Waiting for endpoints to shut down -44:52.805 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_status_in_sidecar -2022-11-07T22:44:52.210Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:52.219Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:52.239 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b820bd0-654d-753a-54a6-7fb77fd31b2e Address:127.0.0.1:11265}]" -44:52.239 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11265 [Follower]" leader= -44:52.239 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e.dc1 127.0.0.1 -44:52.240 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e 127.0.0.1 -2022-11-07T22:44:52.240Z [INFO] agent.router: Initializing LAN area manager -44:52.240 [INFO] TestAgent.server: Adding LAN server: server="Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e (Addr: tcp/127.0.0.1:11265) (DC: dc1)" -44:52.240 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e.dc1 area=wan -44:52.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11260 network=udp -44:52.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11260 network=tcp -44:52.242 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11261 network=tcp -44:52.242 [INFO] TestAgent: started state syncer -44:52.242 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11266 network=tcp -44:52.304 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:52.304 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11265 [Candidate]" term=2 -44:52.312 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:52.312 [DEBUG] TestAgent.server.raft: vote granted: from=8b820bd0-654d-753a-54a6-7fb77fd31b2e term=2 tally=1 -44:52.312 [INFO] TestAgent.server.raft: election won: tally=1 -44:52.312 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11265 [Leader]" -44:52.312 [INFO] TestAgent.server: cluster leadership acquired -44:52.312 [INFO] TestAgent.server: New leader elected: payload=Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e -44:52.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11265 -44:52.328 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:52.345 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:52.345 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:52.345 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:52.345 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:52.345 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:52.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e -44:52.345 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b820bd0-654d-753a-54a6-7fb77fd31b2e -44:52.355 [INFO] TestAgent.server: federation state anti-entropy synced -44:52.360 [INFO] TestAgent: Requesting shutdown -44:52.360 [INFO] TestAgent.server: shutting down server -44:52.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:52.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:52.360 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:52.360 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:52.360 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:52.360 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:52.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:52.360 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:52.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:52.360 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:52.360 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:52.363 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:52.367Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:52.367Z [INFO] agent.router.manager: shutting down -44:52.367 [INFO] TestAgent: consul server down -44:52.367 [INFO] TestAgent: shutdown complete -44:52.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11260 network=tcp -44:52.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11260 network=udp -44:52.367 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11261 network=tcp -44:52.867 [INFO] TestAgent: Waiting for endpoints to shut down -44:52.867 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:52.864Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:52.878Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:52.909 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18a9025f-a6f8-654b-a063-89da6a1474f9 Address:127.0.0.1:29643}]" +29:52.910 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18a9025f-a6f8-654b-a063-89da6a1474f9.dc1 127.0.0.1 +29:52.911 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18a9025f-a6f8-654b-a063-89da6a1474f9 127.0.0.1 +2021-10-05T16:29:52.911Z [INFO] agent.router: Initializing LAN area manager +29:52.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29638 network=udp +29:52.911 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29643 [Follower]" leader= +29:52.911 [INFO] TestAgent.server: Adding LAN server: server="Node-18a9025f-a6f8-654b-a063-89da6a1474f9 (Addr: tcp/127.0.0.1:29643) (DC: dc1)" +29:52.911 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18a9025f-a6f8-654b-a063-89da6a1474f9.dc1 area=wan +29:52.912 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29638 network=tcp +29:52.912 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29639 network=tcp +29:52.912 [INFO] TestAgent: started state syncer +29:52.912 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29644 network=tcp +29:52.968 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:52.968 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29643 [Candidate]" term=2 +29:52.996 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:52.996 [DEBUG] TestAgent.server.raft: vote granted: from=18a9025f-a6f8-654b-a063-89da6a1474f9 term=2 tally=1 +29:52.996 [INFO] TestAgent.server.raft: election won: tally=1 +29:52.996 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29643 [Leader]" +29:52.999 [INFO] TestAgent.server: cluster leadership acquired +29:53.000 [INFO] TestAgent.server: New leader elected: payload=Node-18a9025f-a6f8-654b-a063-89da6a1474f9 +29:53.001 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29643 +29:53.033 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:53.072 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:53.072 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:53.072 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:53.072 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:53.072 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:53.072 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18a9025f-a6f8-654b-a063-89da6a1474f9 +29:53.072 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18a9025f-a6f8-654b-a063-89da6a1474f9 +29:53.090 [INFO] TestAgent.server: federation state anti-entropy synced +29:53.116 [INFO] TestAgent: Requesting shutdown +29:53.116 [INFO] TestAgent.server: shutting down server +29:53.116 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:53.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:53.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:53.116 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:53.116 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:53.116 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:53.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:53.116 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:53.116 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:53.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:53.116 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:53.124 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:53.132Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:53.132Z [INFO] agent.router.manager: shutting down +29:53.132 [INFO] TestAgent: consul server down +29:53.132 [INFO] TestAgent: shutdown complete +29:53.132 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29638 network=tcp +29:53.132 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29638 network=udp +29:53.132 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29639 network=tcp +29:53.632 [INFO] TestAgent: Waiting for endpoints to shut down +29:53.632 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_status_in_sidecar -2022-11-07T22:44:52.879Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:52.888Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:52.907 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:862fa45b-79a9-1101-9d61-6eaa688154fd Address:127.0.0.1:11279}]" -44:52.908 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11279 [Follower]" leader= -44:52.908 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-862fa45b-79a9-1101-9d61-6eaa688154fd.dc1 127.0.0.1 -44:52.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-862fa45b-79a9-1101-9d61-6eaa688154fd 127.0.0.1 -2022-11-07T22:44:52.908Z [INFO] agent.router: Initializing LAN area manager -44:52.908 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-862fa45b-79a9-1101-9d61-6eaa688154fd.dc1 area=wan -44:52.908 [INFO] TestAgent.server: Adding LAN server: server="Node-862fa45b-79a9-1101-9d61-6eaa688154fd (Addr: tcp/127.0.0.1:11279) (DC: dc1)" -44:52.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11274 network=udp -44:52.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11274 network=tcp -44:52.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11275 network=tcp -44:52.909 [INFO] TestAgent: started state syncer -44:52.909 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11280 network=tcp -44:52.953 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:52.953 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11279 [Candidate]" term=2 -44:52.963 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:52.963 [DEBUG] TestAgent.server.raft: vote granted: from=862fa45b-79a9-1101-9d61-6eaa688154fd term=2 tally=1 -44:52.963 [INFO] TestAgent.server.raft: election won: tally=1 -44:52.963 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11279 [Leader]" -44:52.963 [INFO] TestAgent.server: cluster leadership acquired -44:52.963 [INFO] TestAgent.server: New leader elected: payload=Node-862fa45b-79a9-1101-9d61-6eaa688154fd -44:52.969 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11279 -44:52.981 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:52.997 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:52.997 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:52.997 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:52.997 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:52.997 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:52.997 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-862fa45b-79a9-1101-9d61-6eaa688154fd -44:52.997 [INFO] TestAgent.server: member joined, marking health alive: member=Node-862fa45b-79a9-1101-9d61-6eaa688154fd -44:53.009 [INFO] TestAgent.server: federation state anti-entropy synced -44:53.032 [INFO] TestAgent: Requesting shutdown -44:53.032 [INFO] TestAgent.server: shutting down server -44:53.032 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:53.032 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:53.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:53.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:53.032 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:53.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:53.032 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:53.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:53.032 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:53.032 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:53.032 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:53.035 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:53.039Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:53.039Z [INFO] agent.router.manager: shutting down -44:53.039 [INFO] TestAgent: consul server down -44:53.039 [INFO] TestAgent: shutdown complete -44:53.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11274 network=tcp -44:53.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11274 network=udp -44:53.039 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11275 network=tcp -44:53.539 [INFO] TestAgent: Waiting for endpoints to shut down -44:53.539 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:52.826Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:52.851Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:52.909 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:849f38e0-4428-49d0-8d30-8a8eb7734157 Address:127.0.0.1:29636}]" +29:52.909 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29636 [Follower]" leader= +29:52.910 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-849f38e0-4428-49d0-8d30-8a8eb7734157.dc1 127.0.0.1 +29:52.910 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-849f38e0-4428-49d0-8d30-8a8eb7734157 127.0.0.1 +2021-10-05T16:29:52.910Z [INFO] agent.router: Initializing LAN area manager +29:52.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=udp +29:52.911 [INFO] TestAgent.server: Adding LAN server: server="Node-849f38e0-4428-49d0-8d30-8a8eb7734157 (Addr: tcp/127.0.0.1:29636) (DC: dc1)" +29:52.911 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-849f38e0-4428-49d0-8d30-8a8eb7734157.dc1 area=wan +29:52.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=tcp +29:52.911 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29632 network=tcp +29:52.911 [INFO] TestAgent: started state syncer +29:52.912 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29637 network=tcp +29:52.975 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:52.975 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29636 [Candidate]" term=2 +29:53.011 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:53.011 [DEBUG] TestAgent.server.raft: vote granted: from=849f38e0-4428-49d0-8d30-8a8eb7734157 term=2 tally=1 +29:53.011 [INFO] TestAgent.server.raft: election won: tally=1 +29:53.011 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29636 [Leader]" +29:53.012 [INFO] TestAgent.server: cluster leadership acquired +29:53.012 [INFO] TestAgent.server: New leader elected: payload=Node-849f38e0-4428-49d0-8d30-8a8eb7734157 +29:53.031 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29636 +29:53.063 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:53.064 [INFO] TestAgent: Synced node info +29:53.064 [DEBUG] TestAgent: Node info in sync +29:53.080 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:53.080 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:53.080 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:53.080 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:53.080 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:53.080 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-849f38e0-4428-49d0-8d30-8a8eb7734157 +29:53.080 [INFO] TestAgent.server: member joined, marking health alive: member=Node-849f38e0-4428-49d0-8d30-8a8eb7734157 +29:53.092 [INFO] TestAgent.server: federation state anti-entropy synced +29:53.164 [INFO] TestAgent: Requesting shutdown +29:53.164 [INFO] TestAgent.server: shutting down server +29:53.164 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:53.164 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:53.164 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:53.165 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:53.165 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:53.165 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:53.165 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:53.165 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:53.165 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:53.165 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:53.169 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:53.172Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:53.172Z [INFO] agent.router.manager: shutting down +29:53.172 [INFO] TestAgent: consul server down +29:53.172 [INFO] TestAgent: shutdown complete +29:53.172 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=tcp +29:53.172 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=udp +29:53.172 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29632 network=tcp +29:53.673 [INFO] TestAgent: Waiting for endpoints to shut down +29:53.673 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_status_in_sidecar +2021-10-05T16:29:53.647Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:53.675Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:53.741 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63388fa6-6b62-ea08-7267-439ad7451fc9 Address:127.0.0.1:29650}]" +29:53.742 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63388fa6-6b62-ea08-7267-439ad7451fc9.dc1 127.0.0.1 +29:53.742 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63388fa6-6b62-ea08-7267-439ad7451fc9 127.0.0.1 +2021-10-05T16:29:53.742Z [INFO] agent.router: Initializing LAN area manager +29:53.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=udp +29:53.742 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29650 [Follower]" leader= +29:53.743 [INFO] TestAgent.server: Adding LAN server: server="Node-63388fa6-6b62-ea08-7267-439ad7451fc9 (Addr: tcp/127.0.0.1:29650) (DC: dc1)" +29:53.743 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63388fa6-6b62-ea08-7267-439ad7451fc9.dc1 area=wan +29:53.743 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=tcp +29:53.743 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29646 network=tcp +29:53.743 [INFO] TestAgent: started state syncer +29:53.743 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29651 network=tcp +29:53.793 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:53.793 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29650 [Candidate]" term=2 +29:53.803 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:53.803 [DEBUG] TestAgent.server.raft: vote granted: from=63388fa6-6b62-ea08-7267-439ad7451fc9 term=2 tally=1 +29:53.803 [INFO] TestAgent.server.raft: election won: tally=1 +29:53.803 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29650 [Leader]" +29:53.804 [INFO] TestAgent.server: cluster leadership acquired +29:53.804 [INFO] TestAgent.server: New leader elected: payload=Node-63388fa6-6b62-ea08-7267-439ad7451fc9 +29:53.813 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29650 +29:53.826 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:53.878 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:53.878 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:53.878 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:53.878 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:53.878 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:53.878 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63388fa6-6b62-ea08-7267-439ad7451fc9 +29:53.878 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63388fa6-6b62-ea08-7267-439ad7451fc9 +29:53.892 [INFO] TestAgent.server: federation state anti-entropy synced +29:53.928 [INFO] TestAgent: Requesting shutdown +29:53.928 [INFO] TestAgent.server: shutting down server +29:53.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:53.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:53.928 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:53.928 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:53.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:53.928 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:53.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:53.928 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:53.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:53.928 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:53.928 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:53.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:53.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:53.948Z [INFO] agent.router.manager: shutting down +29:53.948 [INFO] TestAgent: consul server down +29:53.948 [INFO] TestAgent: shutdown complete +29:53.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=tcp +29:53.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=udp +29:53.948 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29646 network=tcp +29:54.452 [INFO] TestAgent: Waiting for endpoints to shut down +29:54.452 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_status_in_sidecar -2022-11-07T22:44:52.821Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:52.833Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:52.883 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a97534ba-d678-2858-aed8-7b09849b03a7 Address:127.0.0.1:11272}]" -44:52.883 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11272 [Follower]" leader= -44:52.883 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a97534ba-d678-2858-aed8-7b09849b03a7.dc1 127.0.0.1 -44:52.883 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a97534ba-d678-2858-aed8-7b09849b03a7 127.0.0.1 -2022-11-07T22:44:52.883Z [INFO] agent.router: Initializing LAN area manager -44:52.883 [INFO] TestAgent.server: Adding LAN server: server="Node-a97534ba-d678-2858-aed8-7b09849b03a7 (Addr: tcp/127.0.0.1:11272) (DC: dc1)" -44:52.884 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a97534ba-d678-2858-aed8-7b09849b03a7.dc1 area=wan -44:52.884 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11267 network=udp -44:52.884 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11267 network=tcp -44:52.884 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11268 network=tcp -44:52.884 [INFO] TestAgent: started state syncer -44:52.884 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11273 network=tcp -44:52.926 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:52.926 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11272 [Candidate]" term=2 -44:52.934 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:52.934 [DEBUG] TestAgent.server.raft: vote granted: from=a97534ba-d678-2858-aed8-7b09849b03a7 term=2 tally=1 -44:52.934 [INFO] TestAgent.server.raft: election won: tally=1 -44:52.934 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11272 [Leader]" -44:52.934 [INFO] TestAgent.server: cluster leadership acquired -44:52.934 [INFO] TestAgent.server: New leader elected: payload=Node-a97534ba-d678-2858-aed8-7b09849b03a7 -44:52.940 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11272 -44:52.950 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:52.969 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:52.969 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:52.969 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:52.969 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:52.969 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:52.969 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a97534ba-d678-2858-aed8-7b09849b03a7 -44:52.969 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a97534ba-d678-2858-aed8-7b09849b03a7 -44:52.978 [INFO] TestAgent.server: federation state anti-entropy synced -44:53.170 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:53.283 [INFO] TestAgent: Requesting shutdown -44:53.283 [INFO] TestAgent.server: shutting down server -44:53.283 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:53.283 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:53.283 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:53.283 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:53.283 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:53.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:53.283 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:53.283 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:53.283 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:53.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:53.306 [INFO] TestAgent: Synced node info -44:53.311 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:53.311Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:53.311Z [INFO] agent.router.manager: shutting down -44:53.312 [INFO] TestAgent: consul server down -44:53.312 [INFO] TestAgent: shutdown complete -44:53.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11267 network=tcp -44:53.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11267 network=udp -44:53.312 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11268 network=tcp -44:53.812 [INFO] TestAgent: Waiting for endpoints to shut down -44:53.812 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:53.688Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:53.703Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:53.768 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0d71204f-c968-ced5-1815-18e66aa25111 Address:127.0.0.1:29657}]" +29:53.768 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29657 [Follower]" leader= +29:53.769 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0d71204f-c968-ced5-1815-18e66aa25111.dc1 127.0.0.1 +29:53.769 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0d71204f-c968-ced5-1815-18e66aa25111 127.0.0.1 +2021-10-05T16:29:53.769Z [INFO] agent.router: Initializing LAN area manager +29:53.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=udp +29:53.770 [INFO] TestAgent.server: Adding LAN server: server="Node-0d71204f-c968-ced5-1815-18e66aa25111 (Addr: tcp/127.0.0.1:29657) (DC: dc1)" +29:53.770 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0d71204f-c968-ced5-1815-18e66aa25111.dc1 area=wan +29:53.770 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=tcp +29:53.770 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29653 network=tcp +29:53.770 [INFO] TestAgent: started state syncer +29:53.770 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29658 network=tcp +29:53.836 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:53.836 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29657 [Candidate]" term=2 +29:53.875 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:53.875 [DEBUG] TestAgent.server.raft: vote granted: from=0d71204f-c968-ced5-1815-18e66aa25111 term=2 tally=1 +29:53.875 [INFO] TestAgent.server.raft: election won: tally=1 +29:53.875 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29657 [Leader]" +29:53.876 [INFO] TestAgent.server: cluster leadership acquired +29:53.878 [INFO] TestAgent.server: New leader elected: payload=Node-0d71204f-c968-ced5-1815-18e66aa25111 +29:53.882 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29657 +29:53.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:53.951 [INFO] TestAgent: Synced node info +29:53.951 [DEBUG] TestAgent: Node info in sync +29:53.982 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:53.982 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:53.982 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:53.982 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:53.982 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:53.982 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0d71204f-c968-ced5-1815-18e66aa25111 +29:53.982 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0d71204f-c968-ced5-1815-18e66aa25111 +29:54.016 [INFO] TestAgent.server: federation state anti-entropy synced +29:54.024 [INFO] TestAgent: Requesting shutdown +29:54.024 [INFO] TestAgent.server: shutting down server +29:54.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:54.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:54.024 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:54.024 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:54.024 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:54.024 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:54.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:54.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:54.024 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:54.024 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:54.035 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:54.043Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:54.043Z [INFO] agent.router.manager: shutting down +29:54.043 [INFO] TestAgent: consul server down +29:54.043 [INFO] TestAgent: shutdown complete +29:54.043 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=tcp +29:54.043 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=udp +29:54.043 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29653 network=tcp +29:54.543 [INFO] TestAgent: Waiting for endpoints to shut down +29:54.543 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_status_in_sidecar -2022-11-07T22:44:53.557Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.573Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.594 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:134842f3-7f17-37f7-cb9d-0c5cecec0281 Address:127.0.0.1:11286}]" -44:53.594 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11286 [Follower]" leader= -44:53.594 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-134842f3-7f17-37f7-cb9d-0c5cecec0281.dc1 127.0.0.1 -44:53.595 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-134842f3-7f17-37f7-cb9d-0c5cecec0281 127.0.0.1 -2022-11-07T22:44:53.595Z [INFO] agent.router: Initializing LAN area manager -44:53.595 [INFO] TestAgent.server: Adding LAN server: server="Node-134842f3-7f17-37f7-cb9d-0c5cecec0281 (Addr: tcp/127.0.0.1:11286) (DC: dc1)" -44:53.595 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-134842f3-7f17-37f7-cb9d-0c5cecec0281.dc1 area=wan -44:53.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11281 network=udp -44:53.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11281 network=tcp -44:53.595 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11282 network=tcp -44:53.596 [INFO] TestAgent: started state syncer -44:53.596 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11287 network=tcp -44:53.630 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.630 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11286 [Candidate]" term=2 -44:53.642 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.642 [DEBUG] TestAgent.server.raft: vote granted: from=134842f3-7f17-37f7-cb9d-0c5cecec0281 term=2 tally=1 -44:53.642 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.642 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11286 [Leader]" -44:53.642 [INFO] TestAgent.server: cluster leadership acquired -44:53.642 [INFO] TestAgent.server: New leader elected: payload=Node-134842f3-7f17-37f7-cb9d-0c5cecec0281 -44:53.648 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11286 -44:53.660 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:53.678 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:53.678 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:53.678 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:53.678 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:53.678 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:53.678 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-134842f3-7f17-37f7-cb9d-0c5cecec0281 -44:53.678 [INFO] TestAgent.server: member joined, marking health alive: member=Node-134842f3-7f17-37f7-cb9d-0c5cecec0281 -44:53.688 [INFO] TestAgent.server: federation state anti-entropy synced -44:53.991 [INFO] TestAgent: Requesting shutdown -44:53.991 [INFO] TestAgent.server: shutting down server -44:53.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:53.991 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:53.991 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:53.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:53.991 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:53.991 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:53.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:53.991 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:53.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:53.991 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:53.991 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.000 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.007Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.007Z [INFO] agent.router.manager: shutting down -44:54.007 [INFO] TestAgent: consul server down -44:54.007 [INFO] TestAgent: shutdown complete -44:54.007 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11281 network=tcp -44:54.007 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11281 network=udp -44:54.007 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11282 network=tcp -44:54.507 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.507 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:54.471Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:54.487Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:54.536 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97bc7bd2-84c2-d8ad-79a4-749cf422d77e Address:127.0.0.1:29664}]" +29:54.537 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e.dc1 127.0.0.1 +29:54.537 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e 127.0.0.1 +2021-10-05T16:29:54.537Z [INFO] agent.router: Initializing LAN area manager +29:54.537 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=udp +29:54.537 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29664 [Follower]" leader= +29:54.538 [INFO] TestAgent.server: Adding LAN server: server="Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e (Addr: tcp/127.0.0.1:29664) (DC: dc1)" +29:54.538 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e.dc1 area=wan +29:54.538 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=tcp +29:54.538 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29660 network=tcp +29:54.538 [INFO] TestAgent: started state syncer +29:54.538 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29665 network=tcp +29:54.584 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:54.584 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29664 [Candidate]" term=2 +29:54.632 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:54.632 [DEBUG] TestAgent.server.raft: vote granted: from=97bc7bd2-84c2-d8ad-79a4-749cf422d77e term=2 tally=1 +29:54.632 [INFO] TestAgent.server.raft: election won: tally=1 +29:54.632 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29664 [Leader]" +29:54.632 [INFO] TestAgent.server: cluster leadership acquired +29:54.632 [INFO] TestAgent.server: New leader elected: payload=Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e +29:54.641 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29664 +29:54.661 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:54.688 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:54.688 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:54.688 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:54.688 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:54.688 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:54.688 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e +29:54.688 [INFO] TestAgent.server: member joined, marking health alive: member=Node-97bc7bd2-84c2-d8ad-79a4-749cf422d77e +29:54.708 [INFO] TestAgent.server: federation state anti-entropy synced +29:54.820 [INFO] TestAgent: Requesting shutdown +29:54.820 [INFO] TestAgent.server: shutting down server +29:54.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:54.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:54.820 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:54.820 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:54.820 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:54.820 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:54.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:54.820 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:54.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:54.820 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:54.820 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:54.828 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:54.836Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:54.836Z [INFO] agent.router.manager: shutting down +29:54.836 [INFO] TestAgent: consul server down +29:54.836 [INFO] TestAgent: shutdown complete +29:54.836 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=tcp +29:54.836 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=udp +29:54.836 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29660 network=tcp +29:55.340 [INFO] TestAgent: Waiting for endpoints to shut down +29:55.340 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered -2022-11-07T22:44:53.824Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.833Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.860 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06bfdddc-495a-5bf4-7431-059f38eddfcd Address:127.0.0.1:11293}]" -44:53.860 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11293 [Follower]" leader= -44:53.860 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-06bfdddc-495a-5bf4-7431-059f38eddfcd.dc1 127.0.0.1 -44:53.861 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-06bfdddc-495a-5bf4-7431-059f38eddfcd 127.0.0.1 -2022-11-07T22:44:53.861Z [INFO] agent.router: Initializing LAN area manager -44:53.861 [INFO] TestAgent.server: Adding LAN server: server="Node-06bfdddc-495a-5bf4-7431-059f38eddfcd (Addr: tcp/127.0.0.1:11293) (DC: dc1)" -44:53.861 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-06bfdddc-495a-5bf4-7431-059f38eddfcd.dc1 area=wan -44:53.861 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11288 network=tcp -44:53.861 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11288 network=udp -44:53.861 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11289 network=tcp -44:53.861 [INFO] TestAgent: started state syncer -44:53.861 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11294 network=tcp -44:53.896 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.896 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11293 [Candidate]" term=2 -44:53.913 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.913 [DEBUG] TestAgent.server.raft: vote granted: from=06bfdddc-495a-5bf4-7431-059f38eddfcd term=2 tally=1 -44:53.913 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.913 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11293 [Leader]" -44:53.913 [INFO] TestAgent.server: cluster leadership acquired -44:53.913 [INFO] TestAgent.server: New leader elected: payload=Node-06bfdddc-495a-5bf4-7431-059f38eddfcd -44:53.928 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11293 -44:53.945 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:53.980 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:53.980 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:53.980 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:53.980 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:53.980 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:53.980 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06bfdddc-495a-5bf4-7431-059f38eddfcd -44:53.980 [INFO] TestAgent.server: member joined, marking health alive: member=Node-06bfdddc-495a-5bf4-7431-059f38eddfcd -44:54.001 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.108 [INFO] TestAgent: Requesting shutdown -44:54.108 [INFO] TestAgent.server: shutting down server -44:54.108 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.108 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.108 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.108 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.108 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:54.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.108 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.108 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.111 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.115Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.115Z [INFO] agent.router.manager: shutting down -44:54.115 [INFO] TestAgent: consul server down -44:54.115 [INFO] TestAgent: shutdown complete -44:54.115 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11288 network=tcp -44:54.115 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11288 network=udp -44:54.115 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11289 network=tcp -44:54.615 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.615 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:54.561Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:54.603Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:54.647 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6503926b-2e6a-0712-faf5-7586c43767b1 Address:127.0.0.1:29671}]" +29:54.648 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6503926b-2e6a-0712-faf5-7586c43767b1.dc1 127.0.0.1 +29:54.648 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29671 [Follower]" leader= +29:54.648 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6503926b-2e6a-0712-faf5-7586c43767b1 127.0.0.1 +2021-10-05T16:29:54.648Z [INFO] agent.router: Initializing LAN area manager +29:54.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=udp +29:54.648 [INFO] TestAgent.server: Adding LAN server: server="Node-6503926b-2e6a-0712-faf5-7586c43767b1 (Addr: tcp/127.0.0.1:29671) (DC: dc1)" +29:54.649 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6503926b-2e6a-0712-faf5-7586c43767b1.dc1 area=wan +29:54.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=tcp +29:54.649 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29667 network=tcp +29:54.649 [INFO] TestAgent: started state syncer +29:54.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29672 network=tcp +29:54.689 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:54.689 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29671 [Candidate]" term=2 +29:54.707 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:54.707 [DEBUG] TestAgent.server.raft: vote granted: from=6503926b-2e6a-0712-faf5-7586c43767b1 term=2 tally=1 +29:54.707 [INFO] TestAgent.server.raft: election won: tally=1 +29:54.707 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29671 [Leader]" +29:54.708 [INFO] TestAgent.server: cluster leadership acquired +29:54.708 [INFO] TestAgent.server: New leader elected: payload=Node-6503926b-2e6a-0712-faf5-7586c43767b1 +29:54.714 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29671 +29:54.736 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:54.766 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:54.766 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:54.766 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:54.766 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:54.766 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:54.766 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6503926b-2e6a-0712-faf5-7586c43767b1 +29:54.766 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6503926b-2e6a-0712-faf5-7586c43767b1 +29:54.792 [INFO] TestAgent.server: federation state anti-entropy synced +29:54.836 [INFO] TestAgent: Requesting shutdown +29:54.836 [INFO] TestAgent.server: shutting down server +29:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:54.836 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:54.836 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:54.848 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:54.864Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:54.864Z [INFO] agent.router.manager: shutting down +29:54.864 [INFO] TestAgent: consul server down +29:54.864 [INFO] TestAgent: shutdown complete +29:54.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=tcp +29:54.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=udp +29:54.864 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29667 network=tcp +29:55.365 [INFO] TestAgent: Waiting for endpoints to shut down +29:55.365 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered -2022-11-07T22:44:54.520Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:54.532Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:54.552 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:788011e2-15c8-714f-b7b0-e6146107af10 Address:127.0.0.1:11300}]" -44:54.552 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11300 [Follower]" leader= -44:54.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-788011e2-15c8-714f-b7b0-e6146107af10.dc1 127.0.0.1 -44:54.553 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-788011e2-15c8-714f-b7b0-e6146107af10 127.0.0.1 -2022-11-07T22:44:54.553Z [INFO] agent.router: Initializing LAN area manager -44:54.553 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-788011e2-15c8-714f-b7b0-e6146107af10.dc1 area=wan -44:54.553 [INFO] TestAgent.server: Adding LAN server: server="Node-788011e2-15c8-714f-b7b0-e6146107af10 (Addr: tcp/127.0.0.1:11300) (DC: dc1)" -44:54.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11295 network=udp -44:54.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11295 network=tcp -44:54.553 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11296 network=tcp -44:54.554 [INFO] TestAgent: started state syncer -44:54.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11301 network=tcp -44:54.597 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:54.597 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11300 [Candidate]" term=2 -44:54.605 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:54.605 [DEBUG] TestAgent.server.raft: vote granted: from=788011e2-15c8-714f-b7b0-e6146107af10 term=2 tally=1 -44:54.605 [INFO] TestAgent.server.raft: election won: tally=1 -44:54.605 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11300 [Leader]" -44:54.605 [INFO] TestAgent.server: cluster leadership acquired -44:54.605 [INFO] TestAgent.server: New leader elected: payload=Node-788011e2-15c8-714f-b7b0-e6146107af10 -44:54.610 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11300 -44:54.621 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.644 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.644 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.644 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.644 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.644 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.644 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-788011e2-15c8-714f-b7b0-e6146107af10 -44:54.644 [INFO] TestAgent.server: member joined, marking health alive: member=Node-788011e2-15c8-714f-b7b0-e6146107af10 -44:54.656 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.805 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.808 [INFO] TestAgent: Synced node info -44:54.843 [DEBUG] TestAgent: Node info in sync -44:54.845 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:54.852 [INFO] TestAgent: Synced service: service=web -44:54.852 [DEBUG] TestAgent: removed service: service=web -44:54.852 [DEBUG] TestAgent: Node info in sync -44:54.852 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy -44:54.855 [INFO] TestAgent: Deregistered service: service=web -44:54.855 [INFO] TestAgent: Requesting shutdown -44:54.855 [INFO] TestAgent.server: shutting down server -44:54.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.855 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.855 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.855 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.855 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.855 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.855 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.858 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.861Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.861Z [INFO] agent.router.manager: shutting down -44:54.861 [INFO] TestAgent: consul server down -44:54.861 [INFO] TestAgent: shutdown complete -44:54.861 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11295 network=tcp -44:54.861 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11295 network=udp -44:54.861 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11296 network=tcp -44:55.362 [INFO] TestAgent: Waiting for endpoints to shut down -44:55.362 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:55.387Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:55.402Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:55.434 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:31ef9fd2-6480-7011-b105-272849477105 Address:127.0.0.1:29678}]" +29:55.434 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29678 [Follower]" leader= +29:55.435 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-31ef9fd2-6480-7011-b105-272849477105.dc1 127.0.0.1 +29:55.436 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-31ef9fd2-6480-7011-b105-272849477105 127.0.0.1 +2021-10-05T16:29:55.436Z [INFO] agent.router: Initializing LAN area manager +29:55.436 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-31ef9fd2-6480-7011-b105-272849477105.dc1 area=wan +29:55.436 [INFO] TestAgent.server: Adding LAN server: server="Node-31ef9fd2-6480-7011-b105-272849477105 (Addr: tcp/127.0.0.1:29678) (DC: dc1)" +29:55.436 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=udp +29:55.436 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=tcp +29:55.437 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29674 network=tcp +29:55.437 [INFO] TestAgent: started state syncer +29:55.437 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29679 network=tcp +29:55.506 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:55.506 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29678 [Candidate]" term=2 +29:55.516 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:55.516 [DEBUG] TestAgent.server.raft: vote granted: from=31ef9fd2-6480-7011-b105-272849477105 term=2 tally=1 +29:55.516 [INFO] TestAgent.server.raft: election won: tally=1 +29:55.516 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29678 [Leader]" +29:55.517 [INFO] TestAgent.server: cluster leadership acquired +29:55.517 [INFO] TestAgent.server: New leader elected: payload=Node-31ef9fd2-6480-7011-b105-272849477105 +29:55.528 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29678 +29:55.554 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:55.598 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:55.598 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:55.598 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:55.598 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:55.599 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:55.599 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-31ef9fd2-6480-7011-b105-272849477105 +29:55.599 [INFO] TestAgent.server: member joined, marking health alive: member=Node-31ef9fd2-6480-7011-b105-272849477105 +29:55.612 [INFO] TestAgent.server: federation state anti-entropy synced +29:55.660 [INFO] TestAgent: Synced node info +29:55.665 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:55.670 [INFO] TestAgent: Synced service: service=web +29:55.671 [DEBUG] TestAgent: removed service: service=web +29:55.671 [DEBUG] TestAgent: Node info in sync +29:55.671 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy +29:55.683 [INFO] TestAgent: Deregistered service: service=web +29:55.683 [INFO] TestAgent: Requesting shutdown +29:55.683 [INFO] TestAgent.server: shutting down server +29:55.683 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:55.683 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:55.683 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:55.683 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:55.683 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:55.683 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:55.683 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:55.683 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:55.683 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:55.683 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:55.683 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:55.687 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:55.691Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:55.691Z [INFO] agent.router.manager: shutting down +29:55.691 [INFO] TestAgent: consul server down +29:55.691 [INFO] TestAgent: shutdown complete +29:55.691 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=tcp +29:55.691 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=udp +29:55.691 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29674 network=tcp +29:56.191 [INFO] TestAgent: Waiting for endpoints to shut down +29:56.191 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/updates_to_sidecar_should_work -2022-11-07T22:44:54.626Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:54.635Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:54.667 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1f7b4b22-51af-97e2-903a-227b322e14de Address:127.0.0.1:11307}]" -44:54.667 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11307 [Follower]" leader= -44:54.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1f7b4b22-51af-97e2-903a-227b322e14de.dc1 127.0.0.1 -44:54.668 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1f7b4b22-51af-97e2-903a-227b322e14de 127.0.0.1 -2022-11-07T22:44:54.668Z [INFO] agent.router: Initializing LAN area manager -44:54.668 [INFO] TestAgent.server: Adding LAN server: server="Node-1f7b4b22-51af-97e2-903a-227b322e14de (Addr: tcp/127.0.0.1:11307) (DC: dc1)" -44:54.668 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1f7b4b22-51af-97e2-903a-227b322e14de.dc1 area=wan -44:54.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11302 network=udp -44:54.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11302 network=tcp -44:54.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11303 network=tcp -44:54.668 [INFO] TestAgent: started state syncer -44:54.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11308 network=tcp -44:54.711 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:54.711 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11307 [Candidate]" term=2 -44:54.724 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:54.724 [DEBUG] TestAgent.server.raft: vote granted: from=1f7b4b22-51af-97e2-903a-227b322e14de term=2 tally=1 -44:54.724 [INFO] TestAgent.server.raft: election won: tally=1 -44:54.724 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11307 [Leader]" -44:54.724 [INFO] TestAgent.server: cluster leadership acquired -44:54.724 [INFO] TestAgent.server: New leader elected: payload=Node-1f7b4b22-51af-97e2-903a-227b322e14de -44:54.736 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11307 -44:54.756 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.781 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.781 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.781 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.781 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.781 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.781 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1f7b4b22-51af-97e2-903a-227b322e14de -44:54.781 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1f7b4b22-51af-97e2-903a-227b322e14de -44:54.788 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.890 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.893 [INFO] TestAgent: Synced node info -44:54.893 [DEBUG] TestAgent: Node info in sync -44:55.098 [DEBUG] TestAgent: Node info in sync -44:55.101 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:55.106 [INFO] TestAgent: Synced service: service=web -44:55.107 [DEBUG] TestAgent: removed service: service=web -44:55.107 [DEBUG] TestAgent: Node info in sync -44:55.107 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy -44:55.110 [INFO] TestAgent: Deregistered service: service=web -44:55.110 [INFO] TestAgent: Requesting shutdown -44:55.110 [INFO] TestAgent.server: shutting down server -44:55.110 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:55.110 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:55.110 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:55.110 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:55.110 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:55.110 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:55.110 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:55.110 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:55.110 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:55.110 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:55.114 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:55.117Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:55.117Z [INFO] agent.router.manager: shutting down -44:55.117 [INFO] TestAgent: consul server down -44:55.117 [INFO] TestAgent: shutdown complete -44:55.117 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11302 network=tcp -44:55.117 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11302 network=udp -44:55.117 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11303 network=tcp -44:55.618 [INFO] TestAgent: Waiting for endpoints to shut down -44:55.618 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:55.389Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:55.416Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:55.457 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3d0a3758-4385-6722-5e77-28db26b5014a Address:127.0.0.1:29685}]" +29:55.457 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29685 [Follower]" leader= +29:55.458 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3d0a3758-4385-6722-5e77-28db26b5014a.dc1 127.0.0.1 +29:55.458 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3d0a3758-4385-6722-5e77-28db26b5014a 127.0.0.1 +2021-10-05T16:29:55.459Z [INFO] agent.router: Initializing LAN area manager +29:55.459 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3d0a3758-4385-6722-5e77-28db26b5014a.dc1 area=wan +29:55.459 [INFO] TestAgent.server: Adding LAN server: server="Node-3d0a3758-4385-6722-5e77-28db26b5014a (Addr: tcp/127.0.0.1:29685) (DC: dc1)" +29:55.459 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=udp +29:55.459 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=tcp +29:55.459 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29681 network=tcp +29:55.459 [INFO] TestAgent: started state syncer +29:55.460 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29686 network=tcp +29:55.517 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:55.517 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29685 [Candidate]" term=2 +29:55.540 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:55.540 [DEBUG] TestAgent.server.raft: vote granted: from=3d0a3758-4385-6722-5e77-28db26b5014a term=2 tally=1 +29:55.540 [INFO] TestAgent.server.raft: election won: tally=1 +29:55.540 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29685 [Leader]" +29:55.543 [INFO] TestAgent.server: cluster leadership acquired +29:55.543 [INFO] TestAgent.server: New leader elected: payload=Node-3d0a3758-4385-6722-5e77-28db26b5014a +29:55.551 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29685 +29:55.583 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:55.610 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:55.610 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:55.610 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:55.610 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:55.610 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:55.610 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3d0a3758-4385-6722-5e77-28db26b5014a +29:55.610 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3d0a3758-4385-6722-5e77-28db26b5014a +29:55.622 [INFO] TestAgent.server: federation state anti-entropy synced +29:55.663 [INFO] TestAgent: Synced node info +29:55.669 [INFO] TestAgent: Synced service: service=web +29:55.683 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:55.683 [DEBUG] TestAgent: removed service: service=web +29:55.683 [DEBUG] TestAgent: Node info in sync +29:55.690 [INFO] TestAgent: Deregistered service: service=web +29:55.690 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy +29:55.690 [INFO] TestAgent: Requesting shutdown +29:55.690 [INFO] TestAgent.server: shutting down server +29:55.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:55.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:55.690 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:55.690 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:55.690 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:55.690 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:55.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:55.690 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:55.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:55.690 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:55.690 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:55.695 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:55.698Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:55.698Z [INFO] agent.router.manager: shutting down +29:55.698 [INFO] TestAgent: consul server down +29:55.698 [INFO] TestAgent: shutdown complete +29:55.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=tcp +29:55.698 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=udp +29:55.698 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29681 network=tcp +29:56.200 [INFO] TestAgent: Waiting for endpoints to shut down +29:56.200 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/updates_to_sidecar_should_work -2022-11-07T22:44:55.375Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:55.385Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:55.407 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:123b8d34-3af0-f2b0-1901-4ad01eeacb83 Address:127.0.0.1:11314}]" -44:55.408 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11314 [Follower]" leader= -44:55.408 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83.dc1 127.0.0.1 -44:55.408 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83 127.0.0.1 -2022-11-07T22:44:55.408Z [INFO] agent.router: Initializing LAN area manager -44:55.408 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83.dc1 area=wan -44:55.409 [INFO] TestAgent.server: Adding LAN server: server="Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83 (Addr: tcp/127.0.0.1:11314) (DC: dc1)" -44:55.409 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11309 network=udp -44:55.409 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11309 network=tcp -44:55.409 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11310 network=tcp -44:55.409 [INFO] TestAgent: started state syncer -44:55.409 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11315 network=tcp -44:55.477 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:55.477 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11314 [Candidate]" term=2 -44:55.486 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:55.486 [DEBUG] TestAgent.server.raft: vote granted: from=123b8d34-3af0-f2b0-1901-4ad01eeacb83 term=2 tally=1 -44:55.486 [INFO] TestAgent.server.raft: election won: tally=1 -44:55.486 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11314 [Leader]" -44:55.486 [INFO] TestAgent.server: cluster leadership acquired -44:55.486 [INFO] TestAgent.server: New leader elected: payload=Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83 -44:55.491 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11314 -44:55.503 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:55.519 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:55.519 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:55.519 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:55.519 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:55.519 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:55.519 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83 -44:55.519 [INFO] TestAgent.server: member joined, marking health alive: member=Node-123b8d34-3af0-f2b0-1901-4ad01eeacb83 -44:55.524 [INFO] TestAgent.server: federation state anti-entropy synced -44:55.563 [INFO] TestAgent: Synced node info -44:55.568 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:55.574 [INFO] TestAgent: Synced service: service=web -44:55.574 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:55.574 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:55.574 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.574 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.574 [DEBUG] TestAgent: removed service: service=web -44:55.574 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.574 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.574 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:55.574 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:55.574 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:55.574 [DEBUG] TestAgent: Node info in sync -44:55.578 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:55.580 [INFO] TestAgent: Deregistered service: service=web -44:55.580 [INFO] TestAgent: Requesting shutdown -44:55.581 [INFO] TestAgent.server: shutting down server -44:55.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:55.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:55.581 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:55.581 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:55.581 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:55.581 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:55.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:55.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:55.581 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:55.581 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:55.581 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:55.584 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:55.588Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:55.588Z [INFO] agent.router.manager: shutting down -44:55.588 [INFO] TestAgent: consul server down -44:55.588 [INFO] TestAgent: shutdown complete -44:55.588 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11309 network=tcp -44:55.588 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11309 network=udp -44:55.588 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11310 network=tcp -44:56.088 [INFO] TestAgent: Waiting for endpoints to shut down -44:56.088 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:56.269Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:56.298Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:56.451 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4789e23-ed59-5c56-30af-502c9e96678c Address:127.0.0.1:29692}]" +29:56.452 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e4789e23-ed59-5c56-30af-502c9e96678c.dc1 127.0.0.1 +29:56.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29692 [Follower]" leader= +29:56.452 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e4789e23-ed59-5c56-30af-502c9e96678c 127.0.0.1 +2021-10-05T16:29:56.452Z [INFO] agent.router: Initializing LAN area manager +29:56.452 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=udp +29:56.452 [INFO] TestAgent.server: Adding LAN server: server="Node-e4789e23-ed59-5c56-30af-502c9e96678c (Addr: tcp/127.0.0.1:29692) (DC: dc1)" +29:56.453 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e4789e23-ed59-5c56-30af-502c9e96678c.dc1 area=wan +29:56.453 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=tcp +29:56.453 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29688 network=tcp +29:56.453 [INFO] TestAgent: started state syncer +29:56.453 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29693 network=tcp +29:56.521 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:56.521 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29692 [Candidate]" term=2 +29:56.572 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:56.572 [DEBUG] TestAgent.server.raft: vote granted: from=e4789e23-ed59-5c56-30af-502c9e96678c term=2 tally=1 +29:56.572 [INFO] TestAgent.server.raft: election won: tally=1 +29:56.572 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29692 [Leader]" +29:56.576 [INFO] TestAgent.server: New leader elected: payload=Node-e4789e23-ed59-5c56-30af-502c9e96678c +29:56.576 [INFO] TestAgent.server: cluster leadership acquired +29:56.604 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29692 +29:56.627 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:56.654 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:56.654 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:56.654 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:56.654 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:56.654 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:56.654 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e4789e23-ed59-5c56-30af-502c9e96678c +29:56.654 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e4789e23-ed59-5c56-30af-502c9e96678c +29:56.668 [INFO] TestAgent.server: federation state anti-entropy synced +29:56.824 [INFO] TestAgent: Synced node info +29:56.830 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:56.830 [WARN] TestAgent: Check socket connection failed: check=service:web-sidecar-proxy:1 error="dial tcp 127.0.0.1:6666: connect: connection refused" +29:56.830 [WARN] TestAgent: Check is now critical: check=service:web-sidecar-proxy:1 +29:56.834 [INFO] TestAgent: Synced service: service=web +29:56.834 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:56.834 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:56.835 [DEBUG] TestAgent: removed service: service=web +29:56.835 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:56.835 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:56.835 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:56.835 [DEBUG] TestAgent: Node info in sync +29:56.841 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:56.846 [INFO] TestAgent: Deregistered service: service=web +29:56.847 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:56.847 [INFO] TestAgent: Requesting shutdown +29:56.847 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:56.847 [INFO] TestAgent.server: shutting down server +29:56.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:56.847 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:56.847 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:56.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:56.847 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:56.847 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:56.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:56.847 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:56.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:56.847 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:56.847 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:56.853 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:56.853Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:56.853Z [INFO] agent.router.manager: shutting down +29:56.853 [INFO] TestAgent: consul server down +29:56.853 [INFO] TestAgent: shutdown complete +29:56.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=tcp +29:56.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=udp +29:56.853 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29688 network=tcp +29:57.356 [INFO] TestAgent: Waiting for endpoints to shut down +29:57.356 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/normal/update_that_removes_sidecar_should_NOT_deregister_it -2022-11-07T22:44:55.634Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:55.646Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:55.666 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f4903be3-d4e8-3bfd-886e-048fe2d67bf6 Address:127.0.0.1:11321}]" -44:55.667 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11321 [Follower]" leader= -44:55.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6.dc1 127.0.0.1 -44:55.667 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6 127.0.0.1 -2022-11-07T22:44:55.667Z [INFO] agent.router: Initializing LAN area manager -44:55.667 [INFO] TestAgent.server: Adding LAN server: server="Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6 (Addr: tcp/127.0.0.1:11321) (DC: dc1)" -44:55.667 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6.dc1 area=wan -44:55.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11316 network=udp -44:55.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11316 network=tcp -44:55.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11317 network=tcp -44:55.668 [INFO] TestAgent: started state syncer -44:55.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11322 network=tcp -44:55.728 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:55.728 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11321 [Candidate]" term=2 -44:55.737 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:55.737 [DEBUG] TestAgent.server.raft: vote granted: from=f4903be3-d4e8-3bfd-886e-048fe2d67bf6 term=2 tally=1 -44:55.737 [INFO] TestAgent.server.raft: election won: tally=1 -44:55.737 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11321 [Leader]" -44:55.737 [INFO] TestAgent.server: cluster leadership acquired -44:55.737 [INFO] TestAgent.server: New leader elected: payload=Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6 -44:55.742 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11321 -44:55.753 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:55.769 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:55.769 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:55.769 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:55.769 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:55.769 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:55.769 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6 -44:55.769 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4903be3-d4e8-3bfd-886e-048fe2d67bf6 -44:55.778 [INFO] TestAgent.server: federation state anti-entropy synced -44:55.809 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:55.812 [INFO] TestAgent: Synced node info -44:55.812 [DEBUG] TestAgent: Node info in sync -44:55.937 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -44:55.937 [DEBUG] TestAgent: Node info in sync -44:55.942 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:55.947 [INFO] TestAgent: Synced service: service=web -44:55.947 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 -44:55.947 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 -44:55.960 [DEBUG] TestAgent: removed service: service=web -44:55.961 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.961 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 -44:55.961 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.961 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 -44:55.961 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:55.961 [DEBUG] TestAgent: Node info in sync -44:55.961 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.961 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:55.964 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:55.967 [INFO] TestAgent: Deregistered service: service=web -44:55.967 [INFO] TestAgent: Requesting shutdown -44:55.967 [INFO] TestAgent.server: shutting down server -44:55.967 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:55.967 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:55.967 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:55.967 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:55.967 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:55.967 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:55.967 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:55.967 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:55.967 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:55.967 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:55.970 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:55.973Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:55.973Z [INFO] agent.router.manager: shutting down -44:55.973 [INFO] TestAgent: consul server down -44:55.973 [INFO] TestAgent: shutdown complete -44:55.973 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11316 network=tcp -44:55.973 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11316 network=udp -44:55.973 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11317 network=tcp -44:56.473 [INFO] TestAgent: Waiting for endpoints to shut down -44:56.473 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:56.254Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:56.301Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:56.464 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:df097413-f517-1751-9452-88450ae3bd64 Address:127.0.0.1:29699}]" +29:56.465 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-df097413-f517-1751-9452-88450ae3bd64.dc1 127.0.0.1 +29:56.465 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-df097413-f517-1751-9452-88450ae3bd64 127.0.0.1 +2021-10-05T16:29:56.465Z [INFO] agent.router: Initializing LAN area manager +29:56.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=udp +29:56.465 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29699 [Follower]" leader= +29:56.466 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=tcp +29:56.466 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-df097413-f517-1751-9452-88450ae3bd64.dc1 area=wan +29:56.466 [INFO] TestAgent.server: Adding LAN server: server="Node-df097413-f517-1751-9452-88450ae3bd64 (Addr: tcp/127.0.0.1:29699) (DC: dc1)" +29:56.466 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29695 network=tcp +29:56.466 [INFO] TestAgent: started state syncer +29:56.466 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29700 network=tcp +29:56.508 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:56.508 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29699 [Candidate]" term=2 +29:56.549 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:56.549 [DEBUG] TestAgent.server.raft: vote granted: from=df097413-f517-1751-9452-88450ae3bd64 term=2 tally=1 +29:56.549 [INFO] TestAgent.server.raft: election won: tally=1 +29:56.549 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29699 [Leader]" +29:56.549 [INFO] TestAgent.server: cluster leadership acquired +29:56.550 [INFO] TestAgent.server: New leader elected: payload=Node-df097413-f517-1751-9452-88450ae3bd64 +29:56.584 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29699 +29:56.628 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:56.692 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:56.692 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:56.692 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:56.692 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:56.692 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:56.692 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df097413-f517-1751-9452-88450ae3bd64 +29:56.692 [INFO] TestAgent.server: member joined, marking health alive: member=Node-df097413-f517-1751-9452-88450ae3bd64 +29:56.724 [INFO] TestAgent: Synced node info +29:56.724 [DEBUG] TestAgent: Node info in sync +29:56.724 [INFO] TestAgent.server: federation state anti-entropy synced +29:56.839 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +29:56.839 [DEBUG] TestAgent: Node info in sync +29:56.867 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:56.871 [INFO] TestAgent: Synced service: service=web +29:56.871 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:1 +29:56.871 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy:2 +29:56.892 [DEBUG] TestAgent: removed service: service=web +29:56.893 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:1 +29:56.893 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy:2 +29:56.893 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:56.893 [DEBUG] TestAgent: Node info in sync +29:56.893 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:56.898 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:56.902 [INFO] TestAgent: Deregistered service: service=web +29:56.902 [INFO] TestAgent: Requesting shutdown +29:56.902 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:56.902 [INFO] TestAgent.server: shutting down server +29:56.902 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:56.902 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:56.902 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:56.902 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:56.902 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:56.902 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:56.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:56.902 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:56.902 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:56.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:56.906 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:56.906Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:56.906Z [INFO] agent.router.manager: shutting down +29:56.906 [INFO] TestAgent: consul server down +29:56.906 [INFO] TestAgent: shutdown complete +29:56.906 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=tcp +29:56.906 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=udp +29:56.906 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29695 network=tcp +29:57.406 [INFO] TestAgent: Waiting for endpoints to shut down +29:57.406 [INFO] TestAgent: Endpoints down === RUN TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/update_that_removes_sidecar_should_NOT_deregister_it -2022-11-07T22:44:56.107Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:56.116Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:56.136 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7989d2d5-847a-55bc-2442-233c85f92429 Address:127.0.0.1:11328}]" -44:56.136 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11328 [Follower]" leader= -44:56.137 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7989d2d5-847a-55bc-2442-233c85f92429.dc1 127.0.0.1 -44:56.137 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7989d2d5-847a-55bc-2442-233c85f92429 127.0.0.1 -2022-11-07T22:44:56.137Z [INFO] agent.router: Initializing LAN area manager -44:56.137 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7989d2d5-847a-55bc-2442-233c85f92429.dc1 area=wan -44:56.137 [INFO] TestAgent.server: Adding LAN server: server="Node-7989d2d5-847a-55bc-2442-233c85f92429 (Addr: tcp/127.0.0.1:11328) (DC: dc1)" -44:56.137 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11323 network=udp -44:56.137 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11323 network=tcp -44:56.138 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11324 network=tcp -44:56.138 [INFO] TestAgent: started state syncer -44:56.138 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11329 network=tcp -44:56.176 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:56.176 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11328 [Candidate]" term=2 -44:56.184 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:56.184 [DEBUG] TestAgent.server.raft: vote granted: from=7989d2d5-847a-55bc-2442-233c85f92429 term=2 tally=1 -44:56.184 [INFO] TestAgent.server.raft: election won: tally=1 -44:56.184 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11328 [Leader]" -44:56.184 [INFO] TestAgent.server: cluster leadership acquired -44:56.184 [INFO] TestAgent.server: New leader elected: payload=Node-7989d2d5-847a-55bc-2442-233c85f92429 -44:56.189 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11328 -44:56.200 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:56.217 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:56.217 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:56.217 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:56.217 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:56.217 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:56.217 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7989d2d5-847a-55bc-2442-233c85f92429 -44:56.217 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7989d2d5-847a-55bc-2442-233c85f92429 -44:56.226 [INFO] TestAgent.server: federation state anti-entropy synced -44:56.366 [INFO] TestAgent: Synced node info -44:56.374 [INFO] TestAgent: Synced service: service=web -44:56.377 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:56.377 [DEBUG] TestAgent: removed service: service=web -44:56.377 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:56.377 [DEBUG] TestAgent: Node info in sync -44:56.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:56.380 [INFO] TestAgent: Deregistered service: service=web -44:56.383 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:56.383 [INFO] TestAgent: Requesting shutdown -44:56.383 [INFO] TestAgent.server: shutting down server -44:56.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:56.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:56.383 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:56.383 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:56.383 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:56.383 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:56.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:56.383 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:56.383 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:56.383 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:56.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:56.386 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:56.390Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:56.390Z [INFO] agent.router.manager: shutting down -44:56.390 [INFO] TestAgent: consul server down -44:56.390 [INFO] TestAgent: shutdown complete -44:56.390 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11323 network=tcp -44:56.390 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11323 network=udp -44:56.390 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11324 network=tcp -44:56.890 [INFO] TestAgent: Waiting for endpoints to shut down -44:56.890 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:56.483Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:56.492Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:56.511 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0666a87c-044d-b397-b5b9-bbe01457156b Address:127.0.0.1:11335}]" -44:56.512 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11335 [Follower]" leader= -44:56.512 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0666a87c-044d-b397-b5b9-bbe01457156b.dc1 127.0.0.1 -44:56.512 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0666a87c-044d-b397-b5b9-bbe01457156b 127.0.0.1 -2022-11-07T22:44:56.512Z [INFO] agent.router: Initializing LAN area manager -44:56.512 [INFO] TestAgent.server: Adding LAN server: server="Node-0666a87c-044d-b397-b5b9-bbe01457156b (Addr: tcp/127.0.0.1:11335) (DC: dc1)" -44:56.512 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0666a87c-044d-b397-b5b9-bbe01457156b.dc1 area=wan -44:56.512 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11330 network=udp -44:56.512 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11330 network=tcp -44:56.513 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11331 network=tcp -44:56.513 [INFO] TestAgent: started state syncer -44:56.513 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11336 network=tcp -44:56.572 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:56.572 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11335 [Candidate]" term=2 -44:56.580 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:56.580 [DEBUG] TestAgent.server.raft: vote granted: from=0666a87c-044d-b397-b5b9-bbe01457156b term=2 tally=1 -44:56.580 [INFO] TestAgent.server.raft: election won: tally=1 -44:56.580 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11335 [Leader]" -44:56.580 [INFO] TestAgent.server: cluster leadership acquired -44:56.580 [INFO] TestAgent.server: New leader elected: payload=Node-0666a87c-044d-b397-b5b9-bbe01457156b -44:56.585 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11335 -44:56.596 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:56.615 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:56.615 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:56.615 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:56.615 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:56.615 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:56.615 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0666a87c-044d-b397-b5b9-bbe01457156b -44:56.615 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0666a87c-044d-b397-b5b9-bbe01457156b -44:56.625 [INFO] TestAgent.server: federation state anti-entropy synced -44:56.629 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:56.632 [INFO] TestAgent: Synced node info -44:56.685 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -44:56.689 [DEBUG] TestAgent: Node info in sync -44:56.699 [INFO] TestAgent: Synced service: service=web -44:56.702 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -44:56.702 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:56.702 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:56.702 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:56.702 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -44:56.703 [DEBUG] TestAgent: removed service: service=web -44:56.703 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -44:56.703 [DEBUG] TestAgent: Node info in sync -44:56.706 [INFO] TestAgent: Deregistered service: service=web -44:56.710 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy -44:56.710 [INFO] TestAgent: Requesting shutdown -44:56.710 [INFO] TestAgent.server: shutting down server -44:56.710 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:56.710 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:56.710 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:56.710 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:56.710 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:56.711 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:56.711 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:56.711 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:56.711 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:56.711 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:56.716 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:56.719Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:56.719Z [INFO] agent.router.manager: shutting down -44:56.719 [INFO] TestAgent: consul server down -44:56.719 [INFO] TestAgent: shutdown complete -44:56.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11330 network=tcp -44:56.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11330 network=udp -44:56.719 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11331 network=tcp -44:57.219 [INFO] TestAgent: Waiting for endpoints to shut down -44:57.219 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:57.383Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:57.420Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:57.558 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef539797-a939-7ffe-2120-68bd9faaeb6e Address:127.0.0.1:29706}]" +29:57.558 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef539797-a939-7ffe-2120-68bd9faaeb6e.dc1 127.0.0.1 +29:57.559 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29706 [Follower]" leader= +29:57.559 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef539797-a939-7ffe-2120-68bd9faaeb6e 127.0.0.1 +2021-10-05T16:29:57.559Z [INFO] agent.router: Initializing LAN area manager +29:57.559 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=udp +29:57.559 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef539797-a939-7ffe-2120-68bd9faaeb6e.dc1 area=wan +29:57.560 [INFO] TestAgent.server: Adding LAN server: server="Node-ef539797-a939-7ffe-2120-68bd9faaeb6e (Addr: tcp/127.0.0.1:29706) (DC: dc1)" +29:57.560 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=tcp +29:57.560 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29702 network=tcp +29:57.560 [INFO] TestAgent: started state syncer +29:57.560 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29707 network=tcp +29:57.596 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:57.596 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29706 [Candidate]" term=2 +29:57.646 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:57.646 [DEBUG] TestAgent.server.raft: vote granted: from=ef539797-a939-7ffe-2120-68bd9faaeb6e term=2 tally=1 +29:57.646 [INFO] TestAgent.server.raft: election won: tally=1 +29:57.646 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29706 [Leader]" +29:57.648 [INFO] TestAgent.server: cluster leadership acquired +29:57.648 [INFO] TestAgent.server: New leader elected: payload=Node-ef539797-a939-7ffe-2120-68bd9faaeb6e +29:57.676 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29706 +29:57.704 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:57.752 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:57.752 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:57.752 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:57.752 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:57.752 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:57.752 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef539797-a939-7ffe-2120-68bd9faaeb6e +29:57.752 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef539797-a939-7ffe-2120-68bd9faaeb6e +29:57.788 [INFO] TestAgent.server: federation state anti-entropy synced +29:57.824 [INFO] TestAgent: Synced node info +29:57.848 [INFO] TestAgent: Synced service: service=web +29:57.880 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:57.880 [DEBUG] TestAgent: removed service: service=web +29:57.880 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:57.880 [DEBUG] TestAgent: Node info in sync +29:57.900 [INFO] TestAgent: Deregistered service: service=web +29:57.916 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:57.916 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:57.916 [DEBUG] TestAgent: Node info in sync +29:57.916 [INFO] TestAgent: Requesting shutdown +29:57.916 [INFO] TestAgent.server: shutting down server +29:57.916 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:57.916 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:57.916 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:57.916 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:57.916 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:57.916 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:57.916 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:57.917 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:57.917 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:57.917 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:57.920 [DEBUG] TestAgent: Node info in sync +29:57.927 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:57.927Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:57.928Z [INFO] agent.router.manager: shutting down +29:57.928 [INFO] TestAgent: consul server down +29:57.928 [INFO] TestAgent: shutdown complete +29:57.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=tcp +29:57.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=udp +29:57.928 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29702 network=tcp +29:58.432 [INFO] TestAgent: Waiting for endpoints to shut down +29:58.432 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:57.452Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:57.494Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:57.570 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:737283c2-c031-6faf-91aa-ce8412bf1fc0 Address:127.0.0.1:29713}]" +29:57.570 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29713 [Follower]" leader= +29:57.571 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-737283c2-c031-6faf-91aa-ce8412bf1fc0.dc1 127.0.0.1 +29:57.571 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-737283c2-c031-6faf-91aa-ce8412bf1fc0 127.0.0.1 +2021-10-05T16:29:57.571Z [INFO] agent.router: Initializing LAN area manager +29:57.572 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=udp +29:57.572 [INFO] TestAgent.server: Adding LAN server: server="Node-737283c2-c031-6faf-91aa-ce8412bf1fc0 (Addr: tcp/127.0.0.1:29713) (DC: dc1)" +29:57.572 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-737283c2-c031-6faf-91aa-ce8412bf1fc0.dc1 area=wan +29:57.572 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=tcp +29:57.572 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29709 network=tcp +29:57.572 [INFO] TestAgent: started state syncer +29:57.573 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29714 network=tcp +29:57.607 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:57.607 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29713 [Candidate]" term=2 +29:57.619 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:57.619 [DEBUG] TestAgent.server.raft: vote granted: from=737283c2-c031-6faf-91aa-ce8412bf1fc0 term=2 tally=1 +29:57.619 [INFO] TestAgent.server.raft: election won: tally=1 +29:57.619 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29713 [Leader]" +29:57.620 [INFO] TestAgent.server: cluster leadership acquired +29:57.620 [INFO] TestAgent.server: New leader elected: payload=Node-737283c2-c031-6faf-91aa-ce8412bf1fc0 +29:57.635 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29713 +29:57.651 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:57.664 [INFO] TestAgent: Synced node info +29:57.692 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:57.692 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:57.692 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:57.692 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:57.692 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:57.692 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-737283c2-c031-6faf-91aa-ce8412bf1fc0 +29:57.692 [INFO] TestAgent.server: member joined, marking health alive: member=Node-737283c2-c031-6faf-91aa-ce8412bf1fc0 +29:57.711 [INFO] TestAgent.server: federation state anti-entropy synced +29:57.984 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +29:57.992 [DEBUG] TestAgent: Node info in sync +29:57.999 [INFO] TestAgent: Synced service: service=web +29:58.008 [INFO] TestAgent: Synced service: service=web-sidecar-proxy +29:58.016 [DEBUG] TestAgent: removed service: service=web +29:58.016 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +29:58.016 [DEBUG] TestAgent: Node info in sync +29:58.020 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:58.034 [INFO] TestAgent: Deregistered service: service=web +29:58.040 [INFO] TestAgent: Deregistered service: service=web-sidecar-proxy +29:58.040 [INFO] TestAgent: Requesting shutdown +29:58.040 [INFO] TestAgent.server: shutting down server +29:58.040 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:58.040 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:58.040 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +29:58.040 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:58.040 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:58.040 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:58.040 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:58.040 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:58.040 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:58.040 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:58.040 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:58.058 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:58.061Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:58.061Z [INFO] agent.router.manager: shutting down +29:58.061 [INFO] TestAgent: consul server down +29:58.061 [INFO] TestAgent: shutdown complete +29:58.061 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=tcp +29:58.062 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=udp +29:58.062 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29709 network=tcp +29:58.562 [INFO] TestAgent: Waiting for endpoints to shut down +29:58.562 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar (0.00s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal (13.51s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/sanity_check_no_sidecar_case (0.84s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/default_sidecar (0.78s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_defaults (1.14s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_denied (0.97s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_sidecar (1.05s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination (1.36s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_overridden_sidecar (0.81s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal (14.60s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/sanity_check_no_sidecar_case (1.38s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/default_sidecar (1.29s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_defaults (0.90s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_denied (0.79s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_sidecar (1.09s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination (0.83s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_not_for_overridden_sidecar (0.89s) --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/ACL_OK_for_service_but_and_overridden_for_sidecar (0.99s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_definition_in_sidecar (0.88s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_definitions_in_sidecar (0.67s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_status_in_sidecar (0.67s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_status_in_sidecar (0.97s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered (0.85s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/updates_to_sidecar_should_work (0.73s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/update_that_removes_sidecar_should_NOT_deregister_it (0.80s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager (13.84s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/sanity_check_no_sidecar_case (0.84s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/default_sidecar (0.97s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_definition_in_sidecar (0.84s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_definitions_in_sidecar (0.78s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_check_status_in_sidecar (0.82s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/invalid_checks_status_in_sidecar (0.89s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered (0.84s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/updates_to_sidecar_should_work (1.16s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/normal/update_that_removes_sidecar_should_NOT_deregister_it (1.08s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager (14.72s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/sanity_check_no_sidecar_case (1.32s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/default_sidecar (1.14s) --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_defaults (1.06s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_denied (0.73s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_sidecar (0.98s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination (1.37s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_overridden_sidecar (0.99s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_and_overridden_for_sidecar (0.87s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_definition_in_sidecar (0.70s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_definitions_in_sidecar (0.92s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_status_in_sidecar (1.01s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_status_in_sidecar (0.80s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered (1.00s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/updates_to_sidecar_should_work (0.86s) - --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/update_that_removes_sidecar_should_NOT_deregister_it (0.75s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_denied (0.83s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_sidecar (1.03s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_and_sidecar_but_not_sidecar's_overridden_destination (0.77s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_not_for_overridden_sidecar (0.85s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/ACL_OK_for_service_but_and_overridden_for_sidecar (0.99s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_definition_in_sidecar (0.98s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_definitions_in_sidecar (0.87s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_check_status_in_sidecar (0.87s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/invalid_checks_status_in_sidecar (0.82s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/another_service_registered_with_same_ID_as_a_sidecar_should_not_be_deregistered (0.83s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/updates_to_sidecar_should_work (1.21s) + --- PASS: TestAgent_RegisterServiceDeregisterService_Sidecar/service_manager/update_that_removes_sidecar_should_NOT_deregister_it (1.16s) === RUN TestAgent_RegisterService_UnmanagedConnectProxyInvalid === RUN TestAgent_RegisterService_UnmanagedConnectProxyInvalid/normal === PAUSE TestAgent_RegisterService_UnmanagedConnectProxyInvalid/normal @@ -9126,120 +9184,120 @@ === PAUSE TestAgent_RegisterService_UnmanagedConnectProxyInvalid/service_manager === CONT TestAgent_RegisterService_UnmanagedConnectProxyInvalid/normal === CONT TestAgent_RegisterService_UnmanagedConnectProxyInvalid/service_manager -2022-11-07T22:44:57.230Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:57.239Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:57.268 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ec7e1d28-0215-ec81-9d0d-0892b8d82bdb Address:127.0.0.1:11342}]" -44:57.269 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11342 [Follower]" leader= -44:57.269 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb.dc1 127.0.0.1 -44:57.270 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb 127.0.0.1 -2022-11-07T22:44:57.270Z [INFO] agent.router: Initializing LAN area manager -44:57.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb.dc1 area=wan -44:57.270 [INFO] TestAgent.server: Adding LAN server: server="Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb (Addr: tcp/127.0.0.1:11342) (DC: dc1)" -44:57.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11337 network=tcp -44:57.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11337 network=udp -44:57.270 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11338 network=tcp -44:57.270 [INFO] TestAgent: started state syncer -44:57.270 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11343 network=tcp -44:57.334 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:57.334 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11342 [Candidate]" term=2 -44:57.344 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:57.344 [DEBUG] TestAgent.server.raft: vote granted: from=ec7e1d28-0215-ec81-9d0d-0892b8d82bdb term=2 tally=1 -44:57.344 [INFO] TestAgent.server.raft: election won: tally=1 -44:57.344 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11342 [Leader]" -44:57.344 [INFO] TestAgent.server: cluster leadership acquired -44:57.344 [INFO] TestAgent.server: New leader elected: payload=Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb -44:57.349 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11342 -44:57.362 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:57.382 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:57.382 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:57.382 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:57.382 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:57.382 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:57.382 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb -44:57.382 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec7e1d28-0215-ec81-9d0d-0892b8d82bdb -44:57.392 [INFO] TestAgent.server: federation state anti-entropy synced -44:57.427 [INFO] TestAgent: Requesting shutdown -44:57.427 [INFO] TestAgent.server: shutting down server -44:57.427 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:57.427 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:57.427 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:57.427 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:57.427 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:57.428 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:57.428 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:57.428 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:57.428 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:57.428 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:57.428 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:57.432 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:57.436Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:57.436Z [INFO] agent.router.manager: shutting down -44:57.436 [INFO] TestAgent: consul server down -44:57.436 [INFO] TestAgent: shutdown complete -44:57.436 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11337 network=tcp -44:57.436 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11337 network=udp -44:57.436 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11338 network=tcp -44:57.936 [INFO] TestAgent: Waiting for endpoints to shut down -44:57.936 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:57.231Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:57.240Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:57.267 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b Address:127.0.0.1:11349}]" -44:57.267 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11349 [Follower]" leader= -44:57.267 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b.dc1 127.0.0.1 -44:57.267 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b 127.0.0.1 -2022-11-07T22:44:57.267Z [INFO] agent.router: Initializing LAN area manager -44:57.268 [INFO] TestAgent.server: Adding LAN server: server="Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b (Addr: tcp/127.0.0.1:11349) (DC: dc1)" -44:57.268 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b.dc1 area=wan -44:57.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11344 network=udp -44:57.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11344 network=tcp -44:57.268 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11345 network=tcp -44:57.268 [INFO] TestAgent: started state syncer -44:57.268 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11350 network=tcp -44:57.310 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:57.310 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11349 [Candidate]" term=2 -44:57.318 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:57.318 [DEBUG] TestAgent.server.raft: vote granted: from=3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b term=2 tally=1 -44:57.318 [INFO] TestAgent.server.raft: election won: tally=1 -44:57.318 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11349 [Leader]" -44:57.319 [INFO] TestAgent.server: cluster leadership acquired -44:57.319 [INFO] TestAgent.server: New leader elected: payload=Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b -44:57.323 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11349 -44:57.342 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:57.342 [INFO] TestAgent: Synced node info -44:57.342 [DEBUG] TestAgent: Node info in sync -44:57.354 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:57.355 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:57.355 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:57.355 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:57.355 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:57.355 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b -44:57.355 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ff4aaf5-3680-d0ba-2b4f-8c39faf5374b -44:57.366 [INFO] TestAgent.server: federation state anti-entropy synced -44:57.646 [INFO] TestAgent: Requesting shutdown -44:57.646 [INFO] TestAgent.server: shutting down server -44:57.646 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:57.646 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:57.646 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:57.646 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:57.646 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:57.646 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:57.646 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:57.646 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:57.646 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:57.646 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:57.650 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:57.653Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:57.653Z [INFO] agent.router.manager: shutting down -44:57.653 [INFO] TestAgent: consul server down -44:57.653 [INFO] TestAgent: shutdown complete -44:57.653 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11344 network=tcp -44:57.653 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11344 network=udp -44:57.653 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11345 network=tcp -44:58.153 [INFO] TestAgent: Waiting for endpoints to shut down -44:58.153 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:58.581Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:58.625Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:58.701 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c92344dd-03f1-68bf-de3c-42bf47971e3f Address:127.0.0.1:29720}]" +29:58.701 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29720 [Follower]" leader= +29:58.702 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c92344dd-03f1-68bf-de3c-42bf47971e3f.dc1 127.0.0.1 +29:58.702 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c92344dd-03f1-68bf-de3c-42bf47971e3f 127.0.0.1 +2021-10-05T16:29:58.703Z [INFO] agent.router: Initializing LAN area manager +29:58.703 [INFO] TestAgent.server: Adding LAN server: server="Node-c92344dd-03f1-68bf-de3c-42bf47971e3f (Addr: tcp/127.0.0.1:29720) (DC: dc1)" +29:58.703 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c92344dd-03f1-68bf-de3c-42bf47971e3f.dc1 area=wan +29:58.703 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=udp +29:58.703 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=tcp +29:58.703 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29716 network=tcp +29:58.704 [INFO] TestAgent: started state syncer +29:58.704 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29721 network=tcp +29:58.772 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:58.772 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29720 [Candidate]" term=2 +29:58.820 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:58.820 [DEBUG] TestAgent.server.raft: vote granted: from=c92344dd-03f1-68bf-de3c-42bf47971e3f term=2 tally=1 +29:58.820 [INFO] TestAgent.server.raft: election won: tally=1 +29:58.820 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29720 [Leader]" +29:58.824 [INFO] TestAgent.server: cluster leadership acquired +29:58.824 [INFO] TestAgent.server: New leader elected: payload=Node-c92344dd-03f1-68bf-de3c-42bf47971e3f +29:58.852 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29720 +29:58.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:58.980 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:58.980 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:58.980 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:58.980 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:58.980 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:58.980 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c92344dd-03f1-68bf-de3c-42bf47971e3f +29:58.980 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c92344dd-03f1-68bf-de3c-42bf47971e3f +29:59.016 [INFO] TestAgent.server: federation state anti-entropy synced +29:59.024 [INFO] TestAgent: Requesting shutdown +29:59.024 [INFO] TestAgent.server: shutting down server +29:59.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:59.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:59.024 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:59.024 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:59.024 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:59.024 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:59.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:59.024 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:59.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:59.025 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:59.025 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:59.036 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:59.044Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:59.044Z [INFO] agent.router.manager: shutting down +29:59.044 [INFO] TestAgent: consul server down +29:59.044 [INFO] TestAgent: shutdown complete +29:59.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=tcp +29:59.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=udp +29:59.044 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29716 network=tcp +29:59.548 [INFO] TestAgent: Waiting for endpoints to shut down +29:59.548 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:58.618Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:58.646Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:58.677 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f1d2f437-7791-7335-dfcb-47b48ea86815 Address:127.0.0.1:29727}]" +29:58.678 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29727 [Follower]" leader= +29:58.678 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f1d2f437-7791-7335-dfcb-47b48ea86815.dc1 127.0.0.1 +29:58.678 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f1d2f437-7791-7335-dfcb-47b48ea86815 127.0.0.1 +2021-10-05T16:29:58.679Z [INFO] agent.router: Initializing LAN area manager +29:58.679 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29722 network=udp +29:58.679 [INFO] TestAgent.server: Adding LAN server: server="Node-f1d2f437-7791-7335-dfcb-47b48ea86815 (Addr: tcp/127.0.0.1:29727) (DC: dc1)" +29:58.679 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29722 network=tcp +29:58.679 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f1d2f437-7791-7335-dfcb-47b48ea86815.dc1 area=wan +29:58.680 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29723 network=tcp +29:58.680 [INFO] TestAgent: started state syncer +29:58.680 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29728 network=tcp +29:58.720 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:58.720 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29727 [Candidate]" term=2 +29:58.735 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:58.735 [DEBUG] TestAgent.server.raft: vote granted: from=f1d2f437-7791-7335-dfcb-47b48ea86815 term=2 tally=1 +29:58.735 [INFO] TestAgent.server.raft: election won: tally=1 +29:58.735 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29727 [Leader]" +29:58.735 [INFO] TestAgent.server: cluster leadership acquired +29:58.736 [INFO] TestAgent.server: New leader elected: payload=Node-f1d2f437-7791-7335-dfcb-47b48ea86815 +29:58.741 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29727 +29:58.761 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:58.790 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:58.790 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:58.790 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:58.790 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:58.790 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:58.790 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f1d2f437-7791-7335-dfcb-47b48ea86815 +29:58.790 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f1d2f437-7791-7335-dfcb-47b48ea86815 +29:58.804 [INFO] TestAgent.server: federation state anti-entropy synced +29:58.980 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:58.992 [INFO] TestAgent: Synced node info +29:59.040 [INFO] TestAgent: Requesting shutdown +29:59.040 [INFO] TestAgent.server: shutting down server +29:59.040 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:59.040 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:59.040 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:59.040 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:59.040 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:59.040 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:59.040 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:59.041 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:59.041 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:59.041 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:59.056 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:59.064Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:59.064Z [INFO] agent.router.manager: shutting down +29:59.064 [INFO] TestAgent: consul server down +29:59.064 [INFO] TestAgent: shutdown complete +29:59.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29722 network=tcp +29:59.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29722 network=udp +29:59.064 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29723 network=tcp +29:59.568 [INFO] TestAgent: Waiting for endpoints to shut down +29:59.568 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_UnmanagedConnectProxyInvalid (0.00s) - --- PASS: TestAgent_RegisterService_UnmanagedConnectProxyInvalid/normal (0.72s) - --- PASS: TestAgent_RegisterService_UnmanagedConnectProxyInvalid/service_manager (0.93s) + --- PASS: TestAgent_RegisterService_UnmanagedConnectProxyInvalid/service_manager (0.98s) + --- PASS: TestAgent_RegisterService_UnmanagedConnectProxyInvalid/normal (1.01s) === RUN TestAgent_RegisterService_ConnectNative === RUN TestAgent_RegisterService_ConnectNative/normal === PAUSE TestAgent_RegisterService_ConnectNative/normal @@ -9247,127 +9305,131 @@ === PAUSE TestAgent_RegisterService_ConnectNative/service_manager === CONT TestAgent_RegisterService_ConnectNative/normal === CONT TestAgent_RegisterService_ConnectNative/service_manager -2022-11-07T22:44:58.168Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:58.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:58.211 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18a97615-c08c-e639-1da6-4483cd4bdadc Address:127.0.0.1:11356}]" -44:58.211 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11356 [Follower]" leader= -44:58.212 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18a97615-c08c-e639-1da6-4483cd4bdadc.dc1 127.0.0.1 -44:58.212 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18a97615-c08c-e639-1da6-4483cd4bdadc 127.0.0.1 -2022-11-07T22:44:58.212Z [INFO] agent.router: Initializing LAN area manager -44:58.212 [INFO] TestAgent.server: Adding LAN server: server="Node-18a97615-c08c-e639-1da6-4483cd4bdadc (Addr: tcp/127.0.0.1:11356) (DC: dc1)" -44:58.212 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18a97615-c08c-e639-1da6-4483cd4bdadc.dc1 area=wan -44:58.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11351 network=udp -44:58.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11351 network=tcp -44:58.213 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11352 network=tcp -44:58.213 [INFO] TestAgent: started state syncer -44:58.213 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11357 network=tcp -44:58.268 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:58.268 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11356 [Candidate]" term=2 -44:58.277 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:58.277 [DEBUG] TestAgent.server.raft: vote granted: from=18a97615-c08c-e639-1da6-4483cd4bdadc term=2 tally=1 -44:58.277 [INFO] TestAgent.server.raft: election won: tally=1 -44:58.277 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11356 [Leader]" -44:58.277 [INFO] TestAgent.server: cluster leadership acquired -44:58.277 [INFO] TestAgent.server: New leader elected: payload=Node-18a97615-c08c-e639-1da6-4483cd4bdadc -44:58.285 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11356 -44:58.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:58.313 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:58.313 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:58.313 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:58.313 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:58.313 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:58.313 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18a97615-c08c-e639-1da6-4483cd4bdadc -44:58.313 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18a97615-c08c-e639-1da6-4483cd4bdadc -44:58.323 [INFO] TestAgent.server: federation state anti-entropy synced -44:58.349 [INFO] TestAgent: Synced node info -44:58.353 [INFO] TestAgent: Synced service: service=web -44:58.353 [DEBUG] TestAgent: Check in sync: check=service:web -44:58.353 [INFO] TestAgent: Requesting shutdown -44:58.354 [INFO] TestAgent.server: shutting down server -44:58.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:58.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:58.354 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:58.354 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:58.354 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:58.354 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:58.354 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:58.354 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:58.354 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:58.354 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:58.354 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:58.357 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:58.360Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:58.360Z [INFO] agent.router.manager: shutting down -44:58.360 [INFO] TestAgent: consul server down -44:58.360 [INFO] TestAgent: shutdown complete -44:58.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11351 network=tcp -44:58.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11351 network=udp -44:58.360 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11352 network=tcp -44:58.861 [INFO] TestAgent: Waiting for endpoints to shut down -44:58.861 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:58.171Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:58.185Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:58.209 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:25db0bff-c2ec-9a21-a98f-8a543005950b Address:127.0.0.1:11363}]" -44:58.210 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-25db0bff-c2ec-9a21-a98f-8a543005950b.dc1 127.0.0.1 -44:58.210 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11363 [Follower]" leader= -44:58.210 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-25db0bff-c2ec-9a21-a98f-8a543005950b 127.0.0.1 -2022-11-07T22:44:58.210Z [INFO] agent.router: Initializing LAN area manager -44:58.210 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-25db0bff-c2ec-9a21-a98f-8a543005950b.dc1 area=wan -44:58.210 [INFO] TestAgent.server: Adding LAN server: server="Node-25db0bff-c2ec-9a21-a98f-8a543005950b (Addr: tcp/127.0.0.1:11363) (DC: dc1)" -44:58.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11358 network=tcp -44:58.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11358 network=udp -44:58.211 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11359 network=tcp -44:58.211 [INFO] TestAgent: started state syncer -44:58.211 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11364 network=tcp -44:58.249 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:58.249 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11363 [Candidate]" term=2 -44:58.257 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:58.257 [DEBUG] TestAgent.server.raft: vote granted: from=25db0bff-c2ec-9a21-a98f-8a543005950b term=2 tally=1 -44:58.257 [INFO] TestAgent.server.raft: election won: tally=1 -44:58.257 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11363 [Leader]" -44:58.257 [INFO] TestAgent.server: cluster leadership acquired -44:58.257 [INFO] TestAgent.server: New leader elected: payload=Node-25db0bff-c2ec-9a21-a98f-8a543005950b -44:58.263 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11363 -44:58.274 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:58.295 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:58.295 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:58.295 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:58.295 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:58.295 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:58.295 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-25db0bff-c2ec-9a21-a98f-8a543005950b -44:58.295 [INFO] TestAgent.server: member joined, marking health alive: member=Node-25db0bff-c2ec-9a21-a98f-8a543005950b -44:58.304 [INFO] TestAgent.server: federation state anti-entropy synced -44:58.509 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:58.513 [INFO] TestAgent: Synced node info -44:58.513 [DEBUG] TestAgent: Node info in sync -44:58.544 [DEBUG] TestAgent: Node info in sync -44:58.550 [INFO] TestAgent: Synced service: service=web -44:58.550 [DEBUG] TestAgent: Check in sync: check=service:web -44:58.550 [INFO] TestAgent: Requesting shutdown -44:58.550 [INFO] TestAgent.server: shutting down server -44:58.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:58.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:58.550 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:58.550 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:58.550 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:58.550 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:58.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:58.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:58.550 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:58.550 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:58.553 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:58.558Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:58.558Z [INFO] agent.router.manager: shutting down -44:58.558 [INFO] TestAgent: consul server down -44:58.558 [INFO] TestAgent: shutdown complete -44:58.558 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11358 network=tcp -44:58.558 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11358 network=udp -44:58.558 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11359 network=tcp -44:59.058 [INFO] TestAgent: Waiting for endpoints to shut down -44:59.058 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:59.598Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:59.632Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:59.719 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:539c0931-b977-6fa9-6a6c-fd3242b1fd0d Address:127.0.0.1:29741}]" +29:59.720 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d.dc1 127.0.0.1 +29:59.720 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29741 [Follower]" leader= +29:59.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d 127.0.0.1 +2021-10-05T16:29:59.721Z [INFO] agent.router: Initializing LAN area manager +29:59.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=udp +29:59.721 [INFO] TestAgent.server: Adding LAN server: server="Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d (Addr: tcp/127.0.0.1:29741) (DC: dc1)" +29:59.722 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d.dc1 area=wan +29:59.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=tcp +29:59.723 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29737 network=tcp +29:59.723 [INFO] TestAgent: started state syncer +29:59.723 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29742 network=tcp +29:59.789 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:59.789 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29741 [Candidate]" term=2 +29:59.812 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:59.812 [DEBUG] TestAgent.server.raft: vote granted: from=539c0931-b977-6fa9-6a6c-fd3242b1fd0d term=2 tally=1 +29:59.812 [INFO] TestAgent.server.raft: election won: tally=1 +29:59.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29741 [Leader]" +29:59.815 [INFO] TestAgent.server: cluster leadership acquired +29:59.816 [INFO] TestAgent.server: New leader elected: payload=Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d +29:59.819 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29741 +29:59.858 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:59.892 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:59.892 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:59.892 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:59.892 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:59.892 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:59.892 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d +29:59.892 [INFO] TestAgent.server: member joined, marking health alive: member=Node-539c0931-b977-6fa9-6a6c-fd3242b1fd0d +29:59.912 [INFO] TestAgent.server: federation state anti-entropy synced +29:59.968 [INFO] TestAgent: Synced node info +29:59.980 [INFO] TestAgent: Synced service: service=web +29:59.980 [DEBUG] TestAgent: Check in sync: check=service:web +29:59.980 [INFO] TestAgent: Requesting shutdown +29:59.980 [INFO] TestAgent.server: shutting down server +29:59.980 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:59.980 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:59.980 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:59.980 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:59.980 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:59.980 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:59.980 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:59.981 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:59.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:59.981 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:59.981 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:59.991 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:59.991Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:59.991Z [INFO] agent.router.manager: shutting down +29:59.991 [INFO] TestAgent: consul server down +29:59.991 [INFO] TestAgent: shutdown complete +29:59.991 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=tcp +29:59.991 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=udp +29:59.992 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29737 network=tcp +30:00.492 [INFO] TestAgent: Waiting for endpoints to shut down +30:00.492 [INFO] TestAgent: Endpoints down +2021-10-05T16:29:59.606Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:59.661Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:59.691 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 Address:127.0.0.1:29734}]" +29:59.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726.dc1 127.0.0.1 +29:59.692 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29734 [Follower]" leader= +29:59.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 127.0.0.1 +2021-10-05T16:29:59.692Z [INFO] agent.router: Initializing LAN area manager +29:59.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=udp +29:59.692 [INFO] TestAgent.server: Adding LAN server: server="Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 (Addr: tcp/127.0.0.1:29734) (DC: dc1)" +29:59.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726.dc1 area=wan +29:59.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=tcp +29:59.693 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29730 network=tcp +29:59.693 [INFO] TestAgent: started state syncer +29:59.693 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29735 network=tcp +29:59.733 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:59.734 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29734 [Candidate]" term=2 +29:59.744 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:59.744 [DEBUG] TestAgent.server.raft: vote granted: from=d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 term=2 tally=1 +29:59.744 [INFO] TestAgent.server.raft: election won: tally=1 +29:59.744 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29734 [Leader]" +29:59.744 [INFO] TestAgent.server: cluster leadership acquired +29:59.744 [INFO] TestAgent.server: New leader elected: payload=Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 +29:59.751 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29734 +29:59.816 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:59.885 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:59.885 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:59.885 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:59.885 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:59.885 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:59.885 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 +29:59.885 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d27524a1-0e5c-32e9-09b4-4c5a7ab6b726 +29:59.911 [INFO] TestAgent.server: federation state anti-entropy synced +30:00.092 [INFO] TestAgent: Synced node info +30:00.105 [INFO] TestAgent: Synced service: service=web +30:00.105 [DEBUG] TestAgent: Check in sync: check=service:web +30:00.105 [INFO] TestAgent: Requesting shutdown +30:00.105 [INFO] TestAgent.server: shutting down server +30:00.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:00.105 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:00.105 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:00.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:00.106 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:00.106 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:00.106 [DEBUG] TestAgent: Node info in sync +30:00.106 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:00.106 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:00.106 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:00.106 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:00.106 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:00.116 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +30:00.116 [INFO] TestAgent: Synced service: service=web +30:00.116 [DEBUG] TestAgent: Check in sync: check=service:web +30:00.116 [DEBUG] TestAgent: Node info in sync +30:00.116 [DEBUG] TestAgent: Service in sync: service=web +30:00.116 [DEBUG] TestAgent: Check in sync: check=service:web +2021-10-05T16:30:00.116Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:00.117Z [INFO] agent.router.manager: shutting down +30:00.117 [INFO] TestAgent: consul server down +30:00.117 [INFO] TestAgent: shutdown complete +30:00.117 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=tcp +30:00.117 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=udp +30:00.117 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29730 network=tcp +30:00.620 [INFO] TestAgent: Waiting for endpoints to shut down +30:00.620 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ConnectNative (0.00s) - --- PASS: TestAgent_RegisterService_ConnectNative/normal (0.71s) - --- PASS: TestAgent_RegisterService_ConnectNative/service_manager (0.91s) + --- PASS: TestAgent_RegisterService_ConnectNative/service_manager (0.92s) + --- PASS: TestAgent_RegisterService_ConnectNative/normal (1.05s) === RUN TestAgent_RegisterService_ScriptCheck_ExecDisable === RUN TestAgent_RegisterService_ScriptCheck_ExecDisable/normal === PAUSE TestAgent_RegisterService_ScriptCheck_ExecDisable/normal @@ -9375,121 +9437,119 @@ === PAUSE TestAgent_RegisterService_ScriptCheck_ExecDisable/service_manager === CONT TestAgent_RegisterService_ScriptCheck_ExecDisable/normal === CONT TestAgent_RegisterService_ScriptCheck_ExecDisable/service_manager -2022-11-07T22:44:59.070Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:59.085Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:59.113 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fc3d4a5e-79a8-e45e-ecd2-59453b8997ac Address:127.0.0.1:11370}]" -44:59.113 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11370 [Follower]" leader= -44:59.114 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac.dc1 127.0.0.1 -44:59.114 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac 127.0.0.1 -2022-11-07T22:44:59.114Z [INFO] agent.router: Initializing LAN area manager -44:59.114 [INFO] TestAgent.server: Adding LAN server: server="Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac (Addr: tcp/127.0.0.1:11370) (DC: dc1)" -44:59.115 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11365 network=udp -44:59.115 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac.dc1 area=wan -44:59.115 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11365 network=tcp -44:59.115 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11366 network=tcp -44:59.115 [INFO] TestAgent: started state syncer -44:59.115 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11371 network=tcp -44:59.177 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:59.177 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11370 [Candidate]" term=2 -44:59.186 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:59.186 [DEBUG] TestAgent.server.raft: vote granted: from=fc3d4a5e-79a8-e45e-ecd2-59453b8997ac term=2 tally=1 -44:59.186 [INFO] TestAgent.server.raft: election won: tally=1 -44:59.186 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11370 [Leader]" -44:59.187 [INFO] TestAgent.server: cluster leadership acquired -44:59.187 [INFO] TestAgent.server: New leader elected: payload=Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac -44:59.195 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11370 -44:59.208 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:59.235 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:59.235 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:59.235 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:59.235 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:59.235 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:59.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac -44:59.235 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fc3d4a5e-79a8-e45e-ecd2-59453b8997ac -44:59.241 [INFO] TestAgent.server: federation state anti-entropy synced -44:59.248 [INFO] TestAgent: Requesting shutdown -44:59.248 [INFO] TestAgent.server: shutting down server -44:59.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:59.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:59.248 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:59.248 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:59.248 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:59.248 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:59.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:59.248 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:59.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:59.248 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:59.248 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:59.251 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:59.255Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:59.255Z [INFO] agent.router.manager: shutting down -44:59.255 [INFO] TestAgent: consul server down -44:59.255 [INFO] TestAgent: shutdown complete -44:59.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11365 network=tcp -44:59.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11365 network=udp -44:59.255 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11366 network=tcp -44:59.755 [INFO] TestAgent: Waiting for endpoints to shut down -44:59.755 [INFO] TestAgent: Endpoints down -2022-11-07T22:44:59.070Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:59.087Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:59.115 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1c3302ec-f00b-f1d3-7cbd-560ef1cff99a Address:127.0.0.1:11377}]" -44:59.115 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11377 [Follower]" leader= -44:59.116 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a.dc1 127.0.0.1 -44:59.116 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a 127.0.0.1 -2022-11-07T22:44:59.116Z [INFO] agent.router: Initializing LAN area manager -44:59.116 [INFO] TestAgent.server: Adding LAN server: server="Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a (Addr: tcp/127.0.0.1:11377) (DC: dc1)" -44:59.116 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a.dc1 area=wan -44:59.116 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11372 network=udp -44:59.116 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11372 network=tcp -44:59.116 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11373 network=tcp -44:59.116 [INFO] TestAgent: started state syncer -44:59.116 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11378 network=tcp -44:59.182 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:59.182 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11377 [Candidate]" term=2 -44:59.194 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:59.194 [DEBUG] TestAgent.server.raft: vote granted: from=1c3302ec-f00b-f1d3-7cbd-560ef1cff99a term=2 tally=1 -44:59.194 [INFO] TestAgent.server.raft: election won: tally=1 -44:59.194 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11377 [Leader]" -44:59.194 [INFO] TestAgent.server: cluster leadership acquired -44:59.194 [INFO] TestAgent.server: New leader elected: payload=Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a -44:59.201 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11377 -44:59.214 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:59.236 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:59.236 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:59.236 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:59.236 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:59.236 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:59.236 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a -44:59.236 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1c3302ec-f00b-f1d3-7cbd-560ef1cff99a -44:59.248 [INFO] TestAgent.server: federation state anti-entropy synced -44:59.310 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:59.313 [INFO] TestAgent: Synced node info -44:59.313 [DEBUG] TestAgent: Node info in sync -44:59.538 [INFO] TestAgent: Requesting shutdown -44:59.538 [INFO] TestAgent.server: shutting down server -44:59.538 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:59.538 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:59.538 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:59.538 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:59.538 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:59.538 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:59.538 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:59.538 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:59.538 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:59.538 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:59.542 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:59.545Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:59.545Z [INFO] agent.router.manager: shutting down -44:59.545 [INFO] TestAgent: consul server down -44:59.545 [INFO] TestAgent: shutdown complete -44:59.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11372 network=tcp -44:59.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11372 network=udp -44:59.546 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11373 network=tcp -45:00.046 [INFO] TestAgent: Waiting for endpoints to shut down -45:00.046 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:00.669Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:00.688Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:00.862 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:df053f7a-6ee2-1613-3960-8b755ee6cab3 Address:127.0.0.1:29748}]" +30:00.863 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-df053f7a-6ee2-1613-3960-8b755ee6cab3.dc1 127.0.0.1 +30:00.863 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-df053f7a-6ee2-1613-3960-8b755ee6cab3 127.0.0.1 +2021-10-05T16:30:00.863Z [INFO] agent.router: Initializing LAN area manager +30:00.864 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29748 [Follower]" leader= +30:00.864 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=udp +30:00.864 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-df053f7a-6ee2-1613-3960-8b755ee6cab3.dc1 area=wan +30:00.864 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=tcp +30:00.864 [INFO] TestAgent.server: Adding LAN server: server="Node-df053f7a-6ee2-1613-3960-8b755ee6cab3 (Addr: tcp/127.0.0.1:29748) (DC: dc1)" +30:00.864 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29744 network=tcp +30:00.865 [INFO] TestAgent: started state syncer +30:00.865 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29749 network=tcp +30:00.911 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:00.912 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29748 [Candidate]" term=2 +30:00.923 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:00.923 [DEBUG] TestAgent.server.raft: vote granted: from=df053f7a-6ee2-1613-3960-8b755ee6cab3 term=2 tally=1 +30:00.923 [INFO] TestAgent.server.raft: election won: tally=1 +30:00.923 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29748 [Leader]" +30:00.924 [INFO] TestAgent.server: cluster leadership acquired +30:00.924 [INFO] TestAgent.server: New leader elected: payload=Node-df053f7a-6ee2-1613-3960-8b755ee6cab3 +30:00.931 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29748 +30:00.969 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:01.028 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:01.028 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:01.028 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:01.028 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:01.028 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:01.028 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df053f7a-6ee2-1613-3960-8b755ee6cab3 +30:01.028 [INFO] TestAgent.server: member joined, marking health alive: member=Node-df053f7a-6ee2-1613-3960-8b755ee6cab3 +30:01.041 [INFO] TestAgent: Requesting shutdown +30:01.042 [INFO] TestAgent.server: shutting down server +30:01.042 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:01.042 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:01.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:01.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:01.042 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:01.044 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:01.044 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:01.044 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:01.044 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:01.052 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:01.064Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:01.064Z [INFO] agent.router.manager: shutting down +30:01.066 [INFO] TestAgent: consul server down +30:01.066 [INFO] TestAgent: shutdown complete +30:01.066 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=tcp +30:01.066 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=udp +30:01.066 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29744 network=tcp +30:01.066 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +30:01.066 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:01.568 [INFO] TestAgent: Waiting for endpoints to shut down +30:01.568 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:00.654Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:00.688Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:00.874 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ea66d9c7-adc7-4cd3-50fd-6754560867df Address:127.0.0.1:29755}]" +30:00.874 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29755 [Follower]" leader= +30:00.874 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ea66d9c7-adc7-4cd3-50fd-6754560867df.dc1 127.0.0.1 +30:00.875 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ea66d9c7-adc7-4cd3-50fd-6754560867df 127.0.0.1 +2021-10-05T16:30:00.875Z [INFO] agent.router: Initializing LAN area manager +30:00.875 [INFO] TestAgent.server: Adding LAN server: server="Node-ea66d9c7-adc7-4cd3-50fd-6754560867df (Addr: tcp/127.0.0.1:29755) (DC: dc1)" +30:00.875 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ea66d9c7-adc7-4cd3-50fd-6754560867df.dc1 area=wan +30:00.875 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=udp +30:00.875 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=tcp +30:00.876 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29751 network=tcp +30:00.876 [INFO] TestAgent: started state syncer +30:00.876 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29756 network=tcp +30:00.927 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:00.927 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29755 [Candidate]" term=2 +30:00.957 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:00.957 [DEBUG] TestAgent.server.raft: vote granted: from=ea66d9c7-adc7-4cd3-50fd-6754560867df term=2 tally=1 +30:00.957 [INFO] TestAgent.server.raft: election won: tally=1 +30:00.957 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29755 [Leader]" +30:00.957 [INFO] TestAgent.server: cluster leadership acquired +30:00.958 [INFO] TestAgent.server: New leader elected: payload=Node-ea66d9c7-adc7-4cd3-50fd-6754560867df +30:00.968 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29755 +30:01.004 [INFO] TestAgent: Synced node info +30:01.004 [DEBUG] TestAgent: Node info in sync +30:01.033 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:01.081 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:01.081 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:01.081 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:01.082 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:01.082 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:01.082 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ea66d9c7-adc7-4cd3-50fd-6754560867df +30:01.082 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ea66d9c7-adc7-4cd3-50fd-6754560867df +30:01.113 [INFO] TestAgent.server: federation state anti-entropy synced +30:01.226 [INFO] TestAgent: Requesting shutdown +30:01.226 [INFO] TestAgent.server: shutting down server +30:01.226 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:01.226 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:01.226 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:01.226 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:01.226 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:01.226 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:01.226 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:01.226 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:01.226 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:01.226 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:01.232 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:01.240Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:01.241Z [INFO] agent.router.manager: shutting down +30:01.241 [INFO] TestAgent: consul server down +30:01.241 [INFO] TestAgent: shutdown complete +30:01.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=tcp +30:01.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=udp +30:01.241 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29751 network=tcp +30:01.744 [INFO] TestAgent: Waiting for endpoints to shut down +30:01.744 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ScriptCheck_ExecDisable (0.00s) - --- PASS: TestAgent_RegisterService_ScriptCheck_ExecDisable/normal (0.70s) - --- PASS: TestAgent_RegisterService_ScriptCheck_ExecDisable/service_manager (0.99s) + --- PASS: TestAgent_RegisterService_ScriptCheck_ExecDisable/normal (0.95s) + --- PASS: TestAgent_RegisterService_ScriptCheck_ExecDisable/service_manager (1.12s) === RUN TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable === RUN TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/normal === PAUSE TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/normal @@ -9497,121 +9557,123 @@ === PAUSE TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/service_manager === CONT TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/normal === CONT TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/service_manager -2022-11-07T22:45:00.062Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:00.077Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:00.108 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e944e8d8-0c89-e2ef-59e0-2e8723141987 Address:127.0.0.1:11384}]" -45:00.108 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11384 [Follower]" leader= -45:00.109 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e944e8d8-0c89-e2ef-59e0-2e8723141987.dc1 127.0.0.1 -45:00.110 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e944e8d8-0c89-e2ef-59e0-2e8723141987 127.0.0.1 -2022-11-07T22:45:00.110Z [INFO] agent.router: Initializing LAN area manager -45:00.110 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e944e8d8-0c89-e2ef-59e0-2e8723141987.dc1 area=wan -45:00.110 [INFO] TestAgent.server: Adding LAN server: server="Node-e944e8d8-0c89-e2ef-59e0-2e8723141987 (Addr: tcp/127.0.0.1:11384) (DC: dc1)" -45:00.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11379 network=tcp -45:00.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11379 network=udp -45:00.110 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11380 network=tcp -45:00.110 [INFO] TestAgent: started state syncer -45:00.111 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11385 network=tcp -45:00.147 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:00.147 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11384 [Candidate]" term=2 -45:00.156 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:00.156 [DEBUG] TestAgent.server.raft: vote granted: from=e944e8d8-0c89-e2ef-59e0-2e8723141987 term=2 tally=1 -45:00.156 [INFO] TestAgent.server.raft: election won: tally=1 -45:00.156 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11384 [Leader]" -45:00.157 [INFO] TestAgent.server: cluster leadership acquired -45:00.157 [INFO] TestAgent.server: New leader elected: payload=Node-e944e8d8-0c89-e2ef-59e0-2e8723141987 -45:00.163 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11384 -45:00.175 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:00.195 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:00.195 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:00.195 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:00.195 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:00.195 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:00.195 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e944e8d8-0c89-e2ef-59e0-2e8723141987 -45:00.195 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e944e8d8-0c89-e2ef-59e0-2e8723141987 -45:00.200 [INFO] TestAgent.server: federation state anti-entropy synced -45:00.262 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:00.265 [INFO] TestAgent: Synced node info -45:00.365 [INFO] TestAgent: Requesting shutdown -45:00.365 [INFO] TestAgent.server: shutting down server -45:00.365 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:00.366 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:00.366 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:00.366 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:00.366 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:00.366 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:00.366 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:00.366 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:00.366 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:00.366 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:00.369 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:00.373Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:00.373Z [INFO] agent.router.manager: shutting down -45:00.373 [INFO] TestAgent: consul server down -45:00.373 [INFO] TestAgent: shutdown complete -45:00.373 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11379 network=tcp -45:00.373 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11379 network=udp -45:00.373 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11380 network=tcp -45:00.873 [INFO] TestAgent: Waiting for endpoints to shut down -45:00.873 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:00.062Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:00.077Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:00.110 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7dd28624-99e6-15c4-d4b8-ff7be93b87b8 Address:127.0.0.1:11391}]" -45:00.111 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11391 [Follower]" leader= -45:00.111 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8.dc1 127.0.0.1 -45:00.111 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8 127.0.0.1 -2022-11-07T22:45:00.111Z [INFO] agent.router: Initializing LAN area manager -45:00.111 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8.dc1 area=wan -45:00.111 [INFO] TestAgent.server: Adding LAN server: server="Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8 (Addr: tcp/127.0.0.1:11391) (DC: dc1)" -45:00.111 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11386 network=udp -45:00.111 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11386 network=tcp -45:00.112 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11387 network=tcp -45:00.112 [INFO] TestAgent: started state syncer -45:00.112 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11392 network=tcp -45:00.149 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:00.149 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11391 [Candidate]" term=2 -45:00.159 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:00.159 [DEBUG] TestAgent.server.raft: vote granted: from=7dd28624-99e6-15c4-d4b8-ff7be93b87b8 term=2 tally=1 -45:00.159 [INFO] TestAgent.server.raft: election won: tally=1 -45:00.159 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11391 [Leader]" -45:00.159 [INFO] TestAgent.server: cluster leadership acquired -45:00.159 [INFO] TestAgent.server: New leader elected: payload=Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8 -45:00.165 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11391 -45:00.177 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:00.198 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:00.198 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:00.198 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:00.198 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:00.198 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:00.198 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8 -45:00.198 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7dd28624-99e6-15c4-d4b8-ff7be93b87b8 -45:00.203 [INFO] TestAgent.server: federation state anti-entropy synced -45:00.434 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:00.437 [INFO] TestAgent: Synced node info -45:00.481 [INFO] TestAgent: Requesting shutdown -45:00.481 [INFO] TestAgent.server: shutting down server -45:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:00.481 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:00.481 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:00.485 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:00.488Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:00.488Z [INFO] agent.router.manager: shutting down -45:00.489 [INFO] TestAgent: consul server down -45:00.489 [INFO] TestAgent: shutdown complete -45:00.489 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11386 network=tcp -45:00.489 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11386 network=udp -45:00.489 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11387 network=tcp -45:00.989 [INFO] TestAgent: Waiting for endpoints to shut down -45:00.989 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:01.798Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:01.828Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:01.964 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:881be60b-bbd1-9574-0b0f-76579932e100 Address:127.0.0.1:29762}]" +30:01.964 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-881be60b-bbd1-9574-0b0f-76579932e100.dc1 127.0.0.1 +30:01.965 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-881be60b-bbd1-9574-0b0f-76579932e100 127.0.0.1 +2021-10-05T16:30:01.965Z [INFO] agent.router: Initializing LAN area manager +30:01.965 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=udp +30:01.965 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29762 [Follower]" leader= +30:01.965 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-881be60b-bbd1-9574-0b0f-76579932e100.dc1 area=wan +30:01.966 [INFO] TestAgent.server: Adding LAN server: server="Node-881be60b-bbd1-9574-0b0f-76579932e100 (Addr: tcp/127.0.0.1:29762) (DC: dc1)" +30:01.966 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=tcp +30:01.966 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29758 network=tcp +30:01.966 [INFO] TestAgent: started state syncer +30:01.966 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29763 network=tcp +30:02.008 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:02.008 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29762 [Candidate]" term=2 +30:02.056 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:02.056 [DEBUG] TestAgent.server.raft: vote granted: from=881be60b-bbd1-9574-0b0f-76579932e100 term=2 tally=1 +30:02.056 [INFO] TestAgent.server.raft: election won: tally=1 +30:02.056 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29762 [Leader]" +30:02.057 [INFO] TestAgent.server: cluster leadership acquired +30:02.057 [INFO] TestAgent.server: New leader elected: payload=Node-881be60b-bbd1-9574-0b0f-76579932e100 +30:02.088 [INFO] TestAgent: Synced node info +30:02.088 [DEBUG] TestAgent: Node info in sync +30:02.088 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29762 +30:02.127 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:02.152 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:02.152 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:02.152 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:02.152 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:02.152 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:02.152 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-881be60b-bbd1-9574-0b0f-76579932e100 +30:02.152 [INFO] TestAgent.server: member joined, marking health alive: member=Node-881be60b-bbd1-9574-0b0f-76579932e100 +30:02.184 [INFO] TestAgent.server: federation state anti-entropy synced +30:02.188 [INFO] TestAgent: Requesting shutdown +30:02.188 [INFO] TestAgent.server: shutting down server +30:02.188 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:02.188 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:02.188 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:02.188 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:02.188 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:02.188 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:02.188 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:02.188 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:02.188 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:02.188 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:02.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:02.212Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:02.212Z [INFO] agent.router.manager: shutting down +30:02.212 [INFO] TestAgent: consul server down +30:02.212 [INFO] TestAgent: shutdown complete +30:02.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=tcp +30:02.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=udp +30:02.212 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29758 network=tcp +30:02.713 [INFO] TestAgent: Waiting for endpoints to shut down +30:02.713 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:01.813Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:01.853Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:02.008 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15595ea9-62a8-234a-5122-d37887dd3903 Address:127.0.0.1:29769}]" +30:02.009 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15595ea9-62a8-234a-5122-d37887dd3903.dc1 127.0.0.1 +30:02.009 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15595ea9-62a8-234a-5122-d37887dd3903 127.0.0.1 +2021-10-05T16:30:02.009Z [INFO] agent.router: Initializing LAN area manager +30:02.010 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=udp +30:02.010 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29769 [Follower]" leader= +30:02.010 [INFO] TestAgent.server: Adding LAN server: server="Node-15595ea9-62a8-234a-5122-d37887dd3903 (Addr: tcp/127.0.0.1:29769) (DC: dc1)" +30:02.010 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15595ea9-62a8-234a-5122-d37887dd3903.dc1 area=wan +30:02.010 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=tcp +30:02.011 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29765 network=tcp +30:02.011 [INFO] TestAgent: started state syncer +30:02.011 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29770 network=tcp +30:02.076 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:02.076 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29769 [Candidate]" term=2 +30:02.132 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:02.132 [DEBUG] TestAgent.server.raft: vote granted: from=15595ea9-62a8-234a-5122-d37887dd3903 term=2 tally=1 +30:02.132 [INFO] TestAgent.server.raft: election won: tally=1 +30:02.132 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29769 [Leader]" +30:02.132 [INFO] TestAgent.server: cluster leadership acquired +30:02.132 [INFO] TestAgent.server: New leader elected: payload=Node-15595ea9-62a8-234a-5122-d37887dd3903 +30:02.164 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29769 +30:02.210 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:02.255 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:02.255 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:02.255 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:02.255 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:02.255 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:02.255 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15595ea9-62a8-234a-5122-d37887dd3903 +30:02.255 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15595ea9-62a8-234a-5122-d37887dd3903 +30:02.287 [INFO] TestAgent.server: federation state anti-entropy synced +30:02.408 [INFO] TestAgent: Requesting shutdown +30:02.408 [INFO] TestAgent.server: shutting down server +30:02.408 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:02.408 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:02.408 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:02.408 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:02.408 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:02.408 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:02.409 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:02.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:02.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:02.409 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:02.409 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:02.424 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +30:02.424 [INFO] TestAgent: Synced node info +2021-10-05T16:30:02.440Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:02.440Z [INFO] agent.router.manager: shutting down +30:02.444 [INFO] TestAgent: consul server down +30:02.444 [INFO] TestAgent: shutdown complete +30:02.444 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=tcp +30:02.444 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=udp +30:02.444 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29765 network=tcp +30:02.444 [WARN] TestAgent: Deregistering service failed.: service=test error="leadership lost while committing log" +30:02.444 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="leadership lost while committing log" +30:02.948 [INFO] TestAgent: Waiting for endpoints to shut down +30:02.948 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable (0.00s) - --- PASS: TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/normal (0.83s) - --- PASS: TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/service_manager (0.94s) + --- PASS: TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/normal (0.97s) + --- PASS: TestAgent_RegisterService_ScriptCheck_ExecRemoteDisable/service_manager (1.20s) === RUN TestAgent_DeregisterService === PAUSE TestAgent_DeregisterService === RUN TestAgent_DeregisterService_ACLDeny @@ -9711,716 +9773,694 @@ === PAUSE TestAgent_MultiStartStop/#09 === CONT TestAgent_MultiStartStop/#00 === CONT TestAgent_MultiStartStop/#05 -=== CONT TestAgent_MultiStartStop/#03 +=== CONT TestAgent_MultiStartStop/#08 === CONT TestAgent_MultiStartStop/#04 +=== CONT TestAgent_MultiStartStop/#09 === CONT TestAgent_MultiStartStop/#02 -=== CONT TestAgent_MultiStartStop/#08 -=== CONT TestAgent_MultiStartStop/#01 === CONT TestAgent_MultiStartStop/#07 +=== CONT TestAgent_MultiStartStop/#03 === CONT TestAgent_MultiStartStop/#06 -=== CONT TestAgent_MultiStartStop/#09 -2022-11-07T22:45:01.011Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.068 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15e8c340-798e-7e7c-746d-dc466c1c6d44 Address:127.0.0.1:11412}]" -45:01.068 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11412 [Follower]" leader= -45:01.090 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15e8c340-798e-7e7c-746d-dc466c1c6d44.dc1 127.0.0.1 -45:01.091 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15e8c340-798e-7e7c-746d-dc466c1c6d44 127.0.0.1 -2022-11-07T22:45:01.091Z [INFO] agent.router: Initializing LAN area manager -45:01.091 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15e8c340-798e-7e7c-746d-dc466c1c6d44.dc1 area=wan -45:01.091 [INFO] TestAgent.server: Adding LAN server: server="Node-15e8c340-798e-7e7c-746d-dc466c1c6d44 (Addr: tcp/127.0.0.1:11412) (DC: dc1)" -45:01.091 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11407 network=udp -45:01.091 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11407 network=tcp -45:01.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11408 network=tcp -45:01.092 [INFO] TestAgent: started state syncer -45:01.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11413 network=tcp -45:01.114 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.114 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11412 [Candidate]" term=2 -45:01.132 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.132 [DEBUG] TestAgent.server.raft: vote granted: from=15e8c340-798e-7e7c-746d-dc466c1c6d44 term=2 tally=1 -45:01.132 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.132 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11412 [Leader]" -45:01.132 [INFO] TestAgent.server: cluster leadership acquired -45:01.132 [INFO] TestAgent.server: New leader elected: payload=Node-15e8c340-798e-7e7c-746d-dc466c1c6d44 -45:01.144 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11412 -45:01.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.190 [INFO] TestAgent: Synced node info -45:01.190 [DEBUG] TestAgent: Node info in sync -45:01.205 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.205 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.205 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.205 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.205 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.205 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15e8c340-798e-7e7c-746d-dc466c1c6d44 -45:01.205 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15e8c340-798e-7e7c-746d-dc466c1c6d44 -45:01.225 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.501 [INFO] TestAgent: Requesting shutdown -45:01.501 [INFO] TestAgent.server: shutting down server -45:01.501 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.501 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.501 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.501 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.501 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.502 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.502 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.502 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.502 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.502 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.505 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.509Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.509Z [INFO] agent.router.manager: shutting down -45:01.509 [INFO] TestAgent: consul server down -45:01.509 [INFO] TestAgent: shutdown complete -45:01.509 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11407 network=tcp -45:01.509 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11407 network=udp -45:01.509 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11408 network=tcp -45:02.009 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.009 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.003Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.028Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.067 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e9d47733-9bb2-09a0-3492-bd38576290c8 Address:127.0.0.1:11398}]" -45:01.067 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11398 [Follower]" leader= -45:01.068 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e9d47733-9bb2-09a0-3492-bd38576290c8.dc1 127.0.0.1 -45:01.068 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e9d47733-9bb2-09a0-3492-bd38576290c8 127.0.0.1 -2022-11-07T22:45:01.068Z [INFO] agent.router: Initializing LAN area manager -45:01.068 [INFO] TestAgent.server: Adding LAN server: server="Node-e9d47733-9bb2-09a0-3492-bd38576290c8 (Addr: tcp/127.0.0.1:11398) (DC: dc1)" -45:01.068 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e9d47733-9bb2-09a0-3492-bd38576290c8.dc1 area=wan -45:01.090 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11393 network=tcp -45:01.090 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11393 network=udp -45:01.090 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11394 network=tcp -45:01.090 [INFO] TestAgent: started state syncer -45:01.090 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11399 network=tcp -45:01.119 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.119 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11398 [Candidate]" term=2 -45:01.138 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.138 [DEBUG] TestAgent.server.raft: vote granted: from=e9d47733-9bb2-09a0-3492-bd38576290c8 term=2 tally=1 -45:01.138 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11398 [Leader]" -45:01.138 [INFO] TestAgent.server: cluster leadership acquired -45:01.138 [INFO] TestAgent.server: New leader elected: payload=Node-e9d47733-9bb2-09a0-3492-bd38576290c8 -45:01.147 [INFO] TestAgent: Synced node info -45:01.147 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11398 -45:01.185 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.208 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.208 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.208 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.208 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.208 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.208 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e9d47733-9bb2-09a0-3492-bd38576290c8 -45:01.208 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e9d47733-9bb2-09a0-3492-bd38576290c8 -45:01.223 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.527 [INFO] TestAgent: Requesting shutdown -45:01.527 [INFO] TestAgent.server: shutting down server -45:01.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.527 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.527 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.530 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.535Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.535Z [INFO] agent.router.manager: shutting down -45:01.535 [INFO] TestAgent: consul server down -45:01.535 [INFO] TestAgent: shutdown complete -45:01.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11393 network=tcp -45:01.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11393 network=udp -45:01.535 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11394 network=tcp -45:02.035 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.035 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.000Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.019Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.054 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6347daf7-8105-63e1-671f-72fd8e0cb346 Address:127.0.0.1:11405}]" -45:01.054 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11405 [Follower]" leader= -45:01.055 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6347daf7-8105-63e1-671f-72fd8e0cb346.dc1 127.0.0.1 -45:01.056 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6347daf7-8105-63e1-671f-72fd8e0cb346 127.0.0.1 -2022-11-07T22:45:01.056Z [INFO] agent.router: Initializing LAN area manager -45:01.056 [INFO] TestAgent.server: Adding LAN server: server="Node-6347daf7-8105-63e1-671f-72fd8e0cb346 (Addr: tcp/127.0.0.1:11405) (DC: dc1)" -45:01.056 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6347daf7-8105-63e1-671f-72fd8e0cb346.dc1 area=wan -45:01.056 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11400 network=udp -45:01.056 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11400 network=tcp -45:01.056 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11401 network=tcp -45:01.056 [INFO] TestAgent: started state syncer -45:01.056 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11406 network=tcp -45:01.093 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.093 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11405 [Candidate]" term=2 -45:01.102 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.102 [DEBUG] TestAgent.server.raft: vote granted: from=6347daf7-8105-63e1-671f-72fd8e0cb346 term=2 tally=1 -45:01.102 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.102 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11405 [Leader]" -45:01.102 [INFO] TestAgent.server: cluster leadership acquired -45:01.102 [INFO] TestAgent.server: New leader elected: payload=Node-6347daf7-8105-63e1-671f-72fd8e0cb346 -45:01.107 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11405 -45:01.125 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.162 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.162 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.162 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.162 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.162 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.162 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6347daf7-8105-63e1-671f-72fd8e0cb346 -45:01.162 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6347daf7-8105-63e1-671f-72fd8e0cb346 -45:01.185 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.383 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.386 [INFO] TestAgent: Synced node info -45:01.528 [INFO] TestAgent: Requesting shutdown -45:01.528 [INFO] TestAgent.server: shutting down server -45:01.528 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.528 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.528 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.528 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.528 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.528 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.528 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.528 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.528 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.528 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.532 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.537Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.537Z [INFO] agent.router.manager: shutting down -45:01.537 [INFO] TestAgent: consul server down -45:01.537 [INFO] TestAgent: shutdown complete -45:01.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11400 network=tcp -45:01.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11400 network=udp -45:01.537 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11401 network=tcp -45:02.037 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.037 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.011Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.028Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.062 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5a4e8050-eeea-f528-c145-5eedd2fc5c1c Address:127.0.0.1:11426}]" -45:01.062 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11426 [Follower]" leader= -45:01.063 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c.dc1 127.0.0.1 -45:01.064 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c 127.0.0.1 -2022-11-07T22:45:01.064Z [INFO] agent.router: Initializing LAN area manager -45:01.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11421 network=udp -45:01.064 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c.dc1 area=wan -45:01.064 [INFO] TestAgent.server: Adding LAN server: server="Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c (Addr: tcp/127.0.0.1:11426) (DC: dc1)" -45:01.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11421 network=tcp -45:01.065 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11422 network=tcp -45:01.065 [INFO] TestAgent: started state syncer -45:01.065 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11427 network=tcp -45:01.116 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.116 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11426 [Candidate]" term=2 -45:01.136 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.136 [DEBUG] TestAgent.server.raft: vote granted: from=5a4e8050-eeea-f528-c145-5eedd2fc5c1c term=2 tally=1 -45:01.136 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.136 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11426 [Leader]" -45:01.136 [INFO] TestAgent.server: cluster leadership acquired -45:01.136 [INFO] TestAgent.server: New leader elected: payload=Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c -45:01.150 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11426 -45:01.175 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.204 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.204 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.204 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.204 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.204 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.204 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c -45:01.204 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5a4e8050-eeea-f528-c145-5eedd2fc5c1c -45:01.213 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.343 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.347 [INFO] TestAgent: Synced node info -45:01.347 [DEBUG] TestAgent: Node info in sync -45:01.573 [INFO] TestAgent: Requesting shutdown -45:01.573 [INFO] TestAgent.server: shutting down server -45:01.573 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.573 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.573 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.573 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.573 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.573 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.573 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.573 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.573 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.573 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.577 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.580Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.580Z [INFO] agent.router.manager: shutting down -45:01.580 [INFO] TestAgent: consul server down -45:01.580 [INFO] TestAgent: shutdown complete -45:01.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11421 network=tcp -45:01.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11421 network=udp -45:01.580 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11422 network=tcp -45:02.081 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.081 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.026Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.041Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.100 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c462315b-2ed2-9395-3579-d59dbe45b739 Address:127.0.0.1:11461}]" -45:01.100 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11461 [Follower]" leader= -45:01.101 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c462315b-2ed2-9395-3579-d59dbe45b739.dc1 127.0.0.1 -45:01.101 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c462315b-2ed2-9395-3579-d59dbe45b739 127.0.0.1 -2022-11-07T22:45:01.101Z [INFO] agent.router: Initializing LAN area manager -45:01.101 [INFO] TestAgent.server: Adding LAN server: server="Node-c462315b-2ed2-9395-3579-d59dbe45b739 (Addr: tcp/127.0.0.1:11461) (DC: dc1)" -45:01.101 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c462315b-2ed2-9395-3579-d59dbe45b739.dc1 area=wan -45:01.101 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11456 network=udp -45:01.101 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11456 network=tcp -45:01.102 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11457 network=tcp -45:01.102 [INFO] TestAgent: started state syncer -45:01.102 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11462 network=tcp -45:01.162 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.162 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11461 [Candidate]" term=2 -45:01.178 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.178 [DEBUG] TestAgent.server.raft: vote granted: from=c462315b-2ed2-9395-3579-d59dbe45b739 term=2 tally=1 -45:01.178 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.178 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11461 [Leader]" -45:01.178 [INFO] TestAgent.server: cluster leadership acquired -45:01.178 [INFO] TestAgent.server: New leader elected: payload=Node-c462315b-2ed2-9395-3579-d59dbe45b739 -45:01.187 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11461 -45:01.205 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.235 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.235 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.235 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.235 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.235 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c462315b-2ed2-9395-3579-d59dbe45b739 -45:01.235 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c462315b-2ed2-9395-3579-d59dbe45b739 -45:01.249 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.288 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.291 [INFO] TestAgent: Synced node info -45:01.291 [DEBUG] TestAgent: Node info in sync -45:01.580 [INFO] TestAgent: Requesting shutdown -45:01.580 [INFO] TestAgent.server: shutting down server -45:01.580 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.580 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.580 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.580 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.580 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.580 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.580 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.580 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.580 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.580 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.584 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.587Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.587Z [INFO] agent.router.manager: shutting down -45:01.587 [INFO] TestAgent: consul server down -45:01.587 [INFO] TestAgent: shutdown complete -45:01.587 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11456 network=tcp -45:01.587 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11456 network=udp -45:01.587 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11457 network=tcp -45:02.087 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.087 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.013Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.031Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.090 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22ca5aeb-4afc-7895-d731-4bbb451b58a0 Address:127.0.0.1:11419}]" -45:01.090 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11419 [Follower]" leader= -45:01.091 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0.dc1 127.0.0.1 -45:01.091 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0 127.0.0.1 -2022-11-07T22:45:01.091Z [INFO] agent.router: Initializing LAN area manager -45:01.092 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0.dc1 area=wan -45:01.092 [INFO] TestAgent.server: Adding LAN server: server="Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0 (Addr: tcp/127.0.0.1:11419) (DC: dc1)" -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11414 network=udp -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11414 network=tcp -45:01.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11415 network=tcp -45:01.092 [INFO] TestAgent: started state syncer -45:01.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11420 network=tcp -45:01.157 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.157 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11419 [Candidate]" term=2 -45:01.172 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.172 [DEBUG] TestAgent.server.raft: vote granted: from=22ca5aeb-4afc-7895-d731-4bbb451b58a0 term=2 tally=1 -45:01.172 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.172 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11419 [Leader]" -45:01.172 [INFO] TestAgent.server: cluster leadership acquired -45:01.172 [INFO] TestAgent.server: New leader elected: payload=Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0 -45:01.183 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11419 -45:01.191 [INFO] TestAgent: Synced node info -45:01.191 [DEBUG] TestAgent: Node info in sync -45:01.214 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.236 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.236 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.236 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.236 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.236 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.236 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0 -45:01.236 [INFO] TestAgent.server: member joined, marking health alive: member=Node-22ca5aeb-4afc-7895-d731-4bbb451b58a0 -45:01.251 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.584 [INFO] TestAgent: Requesting shutdown -45:01.584 [INFO] TestAgent.server: shutting down server -45:01.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.584 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.584 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.584 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.584 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.584 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.584 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.589 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.592Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.592Z [INFO] agent.router.manager: shutting down -45:01.592 [INFO] TestAgent: consul server down -45:01.592 [INFO] TestAgent: shutdown complete -45:01.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11414 network=tcp -45:01.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11414 network=udp -45:01.592 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11415 network=tcp -45:02.093 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.093 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.029Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.038Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.090 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9489c71-21e2-e9d2-1e3e-12f618068940 Address:127.0.0.1:11454}]" -45:01.090 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11454 [Follower]" leader= -45:01.091 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9489c71-21e2-e9d2-1e3e-12f618068940.dc1 127.0.0.1 -45:01.091 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9489c71-21e2-e9d2-1e3e-12f618068940 127.0.0.1 -2022-11-07T22:45:01.091Z [INFO] agent.router: Initializing LAN area manager -45:01.092 [INFO] TestAgent.server: Adding LAN server: server="Node-c9489c71-21e2-e9d2-1e3e-12f618068940 (Addr: tcp/127.0.0.1:11454) (DC: dc1)" -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11449 network=udp -45:01.092 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9489c71-21e2-e9d2-1e3e-12f618068940.dc1 area=wan -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11449 network=tcp -45:01.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11450 network=tcp -45:01.092 [INFO] TestAgent: started state syncer -45:01.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11455 network=tcp -45:01.131 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.131 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11454 [Candidate]" term=2 -45:01.147 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.147 [DEBUG] TestAgent.server.raft: vote granted: from=c9489c71-21e2-e9d2-1e3e-12f618068940 term=2 tally=1 -45:01.147 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.147 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11454 [Leader]" -45:01.147 [INFO] TestAgent.server: cluster leadership acquired -45:01.147 [INFO] TestAgent.server: New leader elected: payload=Node-c9489c71-21e2-e9d2-1e3e-12f618068940 -45:01.156 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11454 -45:01.181 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.210 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.210 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.210 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.210 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.210 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.210 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9489c71-21e2-e9d2-1e3e-12f618068940 -45:01.210 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9489c71-21e2-e9d2-1e3e-12f618068940 -45:01.228 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.400 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.403 [INFO] TestAgent: Synced node info -45:01.403 [DEBUG] TestAgent: Node info in sync -45:01.604 [INFO] TestAgent: Requesting shutdown -45:01.604 [INFO] TestAgent.server: shutting down server -45:01.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.604 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.604 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.604 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.604 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.604 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.604 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.607 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.610Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.610Z [INFO] agent.router.manager: shutting down -45:01.610 [INFO] TestAgent: consul server down -45:01.610 [INFO] TestAgent: shutdown complete -45:01.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11449 network=tcp -45:01.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11449 network=udp -45:01.610 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11450 network=tcp -45:02.110 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.110 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.012Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.020Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.055 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:13367595-71b7-cffa-26ad-d1e9fdef129c Address:127.0.0.1:11433}]" -45:01.055 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11433 [Follower]" leader= -45:01.056 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-13367595-71b7-cffa-26ad-d1e9fdef129c.dc1 127.0.0.1 -45:01.056 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-13367595-71b7-cffa-26ad-d1e9fdef129c 127.0.0.1 -2022-11-07T22:45:01.056Z [INFO] agent.router: Initializing LAN area manager -45:01.056 [INFO] TestAgent.server: Adding LAN server: server="Node-13367595-71b7-cffa-26ad-d1e9fdef129c (Addr: tcp/127.0.0.1:11433) (DC: dc1)" -45:01.056 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-13367595-71b7-cffa-26ad-d1e9fdef129c.dc1 area=wan -45:01.056 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11428 network=tcp -45:01.056 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11428 network=udp -45:01.057 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11429 network=tcp -45:01.057 [INFO] TestAgent: started state syncer -45:01.057 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11434 network=tcp -45:01.108 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.108 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11433 [Candidate]" term=2 -45:01.121 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.121 [DEBUG] TestAgent.server.raft: vote granted: from=13367595-71b7-cffa-26ad-d1e9fdef129c term=2 tally=1 -45:01.121 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.121 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11433 [Leader]" -45:01.121 [INFO] TestAgent.server: cluster leadership acquired -45:01.121 [INFO] TestAgent.server: New leader elected: payload=Node-13367595-71b7-cffa-26ad-d1e9fdef129c -45:01.135 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11433 -45:01.146 [INFO] TestAgent: Synced node info -45:01.146 [DEBUG] TestAgent: Node info in sync -45:01.174 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.196 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.196 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.196 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.196 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.196 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.196 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-13367595-71b7-cffa-26ad-d1e9fdef129c -45:01.196 [INFO] TestAgent.server: member joined, marking health alive: member=Node-13367595-71b7-cffa-26ad-d1e9fdef129c -45:01.205 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.650 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.650 [DEBUG] TestAgent: Node info in sync -45:01.696 [INFO] TestAgent: Requesting shutdown -45:01.696 [INFO] TestAgent.server: shutting down server -45:01.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.696 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.696 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.696 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.696 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.696 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.696 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.699 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.702Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.702Z [INFO] agent.router.manager: shutting down -45:01.702 [INFO] TestAgent: consul server down -45:01.702 [INFO] TestAgent: shutdown complete -45:01.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11428 network=tcp -45:01.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11428 network=udp -45:01.703 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11429 network=tcp -45:02.203 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.203 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.015Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.032Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.090 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:feb097c7-915f-6560-2f88-d508749c344d Address:127.0.0.1:11440}]" -45:01.090 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11440 [Follower]" leader= -45:01.091 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-feb097c7-915f-6560-2f88-d508749c344d.dc1 127.0.0.1 -45:01.091 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-feb097c7-915f-6560-2f88-d508749c344d 127.0.0.1 -2022-11-07T22:45:01.091Z [INFO] agent.router: Initializing LAN area manager -45:01.092 [INFO] TestAgent.server: Adding LAN server: server="Node-feb097c7-915f-6560-2f88-d508749c344d (Addr: tcp/127.0.0.1:11440) (DC: dc1)" -45:01.092 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-feb097c7-915f-6560-2f88-d508749c344d.dc1 area=wan -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11435 network=udp -45:01.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11435 network=tcp -45:01.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11436 network=tcp -45:01.092 [INFO] TestAgent: started state syncer -45:01.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11441 network=tcp -45:01.141 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.141 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11440 [Candidate]" term=2 -45:01.156 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.156 [DEBUG] TestAgent.server.raft: vote granted: from=feb097c7-915f-6560-2f88-d508749c344d term=2 tally=1 -45:01.156 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.156 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11440 [Leader]" -45:01.157 [INFO] TestAgent.server: cluster leadership acquired -45:01.157 [INFO] TestAgent.server: New leader elected: payload=Node-feb097c7-915f-6560-2f88-d508749c344d -45:01.166 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11440 -45:01.189 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.224 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.224 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.224 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.224 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.224 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.224 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-feb097c7-915f-6560-2f88-d508749c344d -45:01.224 [INFO] TestAgent.server: member joined, marking health alive: member=Node-feb097c7-915f-6560-2f88-d508749c344d -45:01.238 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.373 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:01.376 [INFO] TestAgent: Synced node info -45:01.376 [DEBUG] TestAgent: Node info in sync -45:01.733 [INFO] TestAgent: Requesting shutdown -45:01.733 [INFO] TestAgent.server: shutting down server -45:01.733 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.733 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.733 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.733 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.733 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.733 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.733 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.733 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.733 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.733 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.737 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.740Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.740Z [INFO] agent.router.manager: shutting down -45:01.740 [INFO] TestAgent: consul server down -45:01.740 [INFO] TestAgent: shutdown complete -45:01.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11435 network=tcp -45:01.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11435 network=udp -45:01.741 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11436 network=tcp -45:02.241 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.241 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:01.021Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.040Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.101 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7657ba43-ffb9-6c18-eefa-b08d4dacdc74 Address:127.0.0.1:11447}]" -45:01.101 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11447 [Follower]" leader= -45:01.101 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74.dc1 127.0.0.1 -45:01.102 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74 127.0.0.1 -2022-11-07T22:45:01.102Z [INFO] agent.router: Initializing LAN area manager -45:01.102 [INFO] TestAgent.server: Adding LAN server: server="Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74 (Addr: tcp/127.0.0.1:11447) (DC: dc1)" -45:01.102 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74.dc1 area=wan -45:01.102 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11442 network=udp -45:01.102 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11442 network=tcp -45:01.102 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11443 network=tcp -45:01.102 [INFO] TestAgent: started state syncer -45:01.102 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11448 network=tcp -45:01.148 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.148 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11447 [Candidate]" term=2 -45:01.165 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.165 [DEBUG] TestAgent.server.raft: vote granted: from=7657ba43-ffb9-6c18-eefa-b08d4dacdc74 term=2 tally=1 -45:01.165 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.165 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11447 [Leader]" -45:01.165 [INFO] TestAgent.server: cluster leadership acquired -45:01.165 [INFO] TestAgent.server: New leader elected: payload=Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74 -45:01.175 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11447 -45:01.191 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.214 [INFO] TestAgent: Synced node info -45:01.214 [DEBUG] TestAgent: Node info in sync -45:01.232 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.232 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.232 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.232 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.232 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.232 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74 -45:01.232 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7657ba43-ffb9-6c18-eefa-b08d4dacdc74 -45:01.251 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.778 [INFO] TestAgent: Requesting shutdown -45:01.778 [INFO] TestAgent.server: shutting down server -45:01.778 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:01.778 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:01.778 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:01.778 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:01.778 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:01.778 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:01.778 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:01.778 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:01.778 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:01.778 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:01.781 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:01.785Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:01.785Z [INFO] agent.router.manager: shutting down -45:01.785 [INFO] TestAgent: consul server down -45:01.785 [INFO] TestAgent: shutdown complete -45:01.785 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11442 network=tcp -45:01.785 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11442 network=udp -45:01.785 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11443 network=tcp -45:02.285 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.285 [INFO] TestAgent: Endpoints down +=== CONT TestAgent_MultiStartStop/#01 +2021-10-05T16:30:02.969Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.051Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b47677f6-7e09-cb98-97b6-9301a9291eec Address:127.0.0.1:29783}]" +30:03.212 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29783 [Follower]" leader= +30:03.215 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b47677f6-7e09-cb98-97b6-9301a9291eec.dc1 127.0.0.1 +30:03.228 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b47677f6-7e09-cb98-97b6-9301a9291eec 127.0.0.1 +2021-10-05T16:30:03.228Z [INFO] agent.router: Initializing LAN area manager +30:03.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=udp +30:03.229 [INFO] TestAgent.server: Adding LAN server: server="Node-b47677f6-7e09-cb98-97b6-9301a9291eec (Addr: tcp/127.0.0.1:29783) (DC: dc1)" +30:03.229 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b47677f6-7e09-cb98-97b6-9301a9291eec.dc1 area=wan +30:03.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=tcp +30:03.230 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29779 network=tcp +30:03.230 [INFO] TestAgent: started state syncer +30:03.230 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29784 network=tcp +30:03.264 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.264 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29783 [Candidate]" term=2 +30:03.312 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.312 [DEBUG] TestAgent.server.raft: vote granted: from=b47677f6-7e09-cb98-97b6-9301a9291eec term=2 tally=1 +30:03.312 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.312 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29783 [Leader]" +30:03.312 [INFO] TestAgent.server: cluster leadership acquired +30:03.312 [INFO] TestAgent.server: New leader elected: payload=Node-b47677f6-7e09-cb98-97b6-9301a9291eec +30:03.344 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29783 +30:03.400 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.465 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.465 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.465 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.465 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.465 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.465 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b47677f6-7e09-cb98-97b6-9301a9291eec +30:03.465 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b47677f6-7e09-cb98-97b6-9301a9291eec +30:03.500 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.596 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:03.612 [INFO] TestAgent: Synced node info +30:03.696 [INFO] TestAgent: Requesting shutdown +30:03.696 [INFO] TestAgent.server: shutting down server +30:03.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.696 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.696 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.696 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.696 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.696 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.696 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.704 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.712Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.712Z [INFO] agent.router.manager: shutting down +30:03.712 [INFO] TestAgent: consul server down +30:03.712 [INFO] TestAgent: shutdown complete +30:03.712 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=tcp +30:03.712 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=udp +30:03.712 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29779 network=tcp +30:04.216 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.216 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.145Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.197Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.301 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eed3757a-feed-911f-5d0c-dacefe0c2e3b Address:127.0.0.1:29790}]" +30:03.301 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b.dc1 127.0.0.1 +30:03.302 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b 127.0.0.1 +2021-10-05T16:30:03.302Z [INFO] agent.router: Initializing LAN area manager +30:03.302 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=udp +30:03.302 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29790 [Follower]" leader= +30:03.302 [INFO] TestAgent.server: Adding LAN server: server="Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b (Addr: tcp/127.0.0.1:29790) (DC: dc1)" +30:03.302 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b.dc1 area=wan +30:03.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=tcp +30:03.303 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29786 network=tcp +30:03.303 [INFO] TestAgent: started state syncer +30:03.303 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29791 network=tcp +30:03.348 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.348 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29790 [Candidate]" term=2 +30:03.372 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.372 [DEBUG] TestAgent.server.raft: vote granted: from=eed3757a-feed-911f-5d0c-dacefe0c2e3b term=2 tally=1 +30:03.372 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.372 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29790 [Leader]" +30:03.374 [INFO] TestAgent.server: cluster leadership acquired +30:03.374 [INFO] TestAgent.server: New leader elected: payload=Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b +30:03.392 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29790 +30:03.426 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.477 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.477 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.477 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.477 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.477 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.477 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b +30:03.477 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eed3757a-feed-911f-5d0c-dacefe0c2e3b +30:03.529 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.692 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:03.702 [INFO] TestAgent: Synced node info +30:03.702 [DEBUG] TestAgent: Node info in sync +30:03.712 [INFO] TestAgent: Requesting shutdown +30:03.712 [INFO] TestAgent.server: shutting down server +30:03.712 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.712 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.712 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.712 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.712 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.712 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.712 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.712 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.712 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.712 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.716 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.720Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.720Z [INFO] agent.router.manager: shutting down +30:03.720 [INFO] TestAgent: consul server down +30:03.720 [INFO] TestAgent: shutdown complete +30:03.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=tcp +30:03.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=udp +30:03.720 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29786 network=tcp +30:04.221 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.221 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.056Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.074Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.167 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cc714bbc-01ff-c537-8ac4-5c14f0f5c461 Address:127.0.0.1:29839}]" +30:03.168 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461.dc1 127.0.0.1 +30:03.169 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461 127.0.0.1 +2021-10-05T16:30:03.169Z [INFO] agent.router: Initializing LAN area manager +30:03.169 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=udp +30:03.169 [INFO] TestAgent.server: Adding LAN server: server="Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461 (Addr: tcp/127.0.0.1:29839) (DC: dc1)" +30:03.170 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461.dc1 area=wan +30:03.170 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=tcp +30:03.188 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29839 [Follower]" leader= +30:03.209 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29835 network=tcp +30:03.209 [INFO] TestAgent: started state syncer +30:03.230 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29840 network=tcp +30:03.232 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.232 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29839 [Candidate]" term=2 +30:03.274 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.274 [DEBUG] TestAgent.server.raft: vote granted: from=cc714bbc-01ff-c537-8ac4-5c14f0f5c461 term=2 tally=1 +30:03.274 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.274 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29839 [Leader]" +30:03.275 [INFO] TestAgent.server: cluster leadership acquired +30:03.275 [INFO] TestAgent.server: New leader elected: payload=Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461 +30:03.296 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29839 +30:03.364 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.425 [INFO] TestAgent: Synced node info +30:03.425 [DEBUG] TestAgent: Node info in sync +30:03.443 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.443 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.443 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.443 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.443 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.443 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461 +30:03.443 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cc714bbc-01ff-c537-8ac4-5c14f0f5c461 +30:03.480 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.762 [INFO] TestAgent: Requesting shutdown +30:03.762 [INFO] TestAgent.server: shutting down server +30:03.762 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.762 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.762 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.762 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.762 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.762 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.762 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.762 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.762 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.762 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.775 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.779Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.779Z [INFO] agent.router.manager: shutting down +30:03.779 [INFO] TestAgent: consul server down +30:03.779 [INFO] TestAgent: shutdown complete +30:03.779 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=tcp +30:03.779 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=udp +30:03.779 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29835 network=tcp +30:04.280 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.280 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.148Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.170Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.272 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0adad1d-1c81-1c16-e05e-700f062843fc Address:127.0.0.1:29825}]" +30:03.272 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0adad1d-1c81-1c16-e05e-700f062843fc.dc1 127.0.0.1 +30:03.273 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0adad1d-1c81-1c16-e05e-700f062843fc 127.0.0.1 +2021-10-05T16:30:03.273Z [INFO] agent.router: Initializing LAN area manager +30:03.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=udp +30:03.273 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29825 [Follower]" leader= +30:03.274 [INFO] TestAgent.server: Adding LAN server: server="Node-b0adad1d-1c81-1c16-e05e-700f062843fc (Addr: tcp/127.0.0.1:29825) (DC: dc1)" +30:03.274 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0adad1d-1c81-1c16-e05e-700f062843fc.dc1 area=wan +30:03.274 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=tcp +30:03.274 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29821 network=tcp +30:03.274 [INFO] TestAgent: started state syncer +30:03.274 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29826 network=tcp +30:03.318 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.318 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29825 [Candidate]" term=2 +30:03.338 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.338 [DEBUG] TestAgent.server.raft: vote granted: from=b0adad1d-1c81-1c16-e05e-700f062843fc term=2 tally=1 +30:03.338 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.338 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29825 [Leader]" +30:03.338 [INFO] TestAgent.server: cluster leadership acquired +30:03.338 [INFO] TestAgent.server: New leader elected: payload=Node-b0adad1d-1c81-1c16-e05e-700f062843fc +30:03.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29825 +30:03.411 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0adad1d-1c81-1c16-e05e-700f062843fc +30:03.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0adad1d-1c81-1c16-e05e-700f062843fc +30:03.538 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.580 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:03.596 [INFO] TestAgent: Synced node info +30:03.791 [INFO] TestAgent: Requesting shutdown +30:03.791 [INFO] TestAgent.server: shutting down server +30:03.791 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.791 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.791 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.791 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.791 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.791 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.791 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.791 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.791 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.791 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.800 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.808Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.808Z [INFO] agent.router.manager: shutting down +30:03.808 [INFO] TestAgent: consul server down +30:03.808 [INFO] TestAgent: shutdown complete +30:03.808 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=tcp +30:03.808 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=udp +30:03.808 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29821 network=tcp +30:04.310 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.310 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.082Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.155Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.254 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:03f2970e-fcee-57e7-2478-4918d5373d68 Address:127.0.0.1:29818}]" +30:03.255 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-03f2970e-fcee-57e7-2478-4918d5373d68.dc1 127.0.0.1 +30:03.255 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-03f2970e-fcee-57e7-2478-4918d5373d68 127.0.0.1 +2021-10-05T16:30:03.255Z [INFO] agent.router: Initializing LAN area manager +30:03.255 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=udp +30:03.255 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29818 [Follower]" leader= +30:03.256 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-03f2970e-fcee-57e7-2478-4918d5373d68.dc1 area=wan +30:03.256 [INFO] TestAgent.server: Adding LAN server: server="Node-03f2970e-fcee-57e7-2478-4918d5373d68 (Addr: tcp/127.0.0.1:29818) (DC: dc1)" +30:03.256 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=tcp +30:03.257 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29814 network=tcp +30:03.257 [INFO] TestAgent: started state syncer +30:03.257 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29819 network=tcp +30:03.304 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.304 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29818 [Candidate]" term=2 +30:03.328 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.328 [DEBUG] TestAgent.server.raft: vote granted: from=03f2970e-fcee-57e7-2478-4918d5373d68 term=2 tally=1 +30:03.328 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.328 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29818 [Leader]" +30:03.329 [INFO] TestAgent.server: cluster leadership acquired +30:03.329 [INFO] TestAgent.server: New leader elected: payload=Node-03f2970e-fcee-57e7-2478-4918d5373d68 +30:03.341 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29818 +30:03.412 [INFO] TestAgent: Synced node info +30:03.412 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.472 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.472 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.472 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.472 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.472 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.472 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-03f2970e-fcee-57e7-2478-4918d5373d68 +30:03.472 [INFO] TestAgent.server: member joined, marking health alive: member=Node-03f2970e-fcee-57e7-2478-4918d5373d68 +30:03.528 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.832 [INFO] TestAgent: Requesting shutdown +30:03.832 [INFO] TestAgent.server: shutting down server +30:03.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.832 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.832 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.832 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.832 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.833 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.833 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.840 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.852Z [INFO] agent.router.manager: shutting down +30:03.852 [INFO] TestAgent: consul server down +30:03.852 [INFO] TestAgent: shutdown complete +30:03.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=tcp +30:03.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=udp +30:03.852 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29814 network=tcp +30:04.352 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.352 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.145Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.168Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.248 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:45e97d39-30cd-bfc3-081f-dba73536d7c8 Address:127.0.0.1:29832}]" +30:03.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29832 [Follower]" leader= +30:03.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-45e97d39-30cd-bfc3-081f-dba73536d7c8.dc1 127.0.0.1 +30:03.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-45e97d39-30cd-bfc3-081f-dba73536d7c8 127.0.0.1 +2021-10-05T16:30:03.249Z [INFO] agent.router: Initializing LAN area manager +30:03.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=udp +30:03.250 [INFO] TestAgent.server: Adding LAN server: server="Node-45e97d39-30cd-bfc3-081f-dba73536d7c8 (Addr: tcp/127.0.0.1:29832) (DC: dc1)" +30:03.250 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-45e97d39-30cd-bfc3-081f-dba73536d7c8.dc1 area=wan +30:03.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=tcp +30:03.251 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29828 network=tcp +30:03.251 [INFO] TestAgent: started state syncer +30:03.251 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29833 network=tcp +30:03.308 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.308 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29832 [Candidate]" term=2 +30:03.352 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.352 [DEBUG] TestAgent.server.raft: vote granted: from=45e97d39-30cd-bfc3-081f-dba73536d7c8 term=2 tally=1 +30:03.352 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.352 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29832 [Leader]" +30:03.353 [INFO] TestAgent.server: cluster leadership acquired +30:03.353 [INFO] TestAgent.server: New leader elected: payload=Node-45e97d39-30cd-bfc3-081f-dba73536d7c8 +30:03.374 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29832 +30:03.396 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.483 [INFO] TestAgent: Synced node info +30:03.483 [DEBUG] TestAgent: Node info in sync +30:03.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45e97d39-30cd-bfc3-081f-dba73536d7c8 +30:03.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-45e97d39-30cd-bfc3-081f-dba73536d7c8 +30:03.512 [DEBUG] TestAgent: Node info in sync +30:03.541 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.864 [INFO] TestAgent: Requesting shutdown +30:03.864 [INFO] TestAgent.server: shutting down server +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.864 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.864 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.864 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.864 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.872 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.880Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.880Z [INFO] agent.router.manager: shutting down +30:03.880 [INFO] TestAgent: consul server down +30:03.880 [INFO] TestAgent: shutdown complete +30:03.880 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=tcp +30:03.880 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=udp +30:03.880 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29828 network=tcp +30:04.384 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.384 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.141Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.214Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.296 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38e9aab9-5496-78ed-e685-cd8fefd761ca Address:127.0.0.1:29797}]" +30:03.296 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29797 [Follower]" leader= +30:03.297 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-38e9aab9-5496-78ed-e685-cd8fefd761ca.dc1 127.0.0.1 +30:03.297 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-38e9aab9-5496-78ed-e685-cd8fefd761ca 127.0.0.1 +2021-10-05T16:30:03.297Z [INFO] agent.router: Initializing LAN area manager +30:03.297 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=udp +30:03.298 [INFO] TestAgent.server: Adding LAN server: server="Node-38e9aab9-5496-78ed-e685-cd8fefd761ca (Addr: tcp/127.0.0.1:29797) (DC: dc1)" +30:03.298 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-38e9aab9-5496-78ed-e685-cd8fefd761ca.dc1 area=wan +30:03.298 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=tcp +30:03.298 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29793 network=tcp +30:03.298 [INFO] TestAgent: started state syncer +30:03.299 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29798 network=tcp +30:03.340 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.340 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29797 [Candidate]" term=2 +30:03.387 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.387 [DEBUG] TestAgent.server.raft: vote granted: from=38e9aab9-5496-78ed-e685-cd8fefd761ca term=2 tally=1 +30:03.387 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.387 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29797 [Leader]" +30:03.387 [INFO] TestAgent.server: cluster leadership acquired +30:03.387 [INFO] TestAgent.server: New leader elected: payload=Node-38e9aab9-5496-78ed-e685-cd8fefd761ca +30:03.428 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29797 +30:03.488 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.556 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.556 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.556 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.556 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.556 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.557 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38e9aab9-5496-78ed-e685-cd8fefd761ca +30:03.557 [INFO] TestAgent.server: member joined, marking health alive: member=Node-38e9aab9-5496-78ed-e685-cd8fefd761ca +30:03.592 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.784 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:03.787 [INFO] TestAgent: Synced node info +30:03.864 [INFO] TestAgent: Requesting shutdown +30:03.864 [INFO] TestAgent.server: shutting down server +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.864 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.864 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.868 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.868 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.868 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.868 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.868 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.876 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.899Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.899Z [INFO] agent.router.manager: shutting down +30:03.899 [INFO] TestAgent: consul server down +30:03.899 [INFO] TestAgent: shutdown complete +30:03.899 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=tcp +30:03.899 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=udp +30:03.899 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29793 network=tcp +30:04.400 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.400 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.078Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.164Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.264 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ee2ea2a-311f-f6df-de78-3b97d7785821 Address:127.0.0.1:29804}]" +30:03.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29804 [Follower]" leader= +30:03.265 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0ee2ea2a-311f-f6df-de78-3b97d7785821.dc1 127.0.0.1 +30:03.266 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0ee2ea2a-311f-f6df-de78-3b97d7785821 127.0.0.1 +2021-10-05T16:30:03.266Z [INFO] agent.router: Initializing LAN area manager +30:03.266 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=udp +30:03.266 [INFO] TestAgent.server: Adding LAN server: server="Node-0ee2ea2a-311f-f6df-de78-3b97d7785821 (Addr: tcp/127.0.0.1:29804) (DC: dc1)" +30:03.266 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0ee2ea2a-311f-f6df-de78-3b97d7785821.dc1 area=wan +30:03.266 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=tcp +30:03.267 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29800 network=tcp +30:03.267 [INFO] TestAgent: started state syncer +30:03.267 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29805 network=tcp +30:03.308 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.308 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29804 [Candidate]" term=2 +30:03.329 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.329 [DEBUG] TestAgent.server.raft: vote granted: from=0ee2ea2a-311f-f6df-de78-3b97d7785821 term=2 tally=1 +30:03.329 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.329 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29804 [Leader]" +30:03.330 [INFO] TestAgent.server: cluster leadership acquired +30:03.330 [INFO] TestAgent.server: New leader elected: payload=Node-0ee2ea2a-311f-f6df-de78-3b97d7785821 +30:03.343 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29804 +30:03.413 [INFO] TestAgent: Synced node info +30:03.413 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.469 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.469 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.469 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.469 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.469 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.469 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0ee2ea2a-311f-f6df-de78-3b97d7785821 +30:03.469 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0ee2ea2a-311f-f6df-de78-3b97d7785821 +30:03.512 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.872 [INFO] TestAgent: Requesting shutdown +30:03.872 [INFO] TestAgent.server: shutting down server +30:03.872 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.872 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.872 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.872 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.872 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.872 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.895 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.900Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.900Z [INFO] agent.router.manager: shutting down +30:03.900 [INFO] TestAgent: consul server down +30:03.900 [INFO] TestAgent: shutdown complete +30:03.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=tcp +30:03.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=udp +30:03.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29800 network=tcp +30:04.400 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.400 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:03.064Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.138Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.250 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e47c2655-2941-db84-0831-b82085c7bd94 Address:127.0.0.1:29811}]" +30:03.251 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29811 [Follower]" leader= +30:03.252 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e47c2655-2941-db84-0831-b82085c7bd94.dc1 127.0.0.1 +30:03.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e47c2655-2941-db84-0831-b82085c7bd94 127.0.0.1 +2021-10-05T16:30:03.257Z [INFO] agent.router: Initializing LAN area manager +30:03.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=udp +30:03.258 [INFO] TestAgent.server: Adding LAN server: server="Node-e47c2655-2941-db84-0831-b82085c7bd94 (Addr: tcp/127.0.0.1:29811) (DC: dc1)" +30:03.258 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e47c2655-2941-db84-0831-b82085c7bd94.dc1 area=wan +30:03.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=tcp +30:03.259 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29807 network=tcp +30:03.259 [INFO] TestAgent: started state syncer +30:03.259 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29812 network=tcp +30:03.292 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.292 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29811 [Candidate]" term=2 +30:03.324 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.324 [DEBUG] TestAgent.server.raft: vote granted: from=e47c2655-2941-db84-0831-b82085c7bd94 term=2 tally=1 +30:03.324 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.324 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29811 [Leader]" +30:03.328 [INFO] TestAgent.server: cluster leadership acquired +30:03.328 [INFO] TestAgent.server: New leader elected: payload=Node-e47c2655-2941-db84-0831-b82085c7bd94 +30:03.352 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29811 +30:03.384 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.403 [INFO] TestAgent: Synced node info +30:03.403 [DEBUG] TestAgent: Node info in sync +30:03.444 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.444 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.444 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.444 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.444 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.444 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e47c2655-2941-db84-0831-b82085c7bd94 +30:03.444 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e47c2655-2941-db84-0831-b82085c7bd94 +30:03.489 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.912 [INFO] TestAgent: Requesting shutdown +30:03.912 [INFO] TestAgent.server: shutting down server +30:03.912 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.912 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.912 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.912 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.912 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.912 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.912 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.912 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.920 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.932Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.932Z [INFO] agent.router.manager: shutting down +30:03.932 [INFO] TestAgent: consul server down +30:03.932 [INFO] TestAgent: shutdown complete +30:03.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=tcp +30:03.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=udp +30:03.932 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29807 network=tcp +30:04.436 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.436 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:02.976Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:03.169Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:03.276 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1039ab93-652f-40db-0e8a-e787eae530ab Address:127.0.0.1:29776}]" +30:03.276 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1039ab93-652f-40db-0e8a-e787eae530ab.dc1 127.0.0.1 +30:03.277 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1039ab93-652f-40db-0e8a-e787eae530ab 127.0.0.1 +2021-10-05T16:30:03.277Z [INFO] agent.router: Initializing LAN area manager +30:03.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=udp +30:03.277 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29776 [Follower]" leader= +30:03.278 [INFO] TestAgent.server: Adding LAN server: server="Node-1039ab93-652f-40db-0e8a-e787eae530ab (Addr: tcp/127.0.0.1:29776) (DC: dc1)" +30:03.278 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1039ab93-652f-40db-0e8a-e787eae530ab.dc1 area=wan +30:03.278 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=tcp +30:03.278 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29772 network=tcp +30:03.278 [INFO] TestAgent: started state syncer +30:03.278 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29777 network=tcp +30:03.328 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:03.328 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29776 [Candidate]" term=2 +30:03.380 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:03.380 [DEBUG] TestAgent.server.raft: vote granted: from=1039ab93-652f-40db-0e8a-e787eae530ab term=2 tally=1 +30:03.380 [INFO] TestAgent.server.raft: election won: tally=1 +30:03.380 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29776 [Leader]" +30:03.380 [INFO] TestAgent.server: cluster leadership acquired +30:03.380 [INFO] TestAgent.server: New leader elected: payload=Node-1039ab93-652f-40db-0e8a-e787eae530ab +30:03.412 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29776 +30:03.492 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:03.556 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.556 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.556 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.556 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.556 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.556 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1039ab93-652f-40db-0e8a-e787eae530ab +30:03.556 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1039ab93-652f-40db-0e8a-e787eae530ab +30:03.600 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.696 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:03.716 [INFO] TestAgent: Synced node info +30:03.948 [INFO] TestAgent: Requesting shutdown +30:03.948 [INFO] TestAgent.server: shutting down server +30:03.948 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.948 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.948 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.948 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.948 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.948 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.948 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.948 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.948 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.948 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.956 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.960Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.960Z [INFO] agent.router.manager: shutting down +30:03.960 [INFO] TestAgent: consul server down +30:03.960 [INFO] TestAgent: shutdown complete +30:03.960 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=tcp +30:03.960 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=udp +30:03.960 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29772 network=tcp +30:04.464 [INFO] TestAgent: Waiting for endpoints to shut down +30:04.464 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_MultiStartStop (0.00s) - --- PASS: TestAgent_MultiStartStop/#08 (1.02s) - --- PASS: TestAgent_MultiStartStop/#00 (1.05s) - --- PASS: TestAgent_MultiStartStop/#05 (1.05s) - --- PASS: TestAgent_MultiStartStop/#01 (1.09s) - --- PASS: TestAgent_MultiStartStop/#06 (1.10s) - --- PASS: TestAgent_MultiStartStop/#09 (1.10s) - --- PASS: TestAgent_MultiStartStop/#03 (1.12s) - --- PASS: TestAgent_MultiStartStop/#04 (1.21s) - --- PASS: TestAgent_MultiStartStop/#02 (1.25s) - --- PASS: TestAgent_MultiStartStop/#07 (1.29s) + --- PASS: TestAgent_MultiStartStop/#05 (1.26s) + --- PASS: TestAgent_MultiStartStop/#08 (1.27s) + --- PASS: TestAgent_MultiStartStop/#03 (1.31s) + --- PASS: TestAgent_MultiStartStop/#01 (1.34s) + --- PASS: TestAgent_MultiStartStop/#07 (1.39s) + --- PASS: TestAgent_MultiStartStop/#06 (1.42s) + --- PASS: TestAgent_MultiStartStop/#04 (1.44s) + --- PASS: TestAgent_MultiStartStop/#09 (1.44s) + --- PASS: TestAgent_MultiStartStop/#02 (1.47s) + --- PASS: TestAgent_MultiStartStop/#00 (1.51s) === RUN TestAgent_ConnectClusterIDConfig === RUN TestAgent_ConnectClusterIDConfig/default_TestAgent_has_fixed_cluster_id -2022-11-07T22:45:02.295Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:02.304Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:02.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a429b416-ed1a-aee9-530f-fb9505191ef9 Address:127.0.0.1:11468}]" -45:02.324 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11468 [Follower]" leader= -45:02.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a429b416-ed1a-aee9-530f-fb9505191ef9.dc1 127.0.0.1 -45:02.325 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a429b416-ed1a-aee9-530f-fb9505191ef9 127.0.0.1 -2022-11-07T22:45:02.325Z [INFO] agent.router: Initializing LAN area manager -45:02.325 [INFO] TestAgent.server: Adding LAN server: server="Node-a429b416-ed1a-aee9-530f-fb9505191ef9 (Addr: tcp/127.0.0.1:11468) (DC: dc1)" -45:02.325 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a429b416-ed1a-aee9-530f-fb9505191ef9.dc1 area=wan -45:02.325 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11463 network=udp -45:02.325 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11463 network=tcp -45:02.326 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11464 network=tcp -45:02.326 [INFO] TestAgent: started state syncer -45:02.326 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11469 network=tcp -45:02.391 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:02.391 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11468 [Candidate]" term=2 -45:02.399 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:02.399 [DEBUG] TestAgent.server.raft: vote granted: from=a429b416-ed1a-aee9-530f-fb9505191ef9 term=2 tally=1 -45:02.399 [INFO] TestAgent.server.raft: election won: tally=1 -45:02.399 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11468 [Leader]" -45:02.399 [INFO] TestAgent.server: cluster leadership acquired -45:02.399 [INFO] TestAgent.server: New leader elected: payload=Node-a429b416-ed1a-aee9-530f-fb9505191ef9 -45:02.405 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11468 -45:02.417 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:02.435 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:02.435 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:02.435 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:02.435 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:02.435 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:02.435 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a429b416-ed1a-aee9-530f-fb9505191ef9 -45:02.435 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a429b416-ed1a-aee9-530f-fb9505191ef9 -45:02.445 [INFO] TestAgent.server: federation state anti-entropy synced -45:02.458 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:02.461 [INFO] TestAgent: Synced node info -45:02.461 [DEBUG] TestAgent: Node info in sync -45:02.608 [INFO] TestAgent: Requesting shutdown -45:02.608 [INFO] TestAgent.server: shutting down server -45:02.608 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:02.608 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:02.608 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:02.608 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:02.608 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:02.608 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:02.608 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:02.608 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:02.608 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:02.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:02.612 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:02.615Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:02.615Z [INFO] agent.router.manager: shutting down -45:02.616 [INFO] TestAgent: consul server down -45:02.616 [INFO] TestAgent: shutdown complete -45:02.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11463 network=tcp -45:02.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11463 network=udp -45:02.616 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11464 network=tcp -45:03.116 [INFO] TestAgent: Waiting for endpoints to shut down -45:03.116 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:04.495Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:04.529Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:04.604 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb16300c-178b-7f74-c4f4-5c261a6c800d Address:127.0.0.1:29846}]" +30:04.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cb16300c-178b-7f74-c4f4-5c261a6c800d.dc1 127.0.0.1 +30:04.605 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cb16300c-178b-7f74-c4f4-5c261a6c800d 127.0.0.1 +2021-10-05T16:30:04.605Z [INFO] agent.router: Initializing LAN area manager +30:04.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=udp +30:04.605 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29846 [Follower]" leader= +30:04.606 [INFO] TestAgent.server: Adding LAN server: server="Node-cb16300c-178b-7f74-c4f4-5c261a6c800d (Addr: tcp/127.0.0.1:29846) (DC: dc1)" +30:04.606 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cb16300c-178b-7f74-c4f4-5c261a6c800d.dc1 area=wan +30:04.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=tcp +30:04.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29842 network=tcp +30:04.606 [INFO] TestAgent: started state syncer +30:04.606 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29847 network=tcp +30:04.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:04.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29846 [Candidate]" term=2 +30:04.701 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:04.701 [DEBUG] TestAgent.server.raft: vote granted: from=cb16300c-178b-7f74-c4f4-5c261a6c800d term=2 tally=1 +30:04.701 [INFO] TestAgent.server.raft: election won: tally=1 +30:04.701 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29846 [Leader]" +30:04.702 [INFO] TestAgent.server: cluster leadership acquired +30:04.702 [INFO] TestAgent.server: New leader elected: payload=Node-cb16300c-178b-7f74-c4f4-5c261a6c800d +30:04.707 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29846 +30:04.720 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:04.793 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:04.793 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:04.793 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:04.793 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:04.793 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:04.793 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cb16300c-178b-7f74-c4f4-5c261a6c800d +30:04.793 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cb16300c-178b-7f74-c4f4-5c261a6c800d +30:04.822 [INFO] TestAgent.server: federation state anti-entropy synced +30:04.964 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:04.980 [INFO] TestAgent: Synced node info +30:04.980 [DEBUG] TestAgent: Node info in sync +30:05.084 [INFO] TestAgent: Requesting shutdown +30:05.084 [INFO] TestAgent.server: shutting down server +30:05.084 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:05.084 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:05.084 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:05.084 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:05.084 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:05.084 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:05.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:05.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:05.084 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:05.084 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:05.092 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:05.100Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:05.100Z [INFO] agent.router.manager: shutting down +30:05.100 [INFO] TestAgent: consul server down +30:05.100 [INFO] TestAgent: shutdown complete +30:05.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=tcp +30:05.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=udp +30:05.100 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29842 network=tcp +30:05.604 [INFO] TestAgent: Waiting for endpoints to shut down +30:05.604 [INFO] TestAgent: Endpoints down === RUN TestAgent_ConnectClusterIDConfig/no_cluster_ID_specified_sets_to_test_ID -2022-11-07T22:45:03.127Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:03.136Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:03.159 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7932d19d-e766-fc2a-d59c-5f1f41703f9e Address:127.0.0.1:11475}]" -45:03.159 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11475 [Follower]" leader= -45:03.159 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e.dc1 127.0.0.1 -45:03.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e 127.0.0.1 -2022-11-07T22:45:03.160Z [INFO] agent.router: Initializing LAN area manager -45:03.160 [INFO] TestAgent.server: Adding LAN server: server="Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e (Addr: tcp/127.0.0.1:11475) (DC: dc1)" -45:03.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11470 network=udp -45:03.160 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e.dc1 area=wan -45:03.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11470 network=tcp -45:03.160 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11471 network=tcp -45:03.160 [INFO] TestAgent: started state syncer -45:03.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11476 network=tcp -45:03.207 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:03.207 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11475 [Candidate]" term=2 -45:03.214 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:03.214 [DEBUG] TestAgent.server.raft: vote granted: from=7932d19d-e766-fc2a-d59c-5f1f41703f9e term=2 tally=1 -45:03.214 [INFO] TestAgent.server.raft: election won: tally=1 -45:03.214 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11475 [Leader]" -45:03.214 [INFO] TestAgent.server: cluster leadership acquired -45:03.214 [INFO] TestAgent.server: New leader elected: payload=Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e -45:03.219 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11475 -45:03.230 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:03.247 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:03.247 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:03.247 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:03.247 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:03.247 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:03.247 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e -45:03.247 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7932d19d-e766-fc2a-d59c-5f1f41703f9e -45:03.259 [INFO] TestAgent.server: federation state anti-entropy synced -45:03.352 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:03.355 [INFO] TestAgent: Synced node info -45:03.355 [DEBUG] TestAgent: Node info in sync -45:03.488 [INFO] TestAgent: Requesting shutdown -45:03.488 [INFO] TestAgent.server: shutting down server -45:03.488 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:03.488 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:03.488 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:03.488 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:03.488 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:03.488 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:03.488 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:03.488 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:03.488 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:03.488 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:03.491 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:03.494Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:03.494Z [INFO] agent.router.manager: shutting down -45:03.495 [INFO] TestAgent: consul server down -45:03.495 [INFO] TestAgent: shutdown complete -45:03.495 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11470 network=tcp -45:03.495 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11470 network=udp -45:03.495 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11471 network=tcp -45:03.995 [INFO] TestAgent: Waiting for endpoints to shut down -45:03.995 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:05.641Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:05.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:05.736 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d043d636-0301-cecf-3807-2e67a201f21c Address:127.0.0.1:29853}]" +30:05.737 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d043d636-0301-cecf-3807-2e67a201f21c.dc1 127.0.0.1 +30:05.737 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d043d636-0301-cecf-3807-2e67a201f21c 127.0.0.1 +2021-10-05T16:30:05.737Z [INFO] agent.router: Initializing LAN area manager +30:05.737 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=udp +30:05.737 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29853 [Follower]" leader= +30:05.738 [INFO] TestAgent.server: Adding LAN server: server="Node-d043d636-0301-cecf-3807-2e67a201f21c (Addr: tcp/127.0.0.1:29853) (DC: dc1)" +30:05.738 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d043d636-0301-cecf-3807-2e67a201f21c.dc1 area=wan +30:05.738 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=tcp +30:05.738 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29849 network=tcp +30:05.738 [INFO] TestAgent: started state syncer +30:05.739 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29854 network=tcp +30:05.788 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:05.788 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29853 [Candidate]" term=2 +30:05.836 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:05.836 [DEBUG] TestAgent.server.raft: vote granted: from=d043d636-0301-cecf-3807-2e67a201f21c term=2 tally=1 +30:05.836 [INFO] TestAgent.server.raft: election won: tally=1 +30:05.836 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29853 [Leader]" +30:05.840 [INFO] TestAgent.server: cluster leadership acquired +30:05.840 [INFO] TestAgent.server: New leader elected: payload=Node-d043d636-0301-cecf-3807-2e67a201f21c +30:05.868 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29853 +30:05.928 [INFO] TestAgent: Requesting shutdown +30:05.928 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:05.928 [INFO] TestAgent.server: shutting down server +30:05.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:05.928 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:05.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:05.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:05.948Z [INFO] agent.router.manager: shutting down +30:05.956 [INFO] TestAgent: consul server down +30:05.956 [INFO] TestAgent: shutdown complete +30:05.956 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=tcp +30:05.956 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=udp +30:05.956 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29849 network=tcp +30:05.956 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +30:05.956 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +30:06.459 [INFO] TestAgent: Waiting for endpoints to shut down +30:06.459 [INFO] TestAgent: Endpoints down === RUN TestAgent_ConnectClusterIDConfig/non-UUID_cluster_id_is_fatal -2022-11-07T22:45:04.007Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.016Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:04.016 [INFO] TestAgent: Requesting shutdown -45:04.016 [INFO] TestAgent: shutdown complete ---- PASS: TestAgent_ConnectClusterIDConfig (1.73s) - --- PASS: TestAgent_ConnectClusterIDConfig/default_TestAgent_has_fixed_cluster_id (0.83s) - --- PASS: TestAgent_ConnectClusterIDConfig/no_cluster_ID_specified_sets_to_test_ID (0.88s) - --- PASS: TestAgent_ConnectClusterIDConfig/non-UUID_cluster_id_is_fatal (0.02s) +2021-10-05T16:30:06.487Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:06.521Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:06.521 [INFO] TestAgent: Requesting shutdown +30:06.521 [INFO] TestAgent: shutdown complete +--- PASS: TestAgent_ConnectClusterIDConfig (2.06s) + --- PASS: TestAgent_ConnectClusterIDConfig/default_TestAgent_has_fixed_cluster_id (1.14s) + --- PASS: TestAgent_ConnectClusterIDConfig/no_cluster_ID_specified_sets_to_test_ID (0.86s) + --- PASS: TestAgent_ConnectClusterIDConfig/non-UUID_cluster_id_is_fatal (0.06s) === RUN TestAgent_StartStop === PAUSE TestAgent_StartStop === RUN TestAgent_RPCPing @@ -10448,10 +10488,10 @@ === RUN TestAgent_AddService/service_manager/multiple_checks/check1 === RUN TestAgent_AddService/service_manager/multiple_checks/check-noname === RUN TestAgent_AddService/service_manager/multiple_checks/service:svcid2:3 -=== RUN TestAgent_AddService/service_manager/multiple_checks/service:svcid2:4_ttl === RUN TestAgent_AddService/service_manager/multiple_checks/check1_ttl === RUN TestAgent_AddService/service_manager/multiple_checks/check-noname_ttl === RUN TestAgent_AddService/service_manager/multiple_checks/service:svcid2:3_ttl +=== RUN TestAgent_AddService/service_manager/multiple_checks/service:svcid2:4_ttl === RUN TestAgent_AddService/normal/one_check === RUN TestAgent_AddService/normal/one_check/svcid1 === RUN TestAgent_AddService/normal/one_check/check1 @@ -10462,124 +10502,106 @@ === RUN TestAgent_AddService/normal/multiple_checks/check-noname === RUN TestAgent_AddService/normal/multiple_checks/service:svcid2:3 === RUN TestAgent_AddService/normal/multiple_checks/service:svcid2:4 -=== RUN TestAgent_AddService/normal/multiple_checks/check1_ttl -=== RUN TestAgent_AddService/normal/multiple_checks/check-noname_ttl === RUN TestAgent_AddService/normal/multiple_checks/service:svcid2:3_ttl === RUN TestAgent_AddService/normal/multiple_checks/service:svcid2:4_ttl -2022-11-07T22:45:04.032Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.054Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:04.079 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c5f79259-7936-f0fd-2b02-3af361535ad6 Address:127.0.0.1:11496}]" -45:04.079 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11496 [Follower]" leader= -45:04.080 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:04.080 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:04.080Z [INFO] agent.router: Initializing LAN area manager -45:04.081 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11496) (DC: dc1)" -45:04.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11491 network=udp -45:04.081 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:04.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11491 network=tcp -45:04.081 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11492 network=tcp -45:04.081 [INFO] TestAgent: started state syncer -45:04.081 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11497 network=tcp -45:04.128 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:04.128 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11496 [Candidate]" term=2 -45:04.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:04.137 [DEBUG] TestAgent.server.raft: vote granted: from=c5f79259-7936-f0fd-2b02-3af361535ad6 term=2 tally=1 -45:04.137 [INFO] TestAgent.server.raft: election won: tally=1 -45:04.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11496 [Leader]" -45:04.137 [INFO] TestAgent.server: cluster leadership acquired -45:04.137 [INFO] TestAgent.server: New leader elected: payload=node1 -45:04.144 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11496 -45:04.155 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:04.178 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:04.178 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:04.178 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:04.178 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:04.178 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:04.178 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:04.178 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:04.188 [INFO] TestAgent.server: federation state anti-entropy synced -45:04.333 [INFO] TestAgent: Requesting shutdown -45:04.333 [INFO] TestAgent.server: shutting down server -45:04.333 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:04.333 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:04.333 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:04.333 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:04.333 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:04.333 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:04.333 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:04.336 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:04.340Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:04.340Z [INFO] agent.router.manager: shutting down -45:04.340 [INFO] TestAgent: consul server down -45:04.340 [INFO] TestAgent: shutdown complete -45:04.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11491 network=tcp -45:04.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11491 network=udp -45:04.340 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11492 network=tcp -45:04.841 [INFO] TestAgent: Waiting for endpoints to shut down -45:04.841 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:04.032Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.052Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:04.077 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6b217fcd-092e-d99f-b45e-4d797508669f Address:127.0.0.1:11489}]" -45:04.077 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11489 [Follower]" leader= -45:04.078 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:04.078 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:04.078Z [INFO] agent.router: Initializing LAN area manager -45:04.078 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:04.078 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11489) (DC: dc1)" -45:04.078 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11484 network=udp -45:04.078 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11484 network=tcp -45:04.079 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11485 network=tcp -45:04.079 [INFO] TestAgent: started state syncer -45:04.079 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11490 network=tcp -45:04.115 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:04.115 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11489 [Candidate]" term=2 -45:04.123 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:04.123 [DEBUG] TestAgent.server.raft: vote granted: from=6b217fcd-092e-d99f-b45e-4d797508669f term=2 tally=1 -45:04.123 [INFO] TestAgent.server.raft: election won: tally=1 -45:04.123 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11489 [Leader]" -45:04.123 [INFO] TestAgent.server: cluster leadership acquired -45:04.123 [INFO] TestAgent.server: New leader elected: payload=node1 -45:04.128 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11489 -45:04.140 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:04.159 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:04.159 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:04.159 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:04.159 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:04.159 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:04.159 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:04.159 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:04.166 [INFO] TestAgent.server: federation state anti-entropy synced -45:04.307 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:04.311 [INFO] TestAgent: Synced node info -45:04.311 [DEBUG] TestAgent: Node info in sync -45:04.464 [INFO] TestAgent: Requesting shutdown -45:04.464 [INFO] TestAgent.server: shutting down server -45:04.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:04.464 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:04.464 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:04.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:04.464 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:04.464 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:04.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:04.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:04.464 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:04.464 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:04.468 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:04.471Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:04.471Z [INFO] agent.router.manager: shutting down -45:04.471 [INFO] TestAgent: consul server down -45:04.471 [INFO] TestAgent: shutdown complete -45:04.471 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11484 network=tcp -45:04.471 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11484 network=udp -45:04.471 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11485 network=tcp -45:04.971 [INFO] TestAgent: Waiting for endpoints to shut down -45:04.971 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_AddService/normal/multiple_checks/check1_ttl +=== RUN TestAgent_AddService/normal/multiple_checks/check-noname_ttl +2021-10-05T16:30:06.552Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:06.581Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:06.692 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7029a7d0-54f5-ae23-6eb2-984d90679eaa Address:127.0.0.1:29874}]" +30:06.693 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:06.693 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:06.693Z [INFO] agent.router: Initializing LAN area manager +30:06.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=udp +30:06.693 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29874 [Follower]" leader= +30:06.694 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29874) (DC: dc1)" +30:06.694 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:06.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=tcp +30:06.694 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29870 network=tcp +30:06.694 [INFO] TestAgent: started state syncer +30:06.695 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29875 network=tcp +30:06.760 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:06.760 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29874 [Candidate]" term=2 +30:06.812 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:06.812 [DEBUG] TestAgent.server.raft: vote granted: from=7029a7d0-54f5-ae23-6eb2-984d90679eaa term=2 tally=1 +30:06.812 [INFO] TestAgent.server.raft: election won: tally=1 +30:06.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29874 [Leader]" +30:06.816 [INFO] TestAgent.server: cluster leadership acquired +30:06.816 [INFO] TestAgent.server: New leader elected: payload=node1 +30:06.829 [INFO] TestAgent: Requesting shutdown +30:06.829 [INFO] TestAgent.server: shutting down server +30:06.829 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:06.829 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:06.842 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29874 +30:06.844 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:06.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:06.852Z [INFO] agent.router.manager: shutting down +30:06.860 [ERROR] TestAgent.server.autopilot: failed to initialize config: error="leadership lost while committing log" +30:06.860 [INFO] TestAgent: consul server down +30:06.860 [ERROR] TestAgent.server: failed to establish leadership: error="raft is already shutdown" +30:06.860 [INFO] TestAgent: shutdown complete +30:06.860 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=tcp +30:06.860 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=udp +30:06.860 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29870 network=tcp +30:07.360 [INFO] TestAgent: Waiting for endpoints to shut down +30:07.360 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:06.580Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:06.619Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:06.716 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:abb7e9e0-5fd7-88c1-61c4-c03bb7db4978 Address:127.0.0.1:29867}]" +30:06.717 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:06.717 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:06.717Z [INFO] agent.router: Initializing LAN area manager +30:06.717 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=udp +30:06.718 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29867 [Follower]" leader= +30:06.718 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29867) (DC: dc1)" +30:06.718 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:06.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=tcp +30:06.719 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29863 network=tcp +30:06.719 [INFO] TestAgent: started state syncer +30:06.719 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29868 network=tcp +30:06.788 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:06.788 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29867 [Candidate]" term=2 +30:06.838 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:06.838 [DEBUG] TestAgent.server.raft: vote granted: from=abb7e9e0-5fd7-88c1-61c4-c03bb7db4978 term=2 tally=1 +30:06.838 [INFO] TestAgent.server.raft: election won: tally=1 +30:06.838 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29867 [Leader]" +30:06.838 [INFO] TestAgent.server: cluster leadership acquired +30:06.839 [INFO] TestAgent.server: New leader elected: payload=node1 +30:06.864 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29867 +30:06.883 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:06.944 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:06.944 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:06.944 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:06.944 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:06.944 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:06.944 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:06.944 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:06.980 [INFO] TestAgent.server: federation state anti-entropy synced +30:06.997 [INFO] TestAgent: Requesting shutdown +30:06.997 [INFO] TestAgent.server: shutting down server +30:06.997 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:06.997 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:06.997 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:06.997 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:06.997 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:06.997 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:06.997 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:06.997 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:06.997 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:06.997 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:06.997 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:07.012 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:07.020Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:07.020Z [INFO] agent.router.manager: shutting down +30:07.020 [INFO] TestAgent: consul server down +30:07.020 [INFO] TestAgent: shutdown complete +30:07.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=tcp +30:07.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=udp +30:07.020 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29863 network=tcp +30:07.524 [INFO] TestAgent: Waiting for endpoints to shut down +30:07.524 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_AddService (0.00s) - --- PASS: TestAgent_AddService/service_manager (0.82s) + --- PASS: TestAgent_AddService/service_manager (0.84s) --- PASS: TestAgent_AddService/service_manager/one_check (0.00s) --- PASS: TestAgent_AddService/service_manager/one_check/svcid1 (0.00s) --- PASS: TestAgent_AddService/service_manager/one_check/check1 (0.00s) @@ -10590,11 +10612,11 @@ --- PASS: TestAgent_AddService/service_manager/multiple_checks/check1 (0.00s) --- PASS: TestAgent_AddService/service_manager/multiple_checks/check-noname (0.00s) --- PASS: TestAgent_AddService/service_manager/multiple_checks/service:svcid2:3 (0.00s) - --- PASS: TestAgent_AddService/service_manager/multiple_checks/service:svcid2:4_ttl (0.00s) --- PASS: TestAgent_AddService/service_manager/multiple_checks/check1_ttl (0.00s) --- PASS: TestAgent_AddService/service_manager/multiple_checks/check-noname_ttl (0.00s) --- PASS: TestAgent_AddService/service_manager/multiple_checks/service:svcid2:3_ttl (0.00s) - --- PASS: TestAgent_AddService/normal (0.96s) + --- PASS: TestAgent_AddService/service_manager/multiple_checks/service:svcid2:4_ttl (0.00s) + --- PASS: TestAgent_AddService/normal (1.00s) --- PASS: TestAgent_AddService/normal/one_check (0.00s) --- PASS: TestAgent_AddService/normal/one_check/svcid1 (0.00s) --- PASS: TestAgent_AddService/normal/one_check/check1 (0.00s) @@ -10605,10 +10627,10 @@ --- PASS: TestAgent_AddService/normal/multiple_checks/check-noname (0.00s) --- PASS: TestAgent_AddService/normal/multiple_checks/service:svcid2:3 (0.00s) --- PASS: TestAgent_AddService/normal/multiple_checks/service:svcid2:4 (0.00s) - --- PASS: TestAgent_AddService/normal/multiple_checks/check1_ttl (0.00s) - --- PASS: TestAgent_AddService/normal/multiple_checks/check-noname_ttl (0.00s) --- PASS: TestAgent_AddService/normal/multiple_checks/service:svcid2:3_ttl (0.00s) --- PASS: TestAgent_AddService/normal/multiple_checks/service:svcid2:4_ttl (0.00s) + --- PASS: TestAgent_AddService/normal/multiple_checks/check1_ttl (0.00s) + --- PASS: TestAgent_AddService/normal/multiple_checks/check-noname_ttl (0.00s) === RUN TestAgent_AddServices_AliasUpdateCheckNotReverted === RUN TestAgent_AddServices_AliasUpdateCheckNotReverted/normal === PAUSE TestAgent_AddServices_AliasUpdateCheckNotReverted/normal @@ -10616,173 +10638,368 @@ === PAUSE TestAgent_AddServices_AliasUpdateCheckNotReverted/service_manager === CONT TestAgent_AddServices_AliasUpdateCheckNotReverted/normal === CONT TestAgent_AddServices_AliasUpdateCheckNotReverted/service_manager -2022-11-07T22:45:04.988Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.999Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:05.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58b5c526-98bc-8006-6c72-ba6b30d9b13c Address:127.0.0.1:11503}]" -45:05.018 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11503 [Follower]" leader= -45:05.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:05.019 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:05.019Z [INFO] agent.router: Initializing LAN area manager -45:05.019 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:05.019 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11503) (DC: dc1)" -45:05.019 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11498 network=udp -45:05.019 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11498 network=tcp -45:05.019 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11499 network=tcp -45:05.020 [INFO] TestAgent: started state syncer -45:05.020 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11504 network=tcp -45:05.073 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:05.073 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11503 [Candidate]" term=2 -45:05.081 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:05.081 [DEBUG] TestAgent.server.raft: vote granted: from=58b5c526-98bc-8006-6c72-ba6b30d9b13c term=2 tally=1 -45:05.081 [INFO] TestAgent.server.raft: election won: tally=1 -45:05.081 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11503 [Leader]" -45:05.081 [INFO] TestAgent.server: cluster leadership acquired -45:05.081 [INFO] TestAgent.server: New leader elected: payload=node1 -45:05.086 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11503 -45:05.097 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:05.118 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:05.118 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:05.118 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:05.118 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:05.118 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:05.118 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:05.118 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:05.126 [INFO] TestAgent.server: federation state anti-entropy synced -45:05.174 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:05.177 [INFO] TestAgent: Synced node info -45:05.227 [INFO] TestAgent: Requesting shutdown -45:05.227 [INFO] TestAgent.server: shutting down server -45:05.227 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:05.227 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:05.227 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:05.227 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:05.227 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:05.227 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:05.227 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:05.227 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:05.227 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:05.227 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:05.230 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:05.234Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:05.234Z [INFO] agent.router.manager: shutting down -45:05.234 [INFO] TestAgent: consul server down -45:05.234 [INFO] TestAgent: shutdown complete -45:05.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11498 network=tcp -45:05.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11498 network=udp -45:05.234 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11499 network=tcp -45:05.734 [INFO] TestAgent: Waiting for endpoints to shut down -45:05.734 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:04.988Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.999Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:05.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5c53805d-36ed-f9d3-1cc4-0dc196cb669c Address:127.0.0.1:11510}]" -45:05.018 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11510 [Follower]" leader= -45:05.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:05.019 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:05.019Z [INFO] agent.router: Initializing LAN area manager -45:05.019 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11510) (DC: dc1)" -45:05.019 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:05.019 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11505 network=udp -45:05.019 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11505 network=tcp -45:05.020 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11506 network=tcp -45:05.020 [INFO] TestAgent: started state syncer -45:05.020 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11511 network=tcp -45:05.088 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:05.088 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11510 [Candidate]" term=2 -45:05.097 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:05.097 [DEBUG] TestAgent.server.raft: vote granted: from=5c53805d-36ed-f9d3-1cc4-0dc196cb669c term=2 tally=1 -45:05.097 [INFO] TestAgent.server.raft: election won: tally=1 -45:05.097 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11510 [Leader]" -45:05.097 [INFO] TestAgent.server: cluster leadership acquired -45:05.097 [INFO] TestAgent.server: New leader elected: payload=node1 -45:05.102 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11510 -45:05.113 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:05.131 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:05.131 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:05.131 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:05.131 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:05.131 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:05.131 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:05.131 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:05.141 [INFO] TestAgent.server: federation state anti-entropy synced -45:05.468 [INFO] TestAgent: Requesting shutdown -45:05.468 [INFO] TestAgent.server: shutting down server -45:05.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:05.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:05.468 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:05.468 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:05.468 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:05.468 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:05.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:05.468 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:05.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:05.468 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:05.468 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:05.473 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:05.476Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:05.476Z [INFO] agent.router.manager: shutting down -45:05.477 [INFO] TestAgent: consul server down -45:05.477 [INFO] TestAgent: shutdown complete -45:05.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11505 network=tcp -45:05.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11505 network=udp -45:05.477 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11506 network=tcp -45:05.977 [INFO] TestAgent: Waiting for endpoints to shut down -45:05.977 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:07.552Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:07.583Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:07.621 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:248a36a3-447e-2aa6-42f7-d99cd0dcbcca Address:127.0.0.1:29888}]" +30:07.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:07.622 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:07.623Z [INFO] agent.router: Initializing LAN area manager +30:07.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=udp +30:07.623 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29888 [Follower]" leader= +30:07.623 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29888) (DC: dc1)" +30:07.623 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:07.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=tcp +30:07.624 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29884 network=tcp +30:07.624 [INFO] TestAgent: started state syncer +30:07.624 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29889 network=tcp +30:07.683 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:07.683 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29888 [Candidate]" term=2 +30:07.720 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:07.720 [DEBUG] TestAgent.server.raft: vote granted: from=248a36a3-447e-2aa6-42f7-d99cd0dcbcca term=2 tally=1 +30:07.720 [INFO] TestAgent.server.raft: election won: tally=1 +30:07.720 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29888 [Leader]" +30:07.720 [INFO] TestAgent.server: cluster leadership acquired +30:07.720 [INFO] TestAgent.server: New leader elected: payload=node1 +30:07.756 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29888 +30:07.788 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:07.820 [INFO] TestAgent: Synced node info +30:07.820 [DEBUG] TestAgent: Node info in sync +30:07.876 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:07.876 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:07.876 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:07.876 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:07.876 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:07.876 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:07.876 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:07.908 [INFO] TestAgent.server: federation state anti-entropy synced +30:07.964 [INFO] TestAgent: Requesting shutdown +30:07.964 [INFO] TestAgent.server: shutting down server +30:07.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:07.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:07.964 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:07.964 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:07.964 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:07.964 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:07.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:07.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:07.964 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:07.964 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:07.973 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:07.975Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:07.975Z [INFO] agent.router.manager: shutting down +30:07.975 [INFO] TestAgent: consul server down +30:07.975 [INFO] TestAgent: shutdown complete +30:07.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=tcp +30:07.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=udp +30:07.975 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29884 network=tcp +30:08.476 [INFO] TestAgent: Waiting for endpoints to shut down +30:08.476 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:07.567Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:07.602Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:07.685 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3974ee9d-320d-b64e-2027-cdfc880b633a Address:127.0.0.1:29881}]" +30:07.686 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:07.686 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:07.686Z [INFO] agent.router: Initializing LAN area manager +30:07.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=udp +30:07.687 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29881 [Follower]" leader= +30:07.687 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29881) (DC: dc1)" +30:07.687 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:07.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=tcp +30:07.688 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29877 network=tcp +30:07.688 [INFO] TestAgent: started state syncer +30:07.688 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29882 network=tcp +30:07.756 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:07.756 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29881 [Candidate]" term=2 +30:07.779 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:07.779 [DEBUG] TestAgent.server.raft: vote granted: from=3974ee9d-320d-b64e-2027-cdfc880b633a term=2 tally=1 +30:07.779 [INFO] TestAgent.server.raft: election won: tally=1 +30:07.779 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29881 [Leader]" +30:07.780 [INFO] TestAgent.server: cluster leadership acquired +30:07.780 [INFO] TestAgent.server: New leader elected: payload=node1 +30:07.820 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29881 +30:07.872 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:07.920 [INFO] TestAgent: Synced node info +30:07.935 [DEBUG] TestAgent: Node info in sync +30:07.935 [DEBUG] TestAgent: Node info in sync +30:07.937 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:07.937 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:07.937 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:07.937 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:07.937 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:07.937 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:07.937 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:07.970 [INFO] TestAgent.server: federation state anti-entropy synced +30:07.999 [DEBUG] TestAgent: Node info in sync +30:08.007 [INFO] TestAgent: Synced service: service=fake +30:08.017 [INFO] TestAgent: Synced service: service=web-1 +30:08.036 [INFO] TestAgent: Synced service: service=web-3 +30:08.060 [INFO] TestAgent: Synced service: service=web-4 +30:08.076 [INFO] TestAgent: Synced service: service=web-5 +30:08.096 [INFO] TestAgent: Synced service: service=web-2 +30:08.112 [INFO] TestAgent: Synced service: service=web-6 +30:08.124 [INFO] TestAgent: Synced service: service=web-7 +30:08.140 [INFO] TestAgent: Synced service: service=web-8 +30:08.152 [INFO] TestAgent: Synced service: service=web-9 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.152 [DEBUG] TestAgent: Node info in sync +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.152 [DEBUG] TestAgent: Service in sync: service=fake +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.152 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.152 [DEBUG] TestAgent: Node info in sync +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.152 [DEBUG] TestAgent: Service in sync: service=fake +30:08.152 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.153 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.153 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.168 [INFO] TestAgent: Synced check: check=service:web-9 +30:08.168 [DEBUG] TestAgent: Node info in sync +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.168 [DEBUG] TestAgent: Service in sync: service=fake +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.168 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.168 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.168 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.168 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.168 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.168 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.181 [INFO] TestAgent: Synced check: check=service:web-8 +30:08.181 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.181 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.181 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.181 [DEBUG] TestAgent: Node info in sync +30:08.181 [DEBUG] TestAgent: Service in sync: service=fake +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.181 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.181 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.181 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.184 [INFO] TestAgent: Synced check: check=service:web-3 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.184 [DEBUG] TestAgent: Node info in sync +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.184 [DEBUG] TestAgent: Service in sync: service=fake +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.184 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.184 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.190 [INFO] TestAgent: Synced check: check=service:web-6 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.190 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.190 [DEBUG] TestAgent: Node info in sync +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.190 [DEBUG] TestAgent: Service in sync: service=fake +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.190 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.193 [INFO] TestAgent: Synced check: check=service:web-2 +30:08.193 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.193 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.193 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.212 [INFO] TestAgent: Synced check: check=service:web-1 +30:08.222 [INFO] TestAgent: Synced check: check=service:web-4 +30:08.244 [INFO] TestAgent: Synced check: check=service:web-5 +30:08.248 [INFO] TestAgent: Synced check: check=service:web-7 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.248 [INFO] TestAgent: Requesting shutdown +30:08.248 [DEBUG] TestAgent: Node info in sync +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-4 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-5 +30:08.248 [DEBUG] TestAgent: Service in sync: service=fake +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-1 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-3 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-8 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-9 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-2 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-6 +30:08.248 [DEBUG] TestAgent: Service in sync: service=web-7 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-2 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-3 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-6 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-1 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-4 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-5 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-7 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-8 +30:08.248 [DEBUG] TestAgent: Check in sync: check=service:web-9 +30:08.248 [INFO] TestAgent.server: shutting down server +30:08.248 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:08.248 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:08.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:08.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:08.248 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:08.248 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:08.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:08.248 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:08.248 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:08.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:08.252 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:08.253Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:08.253Z [INFO] agent.router.manager: shutting down +30:08.253 [INFO] TestAgent: consul server down +30:08.253 [INFO] TestAgent: shutdown complete +30:08.253 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=tcp +30:08.253 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=udp +30:08.253 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29877 network=tcp +30:08.756 [INFO] TestAgent: Waiting for endpoints to shut down +30:08.756 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_AddServices_AliasUpdateCheckNotReverted (0.00s) - --- PASS: TestAgent_AddServices_AliasUpdateCheckNotReverted/normal (0.76s) - --- PASS: TestAgent_AddServices_AliasUpdateCheckNotReverted/service_manager (1.01s) + --- PASS: TestAgent_AddServices_AliasUpdateCheckNotReverted/service_manager (0.95s) + --- PASS: TestAgent_AddServices_AliasUpdateCheckNotReverted/normal (1.23s) === RUN TestAgent_CheckAliasRPC -[DEBUG] Service Removed {serviceAlias-696068023 {}} , err:= -[DEBUG] Service Removed {serviceAlias-324302785 {}} , err:= -[DEBUG] Service Removed {serviceAlias-347855559 {}} , err:= -[DEBUG] Service Removed {serviceAlias-2044937347 {}} , err:= -[DEBUG] STARTING shutdown for TestAgent_CheckAliasRPC 2022-11-07 22:45:06.460038853 +0000 UTC m=+31.418522069 -[DEBUG] DONE shutdown for TestAgent_CheckAliasRPC 2022-11-07 22:45:06.460096609 +0000 UTC m=+31.418579789 -2022-11-07T22:45:05.988Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:05.997Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:06.020 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1bc3f813-f6c0-85b2-f895-355ef4e04e41 Address:127.0.0.1:11517}]" -45:06.020 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11517 [Follower]" leader= -45:06.023 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:06.024 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:06.024Z [INFO] agent.router: Initializing LAN area manager -45:06.024 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11512 network=udp -45:06.024 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11517) (DC: dc1)" -45:06.024 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:06.024 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11512 network=tcp -45:06.025 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11513 network=tcp -45:06.025 [INFO] TestAgent: started state syncer -45:06.025 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11518 network=tcp -45:06.062 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:06.062 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11517 [Candidate]" term=2 -45:06.070 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:06.070 [DEBUG] TestAgent.server.raft: vote granted: from=1bc3f813-f6c0-85b2-f895-355ef4e04e41 term=2 tally=1 -45:06.070 [INFO] TestAgent.server.raft: election won: tally=1 -45:06.070 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11517 [Leader]" -45:06.070 [INFO] TestAgent.server: cluster leadership acquired -45:06.070 [INFO] TestAgent.server: New leader elected: payload=node1 -45:06.075 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11517 -45:06.086 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:06.102 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:06.102 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:06.102 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:06.102 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:06.102 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:06.103 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:06.103 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:06.112 [INFO] TestAgent.server: federation state anti-entropy synced -45:06.336 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:06.339 [INFO] TestAgent: Synced node info -45:06.342 [INFO] TestAgent: Synced service: service=svcid1 -45:06.342 [DEBUG] TestAgent: Node info in sync -45:06.342 [DEBUG] TestAgent: Service in sync: service=svcid1 -45:06.359 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.359248046 +0000 UTC m=+31.317731262" -45:06.459 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.459406143 +0000 UTC m=+31.417889346" -45:06.459 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.459533304 +0000 UTC m=+31.418016464" -45:06.459 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.45966367 +0000 UTC m=+31.418146851" -45:06.459 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.459783786 +0000 UTC m=+31.418266963" -45:06.459 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.459883428 +0000 UTC m=+31.418366603" -45:06.460 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.459978668 +0000 UTC m=+31.418461879" -45:06.460 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2022-11-07 22:45:06.460058012 +0000 UTC m=+31.418541202" ---- PASS: TestAgent_CheckAliasRPC (0.48s) +[DEBUG] Service Removed {serviceAlias-1074727158 {}} , err:= +[DEBUG] Service Removed {serviceAlias-876082266 {}} , err:= +[DEBUG] Service Removed {serviceAlias-1023457197 {}} , err:= +[DEBUG] Service Removed {serviceAlias-1216434316 {}} , err:= +[DEBUG] STARTING shutdown for TestAgent_CheckAliasRPC 2021-10-05 16:30:09.879966966 +0000 UTC m=+35.970061939 +[DEBUG] DONE shutdown for TestAgent_CheckAliasRPC 2021-10-05 16:30:09.880021158 +0000 UTC m=+35.970116147 +2021-10-05T16:30:08.778Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:08.816Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:08.857 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e6174c5d-cc28-2e2a-b094-8e232feffd50 Address:127.0.0.1:29895}]" +30:08.857 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:08.858 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:08.858Z [INFO] agent.router: Initializing LAN area manager +30:08.858 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29890 network=udp +30:08.858 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29895 [Follower]" leader= +30:08.859 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29895) (DC: dc1)" +30:08.859 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:08.859 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29890 network=tcp +30:08.860 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29891 network=tcp +30:08.860 [INFO] TestAgent: started state syncer +30:08.860 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29896 network=tcp +30:08.916 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:08.916 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29895 [Candidate]" term=2 +30:08.964 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:08.964 [DEBUG] TestAgent.server.raft: vote granted: from=e6174c5d-cc28-2e2a-b094-8e232feffd50 term=2 tally=1 +30:08.964 [INFO] TestAgent.server.raft: election won: tally=1 +30:08.964 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29895 [Leader]" +30:08.964 [INFO] TestAgent.server: cluster leadership acquired +30:08.964 [INFO] TestAgent.server: New leader elected: payload=node1 +30:08.996 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29895 +30:09.052 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:09.116 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:09.116 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:09.116 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:09.116 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:09.116 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:09.116 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:09.116 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:09.144 [INFO] TestAgent.server: federation state anti-entropy synced +30:09.176 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:09.192 [INFO] TestAgent: Synced node info +30:09.192 [DEBUG] TestAgent: Node info in sync +30:09.752 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:09.752 [DEBUG] TestAgent: Node info in sync +30:09.768 [INFO] TestAgent: Synced service: service=svcid1 +30:09.768 [DEBUG] TestAgent: Node info in sync +30:09.768 [DEBUG] TestAgent: Service in sync: service=svcid1 +30:09.776 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.776467678 +0000 UTC m=+35.866563199" +30:09.776 [DEBUG] TestAgent: Node info in sync +30:09.796 [INFO] TestAgent: Synced service: service=serviceAlias-1074727158 +30:09.816 [INFO] TestAgent: Synced service: service=serviceAlias-876082266 +30:09.825 [INFO] TestAgent: Synced service: service=serviceAlias-1023457197 +30:09.833 [INFO] TestAgent: Synced service: service=serviceAlias-1216434316 +30:09.852 [INFO] TestAgent: Synced service: service=svcid1 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1074727158 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-876082266 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1023457197 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1216434316 +30:09.852 [DEBUG] TestAgent: Node info in sync +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1074727158 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-876082266 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1023457197 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1216434316 +30:09.852 [DEBUG] TestAgent: Service in sync: service=svcid1 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1074727158 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-876082266 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1023457197 +30:09.852 [DEBUG] TestAgent: Check in sync: check=check-1216434316 +30:09.852 [DEBUG] TestAgent: Node info in sync +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1216434316 +30:09.852 [DEBUG] TestAgent: Service in sync: service=svcid1 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1074727158 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-876082266 +30:09.852 [DEBUG] TestAgent: Service in sync: service=serviceAlias-1023457197 +30:09.879 [INFO] TestAgent: Synced check: check=check-1023457197 +30:09.879 [DEBUG] TestAgent: Check in sync: check=check-1216434316 +30:09.879 [DEBUG] TestAgent: Check in sync: check=check-1074727158 +30:09.879 [DEBUG] TestAgent: Check in sync: check=check-876082266 +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.87838237 +0000 UTC m=+35.968477353" +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.879540462 +0000 UTC m=+35.969635409" +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.8796756 +0000 UTC m=+35.969770519" +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.87977563 +0000 UTC m=+35.969870550" +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.879852832 +0000 UTC m=+35.969947785" +30:09.879 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.879931383 +0000 UTC m=+35.970026339" +30:09.880 [DEBUG] TestAgent: Service tag will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: tag="tag-2021-10-05 16:30:09.879988381 +0000 UTC m=+35.970083364" +--- PASS: TestAgent_CheckAliasRPC (1.12s) === RUN TestAgent_AddServiceNoExec === RUN TestAgent_AddServiceNoExec/normal === PAUSE TestAgent_AddServiceNoExec/normal @@ -10793,121 +11010,121 @@ [WARN] freeport: 6 out of 7 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:06.472Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:06.481Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:06.505 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e7394914-2406-928f-7b5f-76d9cbcaef8f Address:127.0.0.1:11531}]" -45:06.505 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11531 [Follower]" leader= -45:06.506 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:06.506 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:06.506Z [INFO] agent.router: Initializing LAN area manager -45:06.506 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11531) (DC: dc1)" -45:06.506 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:06.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11526 network=tcp -45:06.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11526 network=udp -45:06.507 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11527 network=tcp -45:06.507 [INFO] TestAgent: started state syncer -45:06.507 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11532 network=tcp -45:06.549 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:06.549 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11531 [Candidate]" term=2 -45:06.557 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:06.557 [DEBUG] TestAgent.server.raft: vote granted: from=e7394914-2406-928f-7b5f-76d9cbcaef8f term=2 tally=1 -45:06.557 [INFO] TestAgent.server.raft: election won: tally=1 -45:06.557 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11531 [Leader]" -45:06.557 [INFO] TestAgent.server: cluster leadership acquired -45:06.557 [INFO] TestAgent.server: New leader elected: payload=node1 -45:06.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11531 -45:06.575 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:06.593 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:06.593 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:06.593 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:06.593 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:06.593 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:06.593 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:06.593 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:06.606 [INFO] TestAgent.server: federation state anti-entropy synced -45:06.717 [INFO] TestAgent: Requesting shutdown -45:06.717 [INFO] TestAgent.server: shutting down server -45:06.717 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:06.717 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:06.717 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:06.717 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:06.717 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:06.717 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:06.717 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:06.717 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:06.717 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:06.717 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:06.717 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:06.720 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:06.724Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:06.724Z [INFO] agent.router.manager: shutting down -45:06.724 [INFO] TestAgent: consul server down -45:06.724 [INFO] TestAgent: shutdown complete -45:06.724 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11526 network=tcp -45:06.724 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11526 network=udp -45:06.724 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11527 network=tcp -45:07.224 [INFO] TestAgent: Waiting for endpoints to shut down -45:07.224 [INFO] TestAgent: Endpoints down -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:06.476Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:06.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:06.513 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7b80e8cd-4cb5-8457-544d-b6197b0ab524 Address:127.0.0.1:11524}]" -45:06.513 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11524 [Follower]" leader= -45:06.514 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:06.514 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:06.514Z [INFO] agent.router: Initializing LAN area manager -45:06.514 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11524) (DC: dc1)" -45:06.515 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:06.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11519 network=tcp -45:06.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11519 network=udp -45:06.515 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11520 network=tcp -45:06.515 [INFO] TestAgent: started state syncer -45:06.515 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11525 network=tcp -45:06.555 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:06.555 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11524 [Candidate]" term=2 -45:06.564 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:06.564 [DEBUG] TestAgent.server.raft: vote granted: from=7b80e8cd-4cb5-8457-544d-b6197b0ab524 term=2 tally=1 -45:06.564 [INFO] TestAgent.server.raft: election won: tally=1 -45:06.564 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11524 [Leader]" -45:06.564 [INFO] TestAgent.server: cluster leadership acquired -45:06.564 [INFO] TestAgent.server: New leader elected: payload=node1 -45:06.571 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11524 -45:06.582 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:06.603 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:06.603 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:06.603 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:06.603 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:06.603 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:06.603 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:06.603 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:06.615 [INFO] TestAgent.server: federation state anti-entropy synced -45:06.775 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:06.778 [INFO] TestAgent: Synced node info -45:06.893 [INFO] TestAgent: Requesting shutdown -45:06.893 [INFO] TestAgent.server: shutting down server -45:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:06.893 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:06.893 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:06.897 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:06.900Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:06.900Z [INFO] agent.router.manager: shutting down -45:06.900 [INFO] TestAgent: consul server down -45:06.900 [INFO] TestAgent: shutdown complete -45:06.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11519 network=tcp -45:06.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11519 network=udp -45:06.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11520 network=tcp -45:07.401 [INFO] TestAgent: Waiting for endpoints to shut down -45:07.401 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:09.908Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:09.923Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:09.990 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:324e9673-9f68-4ae6-9334-6c8d8e624a40 Address:127.0.0.1:29909}]" +30:09.991 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:09.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29909 [Follower]" leader= +30:09.992 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:09.992Z [INFO] agent.router: Initializing LAN area manager +30:09.992 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29904 network=udp +30:09.992 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29909) (DC: dc1)" +30:09.992 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:09.992 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29904 network=tcp +30:09.993 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29905 network=tcp +30:09.993 [INFO] TestAgent: started state syncer +30:09.993 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29910 network=tcp +30:10.056 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.056 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29909 [Candidate]" term=2 +30:10.090 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.090 [DEBUG] TestAgent.server.raft: vote granted: from=324e9673-9f68-4ae6-9334-6c8d8e624a40 term=2 tally=1 +30:10.090 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.090 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29909 [Leader]" +30:10.090 [INFO] TestAgent.server: cluster leadership acquired +30:10.090 [INFO] TestAgent.server: New leader elected: payload=node1 +30:10.120 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29909 +30:10.192 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.264 [INFO] TestAgent: Synced node info +30:10.283 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.283 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.283 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.283 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.283 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.283 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:10.283 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:10.302 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.314 [INFO] TestAgent: Requesting shutdown +30:10.314 [INFO] TestAgent.server: shutting down server +30:10.314 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:10.314 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:10.314 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:10.314 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:10.314 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:10.314 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:10.314 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:10.314 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:10.314 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:10.314 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:10.337 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:10.339Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:10.339Z [INFO] agent.router.manager: shutting down +30:10.339 [INFO] TestAgent: consul server down +30:10.339 [INFO] TestAgent: shutdown complete +30:10.339 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29904 network=tcp +30:10.339 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29904 network=udp +30:10.339 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29905 network=tcp +30:10.840 [INFO] TestAgent: Waiting for endpoints to shut down +30:10.840 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:09.904Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:09.926Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.004 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef2a0237-634b-d652-c984-a27822cc80a1 Address:127.0.0.1:29902}]" +30:10.005 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:10.005 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:10.005Z [INFO] agent.router: Initializing LAN area manager +30:10.005 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29902 [Follower]" leader= +30:10.006 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29902) (DC: dc1)" +30:10.006 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29897 network=udp +30:10.006 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:10.006 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29897 network=tcp +30:10.006 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29898 network=tcp +30:10.007 [INFO] TestAgent: started state syncer +30:10.007 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29903 network=tcp +30:10.057 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.057 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29902 [Candidate]" term=2 +30:10.077 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.077 [DEBUG] TestAgent.server.raft: vote granted: from=ef2a0237-634b-d652-c984-a27822cc80a1 term=2 tally=1 +30:10.077 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.077 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29902 [Leader]" +30:10.078 [INFO] TestAgent.server: cluster leadership acquired +30:10.078 [INFO] TestAgent.server: New leader elected: payload=node1 +30:10.083 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29902 +30:10.155 [INFO] TestAgent: Synced node info +30:10.155 [DEBUG] TestAgent: Node info in sync +30:10.155 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.249 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.249 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.249 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.249 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.249 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.249 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:10.249 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:10.287 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.421 [INFO] TestAgent: Requesting shutdown +30:10.421 [INFO] TestAgent.server: shutting down server +30:10.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:10.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:10.421 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:10.421 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:10.421 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:10.422 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:10.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:10.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:10.422 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:10.422 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:10.433 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:10.434Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:10.434Z [INFO] agent.router.manager: shutting down +30:10.434 [INFO] TestAgent: consul server down +30:10.434 [INFO] TestAgent: shutdown complete +30:10.434 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29897 network=tcp +30:10.434 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29897 network=udp +30:10.434 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29898 network=tcp +30:10.935 [INFO] TestAgent: Waiting for endpoints to shut down +30:10.935 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_AddServiceNoExec (0.00s) - --- PASS: TestAgent_AddServiceNoExec/service_manager (0.76s) - --- PASS: TestAgent_AddServiceNoExec/normal (0.94s) + --- PASS: TestAgent_AddServiceNoExec/service_manager (0.96s) + --- PASS: TestAgent_AddServiceNoExec/normal (1.06s) === RUN TestAgent_AddServiceNoRemoteExec === RUN TestAgent_AddServiceNoRemoteExec/normal === PAUSE TestAgent_AddServiceNoRemoteExec/normal @@ -10915,376 +11132,372 @@ === PAUSE TestAgent_AddServiceNoRemoteExec/service_manager === CONT TestAgent_AddServiceNoRemoteExec/normal === CONT TestAgent_AddServiceNoRemoteExec/service_manager -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:07.417Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:07.428Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:07.454 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1f5acf20-4dd3-dcc6-cb82-68970e0f396f Address:127.0.0.1:11538}]" -45:07.454 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11538 [Follower]" leader= -45:07.455 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:07.455 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:07.455Z [INFO] agent.router: Initializing LAN area manager -45:07.456 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11538) (DC: dc1)" -45:07.456 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:07.456 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11533 network=tcp -45:07.456 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11533 network=udp -45:07.456 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11534 network=tcp -45:07.456 [INFO] TestAgent: started state syncer -45:07.456 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11539 network=tcp -45:07.506 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:07.506 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11538 [Candidate]" term=2 -45:07.515 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:07.515 [DEBUG] TestAgent.server.raft: vote granted: from=1f5acf20-4dd3-dcc6-cb82-68970e0f396f term=2 tally=1 -45:07.515 [INFO] TestAgent.server.raft: election won: tally=1 -45:07.515 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11538 [Leader]" -45:07.516 [INFO] TestAgent.server: cluster leadership acquired -45:07.516 [INFO] TestAgent.server: New leader elected: payload=node1 -45:07.522 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11538 -45:07.533 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:07.554 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:07.554 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:07.554 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:07.554 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:07.554 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:07.554 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:07.554 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:07.560 [INFO] TestAgent.server: federation state anti-entropy synced -45:07.575 [INFO] TestAgent: Requesting shutdown -45:07.575 [INFO] TestAgent.server: shutting down server -45:07.575 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:07.575 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:07.575 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:07.575 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:07.575 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:07.575 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:07.575 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:07.575 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:07.575 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:07.575 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:07.575 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:07.578 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:07.581Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:07.581Z [INFO] agent.router.manager: shutting down -45:07.581 [INFO] TestAgent: consul server down -45:07.581 [INFO] TestAgent: shutdown complete -45:07.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11533 network=tcp -45:07.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11533 network=udp -45:07.581 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11534 network=tcp -45:08.081 [INFO] TestAgent: Waiting for endpoints to shut down -45:08.081 [INFO] TestAgent: Endpoints down -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:07.411Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:07.426Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:07.452 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:417cecdb-3d55-7d9d-5028-f52f99aa7e61 Address:127.0.0.1:11545}]" -45:07.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11545 [Follower]" leader= -45:07.453 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:07.454 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:07.454Z [INFO] agent.router: Initializing LAN area manager -45:07.454 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11545) (DC: dc1)" -45:07.454 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:07.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11540 network=tcp -45:07.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11540 network=udp -45:07.454 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11541 network=tcp -45:07.454 [INFO] TestAgent: started state syncer -45:07.454 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11546 network=tcp -45:07.510 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:07.510 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11545 [Candidate]" term=2 -45:07.520 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:07.520 [DEBUG] TestAgent.server.raft: vote granted: from=417cecdb-3d55-7d9d-5028-f52f99aa7e61 term=2 tally=1 -45:07.520 [INFO] TestAgent.server.raft: election won: tally=1 -45:07.520 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11545 [Leader]" -45:07.520 [INFO] TestAgent.server: cluster leadership acquired -45:07.520 [INFO] TestAgent.server: New leader elected: payload=node1 -45:07.526 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11545 -45:07.537 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:07.557 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:07.557 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:07.557 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:07.557 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:07.557 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:07.557 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:07.557 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:07.568 [INFO] TestAgent.server: federation state anti-entropy synced -45:07.570 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:07.573 [INFO] TestAgent: Synced node info -45:07.573 [DEBUG] TestAgent: Node info in sync -45:07.791 [INFO] TestAgent: Requesting shutdown -45:07.791 [INFO] TestAgent.server: shutting down server -45:07.791 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:07.791 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:07.791 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:07.791 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:07.791 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:07.792 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:07.792 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:07.791 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:07.792 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:07.792 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:07.796 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:07.799Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:07.799Z [INFO] agent.router.manager: shutting down -45:07.799 [INFO] TestAgent: consul server down -45:07.799 [INFO] TestAgent: shutdown complete -45:07.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11540 network=tcp -45:07.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11540 network=udp -45:07.799 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11541 network=tcp -45:08.299 [INFO] TestAgent: Waiting for endpoints to shut down -45:08.299 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:11.024Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:11.064Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:11.138 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c199d582-3ead-5635-294d-6a02e78f26b5 Address:127.0.0.1:29916}]" +30:11.138 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:11.139 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:11.139Z [INFO] agent.router: Initializing LAN area manager +30:11.139 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29911 network=udp +30:11.139 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29916 [Follower]" leader= +30:11.140 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29916) (DC: dc1)" +30:11.140 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:11.140 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29911 network=tcp +30:11.141 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29912 network=tcp +30:11.141 [INFO] TestAgent: started state syncer +30:11.141 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29917 network=tcp +30:11.204 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:11.204 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29916 [Candidate]" term=2 +30:11.235 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:11.235 [DEBUG] TestAgent.server.raft: vote granted: from=c199d582-3ead-5635-294d-6a02e78f26b5 term=2 tally=1 +30:11.235 [INFO] TestAgent.server.raft: election won: tally=1 +30:11.235 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29916 [Leader]" +30:11.235 [INFO] TestAgent.server: cluster leadership acquired +30:11.235 [INFO] TestAgent.server: New leader elected: payload=node1 +30:11.252 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29916 +30:11.280 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:11.325 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:11.325 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:11.325 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:11.325 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:11.325 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:11.326 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:11.326 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:11.348 [INFO] TestAgent.server: federation state anti-entropy synced +30:11.364 [INFO] TestAgent: Requesting shutdown +30:11.364 [INFO] TestAgent.server: shutting down server +30:11.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:11.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:11.364 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:11.364 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:11.364 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:11.364 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:11.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:11.364 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:11.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:11.364 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:11.364 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.376 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:11.380Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:11.380Z [INFO] agent.router.manager: shutting down +30:11.380 [INFO] TestAgent: consul server down +30:11.380 [INFO] TestAgent: shutdown complete +30:11.380 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29911 network=tcp +30:11.380 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29911 network=udp +30:11.380 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29912 network=tcp +30:11.880 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.880 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:10.995Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:11.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:11.086 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bebfe1c6-b14c-3eb9-ae3d-f77ad8880164 Address:127.0.0.1:29923}]" +30:11.086 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29923 [Follower]" leader= +30:11.088 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:11.089 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:11.089Z [INFO] agent.router: Initializing LAN area manager +30:11.089 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29923) (DC: dc1)" +30:11.089 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:11.090 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29918 network=tcp +30:11.090 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29918 network=udp +30:11.091 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29919 network=tcp +30:11.091 [INFO] TestAgent: started state syncer +30:11.091 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29924 network=tcp +30:11.156 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:11.156 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29923 [Candidate]" term=2 +30:11.204 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:11.204 [DEBUG] TestAgent.server.raft: vote granted: from=bebfe1c6-b14c-3eb9-ae3d-f77ad8880164 term=2 tally=1 +30:11.204 [INFO] TestAgent.server.raft: election won: tally=1 +30:11.204 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29923 [Leader]" +30:11.205 [INFO] TestAgent.server: cluster leadership acquired +30:11.205 [INFO] TestAgent.server: New leader elected: payload=node1 +30:11.231 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29923 +30:11.260 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:11.330 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:11.330 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:11.330 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:11.330 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:11.330 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:11.330 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:11.330 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:11.330 [INFO] TestAgent: Synced node info +30:11.330 [DEBUG] TestAgent: Node info in sync +30:11.356 [INFO] TestAgent: Requesting shutdown +30:11.356 [INFO] TestAgent.server: shutting down server +30:11.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:11.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:11.356 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:11.356 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:11.356 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:11.356 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:11.356 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:11.356 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.364 [INFO] TestAgent.server: federation state anti-entropy synced +30:11.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:11.384 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:11.392Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:11.392Z [INFO] agent.router.manager: shutting down +30:11.393 [INFO] TestAgent: consul server down +30:11.393 [INFO] TestAgent: shutdown complete +30:11.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29918 network=tcp +30:11.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29918 network=udp +30:11.393 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29919 network=tcp +30:11.896 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.896 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_AddServiceNoRemoteExec (0.00s) - --- PASS: TestAgent_AddServiceNoRemoteExec/normal (0.68s) - --- PASS: TestAgent_AddServiceNoRemoteExec/service_manager (0.90s) + --- PASS: TestAgent_AddServiceNoRemoteExec/normal (0.94s) + --- PASS: TestAgent_AddServiceNoRemoteExec/service_manager (0.95s) === RUN TestCacheRateLimit agent_test.go:749: DM-skipped --- SKIP: TestCacheRateLimit (0.00s) === RUN TestAddServiceIPv4TaggedDefault -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:08.315Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:08.329Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:08.351 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0215d145-c11f-072b-f923-7ac933d94faa Address:127.0.0.1:11552}]" -45:08.351 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11552 [Follower]" leader= -45:08.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0215d145-c11f-072b-f923-7ac933d94faa.dc1 127.0.0.1 -45:08.352 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0215d145-c11f-072b-f923-7ac933d94faa 127.0.0.1 -2022-11-07T22:45:08.352Z [INFO] agent.router: Initializing LAN area manager -45:08.352 [INFO] TestAgent.server: Adding LAN server: server="Node-0215d145-c11f-072b-f923-7ac933d94faa (Addr: tcp/127.0.0.1:11552) (DC: dc1)" -45:08.353 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0215d145-c11f-072b-f923-7ac933d94faa.dc1 area=wan -45:08.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11547 network=udp -45:08.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11547 network=tcp -45:08.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11548 network=tcp -45:08.353 [INFO] TestAgent: started state syncer -45:08.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11553 network=tcp -45:08.412 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:08.412 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11552 [Candidate]" term=2 -45:08.420 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:08.420 [DEBUG] TestAgent.server.raft: vote granted: from=0215d145-c11f-072b-f923-7ac933d94faa term=2 tally=1 -45:08.420 [INFO] TestAgent.server.raft: election won: tally=1 -45:08.420 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11552 [Leader]" -45:08.421 [INFO] TestAgent.server: cluster leadership acquired -45:08.421 [INFO] TestAgent.server: New leader elected: payload=Node-0215d145-c11f-072b-f923-7ac933d94faa -45:08.426 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11552 -45:08.437 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:08.459 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:08.459 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:08.459 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:08.459 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:08.459 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:08.459 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0215d145-c11f-072b-f923-7ac933d94faa -45:08.459 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0215d145-c11f-072b-f923-7ac933d94faa -45:08.466 [INFO] TestAgent.server: federation state anti-entropy synced -45:08.699 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service -45:08.699 [INFO] TestAgent: Requesting shutdown -45:08.699 [INFO] TestAgent.server: shutting down server -45:08.699 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:08.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:08.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:08.699 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:08.699 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:08.699 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:08.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:08.699 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:08.699 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:08.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:08.699 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:08.703 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:08.706Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:08.706Z [INFO] agent.router.manager: shutting down -45:08.706 [INFO] TestAgent: consul server down -45:08.706 [INFO] TestAgent: shutdown complete -45:08.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11547 network=tcp -45:08.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11547 network=udp -45:08.706 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11548 network=tcp -45:09.207 [INFO] TestAgent: Waiting for endpoints to shut down -45:09.207 [INFO] TestAgent: Endpoints down ---- PASS: TestAddServiceIPv4TaggedDefault (0.91s) +2021-10-05T16:30:11.925Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:11.955Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:12.048 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b8e794a-d36f-9e9e-a9a1-a425fc78281d Address:127.0.0.1:29930}]" +30:12.048 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29930 [Follower]" leader= +30:12.049 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d.dc1 127.0.0.1 +30:12.049 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d 127.0.0.1 +2021-10-05T16:30:12.049Z [INFO] agent.router: Initializing LAN area manager +30:12.049 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29925 network=udp +30:12.050 [INFO] TestAgent.server: Adding LAN server: server="Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d (Addr: tcp/127.0.0.1:29930) (DC: dc1)" +30:12.050 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d.dc1 area=wan +30:12.050 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29925 network=tcp +30:12.050 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29926 network=tcp +30:12.050 [INFO] TestAgent: started state syncer +30:12.050 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29931 network=tcp +30:12.090 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:12.090 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29930 [Candidate]" term=2 +30:12.129 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:12.129 [DEBUG] TestAgent.server.raft: vote granted: from=5b8e794a-d36f-9e9e-a9a1-a425fc78281d term=2 tally=1 +30:12.129 [INFO] TestAgent.server.raft: election won: tally=1 +30:12.129 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29930 [Leader]" +30:12.129 [INFO] TestAgent.server: cluster leadership acquired +30:12.129 [INFO] TestAgent.server: New leader elected: payload=Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d +30:12.136 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29930 +30:12.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:12.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:12.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:12.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:12.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:12.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:12.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d +30:12.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5b8e794a-d36f-9e9e-a9a1-a425fc78281d +30:12.213 [INFO] TestAgent.server: federation state anti-entropy synced +30:12.232 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service +30:12.232 [INFO] TestAgent: Requesting shutdown +30:12.232 [INFO] TestAgent.server: shutting down server +30:12.232 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:12.232 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:12.232 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:12.232 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:12.232 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:12.232 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:12.232 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:12.232 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:12.232 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:12.232 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:12.232 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:12.237 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:12.243Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:12.243Z [INFO] agent.router.manager: shutting down +30:12.243 [INFO] TestAgent: consul server down +30:12.243 [INFO] TestAgent: shutdown complete +30:12.243 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29925 network=tcp +30:12.243 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29925 network=udp +30:12.243 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29926 network=tcp +30:12.744 [INFO] TestAgent: Waiting for endpoints to shut down +30:12.744 [INFO] TestAgent: Endpoints down +--- PASS: TestAddServiceIPv4TaggedDefault (0.85s) === RUN TestAddServiceIPv6TaggedDefault [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:09.218Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:09.231Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:09.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ec1fb90-63b2-44e8-6bc2-70f66b226a06 Address:127.0.0.1:11559}]" -45:09.252 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11559 [Follower]" leader= -45:09.252 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06.dc1 127.0.0.1 -45:09.253 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06 127.0.0.1 -2022-11-07T22:45:09.253Z [INFO] agent.router: Initializing LAN area manager -45:09.253 [INFO] TestAgent.server: Adding LAN server: server="Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06 (Addr: tcp/127.0.0.1:11559) (DC: dc1)" -45:09.253 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06.dc1 area=wan -45:09.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11554 network=udp -45:09.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11554 network=tcp -45:09.253 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11555 network=tcp -45:09.253 [INFO] TestAgent: started state syncer -45:09.253 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11560 network=tcp -45:09.298 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:09.298 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11559 [Candidate]" term=2 -45:09.306 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:09.306 [DEBUG] TestAgent.server.raft: vote granted: from=3ec1fb90-63b2-44e8-6bc2-70f66b226a06 term=2 tally=1 -45:09.306 [INFO] TestAgent.server.raft: election won: tally=1 -45:09.306 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11559 [Leader]" -45:09.306 [INFO] TestAgent.server: cluster leadership acquired -45:09.306 [INFO] TestAgent.server: New leader elected: payload=Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06 -45:09.311 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11559 -45:09.321 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:09.340 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:09.340 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:09.340 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:09.340 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:09.340 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:09.340 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06 -45:09.340 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ec1fb90-63b2-44e8-6bc2-70f66b226a06 -45:09.346 [INFO] TestAgent.server: federation state anti-entropy synced -45:09.495 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:09.498 [INFO] TestAgent: Synced node info -45:09.498 [DEBUG] TestAgent: Node info in sync -45:09.651 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service -45:09.651 [INFO] TestAgent: Requesting shutdown -45:09.651 [INFO] TestAgent.server: shutting down server -45:09.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:09.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:09.651 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:09.651 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:09.651 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:09.651 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:09.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:09.651 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:09.651 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:09.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:09.655 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:09.659Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:09.659Z [INFO] agent.router.manager: shutting down -45:09.659 [INFO] TestAgent: consul server down -45:09.659 [INFO] TestAgent: shutdown complete -45:09.659 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11554 network=tcp -45:09.659 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11554 network=udp -45:09.659 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11555 network=tcp -45:10.159 [INFO] TestAgent: Waiting for endpoints to shut down -45:10.159 [INFO] TestAgent: Endpoints down ---- PASS: TestAddServiceIPv6TaggedDefault (0.95s) +2021-10-05T16:30:12.771Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:12.822Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:12.908 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 Address:127.0.0.1:29937}]" +30:12.909 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0.dc1 127.0.0.1 +30:12.909 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 127.0.0.1 +2021-10-05T16:30:12.909Z [INFO] agent.router: Initializing LAN area manager +30:12.909 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29932 network=udp +30:12.909 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29937 [Follower]" leader= +30:12.910 [INFO] TestAgent.server: Adding LAN server: server="Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 (Addr: tcp/127.0.0.1:29937) (DC: dc1)" +30:12.910 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0.dc1 area=wan +30:12.910 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29932 network=tcp +30:12.911 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29933 network=tcp +30:12.911 [INFO] TestAgent: started state syncer +30:12.911 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29938 network=tcp +30:12.948 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:12.948 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29937 [Candidate]" term=2 +30:12.996 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:12.996 [DEBUG] TestAgent.server.raft: vote granted: from=c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 term=2 tally=1 +30:12.996 [INFO] TestAgent.server.raft: election won: tally=1 +30:12.996 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29937 [Leader]" +30:12.996 [INFO] TestAgent.server: cluster leadership acquired +30:12.996 [INFO] TestAgent.server: New leader elected: payload=Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 +30:13.016 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29937 +30:13.040 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:13.067 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:13.067 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:13.067 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:13.067 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:13.067 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:13.067 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 +30:13.067 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7e7e7e0-92a9-52d5-a5ce-a1d5368755a0 +30:13.098 [INFO] TestAgent.server: federation state anti-entropy synced +30:13.192 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service +30:13.192 [INFO] TestAgent: Requesting shutdown +30:13.192 [INFO] TestAgent.server: shutting down server +30:13.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:13.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:13.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:13.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:13.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:13.192 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:13.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:13.192 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:13.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:13.192 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:13.192 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:13.205 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:13.224Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:13.224Z [INFO] agent.router.manager: shutting down +30:13.224 [INFO] TestAgent: consul server down +30:13.224 [INFO] TestAgent: shutdown complete +30:13.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29932 network=tcp +30:13.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29932 network=udp +30:13.224 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29933 network=tcp +30:13.724 [INFO] TestAgent: Waiting for endpoints to shut down +30:13.724 [INFO] TestAgent: Endpoints down +--- PASS: TestAddServiceIPv6TaggedDefault (0.98s) === RUN TestAddServiceIPv4TaggedSet [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:10.170Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:10.180Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:10.202 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a43e7b2d-34ec-3080-2118-d373d8e08535 Address:127.0.0.1:11566}]" -45:10.202 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11566 [Follower]" leader= -45:10.203 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a43e7b2d-34ec-3080-2118-d373d8e08535.dc1 127.0.0.1 -45:10.203 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a43e7b2d-34ec-3080-2118-d373d8e08535 127.0.0.1 -2022-11-07T22:45:10.203Z [INFO] agent.router: Initializing LAN area manager -45:10.203 [INFO] TestAgent.server: Adding LAN server: server="Node-a43e7b2d-34ec-3080-2118-d373d8e08535 (Addr: tcp/127.0.0.1:11566) (DC: dc1)" -45:10.203 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a43e7b2d-34ec-3080-2118-d373d8e08535.dc1 area=wan -45:10.204 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11561 network=udp -45:10.204 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11561 network=tcp -45:10.204 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11562 network=tcp -45:10.204 [INFO] TestAgent: started state syncer -45:10.204 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11567 network=tcp -45:10.253 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:10.253 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11566 [Candidate]" term=2 -45:10.261 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:10.261 [DEBUG] TestAgent.server.raft: vote granted: from=a43e7b2d-34ec-3080-2118-d373d8e08535 term=2 tally=1 -45:10.261 [INFO] TestAgent.server.raft: election won: tally=1 -45:10.261 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11566 [Leader]" -45:10.261 [INFO] TestAgent.server: cluster leadership acquired -45:10.261 [INFO] TestAgent.server: New leader elected: payload=Node-a43e7b2d-34ec-3080-2118-d373d8e08535 -45:10.266 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11566 -45:10.277 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:10.295 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:10.295 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:10.295 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:10.295 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:10.295 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:10.295 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a43e7b2d-34ec-3080-2118-d373d8e08535 -45:10.295 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a43e7b2d-34ec-3080-2118-d373d8e08535 -45:10.305 [INFO] TestAgent.server: federation state anti-entropy synced -45:10.563 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:10.567 [INFO] TestAgent: Synced node info -45:10.595 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service -45:10.595 [INFO] TestAgent: Requesting shutdown -45:10.595 [INFO] TestAgent.server: shutting down server -45:10.595 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:10.595 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:10.595 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:10.595 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:10.595 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:10.595 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:10.595 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:10.595 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:10.595 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:10.595 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:10.599 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:10.602Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:10.602Z [INFO] agent.router.manager: shutting down -45:10.602 [INFO] TestAgent: consul server down -45:10.602 [INFO] TestAgent: shutdown complete -45:10.602 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11561 network=tcp -45:10.602 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11561 network=udp -45:10.602 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11562 network=tcp -45:11.102 [INFO] TestAgent: Waiting for endpoints to shut down -45:11.102 [INFO] TestAgent: Endpoints down ---- PASS: TestAddServiceIPv4TaggedSet (0.94s) +2021-10-05T16:30:13.737Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:13.748Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:13.796 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 Address:127.0.0.1:29944}]" +30:13.797 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8.dc1 127.0.0.1 +30:13.797 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 127.0.0.1 +2021-10-05T16:30:13.797Z [INFO] agent.router: Initializing LAN area manager +30:13.798 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29939 network=udp +30:13.798 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29944 [Follower]" leader= +30:13.798 [INFO] TestAgent.server: Adding LAN server: server="Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 (Addr: tcp/127.0.0.1:29944) (DC: dc1)" +30:13.798 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8.dc1 area=wan +30:13.798 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29939 network=tcp +30:13.799 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29940 network=tcp +30:13.799 [INFO] TestAgent: started state syncer +30:13.799 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29945 network=tcp +30:13.857 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:13.857 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29944 [Candidate]" term=2 +30:13.868 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:13.868 [DEBUG] TestAgent.server.raft: vote granted: from=4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 term=2 tally=1 +30:13.868 [INFO] TestAgent.server.raft: election won: tally=1 +30:13.868 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29944 [Leader]" +30:13.872 [INFO] TestAgent.server: cluster leadership acquired +30:13.872 [INFO] TestAgent.server: New leader elected: payload=Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 +30:13.876 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29944 +30:13.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:13.961 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:13.962 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:13.962 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:13.962 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:13.962 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:13.962 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 +30:13.962 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4d2d7251-5b2f-46f7-1d35-67ab7e46ada8 +30:13.984 [INFO] TestAgent.server: federation state anti-entropy synced +30:13.988 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service +30:13.988 [INFO] TestAgent: Requesting shutdown +30:13.988 [INFO] TestAgent.server: shutting down server +30:13.988 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:13.988 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:13.988 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:13.988 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:13.988 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:13.988 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:13.988 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:13.988 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:13.988 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:13.988 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:13.988 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:13.995 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:14.002Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:14.002Z [INFO] agent.router.manager: shutting down +30:14.002 [INFO] TestAgent: consul server down +30:14.002 [INFO] TestAgent: shutdown complete +30:14.002 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29939 network=tcp +30:14.002 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29939 network=udp +30:14.002 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29940 network=tcp +30:14.502 [INFO] TestAgent: Waiting for endpoints to shut down +30:14.502 [INFO] TestAgent: Endpoints down +--- PASS: TestAddServiceIPv4TaggedSet (0.78s) === RUN TestAddServiceIPv6TaggedSet [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:11.117Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:11.131Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:11.152 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:91979561-10c4-d92e-8fdb-3e788e7ca4dd Address:127.0.0.1:11573}]" -45:11.152 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11573 [Follower]" leader= -45:11.153 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd.dc1 127.0.0.1 -45:11.154 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd 127.0.0.1 -2022-11-07T22:45:11.154Z [INFO] agent.router: Initializing LAN area manager -45:11.154 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd.dc1 area=wan -45:11.154 [INFO] TestAgent.server: Adding LAN server: server="Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd (Addr: tcp/127.0.0.1:11573) (DC: dc1)" -45:11.154 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11568 network=udp -45:11.154 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11568 network=tcp -45:11.154 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11569 network=tcp -45:11.155 [INFO] TestAgent: started state syncer -45:11.155 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11574 network=tcp -45:11.200 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:11.200 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11573 [Candidate]" term=2 -45:11.208 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:11.208 [DEBUG] TestAgent.server.raft: vote granted: from=91979561-10c4-d92e-8fdb-3e788e7ca4dd term=2 tally=1 -45:11.208 [INFO] TestAgent.server.raft: election won: tally=1 -45:11.208 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11573 [Leader]" -45:11.209 [INFO] TestAgent.server: cluster leadership acquired -45:11.209 [INFO] TestAgent.server: New leader elected: payload=Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd -45:11.214 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11573 -45:11.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:11.244 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:11.244 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:11.244 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:11.244 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:11.244 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:11.244 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd -45:11.244 [INFO] TestAgent.server: member joined, marking health alive: member=Node-91979561-10c4-d92e-8fdb-3e788e7ca4dd -45:11.257 [INFO] TestAgent.server: federation state anti-entropy synced -45:11.269 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service -45:11.269 [INFO] TestAgent: Requesting shutdown -45:11.269 [INFO] TestAgent.server: shutting down server -45:11.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:11.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:11.269 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:11.269 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:11.269 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:11.269 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:11.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:11.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:11.269 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:11.269 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:11.269 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:11.273 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:11.277Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:11.277Z [INFO] agent.router.manager: shutting down -45:11.277 [INFO] TestAgent: consul server down -45:11.277 [INFO] TestAgent: shutdown complete -45:11.277 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11568 network=tcp -45:11.277 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11568 network=udp -45:11.277 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11569 network=tcp -45:11.777 [INFO] TestAgent: Waiting for endpoints to shut down -45:11.777 [INFO] TestAgent: Endpoints down ---- PASS: TestAddServiceIPv6TaggedSet (0.67s) +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:14.528Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:14.565Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:14.600 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fe7f7bf-5557-49be-ea1f-f899620817a7 Address:127.0.0.1:29951}]" +30:14.601 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fe7f7bf-5557-49be-ea1f-f899620817a7.dc1 127.0.0.1 +30:14.602 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fe7f7bf-5557-49be-ea1f-f899620817a7 127.0.0.1 +2021-10-05T16:30:14.602Z [INFO] agent.router: Initializing LAN area manager +30:14.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29946 network=udp +30:14.602 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29951 [Follower]" leader= +30:14.602 [INFO] TestAgent.server: Adding LAN server: server="Node-2fe7f7bf-5557-49be-ea1f-f899620817a7 (Addr: tcp/127.0.0.1:29951) (DC: dc1)" +30:14.602 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fe7f7bf-5557-49be-ea1f-f899620817a7.dc1 area=wan +30:14.603 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29946 network=tcp +30:14.603 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29947 network=tcp +30:14.603 [INFO] TestAgent: started state syncer +30:14.603 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29952 network=tcp +30:14.660 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:14.660 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29951 [Candidate]" term=2 +30:14.677 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:14.677 [DEBUG] TestAgent.server.raft: vote granted: from=2fe7f7bf-5557-49be-ea1f-f899620817a7 term=2 tally=1 +30:14.677 [INFO] TestAgent.server.raft: election won: tally=1 +30:14.677 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29951 [Leader]" +30:14.677 [INFO] TestAgent.server: cluster leadership acquired +30:14.677 [INFO] TestAgent.server: New leader elected: payload=Node-2fe7f7bf-5557-49be-ea1f-f899620817a7 +30:14.700 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29951 +30:14.748 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:14.806 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:14.806 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:14.806 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:14.806 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:14.806 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:14.806 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fe7f7bf-5557-49be-ea1f-f899620817a7 +30:14.806 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2fe7f7bf-5557-49be-ea1f-f899620817a7 +30:14.823 [INFO] TestAgent.server: federation state anti-entropy synced +30:14.844 [WARN] TestAgent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=my_service +30:14.844 [INFO] TestAgent: Requesting shutdown +30:14.844 [INFO] TestAgent.server: shutting down server +30:14.844 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:14.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:14.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:14.844 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:14.844 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:14.844 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:14.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:14.844 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:14.844 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:14.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:14.844 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:14.856 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:14.859Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:14.859Z [INFO] agent.router.manager: shutting down +30:14.859 [INFO] TestAgent: consul server down +30:14.859 [INFO] TestAgent: shutdown complete +30:14.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29946 network=tcp +30:14.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29946 network=udp +30:14.860 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29947 network=tcp +30:15.360 [INFO] TestAgent: Waiting for endpoints to shut down +30:15.360 [INFO] TestAgent: Endpoints down +--- PASS: TestAddServiceIPv6TaggedSet (0.86s) === RUN TestAgent_RemoveService === RUN TestAgent_RemoveService/normal === PAUSE TestAgent_RemoveService/normal @@ -11295,118 +11508,137 @@ [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:11.793Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:11.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:11.845 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:34a6dd26-970e-264b-09a3-60d90b130922 Address:127.0.0.1:11580}]" -45:11.845 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11580 [Follower]" leader= -45:11.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-34a6dd26-970e-264b-09a3-60d90b130922.dc1 127.0.0.1 -45:11.846 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-34a6dd26-970e-264b-09a3-60d90b130922 127.0.0.1 -2022-11-07T22:45:11.846Z [INFO] agent.router: Initializing LAN area manager -45:11.846 [INFO] TestAgent.server: Adding LAN server: server="Node-34a6dd26-970e-264b-09a3-60d90b130922 (Addr: tcp/127.0.0.1:11580) (DC: dc1)" -45:11.846 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-34a6dd26-970e-264b-09a3-60d90b130922.dc1 area=wan -45:11.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11575 network=udp -45:11.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11575 network=tcp -45:11.847 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11576 network=tcp -45:11.847 [INFO] TestAgent: started state syncer -45:11.847 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11581 network=tcp -45:11.887 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:11.887 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11580 [Candidate]" term=2 -45:11.897 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:11.897 [DEBUG] TestAgent.server.raft: vote granted: from=34a6dd26-970e-264b-09a3-60d90b130922 term=2 tally=1 -45:11.897 [INFO] TestAgent.server.raft: election won: tally=1 -45:11.897 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11580 [Leader]" -45:11.897 [INFO] TestAgent.server: cluster leadership acquired -45:11.897 [INFO] TestAgent.server: New leader elected: payload=Node-34a6dd26-970e-264b-09a3-60d90b130922 -45:11.904 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11580 -45:11.907 [WARN] TestAgent: Failed to deregister service: service=redis error="Service {"redis" {}} does not exist" -45:11.908 [DEBUG] TestAgent: removed check: check=service:memcache -45:11.908 [DEBUG] TestAgent: removed check: check=check2 -45:11.908 [DEBUG] TestAgent: removed service: service=memcache -45:11.908 [DEBUG] TestAgent: removed check: check=service:redis:2 -45:11.908 [DEBUG] TestAgent: removed check: check=service:redis:1 -45:11.908 [DEBUG] TestAgent: removed service: service=redis -45:11.908 [INFO] TestAgent: Requesting shutdown -45:11.908 [INFO] TestAgent.server: shutting down server -45:11.908 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:11.908 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:11.914 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:11.920 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:45:11.921Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:11.921Z [INFO] agent.router.manager: shutting down -45:11.930 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -45:11.930 [INFO] TestAgent: consul server down -45:11.930 [INFO] TestAgent: shutdown complete -45:11.930 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11575 network=tcp -45:11.930 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -45:11.930 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11575 network=udp -45:11.930 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11576 network=tcp -45:12.430 [INFO] TestAgent: Waiting for endpoints to shut down -45:12.430 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:11.794Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:11.809Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:11.848 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2697a9a1-8860-ff58-6d3c-e30712a112a4 Address:127.0.0.1:11587}]" -45:11.848 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11587 [Follower]" leader= -45:11.848 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2697a9a1-8860-ff58-6d3c-e30712a112a4.dc1 127.0.0.1 -45:11.849 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2697a9a1-8860-ff58-6d3c-e30712a112a4 127.0.0.1 -2022-11-07T22:45:11.849Z [INFO] agent.router: Initializing LAN area manager -45:11.849 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2697a9a1-8860-ff58-6d3c-e30712a112a4.dc1 area=wan -45:11.849 [INFO] TestAgent.server: Adding LAN server: server="Node-2697a9a1-8860-ff58-6d3c-e30712a112a4 (Addr: tcp/127.0.0.1:11587) (DC: dc1)" -45:11.849 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11582 network=tcp -45:11.849 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11582 network=udp -45:11.849 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11583 network=tcp -45:11.849 [INFO] TestAgent: started state syncer -45:11.849 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11588 network=tcp -45:11.901 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:11.901 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11587 [Candidate]" term=2 -45:11.910 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:11.910 [DEBUG] TestAgent.server.raft: vote granted: from=2697a9a1-8860-ff58-6d3c-e30712a112a4 term=2 tally=1 -45:11.910 [INFO] TestAgent.server.raft: election won: tally=1 -45:11.910 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11587 [Leader]" -45:11.910 [INFO] TestAgent.server: cluster leadership acquired -45:11.910 [INFO] TestAgent.server: New leader elected: payload=Node-2697a9a1-8860-ff58-6d3c-e30712a112a4 -45:11.918 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11587 -45:11.935 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:11.960 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:11.960 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:11.960 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:11.960 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:11.960 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:11.960 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2697a9a1-8860-ff58-6d3c-e30712a112a4 -45:11.960 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2697a9a1-8860-ff58-6d3c-e30712a112a4 -45:11.967 [INFO] TestAgent.server: federation state anti-entropy synced -45:11.979 [WARN] TestAgent: Failed to deregister service: service=redis error="Service {"redis" {}} does not exist" -45:11.979 [DEBUG] TestAgent: removed check: check=service:memcache -45:11.980 [DEBUG] TestAgent: removed check: check=check2 -45:11.980 [DEBUG] TestAgent: removed service: service=memcache -45:11.980 [DEBUG] TestAgent: removed check: check=service:redis:1 -45:11.980 [DEBUG] TestAgent: removed check: check=service:redis:2 -45:11.980 [DEBUG] TestAgent: removed service: service=redis -45:11.980 [INFO] TestAgent: Requesting shutdown -45:11.980 [INFO] TestAgent.server: shutting down server -45:11.980 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:11.980 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:11.980 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:11.980 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:11.980 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:11.980 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:11.980 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:11.980 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:11.980 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:11.980 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:11.980 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:11.984 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:11.990Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:11.990Z [INFO] agent.router.manager: shutting down -45:11.990 [INFO] TestAgent: consul server down -45:11.990 [INFO] TestAgent: shutdown complete -45:11.990 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11582 network=tcp -45:11.990 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11582 network=udp -45:11.990 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11583 network=tcp -45:12.490 [INFO] TestAgent: Waiting for endpoints to shut down -45:12.490 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:15.405Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:15.447Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:15.497 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:462625e0-a520-f2cc-79bc-ead1bee61e07 Address:127.0.0.1:29965}]" +30:15.498 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-462625e0-a520-f2cc-79bc-ead1bee61e07.dc1 127.0.0.1 +30:15.498 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-462625e0-a520-f2cc-79bc-ead1bee61e07 127.0.0.1 +2021-10-05T16:30:15.498Z [INFO] agent.router: Initializing LAN area manager +30:15.499 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29960 network=udp +30:15.499 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29965 [Follower]" leader= +30:15.499 [INFO] TestAgent.server: Adding LAN server: server="Node-462625e0-a520-f2cc-79bc-ead1bee61e07 (Addr: tcp/127.0.0.1:29965) (DC: dc1)" +30:15.499 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-462625e0-a520-f2cc-79bc-ead1bee61e07.dc1 area=wan +30:15.499 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29960 network=tcp +30:15.500 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29961 network=tcp +30:15.500 [INFO] TestAgent: started state syncer +30:15.500 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29966 network=tcp +30:15.562 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:15.562 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29965 [Candidate]" term=2 +30:15.574 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:15.574 [DEBUG] TestAgent.server.raft: vote granted: from=462625e0-a520-f2cc-79bc-ead1bee61e07 term=2 tally=1 +30:15.574 [INFO] TestAgent.server.raft: election won: tally=1 +30:15.574 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29965 [Leader]" +30:15.574 [INFO] TestAgent.server: cluster leadership acquired +30:15.574 [INFO] TestAgent.server: New leader elected: payload=Node-462625e0-a520-f2cc-79bc-ead1bee61e07 +30:15.582 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29965 +30:15.600 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:15.635 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:15.635 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:15.635 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:15.635 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:15.635 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:15.635 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-462625e0-a520-f2cc-79bc-ead1bee61e07 +30:15.635 [INFO] TestAgent.server: member joined, marking health alive: member=Node-462625e0-a520-f2cc-79bc-ead1bee61e07 +30:15.660 [INFO] TestAgent.server: federation state anti-entropy synced +30:15.863 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:15.868 [INFO] TestAgent: Synced node info +30:15.868 [DEBUG] TestAgent: Node info in sync +30:15.871 [WARN] TestAgent: Failed to deregister service: service=redis error="Service {"redis" {}} does not exist" +30:15.872 [DEBUG] TestAgent: removed check: check=service:memcache +30:15.872 [DEBUG] TestAgent: removed check: check=check2 +30:15.872 [DEBUG] TestAgent: removed service: service=memcache +30:15.872 [DEBUG] TestAgent: removed check: check=service:redis:1 +30:15.872 [DEBUG] TestAgent: removed check: check=service:redis:2 +30:15.872 [DEBUG] TestAgent: removed service: service=redis +30:15.872 [INFO] TestAgent: Requesting shutdown +30:15.872 [INFO] TestAgent.server: shutting down server +30:15.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:15.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:15.872 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:15.872 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:15.872 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:15.872 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:15.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:15.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:15.872 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:15.872 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:15.878 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:15.883Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:15.883Z [INFO] agent.router.manager: shutting down +30:15.883 [INFO] TestAgent: consul server down +30:15.883 [INFO] TestAgent: shutdown complete +30:15.883 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29960 network=tcp +30:15.883 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29960 network=udp +30:15.883 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29961 network=tcp +30:16.384 [INFO] TestAgent: Waiting for endpoints to shut down +30:16.384 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:15.373Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:15.416Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:15.481 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b1d61526-2e55-a394-6052-352da60921ec Address:127.0.0.1:29958}]" +30:15.482 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b1d61526-2e55-a394-6052-352da60921ec.dc1 127.0.0.1 +30:15.482 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b1d61526-2e55-a394-6052-352da60921ec 127.0.0.1 +2021-10-05T16:30:15.482Z [INFO] agent.router: Initializing LAN area manager +30:15.483 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29953 network=udp +30:15.483 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29958 [Follower]" leader= +30:15.483 [INFO] TestAgent.server: Adding LAN server: server="Node-b1d61526-2e55-a394-6052-352da60921ec (Addr: tcp/127.0.0.1:29958) (DC: dc1)" +30:15.483 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b1d61526-2e55-a394-6052-352da60921ec.dc1 area=wan +30:15.483 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29953 network=tcp +30:15.484 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29954 network=tcp +30:15.484 [INFO] TestAgent: started state syncer +30:15.484 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29959 network=tcp +30:15.552 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:15.552 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29958 [Candidate]" term=2 +30:15.575 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:15.575 [DEBUG] TestAgent.server.raft: vote granted: from=b1d61526-2e55-a394-6052-352da60921ec term=2 tally=1 +30:15.575 [INFO] TestAgent.server.raft: election won: tally=1 +30:15.575 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29958 [Leader]" +30:15.575 [INFO] TestAgent.server: cluster leadership acquired +30:15.575 [INFO] TestAgent.server: New leader elected: payload=Node-b1d61526-2e55-a394-6052-352da60921ec +30:15.583 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29958 +30:15.612 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:15.638 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:15.638 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:15.638 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:15.638 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:15.638 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:15.638 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1d61526-2e55-a394-6052-352da60921ec +30:15.638 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b1d61526-2e55-a394-6052-352da60921ec +30:15.676 [INFO] TestAgent.server: federation state anti-entropy synced +30:15.860 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:15.865 [INFO] TestAgent: Synced node info +30:15.907 [WARN] TestAgent: Failed to deregister service: service=redis error="Service {"redis" {}} does not exist" +30:15.908 [DEBUG] TestAgent: removed check: check=check2 +30:15.908 [DEBUG] TestAgent: removed check: check=service:memcache +30:15.908 [DEBUG] TestAgent: removed service: service=memcache +30:15.908 [DEBUG] TestAgent: removed check: check=service:redis:1 +30:15.908 [DEBUG] TestAgent: removed check: check=service:redis:2 +30:15.908 [DEBUG] TestAgent: removed service: service=redis +30:15.908 [INFO] TestAgent: Requesting shutdown +30:15.908 [INFO] TestAgent.server: shutting down server +30:15.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:15.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:15.908 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:15.908 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:15.908 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:15.908 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:15.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:15.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:15.908 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:15.908 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:15.915 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:15.920Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:15.920Z [INFO] agent.router.manager: shutting down +30:15.920 [INFO] TestAgent: consul server down +30:15.920 [INFO] TestAgent: shutdown complete +30:15.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29953 network=tcp +30:15.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29953 network=udp +30:15.920 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29954 network=tcp +30:16.424 [INFO] TestAgent: Waiting for endpoints to shut down +30:16.424 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RemoveService (0.00s) - --- PASS: TestAgent_RemoveService/normal (0.65s) - --- PASS: TestAgent_RemoveService/service_manager (0.71s) + --- PASS: TestAgent_RemoveService/service_manager (1.02s) + --- PASS: TestAgent_RemoveService/normal (1.06s) === RUN TestAgent_RemoveServiceRemovesAllChecks === RUN TestAgent_RemoveServiceRemovesAllChecks/normal === PAUSE TestAgent_RemoveServiceRemovesAllChecks/normal @@ -11417,128 +11649,110 @@ [WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:12.503Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:12.512Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:12.550 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b1eb2e64-b093-cb58-c686-08452b8c93a8 Address:127.0.0.1:11601}]" -45:12.550 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11601 [Follower]" leader= -45:12.551 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:12.551 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:12.551Z [INFO] agent.router: Initializing LAN area manager -45:12.551 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11601) (DC: dc1)" -45:12.551 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:12.552 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11596 network=udp -45:12.552 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11596 network=tcp -45:12.552 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11597 network=tcp -45:12.552 [INFO] TestAgent: started state syncer -45:12.552 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11602 network=tcp -45:12.600 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:12.600 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11601 [Candidate]" term=2 -45:12.618 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:12.618 [DEBUG] TestAgent.server.raft: vote granted: from=b1eb2e64-b093-cb58-c686-08452b8c93a8 term=2 tally=1 -45:12.618 [INFO] TestAgent.server.raft: election won: tally=1 -45:12.618 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11601 [Leader]" -45:12.618 [INFO] TestAgent.server: cluster leadership acquired -45:12.618 [INFO] TestAgent.server: New leader elected: payload=node1 -45:12.627 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11601 -45:12.643 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:12.667 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:12.667 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:12.667 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:12.667 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:12.667 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:12.667 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:12.667 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:12.682 [INFO] TestAgent.server: federation state anti-entropy synced -45:12.781 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:12.785 [INFO] TestAgent: Synced node info -45:12.835 [DEBUG] TestAgent: removed check: check=chk1 -45:12.835 [DEBUG] TestAgent: removed check: check=chk2 -45:12.835 [DEBUG] TestAgent: removed service: service=redis -45:12.835 [INFO] TestAgent: Requesting shutdown -45:12.835 [INFO] TestAgent.server: shutting down server -45:12.835 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:12.835 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:12.835 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:12.835 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:12.835 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:12.835 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:12.835 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:12.835 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:12.835 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:12.835 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:12.839 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:12.843Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:12.843Z [INFO] agent.router.manager: shutting down -45:12.843 [INFO] TestAgent: consul server down -45:12.843 [INFO] TestAgent: shutdown complete -45:12.843 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11596 network=tcp -45:12.843 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11596 network=udp -45:12.843 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11597 network=tcp -45:13.343 [INFO] TestAgent: Waiting for endpoints to shut down -45:13.343 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:12.503Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:12.512Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:12.553 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b352496c-6992-a294-212b-234fcd04d8f1 Address:127.0.0.1:11594}]" -45:12.553 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11594 [Follower]" leader= -45:12.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 -45:12.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 -2022-11-07T22:45:12.554Z [INFO] agent.router: Initializing LAN area manager -45:12.554 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:11594) (DC: dc1)" -45:12.554 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan -45:12.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11589 network=udp -45:12.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11589 network=tcp -45:12.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11590 network=tcp -45:12.554 [INFO] TestAgent: started state syncer -45:12.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11595 network=tcp -45:12.589 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:12.589 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11594 [Candidate]" term=2 -45:12.602 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:12.602 [DEBUG] TestAgent.server.raft: vote granted: from=b352496c-6992-a294-212b-234fcd04d8f1 term=2 tally=1 -45:12.602 [INFO] TestAgent.server.raft: election won: tally=1 -45:12.602 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11594 [Leader]" -45:12.602 [INFO] TestAgent.server: cluster leadership acquired -45:12.602 [INFO] TestAgent.server: New leader elected: payload=node1 -45:12.614 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11594 -45:12.631 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:12.657 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:12.657 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:12.657 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:12.657 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:12.657 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:12.657 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 -45:12.657 [INFO] TestAgent.server: member joined, marking health alive: member=node1 -45:12.670 [INFO] TestAgent.server: federation state anti-entropy synced -45:12.906 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:12.909 [INFO] TestAgent: Synced node info -45:12.991 [DEBUG] TestAgent: removed check: check=chk2 -45:12.991 [DEBUG] TestAgent: removed check: check=chk1 -45:12.991 [DEBUG] TestAgent: removed service: service=redis -45:12.991 [INFO] TestAgent: Requesting shutdown -45:12.991 [INFO] TestAgent.server: shutting down server -45:12.991 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:12.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:12.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:12.991 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:12.991 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:12.991 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:12.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:12.991 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:12.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:12.991 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:12.998 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:13.002Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:13.002Z [INFO] agent.router.manager: shutting down -45:13.002 [INFO] TestAgent: consul server down -45:13.002 [INFO] TestAgent: shutdown complete -45:13.002 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11589 network=tcp -45:13.002 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11589 network=udp -45:13.002 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11590 network=tcp -45:13.502 [INFO] TestAgent: Waiting for endpoints to shut down -45:13.502 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:16.458Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:16.472Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:16.518 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:baff1461-a458-f6f4-025e-02293d017b05 Address:127.0.0.1:29972}]" +30:16.519 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:16.519 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29972 [Follower]" leader= +30:16.519 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:16.519Z [INFO] agent.router: Initializing LAN area manager +30:16.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29967 network=udp +30:16.520 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29972) (DC: dc1)" +30:16.520 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:16.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29967 network=tcp +30:16.520 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29968 network=tcp +30:16.521 [INFO] TestAgent: started state syncer +30:16.521 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29973 network=tcp +30:16.570 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:16.570 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29972 [Candidate]" term=2 +30:16.616 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:16.616 [DEBUG] TestAgent.server.raft: vote granted: from=baff1461-a458-f6f4-025e-02293d017b05 term=2 tally=1 +30:16.616 [INFO] TestAgent.server.raft: election won: tally=1 +30:16.616 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29972 [Leader]" +30:16.616 [INFO] TestAgent.server: cluster leadership acquired +30:16.616 [INFO] TestAgent.server: New leader elected: payload=node1 +30:16.638 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29972 +30:16.674 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:16.684 [DEBUG] TestAgent: removed check: check=chk1 +30:16.684 [DEBUG] TestAgent: removed check: check=chk2 +30:16.684 [DEBUG] TestAgent: removed service: service=redis +30:16.684 [INFO] TestAgent: Requesting shutdown +30:16.684 [INFO] TestAgent.server: shutting down server +30:16.684 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:16.684 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:16.692 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:16.700Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:16.700Z [INFO] agent.router.manager: shutting down +30:16.704 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +30:16.704 [INFO] TestAgent: consul server down +30:16.704 [INFO] TestAgent: shutdown complete +30:16.704 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29967 network=tcp +30:16.704 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29967 network=udp +30:16.704 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29968 network=tcp +30:17.208 [INFO] TestAgent: Waiting for endpoints to shut down +30:17.208 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:16.454Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:16.487Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:16.576 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:aba7d5e0-b2f7-01a7-d002-a2272faf2735 Address:127.0.0.1:29979}]" +30:16.576 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29979 [Follower]" leader= +30:16.577 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: node1.dc1 127.0.0.1 +30:16.577 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: node1 127.0.0.1 +2021-10-05T16:30:16.577Z [INFO] agent.router: Initializing LAN area manager +30:16.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29974 network=udp +30:16.577 [INFO] TestAgent.server: Adding LAN server: server="node1 (Addr: tcp/127.0.0.1:29979) (DC: dc1)" +30:16.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29974 network=tcp +30:16.578 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=node1.dc1 area=wan +30:16.578 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29975 network=tcp +30:16.578 [INFO] TestAgent: started state syncer +30:16.578 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29980 network=tcp +30:16.626 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:16.626 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29979 [Candidate]" term=2 +30:16.637 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:16.637 [DEBUG] TestAgent.server.raft: vote granted: from=aba7d5e0-b2f7-01a7-d002-a2272faf2735 term=2 tally=1 +30:16.637 [INFO] TestAgent.server.raft: election won: tally=1 +30:16.637 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29979 [Leader]" +30:16.638 [INFO] TestAgent.server: cluster leadership acquired +30:16.638 [INFO] TestAgent.server: New leader elected: payload=node1 +30:16.664 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29979 +30:16.709 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:16.776 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:16.776 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:16.776 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:16.776 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:16.776 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:16.776 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=node1 +30:16.776 [INFO] TestAgent.server: member joined, marking health alive: member=node1 +30:16.790 [INFO] TestAgent.server: federation state anti-entropy synced +30:16.834 [DEBUG] TestAgent: removed check: check=chk2 +30:16.834 [DEBUG] TestAgent: removed check: check=chk1 +30:16.836 [DEBUG] TestAgent: removed service: service=redis +30:16.836 [INFO] TestAgent: Requesting shutdown +30:16.848 [INFO] TestAgent.server: shutting down server +30:16.848 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:16.848 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:16.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:16.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:16.848 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:16.848 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:16.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:16.848 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:16.848 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:16.848 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:16.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:16.864 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:16.871Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:16.871Z [INFO] agent.router.manager: shutting down +30:16.871 [INFO] TestAgent: consul server down +30:16.871 [INFO] TestAgent: shutdown complete +30:16.871 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29974 network=tcp +30:16.871 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29974 network=udp +30:16.872 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29975 network=tcp +30:17.372 [INFO] TestAgent: Waiting for endpoints to shut down +30:17.372 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_RemoveServiceRemovesAllChecks (0.00s) - --- PASS: TestAgent_RemoveServiceRemovesAllChecks/service_manager (0.85s) - --- PASS: TestAgent_RemoveServiceRemovesAllChecks/normal (1.01s) + --- PASS: TestAgent_RemoveServiceRemovesAllChecks/normal (0.78s) + --- PASS: TestAgent_RemoveServiceRemovesAllChecks/service_manager (0.94s) === RUN TestAgent_IndexChurn === PAUSE TestAgent_IndexChurn === RUN TestAgent_AddCheck @@ -11584,95 +11798,95 @@ === PAUSE TestAgent_PersistService/service_manager === CONT TestAgent_PersistService/normal === CONT TestAgent_PersistService/service_manager -[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -45:14.069 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-391c10b5-a3de-0619-66cd-04d0c153e82e 127.0.0.1 -2022-11-07T22:45:14.069Z [INFO] agent.router: Initializing LAN area manager -45:14.069 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-df195187-6a92-45a6-f2c9-3c4894b778cd: 127.0.0.1:11613 -45:14.069 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_PersistService_service_manager-TestAgent-agent054200479/services/86a1b907d54bf7010394bf316e183e67 -45:14.069 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11613: connect: connection refused -45:14.069 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node -45:14.069 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11617 network=tcp -45:14.069 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11617 network=udp -45:14.070 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11618 network=tcp -45:14.070 [INFO] TestAgent: started state syncer -45:14.070 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11623 network=tcp -2022-11-07T22:45:14.070Z [WARN] agent.router.manager: No servers available -45:14.070 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:14.070 [INFO] TestAgent: Requesting shutdown -45:14.070 [INFO] TestAgent.client: shutting down client -45:14.070 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:14.079 [INFO] TestAgent: consul client down -45:14.079 [INFO] TestAgent: shutdown complete -45:14.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11617 network=tcp -45:14.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11617 network=udp -45:14.079 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11618 network=tcp -45:14.579 [INFO] TestAgent: Waiting for endpoints to shut down -45:14.579 [INFO] TestAgent: Endpoints down -45:13.524 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-df195187-6a92-45a6-f2c9-3c4894b778cd 127.0.0.1 -2022-11-07T22:45:13.524Z [INFO] agent.router: Initializing LAN area manager -45:13.525 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11610 network=udp -45:13.525 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11610 network=tcp -45:13.525 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11611 network=tcp -45:13.525 [INFO] TestAgent: started state syncer -45:13.526 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11616 network=tcp -45:13.534 [INFO] TestAgent: Requesting shutdown -45:13.534 [INFO] TestAgent.client: shutting down client -45:13.534 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:13.537 [INFO] TestAgent: consul client down -45:13.537 [INFO] TestAgent: shutdown complete -45:13.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11610 network=tcp -45:13.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11610 network=udp -45:13.537 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11611 network=tcp -45:14.038 [INFO] TestAgent: Waiting for endpoints to shut down -45:14.038 [INFO] TestAgent: Endpoints down -45:14.076 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-6cd42618-79a2-8d5d-c1f9-aff35ddb94f0 127.0.0.1 -2022-11-07T22:45:14.076Z [INFO] agent.router: Initializing LAN area manager -45:14.076 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-90d57dd7-db73-1afa-6992-5366920233ad: 127.0.0.1:11606 -45:14.077 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_PersistService_normal-TestAgent-agent233294912/services/86a1b907d54bf7010394bf316e183e67 -45:14.077 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11606: connect: connection refused -45:14.077 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node -45:14.077 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11624 network=tcp -45:14.077 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11624 network=udp -45:14.077 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11625 network=tcp -45:14.077 [INFO] TestAgent: started state syncer -45:14.077 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11630 network=tcp -2022-11-07T22:45:14.077Z [WARN] agent.router.manager: No servers available -45:14.077 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:14.078 [INFO] TestAgent: Requesting shutdown -45:14.078 [INFO] TestAgent.client: shutting down client -45:14.078 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:14.087 [INFO] TestAgent: consul client down -45:14.087 [INFO] TestAgent: shutdown complete -45:14.087 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11624 network=tcp -45:14.087 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11624 network=udp -45:14.087 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11625 network=tcp -45:14.587 [INFO] TestAgent: Waiting for endpoints to shut down -45:14.587 [INFO] TestAgent: Endpoints down -45:13.523 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-90d57dd7-db73-1afa-6992-5366920233ad 127.0.0.1 -2022-11-07T22:45:13.523Z [INFO] agent.router: Initializing LAN area manager -45:13.523 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11603 network=udp -45:13.523 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11603 network=tcp -45:13.523 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11604 network=tcp -45:13.523 [INFO] TestAgent: started state syncer -45:13.523 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11609 network=tcp -2022-11-07T22:45:13.524Z [WARN] agent.router.manager: No servers available -45:13.524 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:13.534 [INFO] TestAgent: Requesting shutdown -45:13.534 [INFO] TestAgent.client: shutting down client -45:13.534 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:13.542 [INFO] TestAgent: consul client down -45:13.542 [INFO] TestAgent: shutdown complete -45:13.542 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11603 network=tcp -45:13.542 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11603 network=udp -45:13.542 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11604 network=tcp -45:14.042 [INFO] TestAgent: Waiting for endpoints to shut down -45:14.042 [INFO] TestAgent: Endpoints down +30:18.002 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-b6846e22-868e-cc82-3128-88fda72ac395 127.0.0.1 +2021-10-05T16:30:18.002Z [INFO] agent.router: Initializing LAN area manager +30:18.003 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_PersistService_service_manager-TestAgent-agent201364860/services/86a1b907d54bf7010394bf316e183e67 +30:18.003 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29995 network=udp +30:18.003 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-e87d8e9b-2610-66b6-5ea1-410343a6be4f: 127.0.0.1:29991 +30:18.003 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29995 network=tcp +30:18.004 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:29991: connect: connection refused +30:18.004 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node +30:18.012 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29996 network=tcp +30:18.012 [INFO] TestAgent: started state syncer +30:18.012 [INFO] TestAgent: Requesting shutdown +30:18.012 [INFO] TestAgent.client: shutting down client +30:18.012 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:18.012 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30001 network=tcp +30:18.012 [ERROR] TestAgent: gRPC server failed: error="grpc: the server has been stopped" +2021-10-05T16:30:18.012Z [WARN] agent.router.manager: No servers available +30:18.012 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:18.020 [INFO] TestAgent: consul client down +30:18.020 [INFO] TestAgent: shutdown complete +30:18.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29995 network=tcp +30:18.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29995 network=udp +30:18.020 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29996 network=tcp +30:18.520 [INFO] TestAgent: Waiting for endpoints to shut down +30:18.520 [INFO] TestAgent: Endpoints down +30:17.413 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-e87d8e9b-2610-66b6-5ea1-410343a6be4f 127.0.0.1 +2021-10-05T16:30:17.413Z [INFO] agent.router: Initializing LAN area manager +30:17.413 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29988 network=udp +30:17.413 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29988 network=tcp +30:17.414 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29989 network=tcp +30:17.414 [INFO] TestAgent: started state syncer +30:17.416 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29994 network=tcp +30:17.432 [INFO] TestAgent: Requesting shutdown +30:17.432 [INFO] TestAgent.client: shutting down client +30:17.432 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:17.441 [INFO] TestAgent: consul client down +30:17.441 [INFO] TestAgent: shutdown complete +30:17.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29988 network=tcp +30:17.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29988 network=udp +30:17.441 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29989 network=tcp +30:17.944 [INFO] TestAgent: Waiting for endpoints to shut down +30:17.944 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! +30:18.078 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-e8140227-0a8b-ea0c-c0a6-c608b4465a58 127.0.0.1 +2021-10-05T16:30:18.078Z [INFO] agent.router: Initializing LAN area manager +30:18.079 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_PersistService_normal-TestAgent-agent093397585/services/86a1b907d54bf7010394bf316e183e67 +30:18.079 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30002 network=udp +30:18.079 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-b952d61f-8f07-a94d-017d-adc4acc42a6a: 127.0.0.1:29984 +30:18.079 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:29984: connect: connection refused +30:18.079 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node +30:18.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30002 network=tcp +30:18.082 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30003 network=tcp +30:18.082 [INFO] TestAgent: started state syncer +30:18.086 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30008 network=tcp +2021-10-05T16:30:18.088Z [WARN] agent.router.manager: No servers available +30:18.088 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:18.088 [INFO] TestAgent: Requesting shutdown +30:18.088 [INFO] TestAgent.client: shutting down client +30:18.088 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:18.100 [INFO] TestAgent: consul client down +30:18.100 [INFO] TestAgent: shutdown complete +30:18.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30002 network=tcp +30:18.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30002 network=udp +30:18.100 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30003 network=tcp +30:18.601 [INFO] TestAgent: Waiting for endpoints to shut down +30:18.601 [INFO] TestAgent: Endpoints down +30:17.419 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-b952d61f-8f07-a94d-017d-adc4acc42a6a 127.0.0.1 +2021-10-05T16:30:17.419Z [INFO] agent.router: Initializing LAN area manager +30:17.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29981 network=udp +30:17.420 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29981 network=tcp +30:17.420 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29982 network=tcp +30:17.420 [INFO] TestAgent: started state syncer +30:17.428 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29987 network=tcp +30:17.457 [INFO] TestAgent: Requesting shutdown +30:17.457 [INFO] TestAgent.client: shutting down client +30:17.457 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:17.495 [INFO] TestAgent: consul client down +30:17.495 [INFO] TestAgent: shutdown complete +30:17.495 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29981 network=tcp +30:17.495 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29981 network=udp +30:17.495 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29982 network=tcp +30:17.996 [INFO] TestAgent: Waiting for endpoints to shut down +30:17.996 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_PersistService (0.00s) - --- PASS: TestAgent_PersistService/service_manager (1.07s) - --- PASS: TestAgent_PersistService/normal (1.08s) + --- PASS: TestAgent_PersistService/service_manager (1.15s) + --- PASS: TestAgent_PersistService/normal (1.23s) === RUN TestAgent_persistedService_compat === RUN TestAgent_persistedService_compat/normal === PAUSE TestAgent_persistedService_compat/normal @@ -11680,126 +11894,126 @@ === PAUSE TestAgent_persistedService_compat/service_manager === CONT TestAgent_persistedService_compat/normal === CONT TestAgent_persistedService_compat/service_manager -[WARN] freeport: 6 out of 34 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:14.599Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:14.608Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:14.647 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb Address:127.0.0.1:11643}]" -45:14.647 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11643 [Follower]" leader= -45:14.647 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb.dc1 127.0.0.1 -45:14.649 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb 127.0.0.1 -2022-11-07T22:45:14.649Z [INFO] agent.router: Initializing LAN area manager -45:14.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11638 network=udp -45:14.650 [INFO] TestAgent.server: Adding LAN server: server="Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb (Addr: tcp/127.0.0.1:11643) (DC: dc1)" -45:14.650 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb.dc1 area=wan -45:14.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11638 network=tcp -45:14.651 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11639 network=tcp -45:14.651 [INFO] TestAgent: started state syncer -45:14.651 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11644 network=tcp -45:14.702 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:14.702 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11643 [Candidate]" term=2 -45:14.720 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:14.720 [DEBUG] TestAgent.server.raft: vote granted: from=11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb term=2 tally=1 -45:14.720 [INFO] TestAgent.server.raft: election won: tally=1 -45:14.720 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11643 [Leader]" -45:14.720 [INFO] TestAgent.server: cluster leadership acquired -45:14.720 [INFO] TestAgent.server: New leader elected: payload=Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb -45:14.731 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11643 -45:14.752 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:14.779 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:14.779 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:14.779 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:14.779 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:14.779 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:14.779 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb -45:14.779 [INFO] TestAgent.server: member joined, marking health alive: member=Node-11247e5e-ae86-4aa1-5ff6-e8bbea8cefdb -45:14.800 [INFO] TestAgent.server: federation state anti-entropy synced -45:14.801 [INFO] TestAgent: Synced node info -45:14.807 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_persistedService_compat_service_manager-TestAgent-agent126234665/services/86a1b907d54bf7010394bf316e183e67 -45:14.807 [INFO] TestAgent: Requesting shutdown -45:14.807 [INFO] TestAgent.server: shutting down server -45:14.807 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:14.807 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:14.807 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:14.807 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:14.807 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:14.807 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:14.807 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:14.807 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:14.807 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:14.807 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:14.811 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:14.816Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:14.816Z [INFO] agent.router.manager: shutting down -45:14.816 [INFO] TestAgent: consul server down -45:14.816 [INFO] TestAgent: shutdown complete -45:14.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11638 network=tcp -45:14.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11638 network=udp -45:14.816 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11639 network=tcp -45:15.316 [INFO] TestAgent: Waiting for endpoints to shut down -45:15.316 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:18.623Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:18.650Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:18.710 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:322488e7-31fc-a45d-c9ed-3857fc8cad6c Address:127.0.0.1:30021}]" +30:18.711 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c.dc1 127.0.0.1 +30:18.712 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c 127.0.0.1 +2021-10-05T16:30:18.712Z [INFO] agent.router: Initializing LAN area manager +30:18.712 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30021 [Follower]" leader= +30:18.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30016 network=udp +30:18.712 [INFO] TestAgent.server: Adding LAN server: server="Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c (Addr: tcp/127.0.0.1:30021) (DC: dc1)" +30:18.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30016 network=tcp +30:18.712 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c.dc1 area=wan +30:18.713 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30017 network=tcp +30:18.713 [INFO] TestAgent: started state syncer +30:18.713 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30022 network=tcp +30:18.760 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:18.760 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30021 [Candidate]" term=2 +30:18.799 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:18.799 [DEBUG] TestAgent.server.raft: vote granted: from=322488e7-31fc-a45d-c9ed-3857fc8cad6c term=2 tally=1 +30:18.799 [INFO] TestAgent.server.raft: election won: tally=1 +30:18.799 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30021 [Leader]" +30:18.800 [INFO] TestAgent.server: cluster leadership acquired +30:18.800 [INFO] TestAgent.server: New leader elected: payload=Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c +30:18.807 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30021 +30:18.824 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:18.880 [INFO] TestAgent: Synced node info +30:18.880 [DEBUG] TestAgent: Node info in sync +30:18.880 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:18.880 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:18.880 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:18.880 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:18.880 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:18.880 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c +30:18.880 [INFO] TestAgent.server: member joined, marking health alive: member=Node-322488e7-31fc-a45d-c9ed-3857fc8cad6c +30:18.896 [INFO] TestAgent.server: federation state anti-entropy synced +30:18.949 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_persistedService_compat_service_manager-TestAgent-agent886600206/services/86a1b907d54bf7010394bf316e183e67 +30:18.949 [INFO] TestAgent: Requesting shutdown +30:18.949 [INFO] TestAgent.server: shutting down server +30:18.949 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:18.949 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:18.949 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:18.949 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:18.949 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:18.949 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:18.949 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:18.950 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:18.950 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:18.950 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:18.956 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:18.969Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:18.969Z [INFO] agent.router.manager: shutting down +30:18.969 [INFO] TestAgent: consul server down +30:18.969 [INFO] TestAgent: shutdown complete +30:18.969 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30016 network=tcp +30:18.969 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30016 network=udp +30:18.969 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30017 network=tcp +30:19.472 [INFO] TestAgent: Waiting for endpoints to shut down +30:19.472 [INFO] TestAgent: Endpoints down [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:14.602Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:14.611Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:14.651 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:500cf7be-f79f-57ec-d739-5338ae080b7c Address:127.0.0.1:11636}]" -45:14.651 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11636 [Follower]" leader= -45:14.651 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-500cf7be-f79f-57ec-d739-5338ae080b7c.dc1 127.0.0.1 -45:14.652 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-500cf7be-f79f-57ec-d739-5338ae080b7c 127.0.0.1 -2022-11-07T22:45:14.652Z [INFO] agent.router: Initializing LAN area manager -45:14.652 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-500cf7be-f79f-57ec-d739-5338ae080b7c.dc1 area=wan -45:14.652 [INFO] TestAgent.server: Adding LAN server: server="Node-500cf7be-f79f-57ec-d739-5338ae080b7c (Addr: tcp/127.0.0.1:11636) (DC: dc1)" -45:14.652 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11631 network=udp -45:14.652 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11631 network=tcp -45:14.653 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11632 network=tcp -45:14.653 [INFO] TestAgent: started state syncer -45:14.653 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11637 network=tcp -45:14.705 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:14.705 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11636 [Candidate]" term=2 -45:14.721 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:14.721 [DEBUG] TestAgent.server.raft: vote granted: from=500cf7be-f79f-57ec-d739-5338ae080b7c term=2 tally=1 -45:14.721 [INFO] TestAgent.server.raft: election won: tally=1 -45:14.721 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11636 [Leader]" -45:14.721 [INFO] TestAgent.server: cluster leadership acquired -45:14.721 [INFO] TestAgent.server: New leader elected: payload=Node-500cf7be-f79f-57ec-d739-5338ae080b7c -45:14.732 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11636 -45:14.749 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:14.776 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:14.776 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:14.776 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:14.776 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:14.776 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:14.776 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-500cf7be-f79f-57ec-d739-5338ae080b7c -45:14.776 [INFO] TestAgent.server: member joined, marking health alive: member=Node-500cf7be-f79f-57ec-d739-5338ae080b7c -45:14.793 [INFO] TestAgent.server: federation state anti-entropy synced -45:14.999 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:15.003 [INFO] TestAgent: Synced node info -45:15.005 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_persistedService_compat_normal-TestAgent-agent817692530/services/86a1b907d54bf7010394bf316e183e67 -45:15.005 [INFO] TestAgent: Requesting shutdown -45:15.005 [INFO] TestAgent.server: shutting down server -45:15.005 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:15.005 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:15.005 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:15.005 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:15.005 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:15.005 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:15.005 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:15.005 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:15.005 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:15.005 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:15.010 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:15.014Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:15.014Z [INFO] agent.router.manager: shutting down -45:15.014 [INFO] TestAgent: consul server down -45:15.014 [INFO] TestAgent: shutdown complete -45:15.014 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11631 network=tcp -45:15.014 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11631 network=udp -45:15.014 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11632 network=tcp -45:15.514 [INFO] TestAgent: Waiting for endpoints to shut down -45:15.514 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:18.619Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:18.638Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:18.740 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b4c3f842-136a-8b47-3420-2c15da29e582 Address:127.0.0.1:30014}]" +30:18.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b4c3f842-136a-8b47-3420-2c15da29e582.dc1 127.0.0.1 +30:18.741 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30014 [Follower]" leader= +30:18.741 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b4c3f842-136a-8b47-3420-2c15da29e582 127.0.0.1 +2021-10-05T16:30:18.741Z [INFO] agent.router: Initializing LAN area manager +30:18.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30009 network=udp +30:18.742 [INFO] TestAgent.server: Adding LAN server: server="Node-b4c3f842-136a-8b47-3420-2c15da29e582 (Addr: tcp/127.0.0.1:30014) (DC: dc1)" +30:18.742 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b4c3f842-136a-8b47-3420-2c15da29e582.dc1 area=wan +30:18.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30009 network=tcp +30:18.742 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30010 network=tcp +30:18.742 [INFO] TestAgent: started state syncer +30:18.742 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30015 network=tcp +30:18.800 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:18.800 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30014 [Candidate]" term=2 +30:18.813 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:18.813 [DEBUG] TestAgent.server.raft: vote granted: from=b4c3f842-136a-8b47-3420-2c15da29e582 term=2 tally=1 +30:18.813 [INFO] TestAgent.server.raft: election won: tally=1 +30:18.813 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30014 [Leader]" +30:18.813 [INFO] TestAgent.server: cluster leadership acquired +30:18.814 [INFO] TestAgent.server: New leader elected: payload=Node-b4c3f842-136a-8b47-3420-2c15da29e582 +30:18.827 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30014 +30:18.868 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:18.902 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:18.902 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:18.902 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:18.902 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:18.902 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:18.902 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b4c3f842-136a-8b47-3420-2c15da29e582 +30:18.902 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b4c3f842-136a-8b47-3420-2c15da29e582 +30:18.916 [INFO] TestAgent.server: federation state anti-entropy synced +30:19.132 [DEBUG] TestAgent: restored service definition from file: service=redis file=/tmp/consul-test/TestAgent_persistedService_compat_normal-TestAgent-agent652751019/services/86a1b907d54bf7010394bf316e183e67 +30:19.132 [INFO] TestAgent: Requesting shutdown +30:19.132 [INFO] TestAgent.server: shutting down server +30:19.132 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:19.132 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:19.132 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:19.132 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:19.132 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:19.132 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:19.132 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:19.132 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:19.133 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:19.133 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:19.133 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:19.143 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:19.152Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:19.152Z [INFO] agent.router.manager: shutting down +30:19.152 [INFO] TestAgent: consul server down +30:19.152 [INFO] TestAgent: shutdown complete +30:19.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30009 network=tcp +30:19.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30009 network=udp +30:19.152 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30010 network=tcp +30:19.656 [INFO] TestAgent: Waiting for endpoints to shut down +30:19.656 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_persistedService_compat (0.00s) - --- PASS: TestAgent_persistedService_compat/service_manager (0.73s) - --- PASS: TestAgent_persistedService_compat/normal (0.93s) + --- PASS: TestAgent_persistedService_compat/service_manager (0.87s) + --- PASS: TestAgent_persistedService_compat/normal (1.05s) === RUN TestAgent_PurgeService === RUN TestAgent_PurgeService/normal === PAUSE TestAgent_PurgeService/normal @@ -11810,130 +12024,114 @@ [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:15.525Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:15.534Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:15.577 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:69ce23d5-fa1d-5825-43f5-864f449cfd15 Address:127.0.0.1:11657}]" -45:15.577 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11657 [Follower]" leader= -45:15.577 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-69ce23d5-fa1d-5825-43f5-864f449cfd15.dc1 127.0.0.1 -45:15.578 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 127.0.0.1 -2022-11-07T22:45:15.578Z [INFO] agent.router: Initializing LAN area manager -45:15.578 [INFO] TestAgent.server: Adding LAN server: server="Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 (Addr: tcp/127.0.0.1:11657) (DC: dc1)" -45:15.578 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-69ce23d5-fa1d-5825-43f5-864f449cfd15.dc1 area=wan -45:15.578 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11652 network=udp -45:15.578 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11652 network=tcp -45:15.579 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11653 network=tcp -45:15.579 [INFO] TestAgent: started state syncer -45:15.579 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11658 network=tcp -45:15.628 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:15.628 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11657 [Candidate]" term=2 -45:15.642 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:15.642 [DEBUG] TestAgent.server.raft: vote granted: from=69ce23d5-fa1d-5825-43f5-864f449cfd15 term=2 tally=1 -45:15.642 [INFO] TestAgent.server.raft: election won: tally=1 -45:15.642 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11657 [Leader]" -45:15.642 [INFO] TestAgent.server: cluster leadership acquired -45:15.642 [INFO] TestAgent.server: New leader elected: payload=Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 -45:15.650 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11657 -45:15.669 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:15.685 [DEBUG] TestAgent: removed service: service=redis -45:15.690 [DEBUG] TestAgent: removed service: service=redis -45:15.690 [INFO] TestAgent: Requesting shutdown -45:15.691 [INFO] TestAgent.server: shutting down server -45:15.691 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:15.691 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:15.696 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:15.702 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:15.702 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:15.702 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:15.702 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:15.702 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:15.702 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 -45:15.702 [INFO] TestAgent.server: member joined, marking health alive: member=Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 -2022-11-07T22:45:15.702Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:15.702Z [INFO] agent.router.manager: shutting down -45:15.702 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:15.702 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:15.702 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:15.702 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:15.702 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:15.709 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-69ce23d5-fa1d-5825-43f5-864f449cfd15 127.0.0.1 11655 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:69ce23d5-fa1d-5825-43f5-864f449cfd15 port:11657 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11656] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -45:15.709 [INFO] TestAgent: consul server down -45:15.709 [INFO] TestAgent: shutdown complete -45:15.709 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11652 network=tcp -45:15.709 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:15.709 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:15.710 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:15.710 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:15.710 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:15.710 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11652 network=udp -45:15.710 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:15.710 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:15.710 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:15.710 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11653 network=tcp -45:16.210 [INFO] TestAgent: Waiting for endpoints to shut down -45:16.210 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:19.694Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:19.754Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:19.822 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3fa0b64d-8c51-0581-cc73-4becb1ab45e1 Address:127.0.0.1:30035}]" +30:19.822 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3fa0b64d-8c51-0581-cc73-4becb1ab45e1.dc1 127.0.0.1 +30:19.823 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3fa0b64d-8c51-0581-cc73-4becb1ab45e1 127.0.0.1 +2021-10-05T16:30:19.823Z [INFO] agent.router: Initializing LAN area manager +30:19.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30030 network=udp +30:19.823 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30035 [Follower]" leader= +30:19.824 [INFO] TestAgent.server: Adding LAN server: server="Node-3fa0b64d-8c51-0581-cc73-4becb1ab45e1 (Addr: tcp/127.0.0.1:30035) (DC: dc1)" +30:19.824 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3fa0b64d-8c51-0581-cc73-4becb1ab45e1.dc1 area=wan +30:19.824 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30030 network=tcp +30:19.824 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30031 network=tcp +30:19.824 [INFO] TestAgent: started state syncer +30:19.825 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30036 network=tcp +30:19.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:19.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30035 [Candidate]" term=2 +30:19.916 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:19.916 [DEBUG] TestAgent.server.raft: vote granted: from=3fa0b64d-8c51-0581-cc73-4becb1ab45e1 term=2 tally=1 +30:19.916 [INFO] TestAgent.server.raft: election won: tally=1 +30:19.916 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30035 [Leader]" +30:19.920 [INFO] TestAgent.server: cluster leadership acquired +30:19.920 [INFO] TestAgent.server: New leader elected: payload=Node-3fa0b64d-8c51-0581-cc73-4becb1ab45e1 +30:19.935 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30035 +30:19.954 [DEBUG] TestAgent: removed service: service=redis +30:19.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:19.988 [INFO] TestAgent: Synced node info +30:19.996 [INFO] TestAgent: Synced service: service=redis +30:19.996 [DEBUG] TestAgent: removed service: service=redis +30:19.996 [INFO] TestAgent: Requesting shutdown +30:19.996 [INFO] TestAgent.server: shutting down server +30:19.996 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:20.008 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:20.013Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:20.013Z [INFO] agent.router.manager: shutting down +30:20.028 [INFO] TestAgent: consul server down +30:20.028 [INFO] TestAgent: shutdown complete +30:20.028 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30030 network=tcp +30:20.028 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30030 network=udp +30:20.028 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30031 network=tcp +30:20.028 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +30:20.028 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +30:20.532 [INFO] TestAgent: Waiting for endpoints to shut down +30:20.532 [INFO] TestAgent: Endpoints down [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:15.526Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:15.534Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:15.584 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e40e5cc2-8d7a-695e-faa8-24b2a52fce70 Address:127.0.0.1:11650}]" -45:15.585 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11650 [Follower]" leader= -45:15.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70.dc1 127.0.0.1 -45:15.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70 127.0.0.1 -2022-11-07T22:45:15.586Z [INFO] agent.router: Initializing LAN area manager -45:15.586 [INFO] TestAgent.server: Adding LAN server: server="Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70 (Addr: tcp/127.0.0.1:11650) (DC: dc1)" -45:15.586 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70.dc1 area=wan -45:15.586 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11645 network=udp -45:15.586 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11645 network=tcp -45:15.586 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11646 network=tcp -45:15.586 [INFO] TestAgent: started state syncer -45:15.586 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11651 network=tcp -45:15.636 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:15.636 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11650 [Candidate]" term=2 -45:15.649 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:15.649 [DEBUG] TestAgent.server.raft: vote granted: from=e40e5cc2-8d7a-695e-faa8-24b2a52fce70 term=2 tally=1 -45:15.649 [INFO] TestAgent.server.raft: election won: tally=1 -45:15.649 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11650 [Leader]" -45:15.649 [INFO] TestAgent.server: cluster leadership acquired -45:15.649 [INFO] TestAgent.server: New leader elected: payload=Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70 -45:15.657 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11650 -45:15.685 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:15.716 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:15.716 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:15.716 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:15.716 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:15.716 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:15.716 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70 -45:15.716 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e40e5cc2-8d7a-695e-faa8-24b2a52fce70 -45:15.730 [INFO] TestAgent.server: federation state anti-entropy synced -45:15.944 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:15.947 [INFO] TestAgent: Synced node info -45:15.950 [INFO] TestAgent: Synced service: service=redis -45:15.950 [DEBUG] TestAgent: removed service: service=redis -45:15.954 [DEBUG] TestAgent: removed service: service=redis -45:15.954 [INFO] TestAgent: Requesting shutdown -45:15.954 [INFO] TestAgent.server: shutting down server -45:15.954 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:15.954 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:15.954 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:15.954 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:15.954 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:15.954 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:15.954 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:15.954 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:15.954 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:15.954 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:15.957 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:15.961Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:15.961Z [INFO] agent.router.manager: shutting down -45:15.961 [INFO] TestAgent: consul server down -45:15.961 [INFO] TestAgent: shutdown complete -45:15.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11645 network=tcp -45:15.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11645 network=udp -45:15.961 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11646 network=tcp -45:16.461 [INFO] TestAgent: Waiting for endpoints to shut down -45:16.461 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:19.679Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:19.746Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:19.836 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07937cc2-3e9c-93cd-53f8-45e564977e26 Address:127.0.0.1:30028}]" +30:19.836 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30028 [Follower]" leader= +30:19.837 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07937cc2-3e9c-93cd-53f8-45e564977e26.dc1 127.0.0.1 +30:19.837 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07937cc2-3e9c-93cd-53f8-45e564977e26 127.0.0.1 +2021-10-05T16:30:19.837Z [INFO] agent.router: Initializing LAN area manager +30:19.837 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30023 network=udp +30:19.837 [INFO] TestAgent.server: Adding LAN server: server="Node-07937cc2-3e9c-93cd-53f8-45e564977e26 (Addr: tcp/127.0.0.1:30028) (DC: dc1)" +30:19.837 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07937cc2-3e9c-93cd-53f8-45e564977e26.dc1 area=wan +30:19.837 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30023 network=tcp +30:19.838 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30024 network=tcp +30:19.838 [INFO] TestAgent: started state syncer +30:19.838 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30029 network=tcp +30:19.888 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:19.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30028 [Candidate]" term=2 +30:19.900 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:19.900 [DEBUG] TestAgent.server.raft: vote granted: from=07937cc2-3e9c-93cd-53f8-45e564977e26 term=2 tally=1 +30:19.900 [INFO] TestAgent.server.raft: election won: tally=1 +30:19.900 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30028 [Leader]" +30:19.904 [INFO] TestAgent.server: cluster leadership acquired +30:19.904 [INFO] TestAgent.server: New leader elected: payload=Node-07937cc2-3e9c-93cd-53f8-45e564977e26 +30:19.909 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30028 +30:19.931 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:19.976 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:19.976 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:19.976 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:19.976 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:19.976 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:19.976 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07937cc2-3e9c-93cd-53f8-45e564977e26 +30:19.976 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07937cc2-3e9c-93cd-53f8-45e564977e26 +30:19.991 [INFO] TestAgent.server: federation state anti-entropy synced +30:20.116 [DEBUG] TestAgent: removed service: service=redis +30:20.128 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:20.136 [INFO] TestAgent: Synced node info +30:20.140 [INFO] TestAgent: Synced service: service=redis +30:20.140 [DEBUG] TestAgent: Node info in sync +30:20.140 [DEBUG] TestAgent: Service in sync: service=redis +30:20.140 [DEBUG] TestAgent: removed service: service=redis +30:20.140 [INFO] TestAgent: Requesting shutdown +30:20.140 [INFO] TestAgent.server: shutting down server +30:20.140 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:20.140 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:20.140 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:20.140 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:20.140 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:20.140 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:20.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:20.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:20.140 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:20.140 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:20.144 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:20.150Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:20.150Z [INFO] agent.router.manager: shutting down +30:20.150 [INFO] TestAgent: consul server down +30:20.150 [INFO] TestAgent: shutdown complete +30:20.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30023 network=tcp +30:20.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30023 network=udp +30:20.150 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30024 network=tcp +30:20.652 [INFO] TestAgent: Waiting for endpoints to shut down +30:20.652 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_PurgeService (0.00s) - --- PASS: TestAgent_PurgeService/service_manager (0.70s) - --- PASS: TestAgent_PurgeService/normal (0.95s) + --- PASS: TestAgent_PurgeService/service_manager (0.87s) + --- PASS: TestAgent_PurgeService/normal (1.00s) === RUN TestAgent_PurgeServiceOnDuplicate === RUN TestAgent_PurgeServiceOnDuplicate/normal === PAUSE TestAgent_PurgeServiceOnDuplicate/normal @@ -11945,94 +12143,91 @@ [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -45:17.027 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-134cac68-a67f-b7c3-b055-8a49e55f7895 127.0.0.1 -2022-11-07T22:45:17.027Z [INFO] agent.router: Initializing LAN area manager -45:17.027 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-f29030c9-1e6c-3e2d-7a28-1de20544fcdf: 127.0.0.1:11662 -45:17.028 [DEBUG] Agent2: service exists, not restoring from file: service=redis file=/tmp/consul-test/TestAgent_PurgeServiceOnDuplicate_normal-TestAgent-agent405949638/services/86a1b907d54bf7010394bf316e183e67 -45:17.028 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11662: connect: connection refused -45:17.028 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node -45:17.028 [INFO] Agent2: Started DNS server: address=127.0.0.1:11673 network=udp -45:17.028 [INFO] Agent2: Started DNS server: address=127.0.0.1:11673 network=tcp -45:17.028 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11674 network=tcp -45:17.028 [INFO] Agent2: started state syncer -45:17.028 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11679 network=tcp -2022-11-07T22:45:17.028Z [WARN] agent.router.manager: No servers available -45:17.028 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:17.029 [INFO] Agent2: Requesting shutdown -45:17.029 [INFO] Agent2.client: shutting down client -45:17.029 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave -45:17.032 [INFO] Agent2: consul client down -45:17.032 [INFO] Agent2: shutdown complete -45:17.032 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11673 network=tcp -45:17.032 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11673 network=udp -45:17.032 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11674 network=tcp -45:17.532 [INFO] Agent2: Waiting for endpoints to shut down -45:17.532 [INFO] Agent2: Endpoints down -45:16.494 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-f29030c9-1e6c-3e2d-7a28-1de20544fcdf 127.0.0.1 -2022-11-07T22:45:16.494Z [INFO] agent.router: Initializing LAN area manager -45:16.494 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11659 network=udp -45:16.494 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11659 network=tcp -45:16.494 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11660 network=tcp -45:16.494 [INFO] TestAgent: started state syncer -45:16.495 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11665 network=tcp -2022-11-07T22:45:16.495Z [WARN] agent.router.manager: No servers available -45:16.495 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:16.501 [INFO] TestAgent: Requesting shutdown -45:16.501 [INFO] TestAgent.client: shutting down client -45:16.501 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:16.504 [INFO] TestAgent: consul client down -45:16.504 [INFO] TestAgent: shutdown complete -45:16.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11659 network=tcp -45:16.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11659 network=udp -45:16.504 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11660 network=tcp -45:17.004 [INFO] TestAgent: Waiting for endpoints to shut down -45:17.004 [INFO] TestAgent: Endpoints down -45:17.036 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-fdc8e575-f5e6-efcc-f4e7-7a1ace6931d4 127.0.0.1 -2022-11-07T22:45:17.036Z [INFO] agent.router: Initializing LAN area manager -45:17.036 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-8861b639-6a8d-1f49-9924-13df2ce67ddb: 127.0.0.1:11669 -45:17.036 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11669: connect: connection refused -45:17.036 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node -45:17.036 [DEBUG] Agent2: service exists, not restoring from file: service=redis file=/tmp/consul-test/TestAgent_PurgeServiceOnDuplicate_service_manager-TestAgent-agent684319597/services/86a1b907d54bf7010394bf316e183e67 -45:17.036 [INFO] Agent2: Started DNS server: address=127.0.0.1:11680 network=tcp -45:17.036 [INFO] Agent2: Started DNS server: address=127.0.0.1:11680 network=udp -45:17.037 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11681 network=tcp -45:17.037 [INFO] Agent2: started state syncer -45:17.037 [INFO] Agent2: Requesting shutdown -45:17.037 [INFO] Agent2.client: shutting down client -45:17.037 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave -45:17.037 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11686 network=tcp -45:17.037 [ERROR] Agent2: gRPC server failed: error="grpc: the server has been stopped" -2022-11-07T22:45:17.037Z [WARN] agent.router.manager: No servers available -45:17.037 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:17.040 [INFO] Agent2: consul client down -45:17.040 [INFO] Agent2: shutdown complete -45:17.040 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11680 network=tcp -45:17.040 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11680 network=udp -45:17.040 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11681 network=tcp -45:17.540 [INFO] Agent2: Waiting for endpoints to shut down -45:17.540 [INFO] Agent2: Endpoints down -45:16.495 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-8861b639-6a8d-1f49-9924-13df2ce67ddb 127.0.0.1 -2022-11-07T22:45:16.495Z [INFO] agent.router: Initializing LAN area manager -45:16.495 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11666 network=udp -45:16.495 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11666 network=tcp -45:16.495 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11667 network=tcp -45:16.495 [INFO] TestAgent: started state syncer -45:16.495 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11672 network=tcp -2022-11-07T22:45:16.496Z [WARN] agent.router.manager: No servers available -45:16.496 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:16.501 [INFO] TestAgent: Requesting shutdown -45:16.501 [INFO] TestAgent.client: shutting down client -45:16.501 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -45:16.504 [INFO] TestAgent: consul client down -45:16.504 [INFO] TestAgent: shutdown complete -45:16.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11666 network=tcp -45:16.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11666 network=udp -45:16.504 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11667 network=tcp -45:17.004 [INFO] TestAgent: Waiting for endpoints to shut down -45:17.004 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +30:21.385 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-bc762d9b-860b-c259-611d-02ee7997c3c5 127.0.0.1 +2021-10-05T16:30:21.385Z [INFO] agent.router: Initializing LAN area manager +30:21.386 [DEBUG] Agent2: service exists, not restoring from file: service=redis file=/tmp/consul-test/TestAgent_PurgeServiceOnDuplicate_normal-TestAgent-agent623680271/services/86a1b907d54bf7010394bf316e183e67 +30:21.386 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-5936bff4-3e13-6388-1b64-f62992401d95: 127.0.0.1:30040 +30:21.386 [INFO] Agent2: Started DNS server: address=127.0.0.1:30051 network=udp +30:21.386 [INFO] Agent2: Started DNS server: address=127.0.0.1:30051 network=tcp +30:21.386 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:30040: connect: connection refused +30:21.386 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node +30:21.387 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30052 network=tcp +30:21.387 [INFO] Agent2: started state syncer +30:21.387 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30057 network=tcp +2021-10-05T16:30:21.387Z [WARN] agent.router.manager: No servers available +30:21.387 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:21.387 [INFO] Agent2: Requesting shutdown +30:21.387 [INFO] Agent2.client: shutting down client +30:21.387 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave +30:21.412 [INFO] Agent2: consul client down +30:21.412 [INFO] Agent2: shutdown complete +30:21.412 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30051 network=tcp +30:21.412 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30051 network=udp +30:21.412 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30052 network=tcp +30:21.912 [INFO] Agent2: Waiting for endpoints to shut down +30:21.912 [INFO] Agent2: Endpoints down +30:20.721 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-5936bff4-3e13-6388-1b64-f62992401d95 127.0.0.1 +2021-10-05T16:30:20.721Z [INFO] agent.router: Initializing LAN area manager +30:20.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30037 network=udp +30:20.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30037 network=tcp +30:20.722 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30038 network=tcp +30:20.722 [INFO] TestAgent: started state syncer +30:20.723 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30043 network=tcp +30:20.732 [INFO] TestAgent: Requesting shutdown +30:20.732 [INFO] TestAgent.client: shutting down client +30:20.732 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:20.752 [INFO] TestAgent: consul client down +30:20.752 [INFO] TestAgent: shutdown complete +30:20.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30037 network=tcp +30:20.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30037 network=udp +30:20.752 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30038 network=tcp +30:21.253 [INFO] TestAgent: Waiting for endpoints to shut down +30:21.253 [INFO] TestAgent: Endpoints down +30:21.401 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-79153125-292a-de71-f7e6-ff416d63fbb3 127.0.0.1 +2021-10-05T16:30:21.401Z [INFO] agent.router: Initializing LAN area manager +30:21.401 [DEBUG] Agent2: service exists, not restoring from file: service=redis file=/tmp/consul-test/TestAgent_PurgeServiceOnDuplicate_service_manager-TestAgent-agent166841634/services/86a1b907d54bf7010394bf316e183e67 +30:21.401 [INFO] Agent2: Started DNS server: address=127.0.0.1:30058 network=udp +30:21.401 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-0fd4b3dd-600a-d839-b08b-c91d4a8f86d3: 127.0.0.1:30047 +30:21.401 [INFO] Agent2: Started DNS server: address=127.0.0.1:30058 network=tcp +30:21.402 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30059 network=tcp +30:21.402 [INFO] Agent2: started state syncer +30:21.402 [INFO] Agent2: Requesting shutdown +30:21.402 [INFO] Agent2.client: shutting down client +30:21.402 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave +30:21.402 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30064 network=tcp +30:21.402 [ERROR] Agent2: gRPC server failed: error="grpc: the server has been stopped" +2021-10-05T16:30:21.402Z [WARN] agent.router.manager: No servers available +30:21.402 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:21.404 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:30047: connect: connection refused +30:21.404 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node +30:21.412 [INFO] Agent2: consul client down +30:21.412 [INFO] Agent2: shutdown complete +30:21.412 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30058 network=tcp +30:21.412 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30058 network=udp +30:21.412 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30059 network=tcp +30:21.913 [INFO] Agent2: Waiting for endpoints to shut down +30:21.913 [INFO] Agent2: Endpoints down +30:20.727 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-0fd4b3dd-600a-d839-b08b-c91d4a8f86d3 127.0.0.1 +2021-10-05T16:30:20.727Z [INFO] agent.router: Initializing LAN area manager +30:20.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30044 network=udp +30:20.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30044 network=tcp +30:20.728 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30045 network=tcp +30:20.728 [INFO] TestAgent: started state syncer +30:20.728 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30050 network=tcp +30:20.752 [INFO] TestAgent: Requesting shutdown +30:20.752 [INFO] TestAgent.client: shutting down client +30:20.752 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +30:20.776 [INFO] TestAgent: consul client down +30:20.776 [INFO] TestAgent: shutdown complete +30:20.776 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30044 network=tcp +30:20.776 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30044 network=udp +30:20.776 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30045 network=tcp +30:21.277 [INFO] TestAgent: Waiting for endpoints to shut down +30:21.277 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_PurgeServiceOnDuplicate (0.00s) - --- PASS: TestAgent_PurgeServiceOnDuplicate/normal (1.07s) - --- PASS: TestAgent_PurgeServiceOnDuplicate/service_manager (1.08s) + --- PASS: TestAgent_PurgeServiceOnDuplicate/normal (1.26s) + --- PASS: TestAgent_PurgeServiceOnDuplicate/service_manager (1.26s) === RUN TestAgent_PersistCheck === PAUSE TestAgent_PersistCheck === RUN TestAgent_PurgeCheck @@ -12055,123 +12250,121 @@ [WARN] freeport: 6 out of 34 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:17.559Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:17.574Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:17.634 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46b14f93-b344-2bf1-abc7-d98d2f933bd0 Address:127.0.0.1:11699}]" -45:17.634 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11699 [Follower]" leader= -45:17.635 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0.dc1 127.0.0.1 -45:17.635 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0 127.0.0.1 -2022-11-07T22:45:17.635Z [INFO] agent.router: Initializing LAN area manager -45:17.635 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0.dc1 area=wan -45:17.635 [INFO] TestAgent.server: Adding LAN server: server="Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0 (Addr: tcp/127.0.0.1:11699) (DC: dc1)" -45:17.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11694 network=udp -45:17.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11694 network=tcp -45:17.636 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11695 network=tcp -45:17.636 [INFO] TestAgent: started state syncer -45:17.636 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11700 network=tcp -45:17.679 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:17.679 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11699 [Candidate]" term=2 -45:17.688 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:17.688 [DEBUG] TestAgent.server.raft: vote granted: from=46b14f93-b344-2bf1-abc7-d98d2f933bd0 term=2 tally=1 -45:17.688 [INFO] TestAgent.server.raft: election won: tally=1 -45:17.688 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11699 [Leader]" -45:17.688 [INFO] TestAgent.server: cluster leadership acquired -45:17.688 [INFO] TestAgent.server: New leader elected: payload=Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0 -45:17.693 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11699 -45:17.704 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:17.725 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:17.725 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:17.725 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:17.725 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:17.725 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:17.725 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0 -45:17.725 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46b14f93-b344-2bf1-abc7-d98d2f933bd0 -45:17.730 [INFO] TestAgent.server: federation state anti-entropy synced -45:17.744 [INFO] TestAgent: Requesting shutdown -45:17.744 [INFO] TestAgent.server: shutting down server -45:17.744 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:17.744 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:17.744 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:17.744 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:17.744 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:17.744 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:17.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:17.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:17.744 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:17.744 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:17.744 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:17.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:17.753Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:17.753Z [INFO] agent.router.manager: shutting down -45:17.753 [INFO] TestAgent: consul server down -45:17.753 [INFO] TestAgent: shutdown complete -45:17.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11694 network=tcp -45:17.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11694 network=udp -45:17.753 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11695 network=tcp -45:18.254 [INFO] TestAgent: Waiting for endpoints to shut down -45:18.254 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:21.928Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:21.944Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:21.978 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9a04998-4f64-658b-5610-4f3db79bb0ce Address:127.0.0.1:30070}]" +30:21.979 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9a04998-4f64-658b-5610-4f3db79bb0ce.dc1 127.0.0.1 +30:21.980 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9a04998-4f64-658b-5610-4f3db79bb0ce 127.0.0.1 +2021-10-05T16:30:21.980Z [INFO] agent.router: Initializing LAN area manager +30:21.980 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30070 [Follower]" leader= +30:21.980 [INFO] TestAgent.server: Adding LAN server: server="Node-c9a04998-4f64-658b-5610-4f3db79bb0ce (Addr: tcp/127.0.0.1:30070) (DC: dc1)" +30:21.980 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9a04998-4f64-658b-5610-4f3db79bb0ce.dc1 area=wan +30:21.980 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30065 network=udp +30:21.980 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30065 network=tcp +30:21.981 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30066 network=tcp +30:21.981 [INFO] TestAgent: started state syncer +30:21.981 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30071 network=tcp +30:22.045 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:22.045 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30070 [Candidate]" term=2 +30:22.057 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:22.057 [DEBUG] TestAgent.server.raft: vote granted: from=c9a04998-4f64-658b-5610-4f3db79bb0ce term=2 tally=1 +30:22.057 [INFO] TestAgent.server.raft: election won: tally=1 +30:22.057 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30070 [Leader]" +30:22.058 [INFO] TestAgent.server: cluster leadership acquired +30:22.058 [INFO] TestAgent.server: New leader elected: payload=Node-c9a04998-4f64-658b-5610-4f3db79bb0ce +30:22.064 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30070 +30:22.096 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:22.146 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:22.146 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:22.146 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:22.146 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:22.146 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:22.146 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9a04998-4f64-658b-5610-4f3db79bb0ce +30:22.146 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9a04998-4f64-658b-5610-4f3db79bb0ce +30:22.163 [INFO] TestAgent.server: federation state anti-entropy synced +30:22.244 [INFO] TestAgent: Requesting shutdown +30:22.244 [INFO] TestAgent.server: shutting down server +30:22.244 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:22.244 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:22.244 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:22.244 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:22.244 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:22.244 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:22.244 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:22.244 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:22.244 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:22.244 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:22.244 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:22.252 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:22.256Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:22.256Z [INFO] agent.router.manager: shutting down +30:22.256 [INFO] TestAgent: consul server down +30:22.256 [INFO] TestAgent: shutdown complete +30:22.256 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30065 network=tcp +30:22.256 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30065 network=udp +30:22.256 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30066 network=tcp +30:22.760 [INFO] TestAgent: Waiting for endpoints to shut down +30:22.760 [INFO] TestAgent: Endpoints down [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:17.558Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:17.573Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:17.635 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:23aa1c31-9f54-9d70-4ea8-9de1f3d586da Address:127.0.0.1:11692}]" -45:17.635 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11692 [Follower]" leader= -45:17.635 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da.dc1 127.0.0.1 -45:17.636 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da 127.0.0.1 -2022-11-07T22:45:17.636Z [INFO] agent.router: Initializing LAN area manager -45:17.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11687 network=udp -45:17.636 [INFO] TestAgent.server: Adding LAN server: server="Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da (Addr: tcp/127.0.0.1:11692) (DC: dc1)" -45:17.636 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da.dc1 area=wan -45:17.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11687 network=tcp -45:17.637 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11688 network=tcp -45:17.637 [INFO] TestAgent: started state syncer -45:17.637 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11693 network=tcp -45:17.693 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:17.693 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11692 [Candidate]" term=2 -45:17.701 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:17.701 [DEBUG] TestAgent.server.raft: vote granted: from=23aa1c31-9f54-9d70-4ea8-9de1f3d586da term=2 tally=1 -45:17.701 [INFO] TestAgent.server.raft: election won: tally=1 -45:17.701 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11692 [Leader]" -45:17.701 [INFO] TestAgent.server: cluster leadership acquired -45:17.701 [INFO] TestAgent.server: New leader elected: payload=Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da -45:17.708 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11692 -45:17.721 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:17.743 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:17.743 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:17.743 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:17.743 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:17.743 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:17.743 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da -45:17.743 [INFO] TestAgent.server: member joined, marking health alive: member=Node-23aa1c31-9f54-9d70-4ea8-9de1f3d586da -45:17.752 [INFO] TestAgent.server: federation state anti-entropy synced -45:17.788 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:17.791 [INFO] TestAgent: Synced node info -45:17.794 [INFO] TestAgent: Synced service: service=rabbitmq -45:18.036 [INFO] TestAgent: Requesting shutdown -45:18.036 [INFO] TestAgent.server: shutting down server -45:18.036 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:18.036 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:18.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:18.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:18.036 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:18.036 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:18.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:18.036 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:18.036 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:18.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:18.122 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:18.126Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:18.126Z [INFO] agent.router.manager: shutting down -45:18.126 [INFO] TestAgent: consul server down -45:18.126 [INFO] TestAgent: shutdown complete -45:18.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11687 network=tcp -45:18.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11687 network=udp -45:18.126 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11688 network=tcp -45:18.626 [INFO] TestAgent: Waiting for endpoints to shut down -45:18.626 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:21.952Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:21.990Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:22.024 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4faf7ce-2ac1-f9f4-1798-b488234d9abd Address:127.0.0.1:30077}]" +30:22.024 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd.dc1 127.0.0.1 +30:22.025 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd 127.0.0.1 +2021-10-05T16:30:22.025Z [INFO] agent.router: Initializing LAN area manager +30:22.025 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30072 network=udp +30:22.025 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30077 [Follower]" leader= +30:22.026 [INFO] TestAgent.server: Adding LAN server: server="Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd (Addr: tcp/127.0.0.1:30077) (DC: dc1)" +30:22.026 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd.dc1 area=wan +30:22.026 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30072 network=tcp +30:22.026 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30073 network=tcp +30:22.026 [INFO] TestAgent: started state syncer +30:22.026 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30078 network=tcp +30:22.092 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:22.092 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30077 [Candidate]" term=2 +30:22.137 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:22.137 [DEBUG] TestAgent.server.raft: vote granted: from=e4faf7ce-2ac1-f9f4-1798-b488234d9abd term=2 tally=1 +30:22.137 [INFO] TestAgent.server.raft: election won: tally=1 +30:22.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30077 [Leader]" +30:22.140 [INFO] TestAgent.server: cluster leadership acquired +30:22.140 [INFO] TestAgent.server: New leader elected: payload=Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd +30:22.147 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30077 +30:22.166 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:22.208 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:22.208 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:22.208 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:22.208 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:22.208 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:22.208 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd +30:22.208 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e4faf7ce-2ac1-f9f4-1798-b488234d9abd +30:22.209 [INFO] TestAgent: Synced node info +30:22.256 [INFO] TestAgent: Synced service: service=rabbitmq +30:22.265 [INFO] TestAgent.server: federation state anti-entropy synced +30:22.374 [INFO] TestAgent: Requesting shutdown +30:22.374 [INFO] TestAgent.server: shutting down server +30:22.374 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:22.374 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:22.374 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:22.374 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:22.374 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:22.374 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:22.374 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:22.374 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:22.375 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:22.375 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:22.388 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:22.389Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:22.389Z [INFO] agent.router.manager: shutting down +30:22.389 [INFO] TestAgent: consul server down +30:22.389 [INFO] TestAgent: shutdown complete +30:22.389 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30072 network=tcp +30:22.389 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30072 network=udp +30:22.389 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30073 network=tcp +30:22.892 [INFO] TestAgent: Waiting for endpoints to shut down +30:22.892 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_loadServices_token (0.00s) - --- PASS: TestAgent_loadServices_token/service_manager (0.71s) - --- PASS: TestAgent_loadServices_token/normal (1.09s) + --- PASS: TestAgent_loadServices_token/normal (0.85s) + --- PASS: TestAgent_loadServices_token/service_manager (0.98s) === RUN TestAgent_loadServices_sidecar === RUN TestAgent_loadServices_sidecar/normal === PAUSE TestAgent_loadServices_sidecar/normal @@ -12182,125 +12375,130 @@ [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:18.645Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:18.654Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:18.681 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fe644e4e-2ad1-370a-2321-ecc8c83a308a Address:127.0.0.1:11706}]" -45:18.681 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11706 [Follower]" leader= -45:18.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a.dc1 127.0.0.1 -45:18.682 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a 127.0.0.1 -2022-11-07T22:45:18.682Z [INFO] agent.router: Initializing LAN area manager -45:18.682 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a.dc1 area=wan -45:18.682 [INFO] TestAgent.server: Adding LAN server: server="Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a (Addr: tcp/127.0.0.1:11706) (DC: dc1)" -45:18.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11701 network=tcp -45:18.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11701 network=udp -45:18.682 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11702 network=tcp -45:18.682 [INFO] TestAgent: started state syncer -45:18.682 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11707 network=tcp -45:18.732 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:18.732 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11706 [Candidate]" term=2 -45:18.740 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:18.740 [DEBUG] TestAgent.server.raft: vote granted: from=fe644e4e-2ad1-370a-2321-ecc8c83a308a term=2 tally=1 -45:18.740 [INFO] TestAgent.server.raft: election won: tally=1 -45:18.740 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11706 [Leader]" -45:18.741 [INFO] TestAgent.server: cluster leadership acquired -45:18.741 [INFO] TestAgent.server: New leader elected: payload=Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a -45:18.746 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11706 -45:18.758 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:18.775 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:18.775 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:18.775 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:18.775 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:18.775 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:18.775 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a -45:18.775 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fe644e4e-2ad1-370a-2321-ecc8c83a308a -45:18.782 [INFO] TestAgent.server: federation state anti-entropy synced -45:18.967 [INFO] TestAgent: Requesting shutdown -45:18.967 [INFO] TestAgent.server: shutting down server -45:18.967 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:18.967 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:18.967 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:18.967 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:18.967 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:18.967 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:18.967 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:18.967 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:18.967 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:18.967 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:18.967 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:18.971 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:18.975Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:18.975Z [INFO] agent.router.manager: shutting down -45:18.975 [INFO] TestAgent: consul server down -45:18.975 [INFO] TestAgent: shutdown complete -45:18.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11701 network=tcp -45:18.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11701 network=udp -45:18.975 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11702 network=tcp -45:19.475 [INFO] TestAgent: Waiting for endpoints to shut down -45:19.475 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:18.636Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:18.648Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:18.679 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:77c4a491-c44f-9946-c9a6-5c92e6b5af88 Address:127.0.0.1:11713}]" -45:18.679 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11713 [Follower]" leader= -45:18.679 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88.dc1 127.0.0.1 -45:18.679 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88 127.0.0.1 -2022-11-07T22:45:18.679Z [INFO] agent.router: Initializing LAN area manager -45:18.680 [INFO] TestAgent.server: Adding LAN server: server="Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88 (Addr: tcp/127.0.0.1:11713) (DC: dc1)" -45:18.680 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88.dc1 area=wan -45:18.680 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy -45:18.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11708 network=tcp -45:18.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11708 network=udp -45:18.680 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11709 network=tcp -45:18.680 [INFO] TestAgent: started state syncer -45:18.680 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11714 network=tcp -45:18.749 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:18.749 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11713 [Candidate]" term=2 -45:18.758 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:18.758 [DEBUG] TestAgent.server.raft: vote granted: from=77c4a491-c44f-9946-c9a6-5c92e6b5af88 term=2 tally=1 -45:18.758 [INFO] TestAgent.server.raft: election won: tally=1 -45:18.758 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11713 [Leader]" -45:18.758 [INFO] TestAgent.server: cluster leadership acquired -45:18.758 [INFO] TestAgent.server: New leader elected: payload=Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88 -45:18.765 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11713 -45:18.775 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:18.794 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:18.794 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:18.794 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:18.794 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:18.794 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:18.794 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88 -45:18.794 [INFO] TestAgent.server: member joined, marking health alive: member=Node-77c4a491-c44f-9946-c9a6-5c92e6b5af88 -45:18.799 [INFO] TestAgent.server: federation state anti-entropy synced -45:18.845 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:18.848 [INFO] TestAgent: Synced node info -45:18.851 [INFO] TestAgent: Synced service: service=rabbitmq -45:18.857 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy -45:18.857 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:18.857 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:19.050 [INFO] TestAgent: Requesting shutdown -45:19.050 [INFO] TestAgent.server: shutting down server -45:19.050 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.050 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.050 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.050 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.050 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.050 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.050 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.050 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.050 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.050 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.053 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.057Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.057Z [INFO] agent.router.manager: shutting down -45:19.057 [INFO] TestAgent: consul server down -45:19.057 [INFO] TestAgent: shutdown complete -45:19.057 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11708 network=tcp -45:19.057 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11708 network=udp -45:19.057 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11709 network=tcp -45:19.557 [INFO] TestAgent: Waiting for endpoints to shut down -45:19.557 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:22.931Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:22.989Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:23.056 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ce5d9289-0d1c-1043-e643-ddd06ed6d7ab Address:127.0.0.1:30091}]" +30:23.057 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab.dc1 127.0.0.1 +30:23.057 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30091 [Follower]" leader= +30:23.057 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab 127.0.0.1 +2021-10-05T16:30:23.058Z [INFO] agent.router: Initializing LAN area manager +30:23.058 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy +30:23.058 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30086 network=udp +30:23.058 [INFO] TestAgent.server: Adding LAN server: server="Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab (Addr: tcp/127.0.0.1:30091) (DC: dc1)" +30:23.058 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab.dc1 area=wan +30:23.059 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30086 network=tcp +30:23.061 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30087 network=tcp +30:23.061 [INFO] TestAgent: started state syncer +30:23.061 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30092 network=tcp +30:23.095 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:23.095 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30091 [Candidate]" term=2 +30:23.105 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:23.105 [DEBUG] TestAgent.server.raft: vote granted: from=ce5d9289-0d1c-1043-e643-ddd06ed6d7ab term=2 tally=1 +30:23.105 [INFO] TestAgent.server.raft: election won: tally=1 +30:23.105 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30091 [Leader]" +30:23.108 [INFO] TestAgent.server: cluster leadership acquired +30:23.108 [INFO] TestAgent.server: New leader elected: payload=Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab +30:23.128 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30091 +30:23.172 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:23.244 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:23.244 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:23.244 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:23.244 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:23.244 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:23.244 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab +30:23.244 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ce5d9289-0d1c-1043-e643-ddd06ed6d7ab +30:23.280 [INFO] TestAgent.server: federation state anti-entropy synced +30:23.340 [INFO] TestAgent: Requesting shutdown +30:23.340 [INFO] TestAgent.server: shutting down server +30:23.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:23.340 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:23.340 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:23.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:23.340 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:23.340 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:23.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:23.340 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:23.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:23.340 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:23.340 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:23.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:23.368Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:23.368Z [INFO] agent.router.manager: shutting down +30:23.368 [INFO] TestAgent: consul server down +30:23.368 [INFO] TestAgent: shutdown complete +30:23.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30086 network=tcp +30:23.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30086 network=udp +30:23.368 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30087 network=tcp +30:23.868 [INFO] TestAgent: Waiting for endpoints to shut down +30:23.868 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:22.931Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:22.971Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:23.064 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f37af6b4-461a-4474-d637-acec1b2c8c75 Address:127.0.0.1:30084}]" +30:23.064 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f37af6b4-461a-4474-d637-acec1b2c8c75.dc1 127.0.0.1 +30:23.065 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f37af6b4-461a-4474-d637-acec1b2c8c75 127.0.0.1 +2021-10-05T16:30:23.065Z [INFO] agent.router: Initializing LAN area manager +30:23.065 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30079 network=udp +30:23.065 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30084 [Follower]" leader= +30:23.066 [INFO] TestAgent.server: Adding LAN server: server="Node-f37af6b4-461a-4474-d637-acec1b2c8c75 (Addr: tcp/127.0.0.1:30084) (DC: dc1)" +30:23.066 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f37af6b4-461a-4474-d637-acec1b2c8c75.dc1 area=wan +30:23.066 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30079 network=tcp +30:23.066 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30080 network=tcp +30:23.066 [INFO] TestAgent: started state syncer +30:23.067 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30085 network=tcp +30:23.128 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:23.128 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30084 [Candidate]" term=2 +30:23.174 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:23.174 [DEBUG] TestAgent.server.raft: vote granted: from=f37af6b4-461a-4474-d637-acec1b2c8c75 term=2 tally=1 +30:23.174 [INFO] TestAgent.server.raft: election won: tally=1 +30:23.174 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30084 [Leader]" +30:23.174 [INFO] TestAgent.server: cluster leadership acquired +30:23.175 [INFO] TestAgent.server: New leader elected: payload=Node-f37af6b4-461a-4474-d637-acec1b2c8c75 +30:23.204 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30084 +30:23.249 [INFO] TestAgent: Synced node info +30:23.263 [INFO] TestAgent: Synced service: service=rabbitmq +30:23.275 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy +30:23.275 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 +30:23.275 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 +30:23.275 [DEBUG] TestAgent: Node info in sync +30:23.275 [DEBUG] TestAgent: Service in sync: service=rabbitmq +30:23.275 [DEBUG] TestAgent: Service in sync: service=rabbitmq-sidecar-proxy +30:23.275 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 +30:23.275 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 +30:23.275 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:23.300 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:23.300 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:23.301 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:23.301 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:23.301 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:23.301 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f37af6b4-461a-4474-d637-acec1b2c8c75 +30:23.301 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f37af6b4-461a-4474-d637-acec1b2c8c75 +30:23.312 [INFO] TestAgent.server: federation state anti-entropy synced +30:23.412 [INFO] TestAgent: Requesting shutdown +30:23.412 [INFO] TestAgent.server: shutting down server +30:23.412 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:23.412 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:23.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:23.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:23.412 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:23.412 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:23.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:23.412 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:23.412 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:23.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:23.424 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:23.432Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:23.432Z [INFO] agent.router.manager: shutting down +30:23.440 [INFO] TestAgent: consul server down +30:23.440 [INFO] TestAgent: shutdown complete +30:23.440 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30079 network=tcp +30:23.440 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30079 network=udp +30:23.440 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30080 network=tcp +30:23.942 [INFO] TestAgent: Waiting for endpoints to shut down +30:23.942 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_loadServices_sidecar (0.00s) - --- PASS: TestAgent_loadServices_sidecar/normal (0.85s) - --- PASS: TestAgent_loadServices_sidecar/service_manager (0.93s) + --- PASS: TestAgent_loadServices_sidecar/service_manager (0.97s) + --- PASS: TestAgent_loadServices_sidecar/normal (1.05s) === RUN TestAgent_loadServices_sidecarSeparateToken === RUN TestAgent_loadServices_sidecarSeparateToken/normal === PAUSE TestAgent_loadServices_sidecarSeparateToken/normal @@ -12311,125 +12509,121 @@ [WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:19.569Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.580Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.608 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e1865504-469d-326b-cf84-5ad7f5a29b7d Address:127.0.0.1:11727}]" -45:19.608 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11727 [Follower]" leader= -45:19.609 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e1865504-469d-326b-cf84-5ad7f5a29b7d.dc1 127.0.0.1 -45:19.609 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e1865504-469d-326b-cf84-5ad7f5a29b7d 127.0.0.1 -2022-11-07T22:45:19.609Z [INFO] agent.router: Initializing LAN area manager -45:19.609 [INFO] TestAgent.server: Adding LAN server: server="Node-e1865504-469d-326b-cf84-5ad7f5a29b7d (Addr: tcp/127.0.0.1:11727) (DC: dc1)" -45:19.609 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e1865504-469d-326b-cf84-5ad7f5a29b7d.dc1 area=wan -45:19.609 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy -45:19.610 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11722 network=tcp -45:19.610 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11722 network=udp -45:19.610 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11723 network=tcp -45:19.610 [INFO] TestAgent: started state syncer -45:19.610 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11728 network=tcp -45:19.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11727 [Candidate]" term=2 -45:19.674 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.674 [DEBUG] TestAgent.server.raft: vote granted: from=e1865504-469d-326b-cf84-5ad7f5a29b7d term=2 tally=1 -45:19.674 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.674 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11727 [Leader]" -45:19.675 [INFO] TestAgent.server: cluster leadership acquired -45:19.675 [INFO] TestAgent.server: New leader elected: payload=Node-e1865504-469d-326b-cf84-5ad7f5a29b7d -45:19.681 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11727 -45:19.692 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.713 [INFO] TestAgent: Requesting shutdown -45:19.713 [INFO] TestAgent.server: shutting down server -45:19.713 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.713 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:19.716 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.716 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.716 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.716 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.716 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.716 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e1865504-469d-326b-cf84-5ad7f5a29b7d -45:19.716 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e1865504-469d-326b-cf84-5ad7f5a29b7d -45:19.717 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:19.723 [INFO] TestAgent.server: federation state anti-entropy synced -2022-11-07T22:45:19.724Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.724Z [INFO] agent.router.manager: shutting down -45:19.729 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-e1865504-469d-326b-cf84-5ad7f5a29b7d 127.0.0.1 11725 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:e1865504-469d-326b-cf84-5ad7f5a29b7d port:11727 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11726] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -45:19.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.729 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.729 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.729 [INFO] TestAgent: consul server down -45:19.729 [INFO] TestAgent: shutdown complete -45:19.729 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11722 network=tcp -45:19.729 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.729 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.729 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11722 network=udp -45:19.729 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.729 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11723 network=tcp -45:20.229 [INFO] TestAgent: Waiting for endpoints to shut down -45:20.229 [INFO] TestAgent: Endpoints down -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:19.567Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.579Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.605 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6ac06a28-2b23-c690-dd31-553eacd26716 Address:127.0.0.1:11720}]" -45:19.605 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11720 [Follower]" leader= -45:19.606 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6ac06a28-2b23-c690-dd31-553eacd26716.dc1 127.0.0.1 -45:19.606 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6ac06a28-2b23-c690-dd31-553eacd26716 127.0.0.1 -2022-11-07T22:45:19.606Z [INFO] agent.router: Initializing LAN area manager -45:19.606 [INFO] TestAgent.server: Adding LAN server: server="Node-6ac06a28-2b23-c690-dd31-553eacd26716 (Addr: tcp/127.0.0.1:11720) (DC: dc1)" -45:19.607 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6ac06a28-2b23-c690-dd31-553eacd26716.dc1 area=wan -45:19.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11715 network=udp -45:19.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11715 network=tcp -45:19.607 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11716 network=tcp -45:19.607 [INFO] TestAgent: started state syncer -45:19.607 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11721 network=tcp -45:19.647 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.647 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11720 [Candidate]" term=2 -45:19.656 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.656 [DEBUG] TestAgent.server.raft: vote granted: from=6ac06a28-2b23-c690-dd31-553eacd26716 term=2 tally=1 -45:19.656 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.656 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11720 [Leader]" -45:19.656 [INFO] TestAgent.server: cluster leadership acquired -45:19.656 [INFO] TestAgent.server: New leader elected: payload=Node-6ac06a28-2b23-c690-dd31-553eacd26716 -45:19.661 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11720 -45:19.674 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.697 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.697 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.697 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.697 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.697 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.697 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6ac06a28-2b23-c690-dd31-553eacd26716 -45:19.697 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6ac06a28-2b23-c690-dd31-553eacd26716 -45:19.703 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.771 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:19.774 [INFO] TestAgent: Synced node info -45:19.777 [INFO] TestAgent: Synced service: service=rabbitmq -45:19.783 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy -45:19.783 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:19.783 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:19.927 [INFO] TestAgent: Requesting shutdown -45:19.927 [INFO] TestAgent.server: shutting down server -45:19.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.927 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.927 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.927 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.927 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.927 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.927 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.927 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.927 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.931 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.934Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.934Z [INFO] agent.router.manager: shutting down -45:19.934 [INFO] TestAgent: consul server down -45:19.934 [INFO] TestAgent: shutdown complete -45:19.934 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11715 network=tcp -45:19.934 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11715 network=udp -45:19.934 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11716 network=tcp -45:20.434 [INFO] TestAgent: Waiting for endpoints to shut down -45:20.434 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:23.976Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:24.002Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.045 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b4f437b2-d485-22bd-8c18-e5fac9457334 Address:127.0.0.1:30105}]" +30:24.046 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b4f437b2-d485-22bd-8c18-e5fac9457334.dc1 127.0.0.1 +30:24.047 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b4f437b2-d485-22bd-8c18-e5fac9457334 127.0.0.1 +2021-10-05T16:30:24.047Z [INFO] agent.router: Initializing LAN area manager +30:24.047 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy +30:24.047 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30100 network=udp +30:24.047 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30105 [Follower]" leader= +30:24.048 [INFO] TestAgent.server: Adding LAN server: server="Node-b4f437b2-d485-22bd-8c18-e5fac9457334 (Addr: tcp/127.0.0.1:30105) (DC: dc1)" +30:24.048 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b4f437b2-d485-22bd-8c18-e5fac9457334.dc1 area=wan +30:24.048 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30100 network=tcp +30:24.051 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30101 network=tcp +30:24.051 [INFO] TestAgent: started state syncer +30:24.052 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30106 network=tcp +30:24.113 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:24.113 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30105 [Candidate]" term=2 +30:24.133 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:24.133 [DEBUG] TestAgent.server.raft: vote granted: from=b4f437b2-d485-22bd-8c18-e5fac9457334 term=2 tally=1 +30:24.133 [INFO] TestAgent.server.raft: election won: tally=1 +30:24.133 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30105 [Leader]" +30:24.133 [INFO] TestAgent.server: cluster leadership acquired +30:24.133 [INFO] TestAgent.server: New leader elected: payload=Node-b4f437b2-d485-22bd-8c18-e5fac9457334 +30:24.149 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30105 +30:24.169 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:24.216 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:24.216 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:24.216 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:24.216 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:24.216 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:24.217 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b4f437b2-d485-22bd-8c18-e5fac9457334 +30:24.217 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b4f437b2-d485-22bd-8c18-e5fac9457334 +30:24.236 [INFO] TestAgent.server: federation state anti-entropy synced +30:24.348 [INFO] TestAgent: Requesting shutdown +30:24.348 [INFO] TestAgent.server: shutting down server +30:24.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:24.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:24.348 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:24.348 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:24.348 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:24.348 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:24.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:24.348 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:24.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:24.348 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:24.348 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:24.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:24.368Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:24.368Z [INFO] agent.router.manager: shutting down +30:24.368 [INFO] TestAgent: consul server down +30:24.368 [INFO] TestAgent: shutdown complete +30:24.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30100 network=tcp +30:24.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30100 network=udp +30:24.368 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30101 network=tcp +30:24.872 [INFO] TestAgent: Waiting for endpoints to shut down +30:24.872 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:23.964Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:23.980Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.011 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c71e0cc4-de57-0867-f07c-87659a8b38c0 Address:127.0.0.1:30098}]" +30:24.012 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30098 [Follower]" leader= +30:24.012 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c71e0cc4-de57-0867-f07c-87659a8b38c0.dc1 127.0.0.1 +30:24.013 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c71e0cc4-de57-0867-f07c-87659a8b38c0 127.0.0.1 +2021-10-05T16:30:24.013Z [INFO] agent.router: Initializing LAN area manager +30:24.013 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30093 network=udp +30:24.014 [INFO] TestAgent.server: Adding LAN server: server="Node-c71e0cc4-de57-0867-f07c-87659a8b38c0 (Addr: tcp/127.0.0.1:30098) (DC: dc1)" +30:24.014 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c71e0cc4-de57-0867-f07c-87659a8b38c0.dc1 area=wan +30:24.014 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30093 network=tcp +30:24.015 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30094 network=tcp +30:24.015 [INFO] TestAgent: started state syncer +30:24.015 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30099 network=tcp +30:24.061 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:24.062 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30098 [Candidate]" term=2 +30:24.073 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:24.073 [DEBUG] TestAgent.server.raft: vote granted: from=c71e0cc4-de57-0867-f07c-87659a8b38c0 term=2 tally=1 +30:24.073 [INFO] TestAgent.server.raft: election won: tally=1 +30:24.073 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30098 [Leader]" +30:24.076 [INFO] TestAgent.server: cluster leadership acquired +30:24.076 [INFO] TestAgent.server: New leader elected: payload=Node-c71e0cc4-de57-0867-f07c-87659a8b38c0 +30:24.090 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30098 +30:24.107 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:24.156 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:24.156 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:24.156 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:24.156 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:24.156 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:24.156 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c71e0cc4-de57-0867-f07c-87659a8b38c0 +30:24.156 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c71e0cc4-de57-0867-f07c-87659a8b38c0 +30:24.200 [INFO] TestAgent.server: federation state anti-entropy synced +30:24.360 [INFO] TestAgent: Requesting shutdown +30:24.360 [INFO] TestAgent.server: shutting down server +30:24.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:24.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:24.360 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:24.360 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:24.360 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:24.360 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:24.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:24.360 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:24.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:24.360 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:24.360 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:24.376 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:24.388Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:24.388Z [INFO] agent.router.manager: shutting down +30:24.388 [INFO] TestAgent: consul server down +30:24.388 [INFO] TestAgent: shutdown complete +30:24.388 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30093 network=tcp +30:24.388 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30093 network=udp +30:24.388 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30094 network=tcp +30:24.892 [INFO] TestAgent: Waiting for endpoints to shut down +30:24.892 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_loadServices_sidecarSeparateToken (0.00s) - --- PASS: TestAgent_loadServices_sidecarSeparateToken/service_manager (0.67s) - --- PASS: TestAgent_loadServices_sidecarSeparateToken/normal (0.88s) + --- PASS: TestAgent_loadServices_sidecarSeparateToken/service_manager (0.93s) + --- PASS: TestAgent_loadServices_sidecarSeparateToken/normal (0.95s) === RUN TestAgent_loadServices_sidecarInheritMeta === RUN TestAgent_loadServices_sidecarInheritMeta/normal === PAUSE TestAgent_loadServices_sidecarInheritMeta/normal @@ -12437,133 +12631,118 @@ === PAUSE TestAgent_loadServices_sidecarInheritMeta/service_manager === CONT TestAgent_loadServices_sidecarInheritMeta/normal === CONT TestAgent_loadServices_sidecarInheritMeta/service_manager -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:20.445Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:20.454Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:20.480 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c90176c9-c3cc-d731-7ec6-05b44e33a558 Address:127.0.0.1:11741}]" -45:20.480 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11741 [Follower]" leader= -45:20.480 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c90176c9-c3cc-d731-7ec6-05b44e33a558.dc1 127.0.0.1 -45:20.481 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 127.0.0.1 -2022-11-07T22:45:20.481Z [INFO] agent.router: Initializing LAN area manager -45:20.481 [INFO] TestAgent.server: Adding LAN server: server="Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 (Addr: tcp/127.0.0.1:11741) (DC: dc1)" -45:20.481 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c90176c9-c3cc-d731-7ec6-05b44e33a558.dc1 area=wan -45:20.481 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy -45:20.481 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11736 network=udp -45:20.481 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11736 network=tcp -45:20.481 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11737 network=tcp -45:20.481 [INFO] TestAgent: started state syncer -45:20.481 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11742 network=tcp -45:20.539 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:20.539 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11741 [Candidate]" term=2 -45:20.550 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:20.550 [DEBUG] TestAgent.server.raft: vote granted: from=c90176c9-c3cc-d731-7ec6-05b44e33a558 term=2 tally=1 -45:20.550 [INFO] TestAgent.server.raft: election won: tally=1 -45:20.550 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11741 [Leader]" -45:20.551 [INFO] TestAgent.server: cluster leadership acquired -45:20.551 [INFO] TestAgent.server: New leader elected: payload=Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 -45:20.556 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11741 -45:20.576 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:20.592 [INFO] TestAgent: Requesting shutdown -45:20.592 [INFO] TestAgent.server: shutting down server -45:20.592 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:20.592 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:20.596 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:20.600 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:20.600 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:20.600 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:20.600 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:20.600 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:20.600 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 -45:20.600 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 -2022-11-07T22:45:20.601Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:20.601Z [INFO] agent.router.manager: shutting down -45:20.601 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: raft is already shutdown" -45:20.605 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-c90176c9-c3cc-d731-7ec6-05b44e33a558 127.0.0.1 11739 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:c90176c9-c3cc-d731-7ec6-05b44e33a558 port:11741 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11740] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -45:20.605 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:20.605 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:20.605 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:20.605 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:20.605 [INFO] TestAgent: consul server down -45:20.605 [INFO] TestAgent: shutdown complete -45:20.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11736 network=tcp -45:20.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11736 network=udp -45:20.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:20.605 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11737 network=tcp -45:20.605 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:20.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:20.605 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:21.105 [INFO] TestAgent: Waiting for endpoints to shut down -45:21.105 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:20.446Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:20.458Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:20.480 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5c7e746e-f8c8-7ab3-793f-648f5d335be0 Address:127.0.0.1:11734}]" -45:20.480 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11734 [Follower]" leader= -45:20.480 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0.dc1 127.0.0.1 -45:20.481 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0 127.0.0.1 -2022-11-07T22:45:20.481Z [INFO] agent.router: Initializing LAN area manager -45:20.481 [INFO] TestAgent.server: Adding LAN server: server="Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0 (Addr: tcp/127.0.0.1:11734) (DC: dc1)" -45:20.481 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0.dc1 area=wan -45:20.481 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11729 network=udp -45:20.481 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11729 network=tcp -45:20.481 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11730 network=tcp -45:20.481 [INFO] TestAgent: started state syncer -45:20.481 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11735 network=tcp -45:20.529 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:20.529 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11734 [Candidate]" term=2 -45:20.537 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:20.537 [DEBUG] TestAgent.server.raft: vote granted: from=5c7e746e-f8c8-7ab3-793f-648f5d335be0 term=2 tally=1 -45:20.537 [INFO] TestAgent.server.raft: election won: tally=1 -45:20.537 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11734 [Leader]" -45:20.538 [INFO] TestAgent.server: cluster leadership acquired -45:20.538 [INFO] TestAgent.server: New leader elected: payload=Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0 -45:20.544 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11734 -45:20.557 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:20.585 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:20.585 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:20.585 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:20.585 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:20.585 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:20.585 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0 -45:20.585 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5c7e746e-f8c8-7ab3-793f-648f5d335be0 -45:20.595 [INFO] TestAgent.server: federation state anti-entropy synced -45:20.682 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:20.685 [INFO] TestAgent: Synced node info -45:20.690 [INFO] TestAgent: Synced service: service=rabbitmq -45:20.693 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy -45:20.693 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:20.693 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:20.693 [DEBUG] TestAgent: Node info in sync -45:20.693 [DEBUG] TestAgent: Service in sync: service=rabbitmq -45:20.693 [DEBUG] TestAgent: Service in sync: service=rabbitmq-sidecar-proxy -45:20.693 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:20.693 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:20.693 [INFO] TestAgent: Requesting shutdown -45:20.693 [INFO] TestAgent.server: shutting down server -45:20.693 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:20.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:20.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:20.693 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:20.693 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:20.693 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:20.693 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:20.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:20.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:20.693 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:20.696 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:20.700Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:20.700Z [INFO] agent.router.manager: shutting down -45:20.700 [INFO] TestAgent: consul server down -45:20.700 [INFO] TestAgent: shutdown complete -45:20.700 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11729 network=tcp -45:20.700 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11729 network=udp -45:20.700 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11730 network=tcp -45:21.200 [INFO] TestAgent: Waiting for endpoints to shut down -45:21.200 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:24.918Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:24.946Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.986 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a5768502-4cb0-264c-3a10-80e5557be89a Address:127.0.0.1:30112}]" +30:24.986 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a5768502-4cb0-264c-3a10-80e5557be89a.dc1 127.0.0.1 +30:24.987 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a5768502-4cb0-264c-3a10-80e5557be89a 127.0.0.1 +2021-10-05T16:30:24.987Z [INFO] agent.router: Initializing LAN area manager +30:24.987 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30107 network=udp +30:24.987 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30112 [Follower]" leader= +30:24.988 [INFO] TestAgent.server: Adding LAN server: server="Node-a5768502-4cb0-264c-3a10-80e5557be89a (Addr: tcp/127.0.0.1:30112) (DC: dc1)" +30:24.988 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a5768502-4cb0-264c-3a10-80e5557be89a.dc1 area=wan +30:24.988 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30107 network=tcp +30:24.988 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30108 network=tcp +30:24.988 [INFO] TestAgent: started state syncer +30:24.989 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30113 network=tcp +30:25.059 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:25.059 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30112 [Candidate]" term=2 +30:25.076 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:25.076 [DEBUG] TestAgent.server.raft: vote granted: from=a5768502-4cb0-264c-3a10-80e5557be89a term=2 tally=1 +30:25.076 [INFO] TestAgent.server.raft: election won: tally=1 +30:25.076 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30112 [Leader]" +30:25.076 [INFO] TestAgent.server: cluster leadership acquired +30:25.076 [INFO] TestAgent.server: New leader elected: payload=Node-a5768502-4cb0-264c-3a10-80e5557be89a +30:25.086 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30112 +30:25.105 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:25.114 [INFO] TestAgent: Requesting shutdown +30:25.114 [INFO] TestAgent.server: shutting down server +30:25.114 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:25.114 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:25.128 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:25.136Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:25.136Z [INFO] agent.router.manager: shutting down +30:25.150 [INFO] TestAgent: consul server down +30:25.150 [INFO] TestAgent: shutdown complete +30:25.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30107 network=tcp +30:25.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30107 network=udp +30:25.150 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30108 network=tcp +30:25.151 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +30:25.151 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +30:25.151 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +30:25.651 [INFO] TestAgent: Waiting for endpoints to shut down +30:25.652 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:24.920Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:24.937Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.978 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e Address:127.0.0.1:30119}]" +30:24.979 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e.dc1 127.0.0.1 +30:24.980 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30119 [Follower]" leader= +30:24.980 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e 127.0.0.1 +2021-10-05T16:30:24.980Z [INFO] agent.router: Initializing LAN area manager +30:24.980 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e.dc1 area=wan +30:24.980 [INFO] TestAgent.server: Adding LAN server: server="Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e (Addr: tcp/127.0.0.1:30119) (DC: dc1)" +30:24.980 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy +30:24.981 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30114 network=udp +30:24.981 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30114 network=tcp +30:24.981 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30115 network=tcp +30:24.981 [INFO] TestAgent: started state syncer +30:24.981 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30120 network=tcp +30:25.018 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:25.018 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30119 [Candidate]" term=2 +30:25.028 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:25.028 [DEBUG] TestAgent.server.raft: vote granted: from=a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e term=2 tally=1 +30:25.028 [INFO] TestAgent.server.raft: election won: tally=1 +30:25.028 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30119 [Leader]" +30:25.028 [INFO] TestAgent.server: cluster leadership acquired +30:25.028 [INFO] TestAgent.server: New leader elected: payload=Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e +30:25.036 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30119 +30:25.066 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:25.094 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:25.101 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:25.103 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:25.108 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:25.129 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:25.129 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:25.129 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:25.129 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:25.129 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:25.129 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e +30:25.129 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a0b7a037-3d3d-e558-52d9-b3c0f5a0cf7e +30:25.153 [INFO] TestAgent.server: federation state anti-entropy synced +30:25.243 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:25.249 [INFO] TestAgent: Synced node info +30:25.255 [INFO] TestAgent: Synced service: service=rabbitmq +30:25.272 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy +30:25.272 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 +30:25.272 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 +30:25.272 [INFO] TestAgent: Requesting shutdown +30:25.272 [INFO] TestAgent.server: shutting down server +30:25.272 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:25.272 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:25.272 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:25.272 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:25.272 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:25.272 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:25.272 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:25.272 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:25.272 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:25.272 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:25.281 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:25.290Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:25.290Z [INFO] agent.router.manager: shutting down +30:25.290 [INFO] TestAgent: consul server down +30:25.290 [INFO] TestAgent: shutdown complete +30:25.290 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30114 network=tcp +30:25.290 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30114 network=udp +30:25.290 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30115 network=tcp +30:25.790 [INFO] TestAgent: Waiting for endpoints to shut down +30:25.790 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_loadServices_sidecarInheritMeta (0.00s) - --- PASS: TestAgent_loadServices_sidecarInheritMeta/service_manager (0.67s) - --- PASS: TestAgent_loadServices_sidecarInheritMeta/normal (0.77s) + --- PASS: TestAgent_loadServices_sidecarInheritMeta/normal (0.76s) + --- PASS: TestAgent_loadServices_sidecarInheritMeta/service_manager (0.90s) === RUN TestAgent_loadServices_sidecarOverrideMeta === RUN TestAgent_loadServices_sidecarOverrideMeta/normal === PAUSE TestAgent_loadServices_sidecarOverrideMeta/normal @@ -12574,116 +12753,108 @@ [WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:21.211Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:21.220Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:21.247 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bbc3465d-204d-11ea-b19d-b12817e20883 Address:127.0.0.1:11755}]" -45:21.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11755 [Follower]" leader= -45:21.248 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bbc3465d-204d-11ea-b19d-b12817e20883.dc1 127.0.0.1 -45:21.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bbc3465d-204d-11ea-b19d-b12817e20883 127.0.0.1 -2022-11-07T22:45:21.249Z [INFO] agent.router: Initializing LAN area manager -45:21.249 [INFO] TestAgent.server: Adding LAN server: server="Node-bbc3465d-204d-11ea-b19d-b12817e20883 (Addr: tcp/127.0.0.1:11755) (DC: dc1)" -45:21.249 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bbc3465d-204d-11ea-b19d-b12817e20883.dc1 area=wan -45:21.249 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy -45:21.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11750 network=tcp -45:21.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11750 network=udp -45:21.249 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11751 network=tcp -45:21.249 [INFO] TestAgent: started state syncer -45:21.249 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11756 network=tcp -45:21.305 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:21.305 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11755 [Candidate]" term=2 -45:21.313 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:21.313 [DEBUG] TestAgent.server.raft: vote granted: from=bbc3465d-204d-11ea-b19d-b12817e20883 term=2 tally=1 -45:21.313 [INFO] TestAgent.server.raft: election won: tally=1 -45:21.313 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11755 [Leader]" -45:21.313 [INFO] TestAgent.server: cluster leadership acquired -45:21.314 [INFO] TestAgent.server: New leader elected: payload=Node-bbc3465d-204d-11ea-b19d-b12817e20883 -45:21.320 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11755 -45:21.322 [INFO] TestAgent: Requesting shutdown -45:21.322 [INFO] TestAgent.server: shutting down server -45:21.322 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:21.322 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:21.330 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:21.334 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:45:21.335Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:21.335Z [INFO] agent.router.manager: shutting down -45:21.343 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -45:21.343 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -45:21.343 [INFO] TestAgent: consul server down -45:21.343 [INFO] TestAgent: shutdown complete -45:21.343 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11750 network=tcp -45:21.343 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11750 network=udp -45:21.343 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11751 network=tcp -45:21.844 [INFO] TestAgent: Waiting for endpoints to shut down -45:21.844 [INFO] TestAgent: Endpoints down -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:21.212Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:21.221Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:21.245 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2a9360b0-4ba2-b280-308e-4aa0e1210f14 Address:127.0.0.1:11748}]" -45:21.246 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11748 [Follower]" leader= -45:21.246 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14.dc1 127.0.0.1 -45:21.247 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14 127.0.0.1 -2022-11-07T22:45:21.247Z [INFO] agent.router: Initializing LAN area manager -45:21.247 [INFO] TestAgent.server: Adding LAN server: server="Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14 (Addr: tcp/127.0.0.1:11748) (DC: dc1)" -45:21.247 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14.dc1 area=wan -45:21.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11743 network=udp -45:21.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11743 network=tcp -45:21.247 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11744 network=tcp -45:21.247 [INFO] TestAgent: started state syncer -45:21.247 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11749 network=tcp -45:21.282 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:21.282 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11748 [Candidate]" term=2 -45:21.290 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:21.290 [DEBUG] TestAgent.server.raft: vote granted: from=2a9360b0-4ba2-b280-308e-4aa0e1210f14 term=2 tally=1 -45:21.290 [INFO] TestAgent.server.raft: election won: tally=1 -45:21.290 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11748 [Leader]" -45:21.290 [INFO] TestAgent.server: cluster leadership acquired -45:21.290 [INFO] TestAgent.server: New leader elected: payload=Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14 -45:21.296 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11748 -45:21.307 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:21.329 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:21.329 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:21.329 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:21.329 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:21.329 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:21.329 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14 -45:21.329 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2a9360b0-4ba2-b280-308e-4aa0e1210f14 -45:21.336 [INFO] TestAgent.server: federation state anti-entropy synced -45:21.548 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:21.551 [INFO] TestAgent: Synced node info -45:21.556 [INFO] TestAgent: Synced service: service=rabbitmq -45:21.562 [INFO] TestAgent: Synced service: service=rabbitmq-sidecar-proxy -45:21.562 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:21.562 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:21.562 [DEBUG] TestAgent: Node info in sync -45:21.562 [DEBUG] TestAgent: Service in sync: service=rabbitmq -45:21.562 [DEBUG] TestAgent: Service in sync: service=rabbitmq-sidecar-proxy -45:21.562 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:1 -45:21.562 [DEBUG] TestAgent: Check in sync: check=service:rabbitmq-sidecar-proxy:2 -45:21.639 [INFO] TestAgent: Requesting shutdown -45:21.639 [INFO] TestAgent.server: shutting down server -45:21.639 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:21.639 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:21.639 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:21.639 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:21.639 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:21.639 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:21.639 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:21.639 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:21.639 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:21.639 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:21.643 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:21.646Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:21.646Z [INFO] agent.router.manager: shutting down -45:21.646 [INFO] TestAgent: consul server down -45:21.646 [INFO] TestAgent: shutdown complete -45:21.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11743 network=tcp -45:21.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11743 network=udp -45:21.646 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11744 network=tcp -45:22.146 [INFO] TestAgent: Waiting for endpoints to shut down -45:22.146 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:25.841Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:25.903Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:26.000 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a1334de-57a1-5d08-84bf-d69fbfc34429 Address:127.0.0.1:30126}]" +30:26.000 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30126 [Follower]" leader= +30:26.001 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8a1334de-57a1-5d08-84bf-d69fbfc34429.dc1 127.0.0.1 +30:26.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8a1334de-57a1-5d08-84bf-d69fbfc34429 127.0.0.1 +2021-10-05T16:30:26.001Z [INFO] agent.router: Initializing LAN area manager +30:26.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30121 network=udp +30:26.002 [INFO] TestAgent.server: Adding LAN server: server="Node-8a1334de-57a1-5d08-84bf-d69fbfc34429 (Addr: tcp/127.0.0.1:30126) (DC: dc1)" +30:26.002 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8a1334de-57a1-5d08-84bf-d69fbfc34429.dc1 area=wan +30:26.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30121 network=tcp +30:26.003 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30122 network=tcp +30:26.003 [INFO] TestAgent: started state syncer +30:26.003 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30127 network=tcp +30:26.052 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:26.052 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30126 [Candidate]" term=2 +30:26.104 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:26.104 [DEBUG] TestAgent.server.raft: vote granted: from=8a1334de-57a1-5d08-84bf-d69fbfc34429 term=2 tally=1 +30:26.104 [INFO] TestAgent.server.raft: election won: tally=1 +30:26.104 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30126 [Leader]" +30:26.107 [INFO] TestAgent.server: cluster leadership acquired +30:26.107 [INFO] TestAgent.server: New leader elected: payload=Node-8a1334de-57a1-5d08-84bf-d69fbfc34429 +30:26.123 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30126 +30:26.164 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:26.172 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:26.172 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:26.175 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:26.175 [ERROR] TestAgent.proxycfg: watch error: service_id=rabbitmq-sidecar-proxy id=leaf error="error filling agent cache: internal error: CA root is nil" +30:26.188 [INFO] TestAgent: Requesting shutdown +30:26.188 [INFO] TestAgent.server: shutting down server +30:26.188 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:26.188 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:26.196 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:26.203Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:26.203Z [INFO] agent.router.manager: shutting down +30:26.206 [INFO] TestAgent: consul server down +30:26.206 [INFO] TestAgent: shutdown complete +30:26.206 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30121 network=tcp +30:26.206 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30121 network=udp +30:26.206 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30122 network=tcp +30:26.206 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +30:26.712 [INFO] TestAgent: Waiting for endpoints to shut down +30:26.712 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:25.843Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:25.923Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:26.003 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:71a79908-f0f1-31ca-3875-1aa3bd67aa28 Address:127.0.0.1:30133}]" +30:26.004 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30133 [Follower]" leader= +30:26.004 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28.dc1 127.0.0.1 +30:26.005 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 127.0.0.1 +2021-10-05T16:30:26.005Z [INFO] agent.router: Initializing LAN area manager +30:26.005 [DEBUG] TestAgent: added local registration for service: service=rabbitmq-sidecar-proxy +30:26.005 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30128 network=udp +30:26.005 [INFO] TestAgent.server: Adding LAN server: server="Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 (Addr: tcp/127.0.0.1:30133) (DC: dc1)" +30:26.005 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28.dc1 area=wan +30:26.006 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30128 network=tcp +30:26.006 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30129 network=tcp +30:26.006 [INFO] TestAgent: started state syncer +30:26.007 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30134 network=tcp +30:26.052 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:26.052 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30133 [Candidate]" term=2 +30:26.096 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:26.096 [DEBUG] TestAgent.server.raft: vote granted: from=71a79908-f0f1-31ca-3875-1aa3bd67aa28 term=2 tally=1 +30:26.096 [INFO] TestAgent.server.raft: election won: tally=1 +30:26.096 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30133 [Leader]" +30:26.100 [INFO] TestAgent.server: cluster leadership acquired +30:26.100 [INFO] TestAgent.server: New leader elected: payload=Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 +30:26.117 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30133 +30:26.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:26.216 [INFO] TestAgent: Requesting shutdown +30:26.216 [INFO] TestAgent.server: shutting down server +30:26.216 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:26.216 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:26.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +30:26.232 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:26.232 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:26.232 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:26.232 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:26.232 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:26.232 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 +30:26.232 [INFO] TestAgent.server: member joined, marking health alive: member=Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 +2021-10-05T16:30:26.236Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:26.236Z [INFO] agent.router.manager: shutting down +30:26.252 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-71a79908-f0f1-31ca-3875-1aa3bd67aa28 127.0.0.1 30131 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:71a79908-f0f1-31ca-3875-1aa3bd67aa28 port:30133 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30132] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +30:26.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:26.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:26.252 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:26.252 [INFO] TestAgent: consul server down +30:26.252 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:26.252 [INFO] TestAgent: shutdown complete +30:26.252 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:26.252 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30128 network=tcp +30:26.252 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30128 network=udp +30:26.252 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:26.252 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:26.252 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30129 network=tcp +30:26.752 [INFO] TestAgent: Waiting for endpoints to shut down +30:26.752 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! --- PASS: TestAgent_loadServices_sidecarOverrideMeta (0.00s) - --- PASS: TestAgent_loadServices_sidecarOverrideMeta/service_manager (0.64s) - --- PASS: TestAgent_loadServices_sidecarOverrideMeta/normal (0.95s) + --- PASS: TestAgent_loadServices_sidecarOverrideMeta/normal (1.04s) + --- PASS: TestAgent_loadServices_sidecarOverrideMeta/service_manager (1.04s) === RUN TestAgent_unloadServices === RUN TestAgent_unloadServices/normal === PAUSE TestAgent_unloadServices/normal @@ -12691,127 +12862,111 @@ === PAUSE TestAgent_unloadServices/service_manager === CONT TestAgent_unloadServices/normal === CONT TestAgent_unloadServices/service_manager -[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:22.158Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:22.171Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:22.199 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a729ce7f-559b-d17c-7f88-c8cb5ea0c074 Address:127.0.0.1:11769}]" -45:22.199 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11769 [Follower]" leader= -45:22.200 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074.dc1 127.0.0.1 -45:22.200 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074 127.0.0.1 -2022-11-07T22:45:22.200Z [INFO] agent.router: Initializing LAN area manager -45:22.200 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074.dc1 area=wan -45:22.200 [INFO] TestAgent.server: Adding LAN server: server="Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074 (Addr: tcp/127.0.0.1:11769) (DC: dc1)" -45:22.200 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11764 network=tcp -45:22.200 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11764 network=udp -45:22.201 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11765 network=tcp -45:22.201 [INFO] TestAgent: started state syncer -45:22.201 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11770 network=tcp -45:22.237 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:22.237 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11769 [Candidate]" term=2 -45:22.246 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:22.246 [DEBUG] TestAgent.server.raft: vote granted: from=a729ce7f-559b-d17c-7f88-c8cb5ea0c074 term=2 tally=1 -45:22.246 [INFO] TestAgent.server.raft: election won: tally=1 -45:22.246 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11769 [Leader]" -45:22.246 [INFO] TestAgent.server: cluster leadership acquired -45:22.246 [INFO] TestAgent.server: New leader elected: payload=Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074 -45:22.251 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11769 -45:22.267 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:22.286 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:22.286 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:22.286 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:22.286 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:22.286 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:22.287 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074 -45:22.287 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a729ce7f-559b-d17c-7f88-c8cb5ea0c074 -45:22.296 [INFO] TestAgent.server: federation state anti-entropy synced -45:22.431 [DEBUG] TestAgent: removed service: service=redis -45:22.431 [INFO] TestAgent: Requesting shutdown -45:22.431 [INFO] TestAgent.server: shutting down server -45:22.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:22.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:22.431 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:22.431 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:22.431 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:22.431 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:22.431 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:22.431 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:22.431 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:22.431 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:22.431 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:22.435 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:22.438Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:22.438Z [INFO] agent.router.manager: shutting down -45:22.438 [INFO] TestAgent: consul server down -45:22.438 [INFO] TestAgent: shutdown complete -45:22.438 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11764 network=tcp -45:22.438 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11764 network=udp -45:22.438 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11765 network=tcp -45:22.938 [INFO] TestAgent: Waiting for endpoints to shut down -45:22.938 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:26.863Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:26.910Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:26.996 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d Address:127.0.0.1:30147}]" +30:26.997 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d.dc1 127.0.0.1 +30:26.998 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d 127.0.0.1 +2021-10-05T16:30:26.998Z [INFO] agent.router: Initializing LAN area manager +30:26.998 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30142 network=udp +30:26.998 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30147 [Follower]" leader= +30:26.998 [INFO] TestAgent.server: Adding LAN server: server="Node-9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d (Addr: tcp/127.0.0.1:30147) (DC: dc1)" +30:26.998 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d.dc1 area=wan +30:26.998 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30142 network=tcp +30:26.999 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30143 network=tcp +30:26.999 [INFO] TestAgent: started state syncer +30:26.999 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30148 network=tcp +30:27.040 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:27.040 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30147 [Candidate]" term=2 +30:27.095 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:27.095 [DEBUG] TestAgent.server.raft: vote granted: from=9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d term=2 tally=1 +30:27.095 [INFO] TestAgent.server.raft: election won: tally=1 +30:27.095 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30147 [Leader]" +30:27.096 [INFO] TestAgent.server: cluster leadership acquired +30:27.096 [INFO] TestAgent.server: New leader elected: payload=Node-9ea4d602-ed2c-92f2-8c1b-cd6adb6ff29d +30:27.125 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30147 +30:27.176 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:27.188 [DEBUG] TestAgent: removed service: service=redis +30:27.188 [INFO] TestAgent: Requesting shutdown +30:27.188 [INFO] TestAgent.server: shutting down server +30:27.188 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:27.188 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:27.204 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:27.216Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:27.216Z [INFO] agent.router.manager: shutting down +30:27.220 [INFO] TestAgent: consul server down +30:27.220 [INFO] TestAgent: shutdown complete +30:27.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30142 network=tcp +30:27.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30142 network=udp +30:27.220 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30143 network=tcp +30:27.220 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +30:27.220 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +30:27.724 [INFO] TestAgent: Waiting for endpoints to shut down +30:27.724 [INFO] TestAgent: Endpoints down [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:22.158Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:22.172Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:22.201 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2726467e-cf0c-f884-5907-e189febfce2b Address:127.0.0.1:11762}]" -45:22.201 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11762 [Follower]" leader= -45:22.202 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2726467e-cf0c-f884-5907-e189febfce2b.dc1 127.0.0.1 -45:22.202 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2726467e-cf0c-f884-5907-e189febfce2b 127.0.0.1 -2022-11-07T22:45:22.202Z [INFO] agent.router: Initializing LAN area manager -45:22.202 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2726467e-cf0c-f884-5907-e189febfce2b.dc1 area=wan -45:22.202 [INFO] TestAgent.server: Adding LAN server: server="Node-2726467e-cf0c-f884-5907-e189febfce2b (Addr: tcp/127.0.0.1:11762) (DC: dc1)" -45:22.202 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11757 network=udp -45:22.202 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11757 network=tcp -45:22.202 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11758 network=tcp -45:22.202 [INFO] TestAgent: started state syncer -45:22.202 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11763 network=tcp -45:22.258 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:22.258 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11762 [Candidate]" term=2 -45:22.269 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:22.269 [DEBUG] TestAgent.server.raft: vote granted: from=2726467e-cf0c-f884-5907-e189febfce2b term=2 tally=1 -45:22.269 [INFO] TestAgent.server.raft: election won: tally=1 -45:22.269 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11762 [Leader]" -45:22.270 [INFO] TestAgent.server: cluster leadership acquired -45:22.270 [INFO] TestAgent.server: New leader elected: payload=Node-2726467e-cf0c-f884-5907-e189febfce2b -45:22.276 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11762 -45:22.289 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:22.307 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:22.307 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:22.307 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:22.307 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:22.307 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:22.307 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2726467e-cf0c-f884-5907-e189febfce2b -45:22.307 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2726467e-cf0c-f884-5907-e189febfce2b -45:22.311 [INFO] TestAgent.server: federation state anti-entropy synced -45:22.490 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:22.494 [INFO] TestAgent: Synced node info -45:22.494 [DEBUG] TestAgent: Node info in sync -45:22.624 [DEBUG] TestAgent: removed service: service=redis -45:22.624 [INFO] TestAgent: Requesting shutdown -45:22.624 [INFO] TestAgent.server: shutting down server -45:22.624 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:22.624 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:22.624 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:22.624 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:22.624 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:22.624 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:22.624 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:22.624 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:22.624 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:22.624 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:22.628 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:22.631Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:22.631Z [INFO] agent.router.manager: shutting down -45:22.631 [INFO] TestAgent: consul server down -45:22.631 [INFO] TestAgent: shutdown complete -45:22.631 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11757 network=tcp -45:22.631 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11757 network=udp -45:22.631 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11758 network=tcp -45:23.134 [INFO] TestAgent: Waiting for endpoints to shut down -45:23.134 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:26.885Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:26.921Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:27.000 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:864458e4-6e50-275c-6ae0-49c599edcebb Address:127.0.0.1:30140}]" +30:27.001 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-864458e4-6e50-275c-6ae0-49c599edcebb.dc1 127.0.0.1 +30:27.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-864458e4-6e50-275c-6ae0-49c599edcebb 127.0.0.1 +2021-10-05T16:30:27.001Z [INFO] agent.router: Initializing LAN area manager +30:27.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30135 network=udp +30:27.002 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30140 [Follower]" leader= +30:27.002 [INFO] TestAgent.server: Adding LAN server: server="Node-864458e4-6e50-275c-6ae0-49c599edcebb (Addr: tcp/127.0.0.1:30140) (DC: dc1)" +30:27.002 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-864458e4-6e50-275c-6ae0-49c599edcebb.dc1 area=wan +30:27.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30135 network=tcp +30:27.003 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30136 network=tcp +30:27.003 [INFO] TestAgent: started state syncer +30:27.003 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30141 network=tcp +30:27.040 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:27.040 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30140 [Candidate]" term=2 +30:27.096 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:27.096 [DEBUG] TestAgent.server.raft: vote granted: from=864458e4-6e50-275c-6ae0-49c599edcebb term=2 tally=1 +30:27.096 [INFO] TestAgent.server.raft: election won: tally=1 +30:27.096 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30140 [Leader]" +30:27.096 [INFO] TestAgent.server: cluster leadership acquired +30:27.096 [INFO] TestAgent.server: New leader elected: payload=Node-864458e4-6e50-275c-6ae0-49c599edcebb +30:27.124 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30140 +30:27.180 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:27.248 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:27.248 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:27.248 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:27.248 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:27.248 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:27.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-864458e4-6e50-275c-6ae0-49c599edcebb +30:27.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-864458e4-6e50-275c-6ae0-49c599edcebb +30:27.280 [INFO] TestAgent.server: federation state anti-entropy synced +30:27.324 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:27.340 [INFO] TestAgent: Synced node info +30:27.405 [DEBUG] TestAgent: removed service: service=redis +30:27.405 [INFO] TestAgent: Requesting shutdown +30:27.405 [INFO] TestAgent.server: shutting down server +30:27.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:27.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:27.405 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:27.405 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:27.405 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:27.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:27.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:27.405 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:27.405 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:27.405 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:27.412 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:27.419Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:27.419Z [INFO] agent.router.manager: shutting down +30:27.419 [INFO] TestAgent: consul server down +30:27.419 [INFO] TestAgent: shutdown complete +30:27.419 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30135 network=tcp +30:27.419 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30135 network=udp +30:27.419 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30136 network=tcp +30:27.920 [INFO] TestAgent: Waiting for endpoints to shut down +30:27.920 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_unloadServices (0.00s) - --- PASS: TestAgent_unloadServices/service_manager (0.79s) - --- PASS: TestAgent_unloadServices/normal (0.99s) + --- PASS: TestAgent_unloadServices/service_manager (0.88s) + --- PASS: TestAgent_unloadServices/normal (1.08s) === RUN TestAgent_Service_MaintenanceMode === PAUSE TestAgent_Service_MaintenanceMode === RUN TestAgent_Service_Reap @@ -12820,69 +12975,75 @@ [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:23.146Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:23.155Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:23.175 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:043f4963-1bcb-691e-fcbe-131d0842707f Address:127.0.0.1:11776}]" -45:23.175 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11776 [Follower]" leader= -45:23.175 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-043f4963-1bcb-691e-fcbe-131d0842707f.dc1 127.0.0.1 -45:23.176 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-043f4963-1bcb-691e-fcbe-131d0842707f 127.0.0.1 -2022-11-07T22:45:23.176Z [INFO] agent.router: Initializing LAN area manager -45:23.176 [INFO] TestAgent.server: Adding LAN server: server="Node-043f4963-1bcb-691e-fcbe-131d0842707f (Addr: tcp/127.0.0.1:11776) (DC: dc1)" -45:23.176 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-043f4963-1bcb-691e-fcbe-131d0842707f.dc1 area=wan -45:23.176 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11771 network=udp -45:23.176 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11771 network=tcp -45:23.177 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11772 network=tcp -45:23.177 [INFO] TestAgent: started state syncer -45:23.177 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11777 network=tcp -45:23.238 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:23.238 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11776 [Candidate]" term=2 -45:23.245 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:23.245 [DEBUG] TestAgent.server.raft: vote granted: from=043f4963-1bcb-691e-fcbe-131d0842707f term=2 tally=1 -45:23.245 [INFO] TestAgent.server.raft: election won: tally=1 -45:23.245 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11776 [Leader]" -45:23.246 [INFO] TestAgent.server: cluster leadership acquired -45:23.246 [INFO] TestAgent.server: New leader elected: payload=Node-043f4963-1bcb-691e-fcbe-131d0842707f -45:23.250 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11776 -45:23.261 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:23.278 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:23.278 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:23.278 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:23.278 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:23.278 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:23.278 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-043f4963-1bcb-691e-fcbe-131d0842707f -45:23.278 [INFO] TestAgent.server: member joined, marking health alive: member=Node-043f4963-1bcb-691e-fcbe-131d0842707f -45:23.287 [INFO] TestAgent.server: federation state anti-entropy synced -45:23.339 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:23.343 [INFO] TestAgent: Synced node info -45:23.388 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis -45:23.463 [DEBUG] TestAgent: Check status updated: check=service:redis status=passing -45:23.489 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis -45:23.728 [DEBUG] TestAgent: removed check: check=service:redis -45:23.728 [DEBUG] TestAgent: removed service: service=redis -45:23.728 [INFO] TestAgent: Check for service has been critical for too long; deregistered service: service=redis check=service:redis -45:23.964 [INFO] TestAgent: Requesting shutdown -45:23.964 [INFO] TestAgent.server: shutting down server -45:23.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:23.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:23.964 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:23.964 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:23.964 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:23.964 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:23.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:23.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:23.964 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:23.964 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:23.972 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:23.976Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:23.976Z [INFO] agent.router.manager: shutting down -45:23.976 [INFO] TestAgent: consul server down -45:23.976 [INFO] TestAgent: shutdown complete -45:23.976 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11771 network=tcp -45:23.976 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11771 network=udp -45:23.976 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11772 network=tcp -45:24.476 [INFO] TestAgent: Waiting for endpoints to shut down -45:24.476 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Service_Reap (1.34s) +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:27.965Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:28.000Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:28.088 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b9f847f0-7c29-ec0c-8294-009e40e2794f Address:127.0.0.1:30154}]" +30:28.089 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b9f847f0-7c29-ec0c-8294-009e40e2794f.dc1 127.0.0.1 +30:28.089 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b9f847f0-7c29-ec0c-8294-009e40e2794f 127.0.0.1 +2021-10-05T16:30:28.089Z [INFO] agent.router: Initializing LAN area manager +30:28.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30149 network=udp +30:28.089 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30154 [Follower]" leader= +30:28.090 [INFO] TestAgent.server: Adding LAN server: server="Node-b9f847f0-7c29-ec0c-8294-009e40e2794f (Addr: tcp/127.0.0.1:30154) (DC: dc1)" +30:28.090 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b9f847f0-7c29-ec0c-8294-009e40e2794f.dc1 area=wan +30:28.090 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30149 network=tcp +30:28.090 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30150 network=tcp +30:28.090 [INFO] TestAgent: started state syncer +30:28.090 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30155 network=tcp +30:28.132 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:28.132 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30154 [Candidate]" term=2 +30:28.172 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:28.172 [DEBUG] TestAgent.server.raft: vote granted: from=b9f847f0-7c29-ec0c-8294-009e40e2794f term=2 tally=1 +30:28.172 [INFO] TestAgent.server.raft: election won: tally=1 +30:28.172 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30154 [Leader]" +30:28.176 [INFO] TestAgent.server: cluster leadership acquired +30:28.176 [INFO] TestAgent.server: New leader elected: payload=Node-b9f847f0-7c29-ec0c-8294-009e40e2794f +30:28.211 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30154 +30:28.229 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:28.291 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:28.291 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:28.291 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:28.291 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:28.291 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:28.291 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b9f847f0-7c29-ec0c-8294-009e40e2794f +30:28.291 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b9f847f0-7c29-ec0c-8294-009e40e2794f +30:28.304 [INFO] TestAgent.server: federation state anti-entropy synced +30:28.340 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis +30:28.397 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:28.419 [INFO] TestAgent: Synced node info +30:28.428 [INFO] TestAgent: Synced service: service=redis +30:28.428 [DEBUG] TestAgent: Check in sync: check=service:redis +30:28.428 [DEBUG] TestAgent: Check status updated: check=service:redis status=passing +30:28.428 [DEBUG] TestAgent: Node info in sync +30:28.428 [DEBUG] TestAgent: Service in sync: service=redis +30:28.428 [DEBUG] TestAgent: Check in sync: check=service:redis +30:28.456 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis +30:28.684 [DEBUG] TestAgent: removed check: check=service:redis +30:28.684 [DEBUG] TestAgent: removed service: service=redis +30:28.684 [INFO] TestAgent: Check for service has been critical for too long; deregistered service: service=redis check=service:redis +30:28.929 [INFO] TestAgent: Requesting shutdown +30:28.929 [INFO] TestAgent.server: shutting down server +30:28.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:28.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:28.929 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:28.929 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:28.929 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:28.930 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:28.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:28.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:28.930 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:28.930 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:28.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:28.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:28.948Z [INFO] agent.router.manager: shutting down +30:28.948 [INFO] TestAgent: consul server down +30:28.948 [INFO] TestAgent: shutdown complete +30:28.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30149 network=tcp +30:28.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30149 network=udp +30:28.948 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30150 network=tcp +30:29.449 [INFO] TestAgent: Waiting for endpoints to shut down +30:29.449 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Service_Reap (1.53s) === RUN TestAgent_Service_NoReap [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! @@ -12890,64 +13051,72 @@ [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:24.489Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:24.498Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:24.522 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cda5f00d-85e2-dc89-179d-7a6f156330b8 Address:127.0.0.1:11783}]" -45:24.522 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11783 [Follower]" leader= -45:24.523 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cda5f00d-85e2-dc89-179d-7a6f156330b8.dc1 127.0.0.1 -45:24.524 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cda5f00d-85e2-dc89-179d-7a6f156330b8 127.0.0.1 -2022-11-07T22:45:24.524Z [INFO] agent.router: Initializing LAN area manager -45:24.524 [INFO] TestAgent.server: Adding LAN server: server="Node-cda5f00d-85e2-dc89-179d-7a6f156330b8 (Addr: tcp/127.0.0.1:11783) (DC: dc1)" -45:24.524 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cda5f00d-85e2-dc89-179d-7a6f156330b8.dc1 area=wan -45:24.524 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11778 network=udp -45:24.524 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11778 network=tcp -45:24.524 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11779 network=tcp -45:24.524 [INFO] TestAgent: started state syncer -45:24.525 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11784 network=tcp -45:24.574 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:24.574 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11783 [Candidate]" term=2 -45:24.591 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:24.591 [DEBUG] TestAgent.server.raft: vote granted: from=cda5f00d-85e2-dc89-179d-7a6f156330b8 term=2 tally=1 -45:24.591 [INFO] TestAgent.server.raft: election won: tally=1 -45:24.591 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11783 [Leader]" -45:24.591 [INFO] TestAgent.server: cluster leadership acquired -45:24.592 [INFO] TestAgent.server: New leader elected: payload=Node-cda5f00d-85e2-dc89-179d-7a6f156330b8 -45:24.597 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11783 -45:24.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:24.619 [INFO] TestAgent: Synced node info -45:24.619 [DEBUG] TestAgent: Node info in sync -45:24.628 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:24.628 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:24.628 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:24.628 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:24.628 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:24.628 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cda5f00d-85e2-dc89-179d-7a6f156330b8 -45:24.628 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cda5f00d-85e2-dc89-179d-7a6f156330b8 -45:24.634 [INFO] TestAgent.server: federation state anti-entropy synced -45:24.988 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis -45:25.363 [INFO] TestAgent: Requesting shutdown -45:25.363 [INFO] TestAgent.server: shutting down server -45:25.363 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:25.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:25.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:25.363 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:25.363 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:25.363 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:25.363 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:25.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:25.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:25.363 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:25.369 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:25.374Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:25.374Z [INFO] agent.router.manager: shutting down -45:25.374 [INFO] TestAgent: consul server down -45:25.374 [INFO] TestAgent: shutdown complete -45:25.374 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11778 network=tcp -45:25.374 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11778 network=udp -45:25.374 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11779 network=tcp -45:25.878 [INFO] TestAgent: Waiting for endpoints to shut down -45:25.878 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Service_NoReap (1.40s) +2021-10-05T16:30:29.536Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:29.571Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:29.610 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1cc7a8bf-80b9-766f-d94f-0e7723573599 Address:127.0.0.1:30161}]" +30:29.611 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1cc7a8bf-80b9-766f-d94f-0e7723573599.dc1 127.0.0.1 +30:29.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1cc7a8bf-80b9-766f-d94f-0e7723573599 127.0.0.1 +2021-10-05T16:30:29.611Z [INFO] agent.router: Initializing LAN area manager +30:29.611 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30156 network=udp +30:29.611 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30161 [Follower]" leader= +30:29.612 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1cc7a8bf-80b9-766f-d94f-0e7723573599.dc1 area=wan +30:29.612 [INFO] TestAgent.server: Adding LAN server: server="Node-1cc7a8bf-80b9-766f-d94f-0e7723573599 (Addr: tcp/127.0.0.1:30161) (DC: dc1)" +30:29.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30156 network=tcp +30:29.612 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30157 network=tcp +30:29.613 [INFO] TestAgent: started state syncer +30:29.613 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30162 network=tcp +30:29.659 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:29.659 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30161 [Candidate]" term=2 +30:29.668 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:29.668 [DEBUG] TestAgent.server.raft: vote granted: from=1cc7a8bf-80b9-766f-d94f-0e7723573599 term=2 tally=1 +30:29.668 [INFO] TestAgent.server.raft: election won: tally=1 +30:29.668 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30161 [Leader]" +30:29.668 [INFO] TestAgent.server: cluster leadership acquired +30:29.668 [INFO] TestAgent.server: New leader elected: payload=Node-1cc7a8bf-80b9-766f-d94f-0e7723573599 +30:29.674 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30161 +30:29.687 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:29.720 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:29.720 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:29.720 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:29.720 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:29.720 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:29.720 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1cc7a8bf-80b9-766f-d94f-0e7723573599 +30:29.720 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1cc7a8bf-80b9-766f-d94f-0e7723573599 +30:29.733 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:29.751 [INFO] TestAgent: Synced node info +30:29.751 [DEBUG] TestAgent: Node info in sync +30:29.751 [INFO] TestAgent.server: federation state anti-entropy synced +30:30.039 [WARN] TestAgent: Check missed TTL, is now critical: check=service:redis +30:30.106 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:30.106 [DEBUG] TestAgent: Node info in sync +30:30.115 [INFO] TestAgent: Synced service: service=redis +30:30.115 [DEBUG] TestAgent: Check in sync: check=service:redis +30:30.115 [DEBUG] TestAgent: Node info in sync +30:30.115 [DEBUG] TestAgent: Service in sync: service=redis +30:30.115 [DEBUG] TestAgent: Check in sync: check=service:redis +30:30.415 [INFO] TestAgent: Requesting shutdown +30:30.415 [INFO] TestAgent.server: shutting down server +30:30.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:30.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:30.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:30.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:30.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:30.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:30.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:30.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:30.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:30.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:30.420 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:30.421Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:30.421Z [INFO] agent.router.manager: shutting down +30:30.421 [INFO] TestAgent: consul server down +30:30.421 [INFO] TestAgent: shutdown complete +30:30.421 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30156 network=tcp +30:30.421 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30156 network=udp +30:30.421 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30157 network=tcp +30:30.924 [INFO] TestAgent: Waiting for endpoints to shut down +30:30.924 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Service_NoReap (1.47s) === RUN TestAgent_AddService_restoresSnapshot === RUN TestAgent_AddService_restoresSnapshot/normal === PAUSE TestAgent_AddService_restoresSnapshot/normal @@ -12958,121 +13127,119 @@ [WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:25.891Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:25.901Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:25.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c196a002-5e48-8892-eb63-3c1c2ebfc0ff Address:127.0.0.1:11797}]" -45:25.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11797 [Follower]" leader= -45:25.931 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff.dc1 127.0.0.1 -45:25.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff 127.0.0.1 -2022-11-07T22:45:25.931Z [INFO] agent.router: Initializing LAN area manager -45:25.931 [INFO] TestAgent.server: Adding LAN server: server="Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff (Addr: tcp/127.0.0.1:11797) (DC: dc1)" -45:25.931 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff.dc1 area=wan -45:25.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11792 network=tcp -45:25.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11792 network=udp -45:25.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11793 network=tcp -45:25.932 [INFO] TestAgent: started state syncer -45:25.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11798 network=tcp -45:25.982 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:25.982 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11797 [Candidate]" term=2 -45:25.992 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:25.992 [DEBUG] TestAgent.server.raft: vote granted: from=c196a002-5e48-8892-eb63-3c1c2ebfc0ff term=2 tally=1 -45:25.992 [INFO] TestAgent.server.raft: election won: tally=1 -45:25.992 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11797 [Leader]" -45:25.992 [INFO] TestAgent.server: cluster leadership acquired -45:25.992 [INFO] TestAgent.server: New leader elected: payload=Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff -45:25.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11797 -45:26.007 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:26.024 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:26.024 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:26.024 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:26.024 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:26.024 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:26.024 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff -45:26.024 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c196a002-5e48-8892-eb63-3c1c2ebfc0ff -45:26.035 [INFO] TestAgent.server: federation state anti-entropy synced -45:26.073 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:26.076 [INFO] TestAgent: Synced node info -45:26.316 [INFO] TestAgent: Requesting shutdown -45:26.316 [INFO] TestAgent.server: shutting down server -45:26.316 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:26.316 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:26.316 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:26.316 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:26.316 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:26.316 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:26.316 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:26.316 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:26.316 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:26.320 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:26.323Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:26.323Z [INFO] agent.router.manager: shutting down -45:26.323 [INFO] TestAgent: consul server down -45:26.323 [INFO] TestAgent: shutdown complete -45:26.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11792 network=tcp -45:26.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11792 network=udp -45:26.323 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11793 network=tcp -45:26.824 [INFO] TestAgent: Waiting for endpoints to shut down -45:26.824 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:25.890Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:25.900Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:25.928 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb Address:127.0.0.1:11790}]" -45:25.928 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11790 [Follower]" leader= -45:25.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb.dc1 127.0.0.1 -45:25.929 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb 127.0.0.1 -2022-11-07T22:45:25.929Z [INFO] agent.router: Initializing LAN area manager -45:25.929 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb.dc1 area=wan -45:25.929 [INFO] TestAgent.server: Adding LAN server: server="Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb (Addr: tcp/127.0.0.1:11790) (DC: dc1)" -45:25.930 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11785 network=udp -45:25.930 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11785 network=tcp -45:25.930 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11786 network=tcp -45:25.930 [INFO] TestAgent: started state syncer -45:25.930 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11791 network=tcp -45:25.990 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:25.990 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11790 [Candidate]" term=2 -45:25.998 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:25.998 [DEBUG] TestAgent.server.raft: vote granted: from=eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb term=2 tally=1 -45:25.998 [INFO] TestAgent.server.raft: election won: tally=1 -45:25.998 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11790 [Leader]" -45:25.998 [INFO] TestAgent.server: cluster leadership acquired -45:25.998 [INFO] TestAgent.server: New leader elected: payload=Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb -45:26.003 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11790 -45:26.017 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:26.036 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:26.036 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:26.036 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:26.036 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:26.036 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:26.037 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb -45:26.037 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eeaa52b1-ed68-c8a0-e3f1-71cfba8945fb -45:26.045 [INFO] TestAgent.server: federation state anti-entropy synced -45:26.224 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:26.227 [INFO] TestAgent: Synced node info -45:26.227 [DEBUG] TestAgent: Node info in sync -45:26.328 [INFO] TestAgent: Requesting shutdown -45:26.328 [INFO] TestAgent.server: shutting down server -45:26.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:26.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:26.328 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:26.328 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:26.328 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:26.328 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:26.328 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:26.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:26.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:26.328 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:26.331 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:26.335Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:26.335Z [INFO] agent.router.manager: shutting down -45:26.336 [INFO] TestAgent: consul server down -45:26.336 [INFO] TestAgent: shutdown complete -45:26.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11785 network=tcp -45:26.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11785 network=udp -45:26.336 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11786 network=tcp -45:26.836 [INFO] TestAgent: Waiting for endpoints to shut down -45:26.836 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:30.979Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:31.017Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:31.094 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22ffac0f-7e13-a75f-7ea7-8267a4d0f313 Address:127.0.0.1:30168}]" +30:31.094 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313.dc1 127.0.0.1 +30:31.095 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313 127.0.0.1 +2021-10-05T16:30:31.095Z [INFO] agent.router: Initializing LAN area manager +30:31.095 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30163 network=udp +30:31.095 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30168 [Follower]" leader= +30:31.095 [INFO] TestAgent.server: Adding LAN server: server="Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313 (Addr: tcp/127.0.0.1:30168) (DC: dc1)" +30:31.096 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313.dc1 area=wan +30:31.096 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30163 network=tcp +30:31.096 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30164 network=tcp +30:31.096 [INFO] TestAgent: started state syncer +30:31.096 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30169 network=tcp +30:31.158 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:31.158 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30168 [Candidate]" term=2 +30:31.167 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:31.167 [DEBUG] TestAgent.server.raft: vote granted: from=22ffac0f-7e13-a75f-7ea7-8267a4d0f313 term=2 tally=1 +30:31.167 [INFO] TestAgent.server.raft: election won: tally=1 +30:31.167 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30168 [Leader]" +30:31.168 [INFO] TestAgent.server: cluster leadership acquired +30:31.168 [INFO] TestAgent.server: New leader elected: payload=Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313 +30:31.172 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30168 +30:31.185 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:31.207 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:31.207 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:31.207 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:31.207 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:31.207 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:31.207 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313 +30:31.208 [INFO] TestAgent.server: member joined, marking health alive: member=Node-22ffac0f-7e13-a75f-7ea7-8267a4d0f313 +30:31.208 [INFO] TestAgent: Requesting shutdown +30:31.208 [INFO] TestAgent.server: shutting down server +30:31.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:31.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:31.208 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:31.208 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:31.208 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:31.208 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:31.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:31.208 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:31.208 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:31.215 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +30:31.218 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:31.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:30:31.221Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:31.221Z [INFO] agent.router.manager: shutting down +30:31.221 [INFO] TestAgent: consul server down +30:31.221 [INFO] TestAgent: shutdown complete +30:31.221 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30163 network=tcp +30:31.221 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30163 network=udp +30:31.221 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30164 network=tcp +30:31.724 [INFO] TestAgent: Waiting for endpoints to shut down +30:31.724 [INFO] TestAgent: Endpoints down +[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:30.975Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:31.017Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:31.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0e6f2b0e-be92-f432-ee1c-bc6793e4d240 Address:127.0.0.1:30175}]" +30:31.080 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30175 [Follower]" leader= +30:31.082 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240.dc1 127.0.0.1 +30:31.083 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240 127.0.0.1 +2021-10-05T16:30:31.083Z [INFO] agent.router: Initializing LAN area manager +30:31.083 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30170 network=udp +30:31.083 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240.dc1 area=wan +30:31.083 [INFO] TestAgent.server: Adding LAN server: server="Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240 (Addr: tcp/127.0.0.1:30175) (DC: dc1)" +30:31.083 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30170 network=tcp +30:31.084 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30171 network=tcp +30:31.084 [INFO] TestAgent: started state syncer +30:31.084 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30176 network=tcp +30:31.140 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:31.140 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30175 [Candidate]" term=2 +30:31.184 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:31.184 [DEBUG] TestAgent.server.raft: vote granted: from=0e6f2b0e-be92-f432-ee1c-bc6793e4d240 term=2 tally=1 +30:31.184 [INFO] TestAgent.server.raft: election won: tally=1 +30:31.184 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30175 [Leader]" +30:31.188 [INFO] TestAgent.server: cluster leadership acquired +30:31.188 [INFO] TestAgent.server: New leader elected: payload=Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240 +30:31.207 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30175 +30:31.222 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:31.247 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:31.247 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:31.247 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:31.247 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:31.247 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:31.247 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240 +30:31.247 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0e6f2b0e-be92-f432-ee1c-bc6793e4d240 +30:31.268 [INFO] TestAgent.server: federation state anti-entropy synced +30:31.352 [INFO] TestAgent: Requesting shutdown +30:31.352 [INFO] TestAgent.server: shutting down server +30:31.352 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:31.352 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:31.352 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:31.352 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:31.352 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:31.352 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:31.352 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:31.352 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:31.352 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:31.352 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:31.352 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:31.360 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:31.368Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:31.368Z [INFO] agent.router.manager: shutting down +30:31.368 [INFO] TestAgent: consul server down +30:31.368 [INFO] TestAgent: shutdown complete +30:31.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30170 network=tcp +30:31.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30170 network=udp +30:31.368 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30171 network=tcp +30:31.872 [INFO] TestAgent: Waiting for endpoints to shut down +30:31.872 [INFO] TestAgent: Endpoints down --- PASS: TestAgent_AddService_restoresSnapshot (0.00s) - --- PASS: TestAgent_AddService_restoresSnapshot/service_manager (0.95s) - --- PASS: TestAgent_AddService_restoresSnapshot/normal (0.96s) + --- PASS: TestAgent_AddService_restoresSnapshot/normal (0.80s) + --- PASS: TestAgent_AddService_restoresSnapshot/service_manager (0.94s) === RUN TestAgent_AddCheck_restoresSnapshot === PAUSE TestAgent_AddCheck_restoresSnapshot === RUN TestAgent_NodeMaintenanceMode @@ -13105,8 +13272,6 @@ === RUN TestAgent_ReloadConfigAndKeepChecksStatus/service_manager === PAUSE TestAgent_ReloadConfigAndKeepChecksStatus/service_manager === CONT TestAgent_ReloadConfigAndKeepChecksStatus/normal -=== CONT TestAgent_ReloadConfigAndKeepChecksStatus/service_manager -=== CONT TestAgent_ReloadConfigAndKeepChecksStatus/normal agent_test.go:3644: DM-skipped === CONT TestAgent_ReloadConfigAndKeepChecksStatus/service_manager agent_test.go:3644: DM-skipped @@ -13199,10 +13364,10 @@ === RUN TestAgent_JoinWAN_viaMeshGateway === PAUSE TestAgent_JoinWAN_viaMeshGateway === RUN TestAutoConfig_Integration -[WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 6 out of 13 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! - agent_test.go:4731: Forcing roots rotation on the server [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! + agent_test.go:4731: Forcing roots rotation on the server [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! @@ -13216,383 +13381,400 @@ [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:27.153Z [INFO] agent.auto_config: retrieving initial agent auto configuration remotely -45:27.206 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -2022-11-07T22:45:27.206Z [INFO] agent.router: Initializing LAN area manager -2022-11-07T22:45:27.206Z [INFO] agent.auto_config: auto-config started -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -45:27.206 [INFO] test-client: Started DNS server: address=127.0.0.1:11806 network=udp -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -45:27.206 [INFO] test-client: Started DNS server: address=127.0.0.1:11806 network=tcp -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -45:27.206 [INFO] test-client: Started HTTP server: address=127.0.0.1:11807 network=tcp -45:27.206 [INFO] test-client: started state syncer -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -45:27.206 [INFO] test-client: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" -45:27.206 [INFO] test-client: Joining cluster...: cluster=LAN -45:27.206 [INFO] test-client: (LAN) joining: lan_addresses=[127.0.0.1:11802] -2022-11-07T22:45:27.206Z [WARN] agent.router.manager: No servers available -45:27.206 [ERROR] test-client.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:27.206 [INFO] test-client: Started gRPC server: address=127.0.0.1:11812 network=tcp -45:27.207 [DEBUG] test-client.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11802 -45:27.208 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 127.0.0.1 -45:27.208 [INFO] test-client: (LAN) joined: number_of_nodes=1 -45:27.208 [DEBUG] test-client: systemd notify failed: error="No socket" -45:27.208 [INFO] test-client: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 -45:27.208 [INFO] test-client.client: adding server: server="Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 (Addr: tcp/127.0.0.1:11804) (DC: dc1)" -45:27.256 [DEBUG] test-client.client: transitioned out of legacy ACL mode -45:27.256 [INFO] test-client.client.serf.lan: serf: EventMemberUpdate: test-client -45:27.286 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader -45:27.486 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader -45:27.486 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client -45:27.686 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader -45:27.686 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client -45:27.886 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader -45:27.886 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client -45:27.886 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client -45:29.611 [INFO] test-client: Requesting shutdown -45:29.612 [INFO] test-client.client: shutting down client -45:29.612 [WARN] test-client.client.serf.lan: serf: Shutdown without a Leave -45:29.615 [INFO] test-client: consul client down -45:29.615 [INFO] test-client: shutdown complete -45:29.615 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:11806 network=tcp -45:29.615 [ERROR] test-client.client: RPC failed to server: method=ConnectCA.Roots server=127.0.0.1:11804 error="rpc error making call: EOF" -45:29.615 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:11806 network=udp -45:29.615 [INFO] test-client: Stopping server: protocol=HTTP address=127.0.0.1:11807 network=tcp -45:30.115 [INFO] test-client: Waiting for endpoints to shut down -45:30.115 [INFO] test-client: Endpoints down -2022-11-07T22:45:26.852Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:26.862Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:26.884 [INFO] TestAgent-Server.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e6b62e55-ec3f-40ed-9c46-abeae1438d06 Address:127.0.0.1:11804}]" -45:26.884 [INFO] TestAgent-Server.server.raft: entering follower state: follower="Node at 127.0.0.1:11804 [Follower]" leader= -45:26.885 [INFO] TestAgent-Server.server.serf.wan: serf: EventMemberJoin: Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06.dc1 127.0.0.1 -45:26.885 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberJoin: Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 127.0.0.1 -2022-11-07T22:45:26.885Z [INFO] agent.router: Initializing LAN area manager -45:26.886 [INFO] TestAgent-Server.server: Handled event for server in area: event=member-join server=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06.dc1 area=wan -45:26.886 [INFO] TestAgent-Server.server: Adding LAN server: server="Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 (Addr: tcp/127.0.0.1:11804) (DC: dc1)" -45:26.886 [INFO] TestAgent-Server: Started DNS server: address=127.0.0.1:11799 network=udp -45:26.886 [INFO] TestAgent-Server: Started DNS server: address=127.0.0.1:11799 network=tcp -45:26.886 [INFO] TestAgent-Server: Started HTTP server: address=127.0.0.1:11800 network=tcp -45:26.886 [INFO] TestAgent-Server: started state syncer -45:26.886 [INFO] TestAgent-Server: Started gRPC server: address=127.0.0.1:11805 network=tcp -45:26.932 [WARN] TestAgent-Server.server.raft: heartbeat timeout reached, starting election: last-leader= -45:26.932 [INFO] TestAgent-Server.server.raft: entering candidate state: node="Node at 127.0.0.1:11804 [Candidate]" term=2 -45:26.936 [DEBUG] TestAgent-Server.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:26.940 [DEBUG] TestAgent-Server.server.raft: votes: needed=1 -45:26.940 [DEBUG] TestAgent-Server.server.raft: vote granted: from=e6b62e55-ec3f-40ed-9c46-abeae1438d06 term=2 tally=1 -45:26.940 [INFO] TestAgent-Server.server.raft: election won: tally=1 -45:26.940 [INFO] TestAgent-Server.server.raft: entering leader state: leader="Node at 127.0.0.1:11804 [Leader]" -45:26.940 [INFO] TestAgent-Server.server: cluster leadership acquired -45:26.940 [INFO] TestAgent-Server.server: New leader elected: payload=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:26.946 [INFO] TestAgent-Server.server: initializing acls -45:26.949 [INFO] TestAgent-Server.server: Created ACL 'global-management' policy -45:26.954 [INFO] TestAgent-Server.server: Bootstrapped ACL master token from configuration -45:26.962 [INFO] TestAgent-Server.server: Created ACL anonymous token from configuration -45:26.962 [INFO] TestAgent-Server.leader: started routine: routine="legacy ACL token upgrade" -45:26.962 [INFO] TestAgent-Server.leader: started routine: routine="acl token reaping" -45:26.962 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberUpdate: Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:26.962 [INFO] TestAgent-Server.server.serf.wan: serf: EventMemberUpdate: Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06.dc1 -45:26.963 [INFO] TestAgent-Server.server: Updating LAN server: server="Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 (Addr: tcp/127.0.0.1:11804) (DC: dc1)" -45:26.963 [INFO] TestAgent-Server.server: Handled event for server in area: event=member-update server=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06.dc1 area=wan -45:26.971 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:26.985 [INFO] TestAgent-Server.server.connect: initialized primary datacenter CA with provider: provider=consul -45:26.985 [INFO] TestAgent-Server.leader: started routine: routine="federation state anti-entropy" -45:26.985 [INFO] TestAgent-Server.leader: started routine: routine="federation state pruning" -45:26.985 [INFO] TestAgent-Server.leader: started routine: routine="intermediate cert renew watch" -45:26.985 [INFO] TestAgent-Server.leader: started routine: routine="CA root pruning" -45:26.985 [DEBUG] TestAgent-Server.server: Skipping self join check for node since the cluster is too small: node=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:26.985 [INFO] TestAgent-Server.server: member joined, marking health alive: member=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:26.990 [INFO] TestAgent-Server.server: federation state anti-entropy synced -45:26.990 [DEBUG] TestAgent-Server.server: Skipping self join check for node since the cluster is too small: node=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:27.128 [DEBUG] TestAgent-Server.acl: dropping node from result due to ACLs: node=Node-e6b62e55-ec3f-40ed-9c46-abeae1438d06 -45:27.207 [DEBUG] TestAgent-Server.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:52422 -45:27.207 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -45:27.207 [INFO] TestAgent-Server.server: member joined, marking health alive: member=test-client -45:27.249 [DEBUG] connect.ca.consul: consul CA provider configured: id=b9:02:75:9e:23:ce:ed:f6:ce:7b:d0:36:6b:ef:06:80:e8:12:c8:cc:d5:3e:a0:5b:d3:42:4a:3f:c3:be:ca:ca is_primary=true -45:27.268 [INFO] TestAgent-Server.server.connect: CA rotated to new root under provider: provider=consul -45:27.289 [DEBUG] TestAgent-Server: Skipping remote check since it is managed automatically: check=serfHealth -45:27.292 [INFO] TestAgent-Server: Synced node info -45:27.292 [DEBUG] TestAgent-Server: Node info in sync -45:27.406 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberUpdate: test-client -45:27.406 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client -45:27.606 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client -45:27.806 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client -45:27.886 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client -45:29.674 [DEBUG] TestAgent-Server: Skipping remote check since it is managed automatically: check=serfHealth -45:29.674 [DEBUG] TestAgent-Server: Node info in sync -45:30.115 [INFO] TestAgent-Server: Requesting shutdown -45:30.115 [INFO] TestAgent-Server.server: shutting down server -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="legacy ACL token upgrade" -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="acl token reaping" -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="federation state anti-entropy" -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="federation state pruning" -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="intermediate cert renew watch" -45:30.115 [DEBUG] TestAgent-Server.leader: stopping routine: routine="CA root pruning" -45:30.115 [WARN] TestAgent-Server.server.serf.lan: serf: Shutdown without a Leave -45:30.116 [INFO] TestAgent-Server.server: shutting down Connect CA roots monitor -45:30.116 [ERROR] TestAgent-Server.server: error performing anti-entropy sync of federation state: error="context canceled" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="federation state anti-entropy" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="legacy ACL token upgrade" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="acl token reaping" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="federation state pruning" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="intermediate cert renew watch" -45:30.116 [DEBUG] TestAgent-Server.leader: stopped routine: routine="CA root pruning" -45:30.121 [WARN] TestAgent-Server.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:30.124Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:30.124Z [INFO] agent.router.manager: shutting down -45:30.124 [INFO] TestAgent-Server: consul server down -45:30.124 [INFO] TestAgent-Server: shutdown complete -45:30.124 [INFO] TestAgent-Server: Stopping server: protocol=DNS address=127.0.0.1:11799 network=tcp -45:30.124 [INFO] TestAgent-Server: Stopping server: protocol=DNS address=127.0.0.1:11799 network=udp -45:30.124 [INFO] TestAgent-Server: Stopping server: protocol=HTTP address=127.0.0.1:11800 network=tcp -45:30.386 [DEBUG] TestAgent-Server.server.memberlist.lan: memberlist: Failed ping: test-client (timeout reached) -45:30.624 [INFO] TestAgent-Server: Waiting for endpoints to shut down -45:30.624 [INFO] TestAgent-Server: Endpoints down ---- PASS: TestAutoConfig_Integration (3.79s) +2021-10-05T16:30:32.474Z [INFO] agent.auto_config: retrieving initial agent auto configuration remotely +30:32.648 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +2021-10-05T16:30:32.648Z [INFO] agent.router: Initializing LAN area manager +2021-10-05T16:30:32.648Z [INFO] agent.auto_config: auto-config started +30:32.649 [INFO] test-client: Started DNS server: address=127.0.0.1:30184 network=udp +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.649Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.650Z [WARN] agent.router.manager: No servers available +30:32.650 [INFO] test-client: Started DNS server: address=127.0.0.1:30184 network=tcp +30:32.651 [INFO] test-client: Started HTTP server: address=127.0.0.1:30185 network=tcp +30:32.651 [INFO] test-client: started state syncer +2021-10-05T16:30:32.651Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:32.651Z [WARN] agent.router.manager: No servers available +30:32.651 [INFO] test-client: Started gRPC server: address=127.0.0.1:30190 network=tcp +30:32.651 [INFO] test-client: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" +30:32.651 [INFO] test-client: Joining cluster...: cluster=LAN +30:32.651 [INFO] test-client: (LAN) joining: lan_addresses=[127.0.0.1:30180] +2021-10-05T16:30:32.651Z [WARN] agent.router.manager: No servers available +30:32.651 [ERROR] test-client.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:32.652 [DEBUG] test-client.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30180 +30:32.656 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: Node-41aa8851-0d80-6bc4-6a22-a462368701d3 127.0.0.1 +30:32.656 [INFO] test-client: (LAN) joined: number_of_nodes=1 +30:32.656 [DEBUG] test-client: systemd notify failed: error="No socket" +30:32.656 [INFO] test-client: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 +30:32.656 [INFO] test-client.client: adding server: server="Node-41aa8851-0d80-6bc4-6a22-a462368701d3 (Addr: tcp/127.0.0.1:30182) (DC: dc1)" +30:32.700 [DEBUG] test-client.client: transitioned out of legacy ACL mode +30:32.700 [INFO] test-client.client.serf.lan: serf: EventMemberUpdate: test-client +30:32.828 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader +30:33.025 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader +30:33.025 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client +30:33.025 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader +30:33.025 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client +30:33.225 [DEBUG] test-client.client.serf.lan: serf: messageUserEventType: consul:new-leader +30:33.225 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client +30:33.428 [DEBUG] test-client.client.serf.lan: serf: messageJoinType: test-client +30:34.739 [DEBUG] test-client: Skipping remote check since it is managed automatically: check=serfHealth +30:34.744 [INFO] test-client: Synced node info +30:34.744 [DEBUG] test-client: Node info in sync +30:34.925 [INFO] test-client: Requesting shutdown +30:34.925 [INFO] test-client.client: shutting down client +30:34.925 [WARN] test-client.client.serf.lan: serf: Shutdown without a Leave +30:34.929 [INFO] test-client: consul client down +30:34.929 [INFO] test-client: shutdown complete +30:34.929 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:30184 network=tcp +30:34.929 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:30184 network=udp +30:34.929 [INFO] test-client: Stopping server: protocol=HTTP address=127.0.0.1:30185 network=tcp +30:34.929 [ERROR] test-client.client: RPC failed to server: method=ConnectCA.Roots server=127.0.0.1:30182 error="rpc error making call: EOF" +30:35.436 [INFO] test-client: Waiting for endpoints to shut down +30:35.436 [INFO] test-client: Endpoints down +2021-10-05T16:30:31.907Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:31.939Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:32.020 [INFO] TestAgent-Server.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:41aa8851-0d80-6bc4-6a22-a462368701d3 Address:127.0.0.1:30182}]" +30:32.021 [INFO] TestAgent-Server.server.serf.wan: serf: EventMemberJoin: Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 127.0.0.1 +30:32.022 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberJoin: Node-41aa8851-0d80-6bc4-6a22-a462368701d3 127.0.0.1 +2021-10-05T16:30:32.022Z [INFO] agent.router: Initializing LAN area manager +30:32.022 [INFO] TestAgent-Server: Started DNS server: address=127.0.0.1:30177 network=udp +30:32.022 [INFO] TestAgent-Server.server.raft: entering follower state: follower="Node at 127.0.0.1:30182 [Follower]" leader= +30:32.023 [INFO] TestAgent-Server.server: Adding LAN server: server="Node-41aa8851-0d80-6bc4-6a22-a462368701d3 (Addr: tcp/127.0.0.1:30182) (DC: dc1)" +30:32.023 [INFO] TestAgent-Server.server: Handled event for server in area: event=member-join server=Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 area=wan +30:32.023 [INFO] TestAgent-Server: Started DNS server: address=127.0.0.1:30177 network=tcp +30:32.023 [INFO] TestAgent-Server: Started HTTP server: address=127.0.0.1:30178 network=tcp +30:32.023 [INFO] TestAgent-Server: started state syncer +30:32.023 [INFO] TestAgent-Server: Started gRPC server: address=127.0.0.1:30183 network=tcp +30:32.060 [WARN] TestAgent-Server.server.raft: heartbeat timeout reached, starting election: last-leader= +30:32.060 [INFO] TestAgent-Server.server.raft: entering candidate state: node="Node at 127.0.0.1:30182 [Candidate]" term=2 +30:32.076 [DEBUG] TestAgent-Server.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:32.108 [DEBUG] TestAgent-Server.server.raft: votes: needed=1 +30:32.108 [DEBUG] TestAgent-Server.server.raft: vote granted: from=41aa8851-0d80-6bc4-6a22-a462368701d3 term=2 tally=1 +30:32.108 [INFO] TestAgent-Server.server.raft: election won: tally=1 +30:32.108 [INFO] TestAgent-Server.server.raft: entering leader state: leader="Node at 127.0.0.1:30182 [Leader]" +30:32.112 [INFO] TestAgent-Server.server: cluster leadership acquired +30:32.112 [INFO] TestAgent-Server.server: New leader elected: payload=Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.140 [INFO] TestAgent-Server.server: initializing acls +30:32.156 [INFO] TestAgent-Server.server: Created ACL 'global-management' policy +30:32.172 [ERROR] TestAgent-Server.anti_entropy: failed to sync remote state: error="ACL not found" +30:32.172 [INFO] TestAgent-Server.server: Bootstrapped ACL master token from configuration +30:32.176 [INFO] TestAgent-Server.server: initializing acls +30:32.220 [INFO] TestAgent-Server.server: Created ACL anonymous token from configuration +30:32.220 [INFO] TestAgent-Server.leader: started routine: routine="legacy ACL token upgrade" +30:32.220 [INFO] TestAgent-Server.leader: started routine: routine="acl token reaping" +30:32.220 [DEBUG] TestAgent-Server.server: transitioning out of legacy ACL mode +30:32.220 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberUpdate: Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.220 [INFO] TestAgent-Server.server.serf.wan: serf: EventMemberUpdate: Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 +30:32.220 [INFO] TestAgent-Server.server: Handled event for server in area: event=member-update server=Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 area=wan +30:32.220 [INFO] TestAgent-Server.server: Created ACL anonymous token from configuration +30:32.220 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberUpdate: Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.220 [INFO] TestAgent-Server.server.serf.wan: serf: EventMemberUpdate: Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 +30:32.224 [INFO] TestAgent-Server.server: Updating LAN server: server="Node-41aa8851-0d80-6bc4-6a22-a462368701d3 (Addr: tcp/127.0.0.1:30182) (DC: dc1)" +30:32.224 [INFO] TestAgent-Server.server: Updating LAN server: server="Node-41aa8851-0d80-6bc4-6a22-a462368701d3 (Addr: tcp/127.0.0.1:30182) (DC: dc1)" +30:32.224 [INFO] TestAgent-Server.server: Handled event for server in area: event=member-update server=Node-41aa8851-0d80-6bc4-6a22-a462368701d3.dc1 area=wan +30:32.272 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:32.336 [INFO] TestAgent-Server.server.connect: initialized primary datacenter CA with provider: provider=consul +30:32.336 [INFO] TestAgent-Server.leader: started routine: routine="federation state anti-entropy" +30:32.336 [INFO] TestAgent-Server.leader: started routine: routine="federation state pruning" +30:32.336 [INFO] TestAgent-Server.leader: started routine: routine="intermediate cert renew watch" +30:32.336 [INFO] TestAgent-Server.leader: started routine: routine="CA root pruning" +30:32.336 [DEBUG] TestAgent-Server.server: Skipping self join check for node since the cluster is too small: node=Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.336 [INFO] TestAgent-Server.server: member joined, marking health alive: member=Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.352 [DEBUG] TestAgent-Server.server: Skipping self join check for node since the cluster is too small: node=Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.352 [DEBUG] TestAgent-Server.server: Skipping self join check for node since the cluster is too small: node=Node-41aa8851-0d80-6bc4-6a22-a462368701d3 +30:32.368 [INFO] TestAgent-Server.server: federation state anti-entropy synced +30:32.652 [DEBUG] TestAgent-Server.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:36408 +30:32.653 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +30:32.653 [INFO] TestAgent-Server.server: member joined, marking health alive: member=test-client +30:32.752 [DEBUG] connect.ca.consul: consul CA provider configured: id=39:3b:97:f2:f0:0a:8b:98:a0:55:68:1e:c4:04:9b:8d:92:e5:1d:b5:67:e9:0f:f3:ea:4e:7e:93:e0:31:6e:98 is_primary=true +30:32.850 [INFO] TestAgent-Server.server.serf.lan: serf: EventMemberUpdate: test-client +30:32.850 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client +30:32.858 [INFO] TestAgent-Server.server.connect: CA rotated to new root under provider: provider=consul +30:33.025 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client +30:33.056 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client +30:33.253 [DEBUG] TestAgent-Server.server.serf.lan: serf: messageJoinType: test-client +30:34.604 [DEBUG] TestAgent-Server: Skipping remote check since it is managed automatically: check=serfHealth +30:34.624 [INFO] TestAgent-Server: Synced node info +30:34.624 [DEBUG] TestAgent-Server: Node info in sync +30:35.436 [INFO] TestAgent-Server: Requesting shutdown +30:35.436 [INFO] TestAgent-Server.server: shutting down server +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="federation state anti-entropy" +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="federation state pruning" +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="intermediate cert renew watch" +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="CA root pruning" +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="legacy ACL token upgrade" +30:35.436 [DEBUG] TestAgent-Server.leader: stopping routine: routine="acl token reaping" +30:35.436 [WARN] TestAgent-Server.server.serf.lan: serf: Shutdown without a Leave +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="intermediate cert renew watch" +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="legacy ACL token upgrade" +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="CA root pruning" +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="acl token reaping" +30:35.436 [INFO] TestAgent-Server.server: shutting down Connect CA roots monitor +30:35.436 [ERROR] TestAgent-Server.server: error performing anti-entropy sync of federation state: error="context canceled" +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="federation state anti-entropy" +30:35.436 [DEBUG] TestAgent-Server.leader: stopped routine: routine="federation state pruning" +30:35.446 [WARN] TestAgent-Server.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:35.448Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:35.448Z [INFO] agent.router.manager: shutting down +30:35.449 [INFO] TestAgent-Server: consul server down +30:35.449 [INFO] TestAgent-Server: shutdown complete +30:35.449 [INFO] TestAgent-Server: Stopping server: protocol=DNS address=127.0.0.1:30177 network=tcp +30:35.449 [INFO] TestAgent-Server: Stopping server: protocol=DNS address=127.0.0.1:30177 network=udp +30:35.449 [INFO] TestAgent-Server: Stopping server: protocol=HTTP address=127.0.0.1:30178 network=tcp +30:35.524 [DEBUG] TestAgent-Server.server.memberlist.lan: memberlist: Failed ping: test-client (timeout reached) +30:35.949 [INFO] TestAgent-Server: Waiting for endpoints to shut down +30:35.949 [INFO] TestAgent-Server: Endpoints down +--- PASS: TestAutoConfig_Integration (4.08s) === RUN TestAgent_AutoEncrypt [WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:30.932Z [INFO] agent.auto_config: automatically upgraded to TLS -45:30.933 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -2022-11-07T22:45:30.933Z [INFO] agent.router: Initializing LAN area manager -2022-11-07T22:45:30.933Z [INFO] agent.auto_config: auto-config started -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -45:30.933 [INFO] test-client: Started DNS server: address=127.0.0.1:11820 network=udp -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -45:30.933 [INFO] test-client: Started DNS server: address=127.0.0.1:11820 network=tcp -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.933Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -45:30.934 [INFO] test-client: Started HTTPS server: address=127.0.0.1:11822 network=tcp -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -45:30.934 [INFO] test-client: started state syncer -45:30.934 [INFO] test-client: Started gRPC server: address=127.0.0.1:11826 network=tcp -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -45:30.934 [INFO] test-client: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" -45:30.934 [INFO] test-client: Joining cluster...: cluster=LAN -45:30.934 [INFO] test-client: (LAN) joining: lan_addresses=[127.0.0.1:11816] -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -2022-11-07T22:45:30.934Z [WARN] agent.router.manager: No servers available -45:30.934 [ERROR] test-client.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:30.934 [DEBUG] test-client.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11816 -45:30.935 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: Node-876003e6-b7fc-6cbe-a806-95e6cd445594 127.0.0.1 -45:30.935 [INFO] test-client: (LAN) joined: number_of_nodes=1 -45:30.935 [DEBUG] test-client: systemd notify failed: error="No socket" -45:30.935 [INFO] test-client: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 -45:30.935 [INFO] test-client.client: adding server: server="Node-876003e6-b7fc-6cbe-a806-95e6cd445594 (Addr: tcp/127.0.0.1:11818) (DC: dc1)" -45:30.972 [INFO] test-client: Requesting shutdown -45:30.972 [INFO] test-client.client: shutting down client -45:30.972 [WARN] test-client.client.serf.lan: serf: Shutdown without a Leave -45:30.976 [INFO] test-client: consul client down -45:30.976 [INFO] test-client: shutdown complete -45:30.976 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:11820 network=tcp -45:30.976 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:11820 network=udp -45:30.976 [INFO] test-client: Stopping server: protocol=HTTPS address=127.0.0.1:11822 network=tcp -45:31.476 [INFO] test-client: Waiting for endpoints to shut down -45:31.476 [INFO] test-client: Endpoints down -2022-11-07T22:45:30.640Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:30.650Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:30.678 [INFO] test-server.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:876003e6-b7fc-6cbe-a806-95e6cd445594 Address:127.0.0.1:11818}]" -45:30.678 [INFO] test-server.server.raft: entering follower state: follower="Node at 127.0.0.1:11818 [Follower]" leader= -45:30.679 [INFO] test-server.server.serf.wan: serf: EventMemberJoin: Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 127.0.0.1 -45:30.679 [INFO] test-server.server.serf.lan: serf: EventMemberJoin: Node-876003e6-b7fc-6cbe-a806-95e6cd445594 127.0.0.1 -2022-11-07T22:45:30.679Z [INFO] agent.router: Initializing LAN area manager -45:30.679 [INFO] test-server.server: Adding LAN server: server="Node-876003e6-b7fc-6cbe-a806-95e6cd445594 (Addr: tcp/127.0.0.1:11818) (DC: dc1)" -45:30.679 [INFO] test-server: Started DNS server: address=127.0.0.1:11813 network=udp -45:30.679 [INFO] test-server.server: Handled event for server in area: event=member-join server=Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 area=wan -45:30.679 [INFO] test-server: Started DNS server: address=127.0.0.1:11813 network=tcp -45:30.680 [INFO] test-server: Started HTTP server: address=127.0.0.1:11814 network=tcp -45:30.680 [INFO] test-server: started state syncer -45:30.680 [INFO] test-server: Started gRPC server: address=127.0.0.1:11819 network=tcp -45:30.719 [WARN] test-server.server.raft: heartbeat timeout reached, starting election: last-leader= -45:30.719 [INFO] test-server.server.raft: entering candidate state: node="Node at 127.0.0.1:11818 [Candidate]" term=2 -45:30.727 [DEBUG] test-server.server.raft: votes: needed=1 -45:30.727 [DEBUG] test-server.server.raft: vote granted: from=876003e6-b7fc-6cbe-a806-95e6cd445594 term=2 tally=1 -45:30.727 [INFO] test-server.server.raft: election won: tally=1 -45:30.727 [INFO] test-server.server.raft: entering leader state: leader="Node at 127.0.0.1:11818 [Leader]" -45:30.727 [INFO] test-server.server: cluster leadership acquired -45:30.727 [INFO] test-server.server: New leader elected: payload=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.729 [INFO] test-server.server: initializing acls -45:30.733 [INFO] test-server.server: initializing acls -45:30.733 [INFO] test-server.server: Created ACL 'global-management' policy -45:30.738 [INFO] test-server.server: Created ACL 'global-management' policy -45:30.746 [INFO] test-server.server: Bootstrapped ACL master token from configuration -45:30.751 [INFO] test-server.server: Bootstrapped ACL master token from configuration -45:30.751 [INFO] test-server.server: Created ACL anonymous token from configuration -45:30.751 [INFO] test-server.leader: started routine: routine="legacy ACL token upgrade" -45:30.751 [INFO] test-server.leader: started routine: routine="acl token reaping" -45:30.751 [DEBUG] test-server.server: transitioning out of legacy ACL mode -45:30.751 [INFO] test-server.server.serf.lan: serf: EventMemberUpdate: Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.751 [INFO] test-server.server.serf.wan: serf: EventMemberUpdate: Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 -45:30.751 [INFO] test-server.server: Handled event for server in area: event=member-update server=Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 area=wan -45:30.751 [INFO] test-server.server: Updating LAN server: server="Node-876003e6-b7fc-6cbe-a806-95e6cd445594 (Addr: tcp/127.0.0.1:11818) (DC: dc1)" -45:30.754 [INFO] test-server.server: Created ACL anonymous token from configuration -45:30.754 [INFO] test-server.server.serf.lan: serf: EventMemberUpdate: Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.754 [INFO] test-server.server.serf.wan: serf: EventMemberUpdate: Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 -45:30.754 [INFO] test-server.server: Updating LAN server: server="Node-876003e6-b7fc-6cbe-a806-95e6cd445594 (Addr: tcp/127.0.0.1:11818) (DC: dc1)" -45:30.754 [INFO] test-server.server: Handled event for server in area: event=member-update server=Node-876003e6-b7fc-6cbe-a806-95e6cd445594.dc1 area=wan -45:30.764 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:30.772 [INFO] test-server: Synced node info -45:30.772 [DEBUG] test-server: Node info in sync -45:30.782 [INFO] test-server.server.connect: initialized primary datacenter CA with provider: provider=consul -45:30.782 [INFO] test-server.leader: started routine: routine="federation state anti-entropy" -45:30.782 [INFO] test-server.leader: started routine: routine="federation state pruning" -45:30.782 [INFO] test-server.leader: started routine: routine="intermediate cert renew watch" -45:30.782 [INFO] test-server.leader: started routine: routine="CA root pruning" -45:30.783 [DEBUG] test-server.server: Skipping self join check for node since the cluster is too small: node=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.783 [INFO] test-server.server: member joined, marking health alive: member=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.787 [DEBUG] test-server.server: Skipping self join check for node since the cluster is too small: node=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.787 [DEBUG] test-server.server: Skipping self join check for node since the cluster is too small: node=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.791 [INFO] test-server.server: federation state anti-entropy synced -45:30.894 [DEBUG] test-server.acl: dropping node from result due to ACLs: node=Node-876003e6-b7fc-6cbe-a806-95e6cd445594 -45:30.934 [DEBUG] test-server.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:39934 -45:30.935 [INFO] test-server.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -45:30.935 [INFO] test-server.server: member joined, marking health alive: member=test-client -45:31.476 [INFO] test-server: Requesting shutdown -45:31.476 [INFO] test-server.server: shutting down server -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="federation state pruning" -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="intermediate cert renew watch" -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="CA root pruning" -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="legacy ACL token upgrade" -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="acl token reaping" -45:31.476 [DEBUG] test-server.leader: stopping routine: routine="federation state anti-entropy" -45:31.476 [WARN] test-server.server.serf.lan: serf: Shutdown without a Leave -45:31.476 [INFO] test-server.server: shutting down Connect CA roots monitor -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="acl token reaping" -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="CA root pruning" -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="legacy ACL token upgrade" -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="federation state pruning" -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="intermediate cert renew watch" -45:31.476 [ERROR] test-server.server: error performing anti-entropy sync of federation state: error="context canceled" -45:31.476 [DEBUG] test-server.leader: stopped routine: routine="federation state anti-entropy" -45:31.480 [WARN] test-server.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:31.485Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:31.485Z [INFO] agent.router.manager: shutting down -45:31.485 [INFO] test-server: consul server down -45:31.485 [INFO] test-server: shutdown complete -45:31.485 [INFO] test-server: Stopping server: protocol=DNS address=127.0.0.1:11813 network=tcp -45:31.485 [INFO] test-server: Stopping server: protocol=DNS address=127.0.0.1:11813 network=udp -45:31.485 [INFO] test-server: Stopping server: protocol=HTTP address=127.0.0.1:11814 network=tcp -45:31.986 [INFO] test-server: Waiting for endpoints to shut down -45:31.986 [INFO] test-server: Endpoints down ---- PASS: TestAgent_AutoEncrypt (1.36s) +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:36.335Z [INFO] agent.auto_config: automatically upgraded to TLS +30:36.336 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +2021-10-05T16:30:36.336Z [INFO] agent.router: Initializing LAN area manager +2021-10-05T16:30:36.336Z [INFO] agent.auto_config: auto-config started +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.337Z [WARN] agent.router.manager: No servers available +30:36.339 [INFO] test-client: Started DNS server: address=127.0.0.1:30198 network=tcp +30:36.339 [INFO] test-client: Started DNS server: address=127.0.0.1:30198 network=udp +30:36.340 [INFO] test-client: Started HTTPS server: address=127.0.0.1:30200 network=tcp +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +30:36.340 [INFO] test-client: started state syncer +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +30:36.340 [INFO] test-client: Started gRPC server: address=127.0.0.1:30204 network=tcp +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +30:36.340 [INFO] test-client: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" +30:36.340 [INFO] test-client: Joining cluster...: cluster=LAN +30:36.340 [INFO] test-client: (LAN) joining: lan_addresses=[127.0.0.1:30194] +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +30:36.340 [ERROR] test-client.anti_entropy: failed to sync remote state: error="No known Consul servers" +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +2021-10-05T16:30:36.340Z [WARN] agent.router.manager: No servers available +30:36.344 [DEBUG] test-client.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30194 +30:36.348 [INFO] test-client.client.serf.lan: serf: EventMemberJoin: Node-bb656f69-3920-42d3-23b9-f7547cb38d08 127.0.0.1 +30:36.348 [INFO] test-client: (LAN) joined: number_of_nodes=1 +30:36.348 [DEBUG] test-client: systemd notify failed: error="No socket" +30:36.348 [INFO] test-client: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 +30:36.348 [INFO] test-client.client: adding server: server="Node-bb656f69-3920-42d3-23b9-f7547cb38d08 (Addr: tcp/127.0.0.1:30196) (DC: dc1)" +30:36.387 [DEBUG] test-client.client: transitioned out of legacy ACL mode +30:36.387 [INFO] test-client.client.serf.lan: serf: EventMemberUpdate: test-client +30:36.397 [INFO] test-client: Requesting shutdown +30:36.398 [INFO] test-client.client: shutting down client +30:36.398 [WARN] test-client.client.serf.lan: serf: Shutdown without a Leave +30:36.407 [INFO] test-client: consul client down +30:36.407 [INFO] test-client: shutdown complete +30:36.407 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:30198 network=tcp +30:36.407 [INFO] test-client: Stopping server: protocol=DNS address=127.0.0.1:30198 network=udp +30:36.407 [INFO] test-client: Stopping server: protocol=HTTPS address=127.0.0.1:30200 network=tcp +30:36.907 [INFO] test-client: Waiting for endpoints to shut down +30:36.907 [INFO] test-client: Endpoints down +2021-10-05T16:30:35.987Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:36.031Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:36.055 [INFO] test-server.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bb656f69-3920-42d3-23b9-f7547cb38d08 Address:127.0.0.1:30196}]" +30:36.056 [INFO] test-server.server.raft: entering follower state: follower="Node at 127.0.0.1:30196 [Follower]" leader= +30:36.056 [INFO] test-server.server.serf.wan: serf: EventMemberJoin: Node-bb656f69-3920-42d3-23b9-f7547cb38d08.dc1 127.0.0.1 +30:36.057 [INFO] test-server.server.serf.lan: serf: EventMemberJoin: Node-bb656f69-3920-42d3-23b9-f7547cb38d08 127.0.0.1 +2021-10-05T16:30:36.057Z [INFO] agent.router: Initializing LAN area manager +30:36.057 [INFO] test-server: Started DNS server: address=127.0.0.1:30191 network=udp +30:36.057 [INFO] test-server.server: Adding LAN server: server="Node-bb656f69-3920-42d3-23b9-f7547cb38d08 (Addr: tcp/127.0.0.1:30196) (DC: dc1)" +30:36.057 [INFO] test-server.server: Handled event for server in area: event=member-join server=Node-bb656f69-3920-42d3-23b9-f7547cb38d08.dc1 area=wan +30:36.057 [INFO] test-server: Started DNS server: address=127.0.0.1:30191 network=tcp +30:36.058 [INFO] test-server: Started HTTP server: address=127.0.0.1:30192 network=tcp +30:36.058 [INFO] test-server: started state syncer +30:36.058 [INFO] test-server: Started gRPC server: address=127.0.0.1:30197 network=tcp +30:36.104 [WARN] test-server.server.raft: heartbeat timeout reached, starting election: last-leader= +30:36.104 [INFO] test-server.server.raft: entering candidate state: node="Node at 127.0.0.1:30196 [Candidate]" term=2 +30:36.108 [DEBUG] test-server.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:36.113 [DEBUG] test-server.server.raft: votes: needed=1 +30:36.113 [DEBUG] test-server.server.raft: vote granted: from=bb656f69-3920-42d3-23b9-f7547cb38d08 term=2 tally=1 +30:36.113 [INFO] test-server.server.raft: election won: tally=1 +30:36.113 [INFO] test-server.server.raft: entering leader state: leader="Node at 127.0.0.1:30196 [Leader]" +30:36.114 [INFO] test-server.server: cluster leadership acquired +30:36.114 [INFO] test-server.server: New leader elected: payload=Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.120 [INFO] test-server.server: initializing acls +30:36.123 [INFO] test-server.server: Created ACL 'global-management' policy +30:36.130 [INFO] test-server.server: Bootstrapped ACL master token from configuration +30:36.145 [INFO] test-server.server: Created ACL anonymous token from configuration +30:36.145 [INFO] test-server.leader: started routine: routine="legacy ACL token upgrade" +30:36.145 [INFO] test-server.leader: started routine: routine="acl token reaping" +30:36.145 [INFO] test-server.server.serf.lan: serf: EventMemberUpdate: Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.145 [INFO] test-server.server.serf.wan: serf: EventMemberUpdate: Node-bb656f69-3920-42d3-23b9-f7547cb38d08.dc1 +30:36.148 [INFO] test-server.server: Updating LAN server: server="Node-bb656f69-3920-42d3-23b9-f7547cb38d08 (Addr: tcp/127.0.0.1:30196) (DC: dc1)" +30:36.148 [INFO] test-server.server: Handled event for server in area: event=member-update server=Node-bb656f69-3920-42d3-23b9-f7547cb38d08.dc1 area=wan +30:36.166 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:36.184 [INFO] test-server.server.connect: initialized primary datacenter CA with provider: provider=consul +30:36.184 [INFO] test-server.leader: started routine: routine="federation state anti-entropy" +30:36.184 [INFO] test-server.leader: started routine: routine="federation state pruning" +30:36.184 [INFO] test-server.leader: started routine: routine="intermediate cert renew watch" +30:36.184 [INFO] test-server.leader: started routine: routine="CA root pruning" +30:36.184 [DEBUG] test-server.server: Skipping self join check for node since the cluster is too small: node=Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.184 [INFO] test-server.server: member joined, marking health alive: member=Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.192 [DEBUG] test-server.server: Skipping self join check for node since the cluster is too small: node=Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.207 [INFO] test-server.server: federation state anti-entropy synced +30:36.241 [DEBUG] test-server.acl: dropping node from result due to ACLs: node=Node-bb656f69-3920-42d3-23b9-f7547cb38d08 +30:36.292 [DEBUG] test-server: Skipping remote check since it is managed automatically: check=serfHealth +30:36.304 [INFO] test-server: Synced node info +30:36.344 [DEBUG] test-server.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:51560 +30:36.345 [INFO] test-server.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +30:36.345 [INFO] test-server.server: member joined, marking health alive: member=test-client +30:36.798 [DEBUG] test-server: Skipping remote check since it is managed automatically: check=serfHealth +30:36.798 [DEBUG] test-server: Node info in sync +30:36.798 [DEBUG] test-server: Node info in sync +30:36.907 [INFO] test-server: Requesting shutdown +30:36.907 [INFO] test-server.server: shutting down server +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="legacy ACL token upgrade" +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="acl token reaping" +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="federation state anti-entropy" +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="federation state pruning" +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="intermediate cert renew watch" +30:36.907 [DEBUG] test-server.leader: stopping routine: routine="CA root pruning" +30:36.907 [WARN] test-server.server.serf.lan: serf: Shutdown without a Leave +30:36.907 [ERROR] test-server.server: error performing anti-entropy sync of federation state: error="context canceled" +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="federation state anti-entropy" +30:36.907 [INFO] test-server.server: shutting down Connect CA roots monitor +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="legacy ACL token upgrade" +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="acl token reaping" +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="federation state pruning" +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="intermediate cert renew watch" +30:36.907 [DEBUG] test-server.leader: stopped routine: routine="CA root pruning" +30:36.913 [WARN] test-server.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:36.920Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:36.920Z [INFO] agent.router.manager: shutting down +30:36.920 [INFO] test-server: consul server down +30:36.920 [INFO] test-server: shutdown complete +30:36.920 [INFO] test-server: Stopping server: protocol=DNS address=127.0.0.1:30191 network=tcp +30:36.920 [INFO] test-server: Stopping server: protocol=DNS address=127.0.0.1:30191 network=udp +30:36.920 [INFO] test-server: Stopping server: protocol=HTTP address=127.0.0.1:30192 network=tcp +30:37.421 [INFO] test-server: Waiting for endpoints to shut down +30:37.421 [INFO] test-server: Endpoints down +--- PASS: TestAgent_AutoEncrypt (1.47s) === RUN TestSharedRPCRouter [WARN] freeport: 6 out of 20 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! -45:32.155 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-f2a357bb-886a-1ab0-6ce6-33c9a080615b 127.0.0.1 -2022-11-07T22:45:32.155Z [INFO] agent.router: Initializing LAN area manager -45:32.155 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11834 network=udp -45:32.155 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11834 network=tcp -45:32.155 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11835 network=tcp -45:32.155 [INFO] TestAgent: started state syncer -45:32.155 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11840 network=tcp -45:32.155 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" -45:32.155 [INFO] TestAgent: Joining cluster...: cluster=LAN -45:32.155 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11830] -2022-11-07T22:45:32.155Z [WARN] agent.router.manager: No servers available -45:32.155 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -45:32.156 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11830 -2022-11-07T22:45:32.156Z [WARN] agent.router.manager: No servers available -45:32.156 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 127.0.0.1 -45:32.156 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -45:32.156 [DEBUG] TestAgent: systemd notify failed: error="No socket" -45:32.156 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 -45:32.156 [INFO] TestAgent.client: adding server: server="Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 (Addr: tcp/127.0.0.1:11832) (DC: dc1)" -2022-11-07T22:45:31.997Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:32.007Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:32.027 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 Address:127.0.0.1:11832}]" -45:32.027 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11832 [Follower]" leader= -45:32.028 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0.dc1 127.0.0.1 -45:32.028 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 127.0.0.1 -2022-11-07T22:45:32.028Z [INFO] agent.router: Initializing LAN area manager -45:32.028 [INFO] TestAgent.server: Adding LAN server: server="Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 (Addr: tcp/127.0.0.1:11832) (DC: dc1)" -45:32.028 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0.dc1 area=wan -45:32.028 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11827 network=udp -45:32.028 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11827 network=tcp -45:32.029 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11828 network=tcp -45:32.029 [INFO] TestAgent: started state syncer -45:32.029 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11833 network=tcp -45:32.067 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:32.067 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11832 [Candidate]" term=2 -45:32.075 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:32.075 [DEBUG] TestAgent.server.raft: vote granted: from=6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 term=2 tally=1 -45:32.075 [INFO] TestAgent.server.raft: election won: tally=1 -45:32.075 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11832 [Leader]" -45:32.076 [INFO] TestAgent.server: cluster leadership acquired -45:32.076 [INFO] TestAgent.server: New leader elected: payload=Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 -45:32.081 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11832 -45:32.093 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:32.109 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:32.109 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:32.109 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:32.109 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:32.109 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:32.109 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 -45:32.109 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6dd73528-ea4f-1938-2e8d-fdb9d2d9d1b0 -45:32.113 [INFO] TestAgent.server: federation state anti-entropy synced -45:32.156 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:53934 -45:32.156 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f2a357bb-886a-1ab0-6ce6-33c9a080615b 127.0.0.1 -45:32.156 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f2a357bb-886a-1ab0-6ce6-33c9a080615b -45:32.182 [INFO] TestAgent: Requesting shutdown -45:32.182 [INFO] TestAgent.server: shutting down server -45:32.182 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:32.182 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:32.182 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:32.182 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:32.182 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:32.182 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:32.182 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:32.182 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:32.182 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:32.182 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:32.182 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:32.186 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:32.189Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:32.189Z [INFO] agent.router.manager: shutting down -45:32.189 [INFO] TestAgent: consul server down -45:32.189 [INFO] TestAgent: shutdown complete -45:32.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11827 network=tcp -45:32.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11827 network=udp -45:32.189 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11828 network=tcp -45:32.689 [INFO] TestAgent: Waiting for endpoints to shut down -45:32.689 [INFO] TestAgent: Endpoints down ---- PASS: TestSharedRPCRouter (0.70s) +[WARN] freeport: 6 out of 6 pending ports are still in use; something probably didn't wait around for the port to be closed! +30:37.994 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-0ecfc6be-3c01-5a5d-bb8c-d364a8738a86 127.0.0.1 +2021-10-05T16:30:37.994Z [INFO] agent.router: Initializing LAN area manager +30:37.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30212 network=udp +30:37.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30212 network=tcp +30:37.995 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30213 network=tcp +30:37.995 [INFO] TestAgent: started state syncer +2021-10-05T16:30:37.996Z [WARN] agent.router.manager: No servers available +30:37.999 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" +30:37.999 [INFO] TestAgent: Joining cluster...: cluster=LAN +30:37.999 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:30208] +30:37.999 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30218 network=tcp +30:37.999 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30208 +2021-10-05T16:30:38.000Z [WARN] agent.router.manager: No servers available +30:38.000 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +30:38.000 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d 127.0.0.1 +30:38.000 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +30:38.000 [DEBUG] TestAgent: systemd notify failed: error="No socket" +30:38.000 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 +30:38.001 [INFO] TestAgent.client: adding server: server="Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d (Addr: tcp/127.0.0.1:30210) (DC: dc1)" +2021-10-05T16:30:37.470Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:37.494Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:37.549 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7a667c84-c254-4acc-1ffe-9ac7136dbc6d Address:127.0.0.1:30210}]" +30:37.550 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d.dc1 127.0.0.1 +30:37.550 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d 127.0.0.1 +2021-10-05T16:30:37.550Z [INFO] agent.router: Initializing LAN area manager +30:37.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30205 network=udp +30:37.551 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30210 [Follower]" leader= +30:37.551 [INFO] TestAgent.server: Adding LAN server: server="Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d (Addr: tcp/127.0.0.1:30210) (DC: dc1)" +30:37.551 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d.dc1 area=wan +30:37.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30205 network=tcp +30:37.552 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30206 network=tcp +30:37.552 [INFO] TestAgent: started state syncer +30:37.552 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30211 network=tcp +30:37.604 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:37.604 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30210 [Candidate]" term=2 +30:37.640 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:37.640 [DEBUG] TestAgent.server.raft: vote granted: from=7a667c84-c254-4acc-1ffe-9ac7136dbc6d term=2 tally=1 +30:37.640 [INFO] TestAgent.server.raft: election won: tally=1 +30:37.640 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30210 [Leader]" +30:37.643 [INFO] TestAgent.server: cluster leadership acquired +30:37.644 [INFO] TestAgent.server: New leader elected: payload=Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d +30:37.647 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30210 +30:37.676 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:37.728 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:37.728 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:37.728 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:37.728 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:37.728 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:37.728 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d +30:37.728 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7a667c84-c254-4acc-1ffe-9ac7136dbc6d +30:37.772 [INFO] TestAgent.server: federation state anti-entropy synced +30:37.860 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:37.876 [INFO] TestAgent: Synced node info +30:37.876 [DEBUG] TestAgent: Node info in sync +30:37.999 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:46168 +30:38.000 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0ecfc6be-3c01-5a5d-bb8c-d364a8738a86 127.0.0.1 +30:38.000 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0ecfc6be-3c01-5a5d-bb8c-d364a8738a86 +30:38.032 [INFO] TestAgent: Requesting shutdown +30:38.033 [INFO] TestAgent.server: shutting down server +30:38.033 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:38.033 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:38.033 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:38.033 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:38.033 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:38.033 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:38.033 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:38.033 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:38.033 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:38.033 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:38.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:38.052Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:38.052Z [INFO] agent.router.manager: shutting down +30:38.052 [INFO] TestAgent: consul server down +30:38.052 [INFO] TestAgent: shutdown complete +30:38.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30205 network=tcp +30:38.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30205 network=udp +30:38.052 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30206 network=tcp +30:38.556 [INFO] TestAgent: Waiting for endpoints to shut down +30:38.556 [INFO] TestAgent: Endpoints down +--- PASS: TestSharedRPCRouter (1.13s) === RUN TestCatalogRegister_Service_InvalidAddress === PAUSE TestCatalogRegister_Service_InvalidAddress === RUN TestCatalogDeregister @@ -13680,178 +13862,150 @@ [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:33.139Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:33.153Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:33.177 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f60e3a5b-8a1f-7a02-51f5-35ec8945d44a Address:127.0.0.1:11853}]" -45:33.177 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11853 [Follower]" leader= -45:33.178 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a.dc2 127.0.0.1 -45:33.178 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a 127.0.0.1 -2022-11-07T22:45:33.178Z [INFO] agent.router: Initializing LAN area manager -45:33.178 [INFO] TestAgent.server: Adding LAN server: server="Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a (Addr: tcp/127.0.0.1:11853) (DC: dc2)" -45:33.179 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a.dc2 area=wan -45:33.179 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11848 network=udp -45:33.179 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11848 network=tcp -45:33.179 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11849 network=tcp -45:33.179 [INFO] TestAgent: started state syncer -45:33.179 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11854 network=tcp -45:33.179 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" -45:33.179 [INFO] TestAgent: Joining cluster...: cluster=WAN -45:33.179 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11845] -45:33.180 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11845 -45:33.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd.dc1 127.0.0.1 -45:33.181 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -45:33.181 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 -45:33.181 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd.dc1 area=wan -45:33.236 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:33.236 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11853 [Candidate]" term=2 -45:33.244 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:33.244 [DEBUG] TestAgent.server.raft: vote granted: from=f60e3a5b-8a1f-7a02-51f5-35ec8945d44a term=2 tally=1 -45:33.244 [INFO] TestAgent.server.raft: election won: tally=1 -45:33.244 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11853 [Leader]" -45:33.244 [INFO] TestAgent.server: cluster leadership acquired -45:33.244 [INFO] TestAgent.server: New leader elected: payload=Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a -45:33.249 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true -45:33.260 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false -45:33.278 [INFO] TestAgent.server.connect: received new intermediate certificate from primary datacenter -45:33.283 [INFO] TestAgent.server.connect: updated root certificates from primary datacenter -45:33.283 [INFO] TestAgent.server.connect: initialized secondary datacenter CA with provider: provider=consul -45:33.283 [INFO] TestAgent.leader: started routine: routine="config entry replication" -45:33.283 [INFO] TestAgent.leader: started routine: routine="federation state replication" -45:33.283 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:33.283 [INFO] TestAgent.leader: started routine: routine="secondary CA roots watch" -45:33.283 [INFO] TestAgent.leader: started routine: routine="intention replication" -45:33.283 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:33.283 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:33.283 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a -45:33.283 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a -45:33.284 [DEBUG] TestAgent.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 -45:33.284 [DEBUG] TestAgent.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 -45:33.284 [DEBUG] TestAgent.server.replication.config_entry: finished fetching config entries: amount=0 -45:33.284 [DEBUG] TestAgent.server.replication.config_entry: Config Entry replication: local=0 remote=0 -45:33.284 [DEBUG] TestAgent.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 -45:33.284 [DEBUG] TestAgent.server.replication.config_entry: replication completed through remote index: index=1 -45:33.285 [DEBUG] TestAgent.server: finished fetching remote objects: amount=1 -45:33.285 [DEBUG] TestAgent.server: diffing replication state: local_amount=0 remote_amount=1 -45:33.285 [DEBUG] TestAgent.server: diffed replication state: deletions=0 updates=1 -45:33.285 [DEBUG] TestAgent.server: performing updates: updates=1 -45:33.288 [INFO] TestAgent.server: federation state anti-entropy synced -45:33.294 [DEBUG] TestAgent.server: finished updates -45:33.294 [DEBUG] TestAgent.replication.federation_state: replication completed through remote index: index=11 -45:33.294 [INFO] TestAgent.server.connect: updated root certificates from primary datacenter -45:33.294 [DEBUG] TestAgent.server: finished fetching remote objects: amount=2 -45:33.294 [DEBUG] TestAgent.server: diffing replication state: local_amount=1 remote_amount=2 -45:33.294 [DEBUG] TestAgent.server: diffed replication state: deletions=0 updates=1 -45:33.294 [DEBUG] TestAgent.server: performing updates: updates=1 -45:33.297 [DEBUG] TestAgent.server: finished updates -45:33.297 [DEBUG] TestAgent.replication.federation_state: replication completed through remote index: index=14 -45:33.297 [INFO] TestAgent.server: federation state anti-entropy synced -45:33.297 [DEBUG] TestAgent.server: finished fetching remote objects: amount=2 -45:33.297 [DEBUG] TestAgent.server: diffing replication state: local_amount=2 remote_amount=2 -45:33.297 [DEBUG] TestAgent.server: diffed replication state: deletions=0 updates=1 -45:33.297 [DEBUG] TestAgent.server: performing updates: updates=1 -45:33.299 [DEBUG] TestAgent.server: finished updates -45:33.299 [DEBUG] TestAgent.replication.federation_state: replication completed through remote index: index=15 -45:33.434 [INFO] TestAgent: Requesting shutdown -45:33.434 [INFO] TestAgent.server: shutting down server -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="config entry replication" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="federation state replication" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="secondary CA roots watch" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="intention replication" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:33.434 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:33.434 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:33.434 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:33.434 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:33.434 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:33.434 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:33.434 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:33.438 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:33.441Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:33.441Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:33.441Z [INFO] agent.router.manager: shutting down -45:33.441 [INFO] TestAgent: consul server down -45:33.441 [INFO] TestAgent: shutdown complete -45:33.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11848 network=tcp -45:33.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11848 network=udp -45:33.441 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11849 network=tcp -45:33.441 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11846 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" -45:33.442 [WARN] TestAgent.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" -45:33.442 [INFO] TestAgent.replication.federation_state: stopped replication -45:33.442 [DEBUG] TestAgent.leader: stopped routine: routine="federation state replication" -45:33.442 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11846 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" -45:33.442 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11846 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" -45:33.442 [ERROR] TestAgent.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" -45:33.442 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11846 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" -45:33.442 [INFO] TestAgent.server.replication.config_entry: stopped replication -45:33.442 [DEBUG] TestAgent.leader: stopped routine: routine="config entry replication" -45:33.442 [DEBUG] TestAgent.leader: stopped routine: routine="intention replication" -45:33.442 [ERROR] TestAgent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" -45:33.442 [DEBUG] TestAgent.leader: stopped routine: routine="secondary CA roots watch" -45:33.942 [INFO] TestAgent: Waiting for endpoints to shut down -45:33.942 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:32.701Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:32.710Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:32.839 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:77b928bc-67e8-7641-1f72-06e7e3f9ebfd Address:127.0.0.1:11846}]" -45:32.839 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11846 [Follower]" leader= -45:32.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd.dc1 127.0.0.1 -45:32.840 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd 127.0.0.1 -2022-11-07T22:45:32.840Z [INFO] agent.router: Initializing LAN area manager -45:32.840 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd.dc1 area=wan -45:32.840 [INFO] TestAgent.server: Adding LAN server: server="Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd (Addr: tcp/127.0.0.1:11846) (DC: dc1)" -45:32.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11841 network=udp -45:32.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11841 network=tcp -45:32.841 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11842 network=tcp -45:32.841 [INFO] TestAgent: started state syncer -45:32.841 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11847 network=tcp -45:32.887 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:32.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11846 [Candidate]" term=2 -45:32.896 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:32.896 [DEBUG] TestAgent.server.raft: vote granted: from=77b928bc-67e8-7641-1f72-06e7e3f9ebfd term=2 tally=1 -45:32.896 [INFO] TestAgent.server.raft: election won: tally=1 -45:32.896 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11846 [Leader]" -45:32.896 [INFO] TestAgent.server: cluster leadership acquired -45:32.896 [INFO] TestAgent.server: New leader elected: payload=Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd -45:32.901 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11846 -45:32.912 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:32.931 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:32.931 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:32.931 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:32.931 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:32.931 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:32.931 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd -45:32.931 [INFO] TestAgent.server: member joined, marking health alive: member=Node-77b928bc-67e8-7641-1f72-06e7e3f9ebfd -45:32.941 [INFO] TestAgent.server: federation state anti-entropy synced -45:32.944 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:32.947 [INFO] TestAgent: Synced node info -45:32.947 [DEBUG] TestAgent: Node info in sync -45:33.180 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:51954 -45:33.180 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a.dc2 127.0.0.1 -45:33.180 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f60e3a5b-8a1f-7a02-51f5-35ec8945d44a.dc2 area=wan -45:33.942 [INFO] TestAgent: Requesting shutdown -45:33.942 [INFO] TestAgent.server: shutting down server -45:33.942 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:33.942 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:33.942 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:33.942 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:33.942 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:33.942 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:33.942 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:33.942 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:33.942 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:33.942 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:33.947 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:33.952Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:33.952Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:33.952Z [INFO] agent.router.manager: shutting down -45:33.952 [INFO] TestAgent: consul server down -45:33.952 [INFO] TestAgent: shutdown complete -45:33.952 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11841 network=tcp -45:33.952 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11841 network=udp -45:33.952 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11842 network=tcp -45:34.452 [INFO] TestAgent: Waiting for endpoints to shut down -45:34.452 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCARoots_PEMEncoding (1.76s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:38.985Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:39.007Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:39.092 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd2a89ae-a009-0d2e-14d8-d914febc36d0 Address:127.0.0.1:30231}]" +30:39.093 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0.dc2 127.0.0.1 +30:39.093 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0 127.0.0.1 +2021-10-05T16:30:39.093Z [INFO] agent.router: Initializing LAN area manager +30:39.093 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30226 network=udp +30:39.093 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30231 [Follower]" leader= +30:39.094 [INFO] TestAgent.server: Adding LAN server: server="Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0 (Addr: tcp/127.0.0.1:30231) (DC: dc2)" +30:39.094 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0.dc2 area=wan +30:39.094 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30226 network=tcp +30:39.094 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30227 network=tcp +30:39.095 [INFO] TestAgent: started state syncer +30:39.095 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30232 network=tcp +30:39.095 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" +30:39.095 [INFO] TestAgent: Joining cluster...: cluster=WAN +30:39.095 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:30223] +30:39.096 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:30223 +30:39.100 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf.dc1 127.0.0.1 +30:39.100 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +30:39.100 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 +30:39.100 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf.dc1 area=wan +30:39.140 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:39.140 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30231 [Candidate]" term=2 +30:39.188 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:39.188 [DEBUG] TestAgent.server.raft: vote granted: from=dd2a89ae-a009-0d2e-14d8-d914febc36d0 term=2 tally=1 +30:39.188 [INFO] TestAgent.server.raft: election won: tally=1 +30:39.188 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30231 [Leader]" +30:39.192 [INFO] TestAgent.server: cluster leadership acquired +30:39.192 [INFO] TestAgent.server: New leader elected: payload=Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0 +30:39.220 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true +30:39.279 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false +30:39.352 [INFO] TestAgent.server.connect: received new intermediate certificate from primary datacenter +30:39.392 [INFO] TestAgent: Synced node info +30:39.392 [DEBUG] TestAgent: Node info in sync +30:39.392 [INFO] TestAgent.server.connect: updated root certificates from primary datacenter +30:39.392 [INFO] TestAgent.server.connect: initialized secondary datacenter CA with provider: provider=consul +30:39.392 [INFO] TestAgent.leader: started routine: routine="config entry replication" +30:39.392 [INFO] TestAgent.leader: started routine: routine="federation state replication" +30:39.392 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:39.392 [INFO] TestAgent.leader: started routine: routine="secondary CA roots watch" +30:39.392 [INFO] TestAgent.leader: started routine: routine="intention replication" +30:39.392 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:39.392 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:39.392 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0 +30:39.392 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0 +30:39.393 [DEBUG] TestAgent.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 +30:39.393 [DEBUG] TestAgent.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 +30:39.400 [INFO] TestAgent: Requesting shutdown +30:39.400 [INFO] TestAgent.server: shutting down server +30:39.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="secondary CA roots watch" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="intention replication" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="config entry replication" +30:39.401 [DEBUG] TestAgent.leader: stopping routine: routine="federation state replication" +30:39.401 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:39.401 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:39.401 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:39.401 [DEBUG] TestAgent.server: finished fetching remote objects: amount=1 +30:39.401 [DEBUG] TestAgent.server.replication.config_entry: finished fetching config entries: amount=0 +30:39.401 [INFO] TestAgent.replication.federation_state: stopped replication +30:39.401 [INFO] TestAgent.server.replication.config_entry: stopped replication +30:39.401 [DEBUG] TestAgent.leader: stopped routine: routine="federation state replication" +30:39.401 [DEBUG] TestAgent.leader: stopped routine: routine="config entry replication" +30:39.401 [DEBUG] TestAgent.leader: stopped routine: routine="intention replication" +30:39.408 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +30:39.416 [INFO] TestAgent.server: federation state anti-entropy synced +30:39.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:30:39.416Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:39.416Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:39.416Z [INFO] agent.router.manager: shutting down +30:39.418 [INFO] TestAgent: consul server down +30:39.418 [INFO] TestAgent: shutdown complete +30:39.418 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30226 network=tcp +30:39.418 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30226 network=udp +30:39.418 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30227 network=tcp +30:39.419 [ERROR] TestAgent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Failed to initialize the secondary CA: leadership lost while committing log" +30:39.419 [DEBUG] TestAgent.leader: stopped routine: routine="secondary CA roots watch" +30:39.919 [INFO] TestAgent: Waiting for endpoints to shut down +30:39.919 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:38.621Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:38.633Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:38.656 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7c71bbe2-7b4f-3fd4-2caf-364669be07bf Address:127.0.0.1:30224}]" +30:38.657 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf.dc1 127.0.0.1 +30:38.657 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf 127.0.0.1 +2021-10-05T16:30:38.657Z [INFO] agent.router: Initializing LAN area manager +30:38.658 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30219 network=udp +30:38.658 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30224 [Follower]" leader= +30:38.658 [INFO] TestAgent.server: Adding LAN server: server="Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf (Addr: tcp/127.0.0.1:30224) (DC: dc1)" +30:38.658 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf.dc1 area=wan +30:38.658 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30219 network=tcp +30:38.659 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30220 network=tcp +30:38.659 [INFO] TestAgent: started state syncer +30:38.659 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30225 network=tcp +30:38.715 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:38.715 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30224 [Candidate]" term=2 +30:38.723 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:38.723 [DEBUG] TestAgent.server.raft: vote granted: from=7c71bbe2-7b4f-3fd4-2caf-364669be07bf term=2 tally=1 +30:38.723 [INFO] TestAgent.server.raft: election won: tally=1 +30:38.723 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30224 [Leader]" +30:38.724 [INFO] TestAgent.server: cluster leadership acquired +30:38.724 [INFO] TestAgent.server: New leader elected: payload=Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf +30:38.728 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30224 +30:38.743 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:38.789 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:38.789 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:38.789 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:38.789 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:38.789 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:38.789 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf +30:38.789 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7c71bbe2-7b4f-3fd4-2caf-364669be07bf +30:38.801 [INFO] TestAgent.server: federation state anti-entropy synced +30:38.928 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:38.948 [INFO] TestAgent: Synced node info +30:39.096 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:38266 +30:39.097 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0.dc2 127.0.0.1 +30:39.097 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd2a89ae-a009-0d2e-14d8-d914febc36d0.dc2 area=wan +30:39.919 [INFO] TestAgent: Requesting shutdown +30:39.919 [INFO] TestAgent.server: shutting down server +30:39.919 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:39.919 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:39.919 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:39.919 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:39.919 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:39.919 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:39.919 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:39.919 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:39.919 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:39.919 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:39.937 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:39.963Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:39.963Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:39.963Z [INFO] agent.router.manager: shutting down +30:39.963 [INFO] TestAgent: consul server down +30:39.963 [INFO] TestAgent: shutdown complete +30:39.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30219 network=tcp +30:39.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30219 network=udp +30:39.963 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30220 network=tcp +30:40.464 [INFO] TestAgent: Waiting for endpoints to shut down +30:40.464 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCARoots_PEMEncoding (1.91s) === RUN TestCoordinate_Disabled_Response === PAUSE TestCoordinate_Disabled_Response === RUN TestCoordinate_Datacenters @@ -13895,314 +14049,315 @@ [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:34.463Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:34.472Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:34.495 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2bd25b27-2f16-3258-c11d-b25f2faab3e8 Address:127.0.0.1:11860}]" -45:34.495 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11860 [Follower]" leader= -45:34.496 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8.dc1 127.0.0.1 -45:34.496 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8 127.0.0.1 -2022-11-07T22:45:34.496Z [INFO] agent.router: Initializing LAN area manager -45:34.496 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8.dc1 area=wan -45:34.496 [INFO] TestAgent.server: Adding LAN server: server="Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8 (Addr: tcp/127.0.0.1:11860) (DC: dc1)" -45:34.497 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11855 network=udp -45:34.497 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11855 network=tcp -45:34.497 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11856 network=tcp -45:34.497 [INFO] TestAgent: started state syncer -45:34.497 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11861 network=tcp -45:34.541 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:34.541 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11860 [Candidate]" term=2 -45:34.551 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:34.551 [DEBUG] TestAgent.server.raft: vote granted: from=2bd25b27-2f16-3258-c11d-b25f2faab3e8 term=2 tally=1 -45:34.551 [INFO] TestAgent.server.raft: election won: tally=1 -45:34.551 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11860 [Leader]" -45:34.552 [INFO] TestAgent.server: cluster leadership acquired -45:34.552 [INFO] TestAgent.server: New leader elected: payload=Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8 -45:34.557 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11860 -45:34.569 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:34.588 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:34.588 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:34.588 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:34.588 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:34.588 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:34.588 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8 -45:34.588 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2bd25b27-2f16-3258-c11d-b25f2faab3e8 -45:34.600 [INFO] TestAgent.server: federation state anti-entropy synced -45:34.609 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=TXT class=IN latency=127.686µs client=127.0.0.1:53330 client_network=udp -45:34.609 [INFO] TestAgent: Requesting shutdown -45:34.609 [INFO] TestAgent.server: shutting down server -45:34.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:34.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:34.609 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:34.609 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:34.609 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:34.609 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:34.609 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:34.609 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:34.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:34.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:34.609 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:34.612 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:34.618Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:34.618Z [INFO] agent.router.manager: shutting down -45:34.618 [INFO] TestAgent: consul server down -45:34.618 [INFO] TestAgent: shutdown complete -45:34.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11855 network=tcp -45:34.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11855 network=udp -45:34.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11856 network=tcp -45:35.118 [INFO] TestAgent: Waiting for endpoints to shut down -45:35.118 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_TXT (0.67s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:40.495Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:40.518Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:40.606 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5decdacf-196e-bd69-f214-f5cbd35b2579 Address:127.0.0.1:30238}]" +30:40.606 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30238 [Follower]" leader= +30:40.606 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5decdacf-196e-bd69-f214-f5cbd35b2579.dc1 127.0.0.1 +30:40.607 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5decdacf-196e-bd69-f214-f5cbd35b2579 127.0.0.1 +2021-10-05T16:30:40.607Z [INFO] agent.router: Initializing LAN area manager +30:40.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30233 network=udp +30:40.607 [INFO] TestAgent.server: Adding LAN server: server="Node-5decdacf-196e-bd69-f214-f5cbd35b2579 (Addr: tcp/127.0.0.1:30238) (DC: dc1)" +30:40.607 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5decdacf-196e-bd69-f214-f5cbd35b2579.dc1 area=wan +30:40.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30233 network=tcp +30:40.608 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30234 network=tcp +30:40.608 [INFO] TestAgent: started state syncer +30:40.608 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30239 network=tcp +30:40.671 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:40.671 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30238 [Candidate]" term=2 +30:40.714 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:40.714 [DEBUG] TestAgent.server.raft: vote granted: from=5decdacf-196e-bd69-f214-f5cbd35b2579 term=2 tally=1 +30:40.714 [INFO] TestAgent.server.raft: election won: tally=1 +30:40.714 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30238 [Leader]" +30:40.714 [INFO] TestAgent.server: cluster leadership acquired +30:40.714 [INFO] TestAgent.server: New leader elected: payload=Node-5decdacf-196e-bd69-f214-f5cbd35b2579 +30:40.732 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30238 +30:40.784 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:40.856 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:40.856 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:40.856 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:40.856 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:40.856 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:40.856 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5decdacf-196e-bd69-f214-f5cbd35b2579 +30:40.856 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5decdacf-196e-bd69-f214-f5cbd35b2579 +30:40.879 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:40.896 [INFO] TestAgent.server: federation state anti-entropy synced +30:40.909 [INFO] TestAgent: Synced node info +30:40.909 [DEBUG] TestAgent: Node info in sync +30:40.910 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=TXT class=IN latency=201.361µs client=127.0.0.1:49665 client_network=udp +30:40.910 [INFO] TestAgent: Requesting shutdown +30:40.910 [INFO] TestAgent.server: shutting down server +30:40.910 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:40.910 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:40.910 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:40.910 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:40.910 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:40.910 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:40.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:40.910 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:40.910 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:40.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:40.914 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:40.928Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:40.928Z [INFO] agent.router.manager: shutting down +30:40.928 [INFO] TestAgent: consul server down +30:40.928 [INFO] TestAgent: shutdown complete +30:40.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30233 network=tcp +30:40.929 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30233 network=udp +30:40.929 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30234 network=tcp +30:41.430 [INFO] TestAgent: Waiting for endpoints to shut down +30:41.430 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_TXT (0.96s) === RUN TestDNS_NodeLookup_TXT_DontSuppress [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:35.128Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:35.137Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:35.159 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4db43e8c-8b9c-751b-ec68-9d89cf090c2e Address:127.0.0.1:11867}]" -45:35.159 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11867 [Follower]" leader= -45:35.160 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e.dc1 127.0.0.1 -45:35.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e 127.0.0.1 -2022-11-07T22:45:35.160Z [INFO] agent.router: Initializing LAN area manager -45:35.161 [INFO] TestAgent.server: Adding LAN server: server="Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e (Addr: tcp/127.0.0.1:11867) (DC: dc1)" -45:35.161 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e.dc1 area=wan -45:35.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11862 network=udp -45:35.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11862 network=tcp -45:35.161 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11863 network=tcp -45:35.161 [INFO] TestAgent: started state syncer -45:35.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11868 network=tcp -45:35.229 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:35.229 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11867 [Candidate]" term=2 -45:35.238 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:35.238 [DEBUG] TestAgent.server.raft: vote granted: from=4db43e8c-8b9c-751b-ec68-9d89cf090c2e term=2 tally=1 -45:35.238 [INFO] TestAgent.server.raft: election won: tally=1 -45:35.238 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11867 [Leader]" -45:35.238 [INFO] TestAgent.server: cluster leadership acquired -45:35.238 [INFO] TestAgent.server: New leader elected: payload=Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e -45:35.243 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11867 -45:35.254 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:35.271 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:35.271 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:35.271 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:35.271 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:35.271 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:35.271 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e -45:35.271 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4db43e8c-8b9c-751b-ec68-9d89cf090c2e -45:35.275 [INFO] TestAgent.server: federation state anti-entropy synced -45:35.403 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:35.406 [INFO] TestAgent: Synced node info -45:35.406 [DEBUG] TestAgent: Node info in sync -45:35.409 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=TXT class=IN latency=106.124µs client=127.0.0.1:46856 client_network=udp -45:35.409 [INFO] TestAgent: Requesting shutdown -45:35.409 [INFO] TestAgent.server: shutting down server -45:35.409 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:35.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:35.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:35.409 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:35.409 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:35.409 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:35.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:35.409 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:35.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:35.409 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:35.413 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:35.416Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:35.416Z [INFO] agent.router.manager: shutting down -45:35.416 [INFO] TestAgent: consul server down -45:35.416 [INFO] TestAgent: shutdown complete -45:35.416 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11862 network=tcp -45:35.416 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11862 network=udp -45:35.416 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11863 network=tcp -45:35.916 [INFO] TestAgent: Waiting for endpoints to shut down -45:35.916 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_TXT_DontSuppress (0.80s) +2021-10-05T16:30:41.447Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:41.485Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:41.524 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:533a2013-424d-132c-c95b-2b39e47ffe7a Address:127.0.0.1:30245}]" +30:41.525 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30245 [Follower]" leader= +30:41.525 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-533a2013-424d-132c-c95b-2b39e47ffe7a.dc1 127.0.0.1 +30:41.526 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-533a2013-424d-132c-c95b-2b39e47ffe7a 127.0.0.1 +2021-10-05T16:30:41.526Z [INFO] agent.router: Initializing LAN area manager +30:41.526 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30240 network=udp +30:41.526 [INFO] TestAgent.server: Adding LAN server: server="Node-533a2013-424d-132c-c95b-2b39e47ffe7a (Addr: tcp/127.0.0.1:30245) (DC: dc1)" +30:41.526 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-533a2013-424d-132c-c95b-2b39e47ffe7a.dc1 area=wan +30:41.526 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30240 network=tcp +30:41.527 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30241 network=tcp +30:41.527 [INFO] TestAgent: started state syncer +30:41.527 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30246 network=tcp +30:41.568 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:41.568 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30245 [Candidate]" term=2 +30:41.603 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:41.603 [DEBUG] TestAgent.server.raft: vote granted: from=533a2013-424d-132c-c95b-2b39e47ffe7a term=2 tally=1 +30:41.603 [INFO] TestAgent.server.raft: election won: tally=1 +30:41.603 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30245 [Leader]" +30:41.604 [INFO] TestAgent.server: cluster leadership acquired +30:41.604 [INFO] TestAgent.server: New leader elected: payload=Node-533a2013-424d-132c-c95b-2b39e47ffe7a +30:41.613 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30245 +30:41.638 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:41.676 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:41.676 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:41.676 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:41.676 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:41.676 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:41.676 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-533a2013-424d-132c-c95b-2b39e47ffe7a +30:41.676 [INFO] TestAgent.server: member joined, marking health alive: member=Node-533a2013-424d-132c-c95b-2b39e47ffe7a +30:41.718 [INFO] TestAgent.server: federation state anti-entropy synced +30:41.727 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=TXT class=IN latency=125.083µs client=127.0.0.1:56963 client_network=udp +30:41.727 [INFO] TestAgent: Requesting shutdown +30:41.727 [INFO] TestAgent.server: shutting down server +30:41.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:41.727 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:41.727 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:41.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:41.727 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:41.727 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:41.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:41.727 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:41.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:41.727 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:41.727 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:41.744 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:41.752Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:41.752Z [INFO] agent.router.manager: shutting down +30:41.752 [INFO] TestAgent: consul server down +30:41.752 [INFO] TestAgent: shutdown complete +30:41.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30240 network=tcp +30:41.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30240 network=udp +30:41.752 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30241 network=tcp +30:42.256 [INFO] TestAgent: Waiting for endpoints to shut down +30:42.256 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_TXT_DontSuppress (0.83s) === RUN TestDNS_NodeLookup_ANY [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:35.926Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:35.935Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:35.953 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b63ca525-8a83-b86a-0480-f1d4a5140a8a Address:127.0.0.1:11874}]" -45:35.953 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11874 [Follower]" leader= -45:35.954 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a.dc1 127.0.0.1 -45:35.954 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a 127.0.0.1 -2022-11-07T22:45:35.954Z [INFO] agent.router: Initializing LAN area manager -45:35.954 [INFO] TestAgent.server: Adding LAN server: server="Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a (Addr: tcp/127.0.0.1:11874) (DC: dc1)" -45:35.954 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a.dc1 area=wan -45:35.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11869 network=udp -45:35.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11869 network=tcp -45:35.955 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11870 network=tcp -45:35.955 [INFO] TestAgent: started state syncer -45:35.955 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11875 network=tcp -45:36.022 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:36.022 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11874 [Candidate]" term=2 -45:36.030 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:36.030 [DEBUG] TestAgent.server.raft: vote granted: from=b63ca525-8a83-b86a-0480-f1d4a5140a8a term=2 tally=1 -45:36.030 [INFO] TestAgent.server.raft: election won: tally=1 -45:36.030 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11874 [Leader]" -45:36.030 [INFO] TestAgent.server: cluster leadership acquired -45:36.030 [INFO] TestAgent.server: New leader elected: payload=Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a -45:36.035 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11874 -45:36.044 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:36.062 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:36.062 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:36.062 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:36.062 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:36.062 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:36.062 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a -45:36.062 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b63ca525-8a83-b86a-0480-f1d4a5140a8a -45:36.066 [INFO] TestAgent.server: federation state anti-entropy synced -45:36.125 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:36.128 [INFO] TestAgent: Synced node info -45:36.377 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=104.673µs client=127.0.0.1:41485 client_network=udp -45:36.377 [INFO] TestAgent: Requesting shutdown -45:36.377 [INFO] TestAgent.server: shutting down server -45:36.377 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:36.377 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:36.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:36.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:36.377 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:36.377 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:36.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:36.377 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:36.377 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:36.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:36.381 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:36.385Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:36.385Z [INFO] agent.router.manager: shutting down -45:36.385 [INFO] TestAgent: consul server down -45:36.385 [INFO] TestAgent: shutdown complete -45:36.385 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11869 network=tcp -45:36.385 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11869 network=udp -45:36.385 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11870 network=tcp -45:36.885 [INFO] TestAgent: Waiting for endpoints to shut down -45:36.885 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_ANY (0.97s) +2021-10-05T16:30:42.284Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:42.303Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:42.396 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1268e432-dd95-c84c-c528-1d29bc8bdd06 Address:127.0.0.1:30252}]" +30:42.396 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30252 [Follower]" leader= +30:42.397 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1268e432-dd95-c84c-c528-1d29bc8bdd06.dc1 127.0.0.1 +30:42.397 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1268e432-dd95-c84c-c528-1d29bc8bdd06 127.0.0.1 +2021-10-05T16:30:42.397Z [INFO] agent.router: Initializing LAN area manager +30:42.397 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30247 network=udp +30:42.398 [INFO] TestAgent.server: Adding LAN server: server="Node-1268e432-dd95-c84c-c528-1d29bc8bdd06 (Addr: tcp/127.0.0.1:30252) (DC: dc1)" +30:42.398 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1268e432-dd95-c84c-c528-1d29bc8bdd06.dc1 area=wan +30:42.398 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30247 network=tcp +30:42.398 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30248 network=tcp +30:42.398 [INFO] TestAgent: started state syncer +30:42.398 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30253 network=tcp +30:42.468 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:42.468 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30252 [Candidate]" term=2 +30:42.511 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:42.511 [DEBUG] TestAgent.server.raft: vote granted: from=1268e432-dd95-c84c-c528-1d29bc8bdd06 term=2 tally=1 +30:42.511 [INFO] TestAgent.server.raft: election won: tally=1 +30:42.512 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30252 [Leader]" +30:42.512 [INFO] TestAgent.server: cluster leadership acquired +30:42.512 [INFO] TestAgent.server: New leader elected: payload=Node-1268e432-dd95-c84c-c528-1d29bc8bdd06 +30:42.518 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30252 +30:42.540 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:42.588 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:42.588 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:42.588 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:42.588 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:42.588 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:42.588 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1268e432-dd95-c84c-c528-1d29bc8bdd06 +30:42.588 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1268e432-dd95-c84c-c528-1d29bc8bdd06 +30:42.628 [INFO] TestAgent.server: federation state anti-entropy synced +30:42.726 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=105.271µs client=127.0.0.1:45701 client_network=udp +30:42.726 [INFO] TestAgent: Requesting shutdown +30:42.726 [INFO] TestAgent.server: shutting down server +30:42.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:42.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:42.726 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:42.726 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:42.726 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:42.726 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:42.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:42.726 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:42.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:42.726 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:42.726 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:42.734 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:42.744Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:42.744Z [INFO] agent.router.manager: shutting down +30:42.744 [INFO] TestAgent: consul server down +30:42.744 [INFO] TestAgent: shutdown complete +30:42.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30247 network=tcp +30:42.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30247 network=udp +30:42.744 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30248 network=tcp +30:43.248 [INFO] TestAgent: Waiting for endpoints to shut down +30:43.248 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_ANY (0.99s) === RUN TestDNS_NodeLookup_ANY_DontSuppressTXT [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:36.895Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:36.904Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:36.992 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a Address:127.0.0.1:11881}]" -45:36.992 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11881 [Follower]" leader= -45:36.992 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a.dc1 127.0.0.1 -45:36.993 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a 127.0.0.1 -2022-11-07T22:45:36.993Z [INFO] agent.router: Initializing LAN area manager -45:36.993 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11876 network=udp -45:36.993 [INFO] TestAgent.server: Adding LAN server: server="Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a (Addr: tcp/127.0.0.1:11881) (DC: dc1)" -45:36.993 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a.dc1 area=wan -45:36.993 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11876 network=tcp -45:36.994 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11877 network=tcp -45:36.994 [INFO] TestAgent: started state syncer -45:36.994 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11882 network=tcp -45:37.029 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:37.029 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11881 [Candidate]" term=2 -45:37.037 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:37.037 [DEBUG] TestAgent.server.raft: vote granted: from=1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a term=2 tally=1 -45:37.037 [INFO] TestAgent.server.raft: election won: tally=1 -45:37.037 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11881 [Leader]" -45:37.037 [INFO] TestAgent.server: cluster leadership acquired -45:37.037 [INFO] TestAgent.server: New leader elected: payload=Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a -45:37.042 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11881 -45:37.053 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:37.074 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:37.074 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:37.074 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:37.074 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:37.074 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:37.074 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a -45:37.074 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1ff8c4c5-fb6a-02fb-2fe8-d65979f5839a -45:37.083 [INFO] TestAgent.server: federation state anti-entropy synced -45:37.191 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:37.194 [INFO] TestAgent: Synced node info -45:37.194 [DEBUG] TestAgent: Node info in sync -45:37.379 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=94.493µs client=127.0.0.1:34034 client_network=udp -45:37.379 [INFO] TestAgent: Requesting shutdown -45:37.379 [INFO] TestAgent.server: shutting down server -45:37.379 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:37.379 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:37.379 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:37.379 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:37.379 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:37.379 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:37.379 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:37.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:37.379 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:37.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:37.382 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:37.386Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:37.386Z [INFO] agent.router.manager: shutting down -45:37.386 [INFO] TestAgent: consul server down -45:37.386 [INFO] TestAgent: shutdown complete -45:37.386 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11876 network=tcp -45:37.386 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11876 network=udp -45:37.386 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11877 network=tcp -45:37.886 [INFO] TestAgent: Waiting for endpoints to shut down -45:37.886 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_ANY_DontSuppressTXT (1.00s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:43.287Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:43.364Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:43.460 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc22ac75-543b-24c9-3631-2e53fb9eca61 Address:127.0.0.1:30259}]" +30:43.460 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30259 [Follower]" leader= +30:43.460 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc22ac75-543b-24c9-3631-2e53fb9eca61.dc1 127.0.0.1 +30:43.461 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc22ac75-543b-24c9-3631-2e53fb9eca61 127.0.0.1 +2021-10-05T16:30:43.461Z [INFO] agent.router: Initializing LAN area manager +30:43.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30254 network=udp +30:43.461 [INFO] TestAgent.server: Adding LAN server: server="Node-bc22ac75-543b-24c9-3631-2e53fb9eca61 (Addr: tcp/127.0.0.1:30259) (DC: dc1)" +30:43.461 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc22ac75-543b-24c9-3631-2e53fb9eca61.dc1 area=wan +30:43.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30254 network=tcp +30:43.462 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30255 network=tcp +30:43.462 [INFO] TestAgent: started state syncer +30:43.462 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30260 network=tcp +30:43.519 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:43.519 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30259 [Candidate]" term=2 +30:43.544 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:43.544 [DEBUG] TestAgent.server.raft: vote granted: from=bc22ac75-543b-24c9-3631-2e53fb9eca61 term=2 tally=1 +30:43.544 [INFO] TestAgent.server.raft: election won: tally=1 +30:43.544 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30259 [Leader]" +30:43.548 [INFO] TestAgent.server: cluster leadership acquired +30:43.548 [INFO] TestAgent.server: New leader elected: payload=Node-bc22ac75-543b-24c9-3631-2e53fb9eca61 +30:43.573 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30259 +30:43.616 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:43.659 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:43.659 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:43.659 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:43.659 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:43.659 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:43.659 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc22ac75-543b-24c9-3631-2e53fb9eca61 +30:43.659 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc22ac75-543b-24c9-3631-2e53fb9eca61 +30:43.684 [INFO] TestAgent.server: federation state anti-entropy synced +30:43.856 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=171.016µs client=127.0.0.1:44725 client_network=udp +30:43.857 [INFO] TestAgent: Requesting shutdown +30:43.857 [INFO] TestAgent.server: shutting down server +30:43.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:43.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:43.857 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:43.857 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:43.857 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:43.857 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:43.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:43.857 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:43.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:43.857 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:43.857 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:43.872 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:43.872Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:43.872Z [INFO] agent.router.manager: shutting down +30:43.872 [INFO] TestAgent: consul server down +30:43.872 [INFO] TestAgent: shutdown complete +30:43.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30254 network=tcp +30:43.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30254 network=udp +30:43.872 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30255 network=tcp +30:44.377 [INFO] TestAgent: Waiting for endpoints to shut down +30:44.377 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_ANY_DontSuppressTXT (1.13s) === RUN TestDNS_NodeLookup_A_SuppressTXT [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:37.902Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:37.910Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:37.934 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8c5f277-5169-8e9f-9b16-f23818479d71 Address:127.0.0.1:11888}]" -45:37.935 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8c5f277-5169-8e9f-9b16-f23818479d71.dc1 127.0.0.1 -45:37.935 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8c5f277-5169-8e9f-9b16-f23818479d71 127.0.0.1 -2022-11-07T22:45:37.935Z [INFO] agent.router: Initializing LAN area manager -45:37.935 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11883 network=udp -45:37.935 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11888 [Follower]" leader= -45:37.936 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8c5f277-5169-8e9f-9b16-f23818479d71.dc1 area=wan -45:37.936 [INFO] TestAgent.server: Adding LAN server: server="Node-f8c5f277-5169-8e9f-9b16-f23818479d71 (Addr: tcp/127.0.0.1:11888) (DC: dc1)" -45:37.936 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11883 network=tcp -45:37.936 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11884 network=tcp -45:37.936 [INFO] TestAgent: started state syncer -45:37.936 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11889 network=tcp -45:37.986 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:37.986 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11888 [Candidate]" term=2 -45:37.994 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:37.994 [DEBUG] TestAgent.server.raft: vote granted: from=f8c5f277-5169-8e9f-9b16-f23818479d71 term=2 tally=1 -45:37.994 [INFO] TestAgent.server.raft: election won: tally=1 -45:37.994 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11888 [Leader]" -45:37.994 [INFO] TestAgent.server: cluster leadership acquired -45:37.994 [INFO] TestAgent.server: New leader elected: payload=Node-f8c5f277-5169-8e9f-9b16-f23818479d71 -45:37.999 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11888 -45:38.011 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:38.028 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:38.028 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:38.028 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:38.028 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:38.028 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:38.028 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8c5f277-5169-8e9f-9b16-f23818479d71 -45:38.028 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8c5f277-5169-8e9f-9b16-f23818479d71 -45:38.037 [INFO] TestAgent.server: federation state anti-entropy synced -45:38.166 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=A class=IN latency=82.509µs client=127.0.0.1:38980 client_network=udp -45:38.166 [INFO] TestAgent: Requesting shutdown -45:38.166 [INFO] TestAgent.server: shutting down server -45:38.166 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:38.166 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:38.166 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:38.166 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:38.166 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:38.166 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:38.166 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:38.166 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:38.166 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:38.166 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:38.166 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:38.169 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:38.173Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:38.173Z [INFO] agent.router.manager: shutting down -45:38.173 [INFO] TestAgent: consul server down -45:38.173 [INFO] TestAgent: shutdown complete -45:38.173 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11883 network=tcp -45:38.173 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11883 network=udp -45:38.173 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11884 network=tcp -45:38.673 [INFO] TestAgent: Waiting for endpoints to shut down -45:38.673 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_A_SuppressTXT (0.79s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:44.412Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:44.434Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:44.528 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f6119892-7e09-93f0-b0ce-201dbaa751ff Address:127.0.0.1:30266}]" +30:44.529 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f6119892-7e09-93f0-b0ce-201dbaa751ff.dc1 127.0.0.1 +30:44.529 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f6119892-7e09-93f0-b0ce-201dbaa751ff 127.0.0.1 +2021-10-05T16:30:44.529Z [INFO] agent.router: Initializing LAN area manager +30:44.529 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30261 network=udp +30:44.530 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30266 [Follower]" leader= +30:44.530 [INFO] TestAgent.server: Adding LAN server: server="Node-f6119892-7e09-93f0-b0ce-201dbaa751ff (Addr: tcp/127.0.0.1:30266) (DC: dc1)" +30:44.530 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f6119892-7e09-93f0-b0ce-201dbaa751ff.dc1 area=wan +30:44.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30261 network=tcp +30:44.531 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30262 network=tcp +30:44.531 [INFO] TestAgent: started state syncer +30:44.531 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30267 network=tcp +30:44.590 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:44.590 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30266 [Candidate]" term=2 +30:44.600 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:44.600 [DEBUG] TestAgent.server.raft: vote granted: from=f6119892-7e09-93f0-b0ce-201dbaa751ff term=2 tally=1 +30:44.600 [INFO] TestAgent.server.raft: election won: tally=1 +30:44.600 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30266 [Leader]" +30:44.600 [INFO] TestAgent.server: cluster leadership acquired +30:44.600 [INFO] TestAgent.server: New leader elected: payload=Node-f6119892-7e09-93f0-b0ce-201dbaa751ff +30:44.608 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30266 +30:44.626 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:44.704 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:44.704 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:44.704 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:44.704 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:44.704 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:44.704 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f6119892-7e09-93f0-b0ce-201dbaa751ff +30:44.704 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f6119892-7e09-93f0-b0ce-201dbaa751ff +30:44.708 [INFO] TestAgent: Synced node info +30:44.708 [DEBUG] TestAgent: Node info in sync +30:44.736 [INFO] TestAgent.server: federation state anti-entropy synced +30:44.808 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=A class=IN latency=137.388µs client=127.0.0.1:46242 client_network=udp +30:44.808 [INFO] TestAgent: Requesting shutdown +30:44.808 [INFO] TestAgent.server: shutting down server +30:44.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:44.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:44.808 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:44.808 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:44.808 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:44.808 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:44.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:44.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:44.808 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:44.808 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:44.816 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:44.820Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:44.820Z [INFO] agent.router.manager: shutting down +30:44.820 [INFO] TestAgent: consul server down +30:44.820 [INFO] TestAgent: shutdown complete +30:44.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30261 network=tcp +30:44.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30261 network=udp +30:44.820 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30262 network=tcp +30:45.320 [INFO] TestAgent: Waiting for endpoints to shut down +30:45.320 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_A_SuppressTXT (0.94s) === RUN TestDNS_EDNS0 === PAUSE TestDNS_EDNS0 === RUN TestDNS_EDNS0_ECS @@ -14323,127 +14478,128 @@ [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:38.684Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:38.694Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:38.716 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c7f2d6c-4271-1734-a9b6-0135f9b13184 Address:127.0.0.1:11895}]" -45:38.716 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11895 [Follower]" leader= -45:38.716 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184.dc1 127.0.0.1 -45:38.717 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184 127.0.0.1 -2022-11-07T22:45:38.717Z [INFO] agent.router: Initializing LAN area manager -45:38.717 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184.dc1 area=wan -45:38.717 [INFO] TestAgent.server: Adding LAN server: server="Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184 (Addr: tcp/127.0.0.1:11895) (DC: dc1)" -45:38.717 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11890 network=tcp -45:38.717 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11890 network=udp -45:38.718 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11891 network=tcp -45:38.718 [INFO] TestAgent: started state syncer -45:38.718 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11896 network=tcp -45:38.780 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:38.780 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11895 [Candidate]" term=2 -45:38.788 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:38.788 [DEBUG] TestAgent.server.raft: vote granted: from=0c7f2d6c-4271-1734-a9b6-0135f9b13184 term=2 tally=1 -45:38.788 [INFO] TestAgent.server.raft: election won: tally=1 -45:38.788 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11895 [Leader]" -45:38.788 [INFO] TestAgent.server: cluster leadership acquired -45:38.788 [INFO] TestAgent.server: New leader elected: payload=Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184 -45:38.793 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11895 -45:38.803 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:38.821 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:38.821 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:38.821 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:38.821 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:38.821 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:38.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184 -45:38.821 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c7f2d6c-4271-1734-a9b6-0135f9b13184 -45:38.830 [INFO] TestAgent.server: federation state anti-entropy synced -45:38.930 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:38.933 [INFO] TestAgent: Synced node info -45:38.933 [DEBUG] TestAgent: Node info in sync -45:38.953 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=192.891µs client=127.0.0.1:59969 client_network=udp -45:38.953 [INFO] TestAgent: Requesting shutdown -45:38.953 [INFO] TestAgent.server: shutting down server -45:38.953 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:38.953 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:38.953 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:38.953 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:38.953 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:38.953 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:38.953 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:38.953 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:38.953 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:38.953 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:38.956 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:38.959Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:38.959Z [INFO] agent.router.manager: shutting down -45:38.959 [INFO] TestAgent: consul server down -45:38.959 [INFO] TestAgent: shutdown complete -45:38.959 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11890 network=tcp -45:38.959 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11890 network=udp -45:38.959 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11891 network=tcp -45:39.459 [INFO] TestAgent: Waiting for endpoints to shut down -45:39.459 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_MetaTXT (0.79s) +2021-10-05T16:30:45.343Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:45.370Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:45.460 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:86a57538-e360-1dc7-5859-39201a841974 Address:127.0.0.1:30273}]" +30:45.460 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30273 [Follower]" leader= +30:45.461 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-86a57538-e360-1dc7-5859-39201a841974.dc1 127.0.0.1 +30:45.461 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-86a57538-e360-1dc7-5859-39201a841974 127.0.0.1 +2021-10-05T16:30:45.461Z [INFO] agent.router: Initializing LAN area manager +30:45.462 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30268 network=udp +30:45.462 [INFO] TestAgent.server: Adding LAN server: server="Node-86a57538-e360-1dc7-5859-39201a841974 (Addr: tcp/127.0.0.1:30273) (DC: dc1)" +30:45.462 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-86a57538-e360-1dc7-5859-39201a841974.dc1 area=wan +30:45.462 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30268 network=tcp +30:45.462 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30269 network=tcp +30:45.463 [INFO] TestAgent: started state syncer +30:45.463 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30274 network=tcp +30:45.506 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:45.506 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30273 [Candidate]" term=2 +30:45.532 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:45.532 [DEBUG] TestAgent.server.raft: vote granted: from=86a57538-e360-1dc7-5859-39201a841974 term=2 tally=1 +30:45.532 [INFO] TestAgent.server.raft: election won: tally=1 +30:45.532 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30273 [Leader]" +30:45.536 [INFO] TestAgent.server: cluster leadership acquired +30:45.536 [INFO] TestAgent.server: New leader elected: payload=Node-86a57538-e360-1dc7-5859-39201a841974 +30:45.561 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30273 +30:45.604 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:45.616 [INFO] TestAgent: Synced node info +30:45.634 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=304.744µs client=127.0.0.1:58754 client_network=udp +30:45.634 [INFO] TestAgent: Requesting shutdown +30:45.634 [INFO] TestAgent.server: shutting down server +30:45.634 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:45.645 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:45.660Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:45.660Z [INFO] agent.router.manager: shutting down +30:45.660 [INFO] TestAgent: consul server down +30:45.660 [INFO] TestAgent: shutdown complete +30:45.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30268 network=tcp +30:45.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30268 network=udp +30:45.660 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30269 network=tcp +30:45.660 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:45.660 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:45.660 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:45.660 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:45.660 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:45.660 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-86a57538-e360-1dc7-5859-39201a841974 +30:45.660 [INFO] TestAgent.server: member joined, marking health alive: member=Node-86a57538-e360-1dc7-5859-39201a841974 +30:45.661 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +30:45.661 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +30:45.661 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +30:45.661 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +30:45.661 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +30:45.661 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-86a57538-e360-1dc7-5859-39201a841974 127.0.0.1 30271 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:86a57538-e360-1dc7-5859-39201a841974 port:30273 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30272] alive 1 5 2 2 5 4}" error="raft is already shutdown" +30:45.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:45.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:45.661 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:45.661 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:45.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:45.661 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:45.661 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:45.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:46.162 [INFO] TestAgent: Waiting for endpoints to shut down +30:46.162 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_MetaTXT (0.84s) === RUN TestDNS_ServiceLookup_SuppressTXT [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:39.470Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:39.480Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:39.500 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8cdb1a2d-4566-466d-4880-597909e564ce Address:127.0.0.1:11902}]" -45:39.500 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11902 [Follower]" leader= -45:39.500 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8cdb1a2d-4566-466d-4880-597909e564ce.dc1 127.0.0.1 -45:39.501 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8cdb1a2d-4566-466d-4880-597909e564ce 127.0.0.1 -2022-11-07T22:45:39.501Z [INFO] agent.router: Initializing LAN area manager -45:39.501 [INFO] TestAgent.server: Adding LAN server: server="Node-8cdb1a2d-4566-466d-4880-597909e564ce (Addr: tcp/127.0.0.1:11902) (DC: dc1)" -45:39.501 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8cdb1a2d-4566-466d-4880-597909e564ce.dc1 area=wan -45:39.501 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11897 network=udp -45:39.501 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11897 network=tcp -45:39.501 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11898 network=tcp -45:39.501 [INFO] TestAgent: started state syncer -45:39.501 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11903 network=tcp -45:39.550 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:39.550 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11902 [Candidate]" term=2 -45:39.558 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:39.558 [DEBUG] TestAgent.server.raft: vote granted: from=8cdb1a2d-4566-466d-4880-597909e564ce term=2 tally=1 -45:39.558 [INFO] TestAgent.server.raft: election won: tally=1 -45:39.558 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11902 [Leader]" -45:39.558 [INFO] TestAgent.server: cluster leadership acquired -45:39.558 [INFO] TestAgent.server: New leader elected: payload=Node-8cdb1a2d-4566-466d-4880-597909e564ce -45:39.563 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11902 -45:39.574 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:39.591 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:39.591 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:39.591 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:39.591 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:39.591 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:39.591 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8cdb1a2d-4566-466d-4880-597909e564ce -45:39.591 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8cdb1a2d-4566-466d-4880-597909e564ce -45:39.596 [INFO] TestAgent.server: federation state anti-entropy synced -45:39.855 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=127.138µs client=127.0.0.1:39256 client_network=udp -45:39.855 [INFO] TestAgent: Requesting shutdown -45:39.855 [INFO] TestAgent.server: shutting down server -45:39.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:39.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:39.855 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:39.855 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:39.855 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:39.855 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:39.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:39.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:39.855 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:39.855 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:39.855 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:39.858 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:39.862Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:39.862Z [INFO] agent.router.manager: shutting down -45:39.862 [INFO] TestAgent: consul server down -45:39.862 [INFO] TestAgent: shutdown complete -45:39.862 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11897 network=tcp -45:39.862 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11897 network=udp -45:39.862 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11898 network=tcp -45:40.362 [INFO] TestAgent: Waiting for endpoints to shut down -45:40.362 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_SuppressTXT (0.90s) +2021-10-05T16:30:46.177Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:46.202Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:46.232 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 Address:127.0.0.1:30280}]" +30:46.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4.dc1 127.0.0.1 +30:46.233 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 127.0.0.1 +2021-10-05T16:30:46.233Z [INFO] agent.router: Initializing LAN area manager +30:46.233 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30275 network=udp +30:46.233 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30280 [Follower]" leader= +30:46.234 [INFO] TestAgent.server: Adding LAN server: server="Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 (Addr: tcp/127.0.0.1:30280) (DC: dc1)" +30:46.234 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4.dc1 area=wan +30:46.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30275 network=tcp +30:46.234 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30276 network=tcp +30:46.234 [INFO] TestAgent: started state syncer +30:46.235 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30281 network=tcp +30:46.277 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:46.277 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30280 [Candidate]" term=2 +30:46.287 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:46.287 [DEBUG] TestAgent.server.raft: vote granted: from=3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 term=2 tally=1 +30:46.287 [INFO] TestAgent.server.raft: election won: tally=1 +30:46.287 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30280 [Leader]" +30:46.287 [INFO] TestAgent.server: cluster leadership acquired +30:46.287 [INFO] TestAgent.server: New leader elected: payload=Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 +30:46.304 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30280 +30:46.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:46.389 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:46.389 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:46.389 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:46.389 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:46.389 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:46.389 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 +30:46.389 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ad32ff5-c1cc-6b87-dba2-47246a0e9fb4 +30:46.448 [INFO] TestAgent.server: federation state anti-entropy synced +30:46.449 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=183.997µs client=127.0.0.1:58268 client_network=udp +30:46.449 [INFO] TestAgent: Requesting shutdown +30:46.449 [INFO] TestAgent.server: shutting down server +30:46.449 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:46.449 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:46.449 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:46.449 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:46.449 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:46.449 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:46.449 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:46.449 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:46.449 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:46.449 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:46.449 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:46.453 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:46.457Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:46.457Z [INFO] agent.router.manager: shutting down +30:46.457 [INFO] TestAgent: consul server down +30:46.457 [INFO] TestAgent: shutdown complete +30:46.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30275 network=tcp +30:46.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30275 network=udp +30:46.457 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30276 network=tcp +30:46.958 [INFO] TestAgent: Waiting for endpoints to shut down +30:46.958 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_SuppressTXT (0.80s) === RUN TestDNS_AddressLookup === PAUSE TestDNS_AddressLookup === RUN TestDNS_AddressLookupIPV6 @@ -14800,7 +14956,7 @@ === RUN TestDecodeAgentRegisterService/serviceID:_first_set === RUN TestDecodeAgentRegisterService/serviceID:_second_set === RUN TestDecodeAgentRegisterService/serviceID:_neither_set ---- PASS: TestDecodeAgentRegisterService (0.01s) +--- PASS: TestDecodeAgentRegisterService (0.02s) --- PASS: TestDecodeAgentRegisterService/translateEnableTagTCs:_both_set (0.00s) --- PASS: TestDecodeAgentRegisterService/translateEnableTagTCs:_first_set (0.00s) --- PASS: TestDecodeAgentRegisterService/translateEnableTagTCs:_second_set (0.00s) @@ -15006,12 +15162,7 @@ --- PASS: TestDecodeTxnConvertOps/durations_negative (0.00s) --- PASS: TestDecodeTxnConvertOps/durations_numeric_and_negative (0.00s) === RUN TestHTTPAPI_MethodNotAllowed_OSS -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query/ @@ -15030,1949 +15181,1948 @@ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query/xxx/explain === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query/xxx/explain === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query/xxx/explain -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/create -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/update/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/leaf/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/leaf/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/leaf/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/leaf/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/leaf/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policies -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/ingress/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/kv/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/destroy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/leave +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/leave +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/leave +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/leave +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/leave +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/leave +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/warn/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/deregister/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/deregister/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/deregister/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/deregister/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/deregister/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rules -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rules -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rules -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rules -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rules -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/self +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/tokens +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/tokens +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/tokens +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/tokens +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/tokens +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/tokens === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/checks === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/checks === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/checks === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/checks === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/checks === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/checks +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/name/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/name/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/name/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/name/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/name/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/info/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/update/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/update/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/update/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/update/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/update/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/update/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/id/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/datacenters === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/info/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/info/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/info/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/info/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/info/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/roles +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/join/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/fire/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/checks/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/host +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/node/ +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/connect/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/kv/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/logout +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policies +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/roots +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/ingress/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rules +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/list +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/snapshot +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/metrics +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/services +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/keyring +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/update === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/clone/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/clone/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/clone/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/clone/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/clone/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/clone/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/datacenters -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/leave -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/replication -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/replication -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/replication -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/replication -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/replication -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/token/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/token/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/token/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/token/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/token/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/maintenance -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/update +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/service/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/roots === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/roots === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/roots === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/roots === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/roots === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/roots -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/checks/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/services -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/match -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/check -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/check -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/check -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/check -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/check -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/check === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/renew/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/bootstrap -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/bootstrap -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/bootstrap -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/bootstrap -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/bootstrap -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/join/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/service/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/connect/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/gateway-services/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/configuration === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/configuration === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/configuration === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/configuration === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/configuration === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/host -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/metrics -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/service/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/destroy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/renew/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/bootstrap +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/node/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/node/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/status/leader === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/status/leader === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/status/leader === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/status/leader === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/status/leader === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/status/leader -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/login -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/token/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/register -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/update -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/members === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/members === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/members === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/members === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/members === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/members -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/deregister -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/nodes === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/nodes === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/nodes === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/nodes === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/nodes === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/nodes -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/keyring -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/logout -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/fire/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/snapshot -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/txn -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/roles -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/discovery-chain/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/list === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/list -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-methods -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-methods -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-methods -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-methods -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-methods -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-methods +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/login +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/nodes +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/replication +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/name/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/deregister +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token/self === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token/self === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token/self === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token/self === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token/self === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token/self -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/create +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/check +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/discovery-chain/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/discovery-chain/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/discovery-chain/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/discovery-chain/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/discovery-chain/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/discovery-chain/ === RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/peer === RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/peer === RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/peer === RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/peer === RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/peer === RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/peer -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/info/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/tokens -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/txn +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/txn +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/txn [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/txn +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/txn +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/txn +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/maintenance +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/register +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/match +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/match +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/match +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/match +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/match +=== RUN TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/match [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:40.388Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:40.400Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:40.420 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fa229894-f1f9-233c-3b70-2308831768f1 Address:127.0.0.1:11909}]" -45:40.420 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11909 [Follower]" leader= -45:40.421 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fa229894-f1f9-233c-3b70-2308831768f1.dc1 127.0.0.1 -45:40.421 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fa229894-f1f9-233c-3b70-2308831768f1 127.0.0.1 -2022-11-07T22:45:40.421Z [INFO] agent.router: Initializing LAN area manager -45:40.421 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fa229894-f1f9-233c-3b70-2308831768f1.dc1 area=wan -45:40.421 [INFO] TestAgent.server: Adding LAN server: server="Node-fa229894-f1f9-233c-3b70-2308831768f1 (Addr: tcp/127.0.0.1:11909) (DC: dc1)" -45:40.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11904 network=udp -45:40.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11904 network=tcp -45:40.422 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11905 network=tcp -45:40.422 [INFO] TestAgent: started state syncer -45:40.422 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11910 network=tcp -45:40.471 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:40.489 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:40.489 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11909 [Candidate]" term=2 -45:40.500 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:40.500 [DEBUG] TestAgent.server.raft: vote granted: from=fa229894-f1f9-233c-3b70-2308831768f1 term=2 tally=1 -45:40.500 [INFO] TestAgent.server.raft: election won: tally=1 -45:40.500 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11909 [Leader]" -45:40.500 [INFO] TestAgent.server: cluster leadership acquired -45:40.500 [INFO] TestAgent.server: New leader elected: payload=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.505 [INFO] TestAgent.server: initializing acls -45:40.508 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:40.508 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:40.512 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:40.520 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:40.520 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:40.520 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:40.520 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.520 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-fa229894-f1f9-233c-3b70-2308831768f1.dc1 -45:40.520 [INFO] TestAgent.server: Updating LAN server: server="Node-fa229894-f1f9-233c-3b70-2308831768f1 (Addr: tcp/127.0.0.1:11909) (DC: dc1)" -45:40.520 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-fa229894-f1f9-233c-3b70-2308831768f1.dc1 area=wan -45:40.529 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:40.543 [INFO] TestAgent: Synced node info -45:40.543 [DEBUG] TestAgent: Node info in sync -45:40.547 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.548 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:40.548 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:40.548 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:40.548 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:40.548 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:40.548 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.548 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.553 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.557 [INFO] TestAgent.server: federation state anti-entropy synced -45:40.574 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query from=127.0.0.1:59024 latency=525.3µs -45:40.574 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query from=127.0.0.1:59026 error="method PUT not allowed" -45:40.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query from=127.0.0.1:59026 latency=60.32µs -45:40.575 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query from=127.0.0.1:59028 latency=277.363µs -45:40.576 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query from=127.0.0.1:59030 error="method DELETE not allowed" -45:40.576 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query from=127.0.0.1:59030 latency=78.846µs -45:40.576 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query from=127.0.0.1:59032 error="method HEAD not allowed" -45:40.576 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query from=127.0.0.1:59032 latency=17.241µs -45:40.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query from=127.0.0.1:59032 latency=1.781µs -45:40.577 [ERROR] TestAgent.http: Request error: method=GET url=/v1/query/ from=127.0.0.1:59032 error="failed prepared query lookup: index error: UUID must be 36 characters" -45:40.577 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/ from=127.0.0.1:59032 latency=781.845µs -45:40.578 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/ from=127.0.0.1:59034 error="Prepared Query lookup failed: failed prepared query lookup: index error: UUID must be 36 characters" -45:40.578 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/ from=127.0.0.1:59034 latency=152.65µs -45:40.579 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/ from=127.0.0.1:59036 error="method POST not allowed" -45:40.579 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/ from=127.0.0.1:59036 latency=66.443µs -45:40.579 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/ from=127.0.0.1:59038 error="Prepared Query lookup failed: failed prepared query lookup: index error: UUID must be 36 characters" -45:40.579 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/ from=127.0.0.1:59038 latency=193.793µs -45:40.580 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/ from=127.0.0.1:59040 error="method HEAD not allowed" -45:40.580 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/ from=127.0.0.1:59040 latency=17.402µs -45:40.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/ from=127.0.0.1:59040 latency=123.557µs -45:40.581 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/xxx/execute from=127.0.0.1:59042 latency=104.744µs -45:40.581 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/xxx/execute from=127.0.0.1:59044 error="method PUT not allowed" -45:40.581 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/xxx/execute from=127.0.0.1:59044 latency=150.877µs -45:40.582 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/xxx/execute from=127.0.0.1:59046 error="method POST not allowed" -45:40.582 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/xxx/execute from=127.0.0.1:59046 latency=49.705µs -45:40.582 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/xxx/execute from=127.0.0.1:59048 error="method DELETE not allowed" -45:40.582 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/xxx/execute from=127.0.0.1:59048 latency=81.251µs -45:40.583 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/xxx/execute from=127.0.0.1:59050 error="method HEAD not allowed" -45:40.583 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/xxx/execute from=127.0.0.1:59050 latency=70.187µs -45:40.583 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/xxx/execute from=127.0.0.1:59050 latency=147.408µs -45:40.584 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/xxx/explain from=127.0.0.1:59052 latency=120.554µs -45:40.584 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/xxx/explain from=127.0.0.1:59054 error="method PUT not allowed" -45:40.584 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/xxx/explain from=127.0.0.1:59054 latency=58.562µs -45:40.585 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/xxx/explain from=127.0.0.1:59056 error="method POST not allowed" -45:40.585 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/xxx/explain from=127.0.0.1:59056 latency=60.773µs -45:40.585 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/xxx/explain from=127.0.0.1:59058 error="method DELETE not allowed" -45:40.585 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/xxx/explain from=127.0.0.1:59058 latency=67.147µs -45:40.586 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/xxx/explain from=127.0.0.1:59060 error="method HEAD not allowed" -45:40.586 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/xxx/explain from=127.0.0.1:59060 latency=17.929µs -45:40.586 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/xxx/explain from=127.0.0.1:59060 latency=48.858µs -45:40.586 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/create from=127.0.0.1:59062 error="method GET not allowed" -45:40.586 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/create from=127.0.0.1:59062 latency=68.967µs -45:40.587 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/create from=127.0.0.1:59064 error="Permission denied" -45:40.587 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/create from=127.0.0.1:59064 latency=134.273µs -45:40.587 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/create from=127.0.0.1:59066 error="method POST not allowed" -45:40.587 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/create from=127.0.0.1:59066 latency=62.488µs -45:40.588 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/create from=127.0.0.1:59068 error="method DELETE not allowed" -45:40.588 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/create from=127.0.0.1:59068 latency=61.612µs -45:40.588 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/create from=127.0.0.1:59070 error="method HEAD not allowed" -45:40.588 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/create from=127.0.0.1:59070 latency=18.522µs -45:40.588 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/create from=127.0.0.1:59070 latency=1.959µs -45:40.589 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/rules/translate from=127.0.0.1:59070 error="method GET not allowed" -45:40.590 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/rules/translate from=127.0.0.1:59070 latency=1.853418ms -45:40.591 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/rules/translate from=127.0.0.1:59072 error="method PUT not allowed" -45:40.591 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/rules/translate from=127.0.0.1:59072 latency=105.097µs -45:40.591 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/rules/translate from=127.0.0.1:59074 error="Permission denied" -45:40.591 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/rules/translate from=127.0.0.1:59074 latency=98.644µs -45:40.592 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/rules/translate from=127.0.0.1:59076 error="method DELETE not allowed" -45:40.592 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/rules/translate from=127.0.0.1:59076 latency=97.261µs -45:40.592 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/rules/translate from=127.0.0.1:59078 error="method HEAD not allowed" -45:40.592 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/rules/translate from=127.0.0.1:59078 latency=19.458µs -45:40.593 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/rules/translate from=127.0.0.1:59078 latency=2.004µs -45:40.593 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/rules/translate/ from=127.0.0.1:59078 error="Bad request: Missing token ID" -45:40.593 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/rules/translate/ from=127.0.0.1:59078 latency=97.945µs -45:40.593 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/rules/translate/ from=127.0.0.1:59080 error="method PUT not allowed" -45:40.593 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/rules/translate/ from=127.0.0.1:59080 latency=64.088µs -45:40.594 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/rules/translate/ from=127.0.0.1:59082 error="method POST not allowed" -45:40.594 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/rules/translate/ from=127.0.0.1:59082 latency=103.798µs -45:40.594 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/rules/translate/ from=127.0.0.1:59084 error="method DELETE not allowed" -45:40.595 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/rules/translate/ from=127.0.0.1:59084 latency=76.68µs -45:40.595 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/rules/translate/ from=127.0.0.1:59086 error="method HEAD not allowed" -45:40.595 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/rules/translate/ from=127.0.0.1:59086 latency=18.072µs -45:40.595 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/rules/translate/ from=127.0.0.1:59086 latency=1.719µs -45:40.596 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:59086 latency=290.828µs -45:40.596 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/datacenters from=127.0.0.1:59088 error="method PUT not allowed" -45:40.596 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/datacenters from=127.0.0.1:59088 latency=62.85µs -45:40.597 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/datacenters from=127.0.0.1:59090 error="method POST not allowed" -45:40.597 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/datacenters from=127.0.0.1:59090 latency=63.081µs -45:40.597 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/datacenters from=127.0.0.1:59092 error="method DELETE not allowed" -45:40.597 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/datacenters from=127.0.0.1:59092 latency=65.397µs -45:40.598 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/datacenters from=127.0.0.1:59094 error="method HEAD not allowed" -45:40.598 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/datacenters from=127.0.0.1:59094 latency=79.446µs -45:40.599 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/datacenters from=127.0.0.1:59094 latency=1.992µs -45:40.599 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/node/ from=127.0.0.1:59094 latency=201.275µs -45:40.599 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/node/ from=127.0.0.1:59096 error="method PUT not allowed" -45:40.600 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/node/ from=127.0.0.1:59096 latency=124.476µs -45:40.600 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/node/ from=127.0.0.1:59098 error="method POST not allowed" -45:40.600 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/node/ from=127.0.0.1:59098 latency=203.136µs -45:40.601 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/node/ from=127.0.0.1:59100 error="method DELETE not allowed" -45:40.601 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/node/ from=127.0.0.1:59100 latency=112.232µs -45:40.601 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/node/ from=127.0.0.1:59102 error="method HEAD not allowed" -45:40.601 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/node/ from=127.0.0.1:59102 latency=18.91µs -45:40.601 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/node/ from=127.0.0.1:59102 latency=1.945µs -45:40.602 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/create from=127.0.0.1:59102 error="method GET not allowed" -45:40.602 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/create from=127.0.0.1:59102 latency=113.17µs -45:40.602 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/create from=127.0.0.1:59104 error="Permission denied" -45:40.602 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:59104 latency=167.407µs -45:40.603 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/create from=127.0.0.1:59106 error="method POST not allowed" -45:40.603 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/create from=127.0.0.1:59106 latency=112.167µs -45:40.603 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/create from=127.0.0.1:59108 error="method DELETE not allowed" -45:40.603 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/create from=127.0.0.1:59108 latency=65.783µs -45:40.604 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/create from=127.0.0.1:59110 error="method HEAD not allowed" -45:40.604 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/create from=127.0.0.1:59110 latency=18.613µs -45:40.604 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/create from=127.0.0.1:59110 latency=1.927µs -45:40.606 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.606 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/services from=127.0.0.1:59110 latency=326.504µs -45:40.607 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/services from=127.0.0.1:59112 error="method PUT not allowed" -45:40.607 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/services from=127.0.0.1:59112 latency=65.915µs -45:40.607 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/services from=127.0.0.1:59114 error="method POST not allowed" -45:40.607 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/services from=127.0.0.1:59114 latency=65.414µs -45:40.608 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/services from=127.0.0.1:59116 error="method DELETE not allowed" -45:40.608 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/services from=127.0.0.1:59116 latency=111.4µs -45:40.609 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/services from=127.0.0.1:59118 error="method HEAD not allowed" -45:40.609 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/services from=127.0.0.1:59118 latency=16.526µs -45:40.609 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/services from=127.0.0.1:59118 latency=1.833µs -45:40.609 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/update/ from=127.0.0.1:59118 error="method GET not allowed" -45:40.609 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/update/ from=127.0.0.1:59118 latency=61.619µs -45:40.610 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/update/ from=127.0.0.1:59120 latency=803.298µs -45:40.611 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/update/ from=127.0.0.1:59122 error="method POST not allowed" -45:40.611 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/update/ from=127.0.0.1:59122 latency=66.485µs -45:40.611 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/update/ from=127.0.0.1:59124 error="method DELETE not allowed" -45:40.612 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/update/ from=127.0.0.1:59124 latency=106.457µs -45:40.612 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/update/ from=127.0.0.1:59126 error="method HEAD not allowed" -45:40.612 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/update/ from=127.0.0.1:59126 latency=13.893µs -45:40.612 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/update/ from=127.0.0.1:59126 latency=1.817µs -45:40.613 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59126 error="URI must be either service or agent" -45:40.613 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59126 latency=295.435µs -45:40.613 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59128 error="method PUT not allowed" -45:40.613 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59128 latency=60.863µs -45:40.614 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59130 error="method POST not allowed" -45:40.614 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59130 latency=99.966µs -45:40.614 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59132 error="method DELETE not allowed" -45:40.614 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59132 latency=86.466µs -45:40.615 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59134 error="method HEAD not allowed" -45:40.615 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59134 latency=16.508µs -45:40.615 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:59134 latency=1.96µs -45:40.615 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node-services/ from=127.0.0.1:59134 latency=62.722µs -45:40.616 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/node-services/ from=127.0.0.1:59136 error="method PUT not allowed" -45:40.616 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/node-services/ from=127.0.0.1:59136 latency=81.568µs -45:40.616 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/node-services/ from=127.0.0.1:59138 error="method POST not allowed" -45:40.616 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/node-services/ from=127.0.0.1:59138 latency=59.73µs -45:40.617 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/node-services/ from=127.0.0.1:59140 error="method DELETE not allowed" -45:40.617 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/node-services/ from=127.0.0.1:59140 latency=80.561µs -45:40.617 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/node-services/ from=127.0.0.1:59142 error="method HEAD not allowed" -45:40.617 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/node-services/ from=127.0.0.1:59142 latency=18.309µs -45:40.617 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/node-services/ from=127.0.0.1:59142 latency=1.768µs -45:40.618 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/node/ from=127.0.0.1:59142 latency=53.689µs -45:40.618 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/node/ from=127.0.0.1:59144 error="method PUT not allowed" -45:40.618 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/node/ from=127.0.0.1:59144 latency=60.899µs -45:40.618 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/node/ from=127.0.0.1:59146 error="method POST not allowed" -45:40.619 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/node/ from=127.0.0.1:59146 latency=59.315µs -45:40.619 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/node/ from=127.0.0.1:59148 error="method DELETE not allowed" -45:40.619 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/node/ from=127.0.0.1:59148 latency=59.282µs -45:40.619 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/node/ from=127.0.0.1:59150 error="method HEAD not allowed" -45:40.619 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/node/ from=127.0.0.1:59150 latency=16.598µs -45:40.620 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/node/ from=127.0.0.1:59150 latency=1.954µs -45:40.620 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policies from=127.0.0.1:59150 error="Permission denied" -45:40.620 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:59150 latency=184.906µs -45:40.620 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policies from=127.0.0.1:59152 error="method PUT not allowed" -45:40.621 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policies from=127.0.0.1:59152 latency=92.384µs -45:40.621 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policies from=127.0.0.1:59154 error="method POST not allowed" -45:40.621 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policies from=127.0.0.1:59154 latency=123.47µs -45:40.621 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policies from=127.0.0.1:59156 error="method DELETE not allowed" -45:40.621 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policies from=127.0.0.1:59156 latency=60.366µs -45:40.622 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policies from=127.0.0.1:59158 error="method HEAD not allowed" -45:40.622 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policies from=127.0.0.1:59158 latency=21.656µs -45:40.622 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policies from=127.0.0.1:59158 latency=1.804µs -45:40.623 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:59158 latency=221.145µs -45:40.623 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions from=127.0.0.1:59160 error="method PUT not allowed" -45:40.623 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions from=127.0.0.1:59160 latency=62.666µs -45:40.624 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions from=127.0.0.1:59162 error="Failed to decode request body: EOF" -45:40.624 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:59162 latency=71.585µs -45:40.624 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions from=127.0.0.1:59164 error="method DELETE not allowed" -45:40.624 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions from=127.0.0.1:59164 latency=93.348µs -45:40.625 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions from=127.0.0.1:59166 error="method HEAD not allowed" -45:40.625 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions from=127.0.0.1:59166 latency=16.759µs -45:40.625 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions from=127.0.0.1:59166 latency=1.922µs -45:40.626 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/ingress/ from=127.0.0.1:59166 latency=69.879µs -45:40.626 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/ingress/ from=127.0.0.1:59168 error="method PUT not allowed" -45:40.626 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/ingress/ from=127.0.0.1:59168 latency=70.328µs -45:40.627 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/ingress/ from=127.0.0.1:59170 error="method POST not allowed" -45:40.627 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/ingress/ from=127.0.0.1:59170 latency=89.035µs -45:40.627 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/ingress/ from=127.0.0.1:59172 error="method DELETE not allowed" -45:40.627 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/ingress/ from=127.0.0.1:59172 latency=76.463µs -45:40.628 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/ingress/ from=127.0.0.1:59174 error="method HEAD not allowed" -45:40.628 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/ingress/ from=127.0.0.1:59174 latency=18.086µs -45:40.628 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/ingress/ from=127.0.0.1:59174 latency=1.642µs -45:40.628 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/destroy/ from=127.0.0.1:59174 error="method GET not allowed" -45:40.628 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/destroy/ from=127.0.0.1:59174 latency=57.311µs -45:40.629 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/destroy/ from=127.0.0.1:59176 latency=71.993µs -45:40.629 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/destroy/ from=127.0.0.1:59178 error="method POST not allowed" -45:40.629 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/destroy/ from=127.0.0.1:59178 latency=61.154µs -45:40.630 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/destroy/ from=127.0.0.1:59180 error="method DELETE not allowed" -45:40.630 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/destroy/ from=127.0.0.1:59180 latency=61.512µs -45:40.630 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/destroy/ from=127.0.0.1:59182 error="method HEAD not allowed" -45:40.630 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/destroy/ from=127.0.0.1:59182 latency=15.845µs -45:40.630 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/destroy/ from=127.0.0.1:59182 latency=1.683µs -45:40.631 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/roots from=127.0.0.1:59182 latency=309.514µs -45:40.631 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/ca/roots from=127.0.0.1:59184 error="method PUT not allowed" -45:40.631 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/roots from=127.0.0.1:59184 latency=85.14µs -45:40.632 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/ca/roots from=127.0.0.1:59186 error="method POST not allowed" -45:40.632 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/ca/roots from=127.0.0.1:59186 latency=61.41µs -45:40.632 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/ca/roots from=127.0.0.1:59188 error="method DELETE not allowed" -45:40.632 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/ca/roots from=127.0.0.1:59188 latency=107.684µs -45:40.633 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/ca/roots from=127.0.0.1:59190 error="method HEAD not allowed" -45:40.633 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/ca/roots from=127.0.0.1:59190 latency=18.926µs -45:40.633 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/ca/roots from=127.0.0.1:59190 latency=1.88µs -45:40.634 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/ from=127.0.0.1:59190 latency=64.505µs -45:40.634 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/ from=127.0.0.1:59192 latency=53.218µs -45:40.634 [ERROR] TestAgent.http: Request error: method=POST url=/v1/kv/ from=127.0.0.1:59194 error="method POST not allowed" -45:40.635 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/kv/ from=127.0.0.1:59194 latency=64.057µs -45:40.635 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/ from=127.0.0.1:59196 latency=102.805µs -45:40.635 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/kv/ from=127.0.0.1:59198 error="method HEAD not allowed" -45:40.635 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/kv/ from=127.0.0.1:59198 latency=21.78µs -45:40.636 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/kv/ from=127.0.0.1:59198 latency=2.085µs -45:40.636 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/destroy/ from=127.0.0.1:59198 error="method GET not allowed" -45:40.636 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/destroy/ from=127.0.0.1:59198 latency=67.461µs -45:40.636 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/ from=127.0.0.1:59200 latency=52.21µs -45:40.637 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/destroy/ from=127.0.0.1:59202 error="method POST not allowed" -45:40.637 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/destroy/ from=127.0.0.1:59202 latency=68.618µs -45:40.637 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/destroy/ from=127.0.0.1:59204 error="method DELETE not allowed" -45:40.637 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/destroy/ from=127.0.0.1:59204 latency=52.869µs -45:40.638 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/destroy/ from=127.0.0.1:59206 error="method HEAD not allowed" -45:40.638 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/destroy/ from=127.0.0.1:59206 latency=16.836µs -45:40.638 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/destroy/ from=127.0.0.1:59206 latency=1.554µs -45:40.638 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/deregister/ from=127.0.0.1:59206 error="method GET not allowed" -45:40.638 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/deregister/ from=127.0.0.1:59206 latency=60.249µs -45:40.638 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/service/deregister/ from=127.0.0.1:59208 error="Unknown service {"" {}}" -45:40.638 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/ from=127.0.0.1:59208 latency=140.846µs -45:40.639 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/deregister/ from=127.0.0.1:59210 error="method POST not allowed" -45:40.639 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/deregister/ from=127.0.0.1:59210 latency=61.47µs -45:40.639 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/deregister/ from=127.0.0.1:59212 error="method DELETE not allowed" -45:40.639 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/deregister/ from=127.0.0.1:59212 latency=54.432µs -45:40.640 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/deregister/ from=127.0.0.1:59214 error="method HEAD not allowed" -45:40.640 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/deregister/ from=127.0.0.1:59214 latency=17.315µs -45:40.640 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/deregister/ from=127.0.0.1:59214 latency=1.854µs -45:40.640 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/maintenance/ from=127.0.0.1:59214 error="method GET not allowed" -45:40.640 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/maintenance/ from=127.0.0.1:59214 latency=62.092µs -45:40.641 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/ from=127.0.0.1:59216 latency=49.401µs -45:40.641 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/maintenance/ from=127.0.0.1:59218 error="method POST not allowed" -45:40.641 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/maintenance/ from=127.0.0.1:59218 latency=64.371µs -45:40.641 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/maintenance/ from=127.0.0.1:59220 error="method DELETE not allowed" -45:40.642 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/maintenance/ from=127.0.0.1:59220 latency=66.045µs -45:40.642 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/maintenance/ from=127.0.0.1:59222 error="method HEAD not allowed" -45:40.642 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/maintenance/ from=127.0.0.1:59222 latency=17.809µs -45:40.642 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/maintenance/ from=127.0.0.1:59222 latency=2.118µs -45:40.643 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/ from=127.0.0.1:59222 latency=59.161µs -45:40.643 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/service/ from=127.0.0.1:59224 error="method PUT not allowed" -45:40.643 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/service/ from=127.0.0.1:59224 latency=67.97µs -45:40.643 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/service/ from=127.0.0.1:59226 error="method POST not allowed" -45:40.643 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/service/ from=127.0.0.1:59226 latency=61.509µs -45:40.644 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/service/ from=127.0.0.1:59228 error="method DELETE not allowed" -45:40.644 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/service/ from=127.0.0.1:59228 latency=65.697µs -45:40.644 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/service/ from=127.0.0.1:59230 error="method HEAD not allowed" -45:40.644 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/service/ from=127.0.0.1:59230 latency=17.899µs -45:40.649 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/service/ from=127.0.0.1:59230 latency=2.603µs -45:40.650 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/update from=127.0.0.1:59230 error="method GET not allowed" -45:40.650 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/update from=127.0.0.1:59230 latency=169.386µs -45:40.651 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/update from=127.0.0.1:59232 latency=106.405µs -45:40.652 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/update from=127.0.0.1:59234 error="method POST not allowed" -45:40.652 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/update from=127.0.0.1:59234 latency=84.55µs -45:40.652 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/update from=127.0.0.1:59236 error="method DELETE not allowed" -45:40.653 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/update from=127.0.0.1:59236 latency=311.575µs -45:40.653 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/update from=127.0.0.1:59238 error="method HEAD not allowed" -45:40.653 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/update from=127.0.0.1:59238 latency=48.758µs -45:40.654 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/update from=127.0.0.1:59238 latency=2.453µs -45:40.654 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rules from=127.0.0.1:59238 error="Permission denied" -45:40.654 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:59238 latency=104.987µs -45:40.654 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rules from=127.0.0.1:59240 error="method PUT not allowed" -45:40.654 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rules from=127.0.0.1:59240 latency=65.651µs -45:40.655 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rules from=127.0.0.1:59242 error="method POST not allowed" -45:40.655 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rules from=127.0.0.1:59242 latency=166.357µs -45:40.655 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rules from=127.0.0.1:59244 error="method DELETE not allowed" -45:40.655 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rules from=127.0.0.1:59244 latency=64.865µs -45:40.656 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rules from=127.0.0.1:59246 error="method HEAD not allowed" -45:40.656 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rules from=127.0.0.1:59246 latency=19.404µs -45:40.656 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rules from=127.0.0.1:59246 latency=1.733µs -45:40.656 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:59246 latency=264.171µs -45:40.657 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/checks from=127.0.0.1:59248 error="method PUT not allowed" -45:40.657 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/checks from=127.0.0.1:59248 latency=95.199µs -45:40.657 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/checks from=127.0.0.1:59250 error="method POST not allowed" -45:40.657 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/checks from=127.0.0.1:59250 latency=52.922µs -45:40.658 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/checks from=127.0.0.1:59252 error="method DELETE not allowed" -45:40.658 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/checks from=127.0.0.1:59252 latency=108.998µs -45:40.658 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/checks from=127.0.0.1:59254 error="method HEAD not allowed" -45:40.658 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/checks from=127.0.0.1:59254 latency=11.922µs -45:40.659 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/checks from=127.0.0.1:59254 latency=1.745µs -45:40.659 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/health/service/name/ from=127.0.0.1:59254 error="Bad request: Missing service Name" -45:40.659 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/health/service/name/ from=127.0.0.1:59254 latency=67.868µs -45:40.659 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/health/service/name/ from=127.0.0.1:59256 error="method PUT not allowed" -45:40.659 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/health/service/name/ from=127.0.0.1:59256 latency=60.347µs -45:40.660 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/health/service/name/ from=127.0.0.1:59258 error="method POST not allowed" -45:40.660 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/health/service/name/ from=127.0.0.1:59258 latency=48.567µs -45:40.660 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/health/service/name/ from=127.0.0.1:59260 error="method DELETE not allowed" -45:40.660 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/health/service/name/ from=127.0.0.1:59260 latency=90.821µs -45:40.661 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/health/service/name/ from=127.0.0.1:59262 error="method HEAD not allowed" -45:40.661 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/health/service/name/ from=127.0.0.1:59262 latency=14.42µs -45:40.661 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/health/service/name/ from=127.0.0.1:59262 latency=1.368µs -45:40.661 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/list from=127.0.0.1:59262 error="Permission denied" -45:40.661 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/list from=127.0.0.1:59262 latency=164.477µs -45:40.661 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/list from=127.0.0.1:59264 error="method PUT not allowed" -45:40.661 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/list from=127.0.0.1:59264 latency=59.698µs -45:40.662 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/list from=127.0.0.1:59266 error="method POST not allowed" -45:40.662 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/list from=127.0.0.1:59266 latency=62.584µs -45:40.662 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/list from=127.0.0.1:59268 error="method DELETE not allowed" -45:40.662 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/list from=127.0.0.1:59268 latency=60.272µs -45:40.663 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/list from=127.0.0.1:59270 error="method HEAD not allowed" -45:40.663 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/list from=127.0.0.1:59270 latency=13.922µs -45:40.663 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/list from=127.0.0.1:59270 latency=1.615µs -45:40.663 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config from=127.0.0.1:59270 error="method GET not allowed" -45:40.663 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config from=127.0.0.1:59270 latency=61.014µs -45:40.664 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/config from=127.0.0.1:59272 error="Bad request: Request decoding failed: EOF" -45:40.664 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:59272 latency=61.148µs -45:40.664 [ERROR] TestAgent.http: Request error: method=POST url=/v1/config from=127.0.0.1:59274 error="method POST not allowed" -45:40.664 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/config from=127.0.0.1:59274 latency=50.029µs -45:40.664 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/config from=127.0.0.1:59276 error="method DELETE not allowed" -45:40.664 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config from=127.0.0.1:59276 latency=48.124µs -45:40.665 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/config from=127.0.0.1:59278 error="method HEAD not allowed" -45:40.665 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/config from=127.0.0.1:59278 latency=13.755µs -45:40.665 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/config from=127.0.0.1:59278 latency=1.853µs -45:40.665 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59278 latency=126.145µs -45:40.666 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/nodes from=127.0.0.1:59280 error="method PUT not allowed" -45:40.666 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/nodes from=127.0.0.1:59280 latency=49.873µs -45:40.666 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/nodes from=127.0.0.1:59282 error="method POST not allowed" -45:40.666 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/nodes from=127.0.0.1:59282 latency=57.611µs -45:40.666 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/nodes from=127.0.0.1:59284 error="method DELETE not allowed" -45:40.666 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/nodes from=127.0.0.1:59284 latency=51.357µs -45:40.667 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/nodes from=127.0.0.1:59286 error="method HEAD not allowed" -45:40.667 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/nodes from=127.0.0.1:59286 latency=18.395µs -45:40.667 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/nodes from=127.0.0.1:59286 latency=1.719µs -45:40.667 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/ from=127.0.0.1:59286 latency=43.347µs -45:40.668 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/info/ from=127.0.0.1:59288 error="method PUT not allowed" -45:40.668 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/info/ from=127.0.0.1:59288 latency=49.661µs -45:40.668 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/info/ from=127.0.0.1:59290 error="method POST not allowed" -45:40.668 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/info/ from=127.0.0.1:59290 latency=64.653µs -45:40.669 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/info/ from=127.0.0.1:59292 error="method DELETE not allowed" -45:40.669 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/info/ from=127.0.0.1:59292 latency=60.497µs -45:40.669 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/info/ from=127.0.0.1:59294 error="method HEAD not allowed" -45:40.669 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/info/ from=127.0.0.1:59294 latency=13.394µs -45:40.669 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/info/ from=127.0.0.1:59294 latency=1.435µs -45:40.669 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/warn/ from=127.0.0.1:59294 error="method GET not allowed" -45:40.669 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/warn/ from=127.0.0.1:59294 latency=63.706µs -45:40.670 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/warn/ from=127.0.0.1:59296 error="Unknown check """ -45:40.670 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/warn/ from=127.0.0.1:59296 latency=108.162µs -45:40.670 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/warn/ from=127.0.0.1:59298 error="method POST not allowed" -45:40.670 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/warn/ from=127.0.0.1:59298 latency=61.685µs -45:40.671 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/warn/ from=127.0.0.1:59300 error="method DELETE not allowed" -45:40.671 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/warn/ from=127.0.0.1:59300 latency=64.505µs -45:40.671 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/warn/ from=127.0.0.1:59302 error="method HEAD not allowed" -45:40.671 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/warn/ from=127.0.0.1:59302 latency=17.817µs -45:40.671 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/warn/ from=127.0.0.1:59302 latency=1.403µs -45:40.672 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/event/list from=127.0.0.1:59302 latency=197.487µs -45:40.672 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/event/list from=127.0.0.1:59304 error="method PUT not allowed" -45:40.672 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/list from=127.0.0.1:59304 latency=50.679µs -45:40.673 [ERROR] TestAgent.http: Request error: method=POST url=/v1/event/list from=127.0.0.1:59306 error="method POST not allowed" -45:40.673 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/event/list from=127.0.0.1:59306 latency=48.596µs -45:40.673 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/event/list from=127.0.0.1:59308 error="method DELETE not allowed" -45:40.673 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/event/list from=127.0.0.1:59308 latency=52.554µs -45:40.673 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/event/list from=127.0.0.1:59310 error="method HEAD not allowed" -45:40.673 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/event/list from=127.0.0.1:59310 latency=13.611µs -45:40.674 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/event/list from=127.0.0.1:59310 latency=1.192µs -45:40.674 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/clone/ from=127.0.0.1:59310 error="method GET not allowed" -45:40.674 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/clone/ from=127.0.0.1:59310 latency=46.798µs -45:40.674 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/clone/ from=127.0.0.1:59312 latency=48.999µs -45:40.674 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/clone/ from=127.0.0.1:59314 error="method POST not allowed" -45:40.675 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/clone/ from=127.0.0.1:59314 latency=50.524µs -45:40.675 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/clone/ from=127.0.0.1:59316 error="method DELETE not allowed" -45:40.675 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/clone/ from=127.0.0.1:59316 latency=51.83µs -45:40.675 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/clone/ from=127.0.0.1:59318 error="method HEAD not allowed" -45:40.675 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/clone/ from=127.0.0.1:59318 latency=18.385µs -45:40.675 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/clone/ from=127.0.0.1:59318 latency=1.256µs -45:40.676 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/datacenters from=127.0.0.1:59318 latency=115.7µs -45:40.676 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/datacenters from=127.0.0.1:59320 error="method PUT not allowed" -45:40.676 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/datacenters from=127.0.0.1:59320 latency=51.336µs -45:40.677 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/datacenters from=127.0.0.1:59322 error="method POST not allowed" -45:40.677 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/datacenters from=127.0.0.1:59322 latency=63.11µs -45:40.677 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/datacenters from=127.0.0.1:59324 error="method DELETE not allowed" -45:40.677 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/datacenters from=127.0.0.1:59324 latency=56.811µs -45:40.677 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/datacenters from=127.0.0.1:59326 error="method HEAD not allowed" -45:40.677 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/datacenters from=127.0.0.1:59326 latency=13.25µs -45:40.678 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/datacenters from=127.0.0.1:59326 latency=2.472µs -45:40.678 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/leave from=127.0.0.1:59326 error="method GET not allowed" -45:40.678 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/leave from=127.0.0.1:59326 latency=62.359µs -45:40.678 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/leave from=127.0.0.1:59328 error="Permission denied" -45:40.678 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/leave from=127.0.0.1:59328 latency=103.852µs -45:40.679 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/leave from=127.0.0.1:59330 error="method POST not allowed" -45:40.679 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/leave from=127.0.0.1:59330 latency=58.84µs -45:40.679 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/leave from=127.0.0.1:59332 error="method DELETE not allowed" -45:40.679 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/leave from=127.0.0.1:59332 latency=50.43µs -45:40.679 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/leave from=127.0.0.1:59334 error="method HEAD not allowed" -45:40.679 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/leave from=127.0.0.1:59334 latency=17.406µs -45:40.680 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/leave from=127.0.0.1:59334 latency=1.218µs -45:40.680 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/replication from=127.0.0.1:59334 latency=134.816µs -45:40.680 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/replication from=127.0.0.1:59336 error="method PUT not allowed" -45:40.680 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/replication from=127.0.0.1:59336 latency=55.489µs -45:40.681 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/replication from=127.0.0.1:59338 error="method POST not allowed" -45:40.681 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/replication from=127.0.0.1:59338 latency=51.62µs -45:40.681 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/replication from=127.0.0.1:59340 error="method DELETE not allowed" -45:40.681 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/replication from=127.0.0.1:59340 latency=60.886µs -45:40.681 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/replication from=127.0.0.1:59342 error="method HEAD not allowed" -45:40.681 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/replication from=127.0.0.1:59342 latency=12.463µs -45:40.682 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/replication from=127.0.0.1:59342 latency=1.444µs -45:40.682 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/ from=127.0.0.1:59342 error="Bad request: Missing token ID" -45:40.682 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/ from=127.0.0.1:59342 latency=51.409µs -45:40.682 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token/ from=127.0.0.1:59344 error="Bad request: Token decoding failed: EOF" -45:40.682 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/ from=127.0.0.1:59344 latency=75.973µs -45:40.683 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token/ from=127.0.0.1:59346 error="method POST not allowed" -45:40.683 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token/ from=127.0.0.1:59346 latency=56.17µs -45:40.683 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token/ from=127.0.0.1:59348 error="Bad request: Missing token ID" -45:40.683 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/ from=127.0.0.1:59348 latency=76.152µs -45:40.684 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token/ from=127.0.0.1:59350 error="method HEAD not allowed" -45:40.684 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token/ from=127.0.0.1:59350 latency=13.498µs -45:40.684 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token/ from=127.0.0.1:59350 latency=2.01µs -45:40.684 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/gateway-services/ from=127.0.0.1:59350 latency=54.041µs -45:40.684 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/gateway-services/ from=127.0.0.1:59352 error="method PUT not allowed" -45:40.684 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/gateway-services/ from=127.0.0.1:59352 latency=49.658µs -45:40.685 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/gateway-services/ from=127.0.0.1:59354 error="method POST not allowed" -45:40.685 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/gateway-services/ from=127.0.0.1:59354 latency=50.685µs -45:40.685 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/gateway-services/ from=127.0.0.1:59356 error="method DELETE not allowed" -45:40.685 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/gateway-services/ from=127.0.0.1:59356 latency=60.66µs -45:40.686 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/gateway-services/ from=127.0.0.1:59358 error="method HEAD not allowed" -45:40.686 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/gateway-services/ from=127.0.0.1:59358 latency=14µs -45:40.686 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/gateway-services/ from=127.0.0.1:59358 latency=1.902µs -45:40.686 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/maintenance from=127.0.0.1:59358 error="method GET not allowed" -45:40.686 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/maintenance from=127.0.0.1:59358 latency=60.585µs -45:40.687 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance from=127.0.0.1:59360 latency=42.324µs -45:40.687 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/maintenance from=127.0.0.1:59362 error="method POST not allowed" -45:40.687 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/maintenance from=127.0.0.1:59362 latency=51.574µs -45:40.687 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/maintenance from=127.0.0.1:59364 error="method DELETE not allowed" -45:40.687 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/maintenance from=127.0.0.1:59364 latency=58.771µs -45:40.688 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/maintenance from=127.0.0.1:59366 error="method HEAD not allowed" -45:40.688 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/maintenance from=127.0.0.1:59366 latency=18.458µs -45:40.688 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/maintenance from=127.0.0.1:59366 latency=1.907µs -45:40.688 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/ from=127.0.0.1:59366 latency=52.08µs -45:40.689 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/node/ from=127.0.0.1:59368 error="method PUT not allowed" -45:40.689 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/node/ from=127.0.0.1:59368 latency=93.268µs -45:40.689 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/node/ from=127.0.0.1:59370 error="method POST not allowed" -45:40.689 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/node/ from=127.0.0.1:59370 latency=55.616µs -45:40.690 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/node/ from=127.0.0.1:59372 error="method DELETE not allowed" -45:40.690 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/node/ from=127.0.0.1:59372 latency=47.84µs -45:40.690 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/node/ from=127.0.0.1:59374 error="method HEAD not allowed" -45:40.690 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/node/ from=127.0.0.1:59374 latency=12.081µs -45:40.690 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/node/ from=127.0.0.1:59374 latency=1.354µs -45:40.690 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:59374 error="Permission denied" -45:40.690 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:59374 latency=123.614µs -45:40.691 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/raft/configuration from=127.0.0.1:59376 error="method PUT not allowed" -45:40.691 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/raft/configuration from=127.0.0.1:59376 latency=94.041µs -45:40.691 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/raft/configuration from=127.0.0.1:59378 error="method POST not allowed" -45:40.691 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/raft/configuration from=127.0.0.1:59378 latency=58.295µs -45:40.692 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/configuration from=127.0.0.1:59380 error="method DELETE not allowed" -45:40.692 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/configuration from=127.0.0.1:59380 latency=64.045µs -45:40.692 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/raft/configuration from=127.0.0.1:59382 error="method HEAD not allowed" -45:40.692 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/raft/configuration from=127.0.0.1:59382 latency=17.739µs -45:40.692 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/raft/configuration from=127.0.0.1:59382 latency=1.755µs -45:40.692 [ERROR] TestAgent.http: Request error: method=GET url=/v1/catalog/register from=127.0.0.1:59382 error="method GET not allowed" -45:40.693 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/register from=127.0.0.1:59382 latency=110.208µs -45:40.693 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/register from=127.0.0.1:59384 latency=64.2µs -45:40.693 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/register from=127.0.0.1:59386 error="method POST not allowed" -45:40.693 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/register from=127.0.0.1:59386 latency=64.548µs -45:40.694 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/register from=127.0.0.1:59388 error="method DELETE not allowed" -45:40.694 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/register from=127.0.0.1:59388 latency=66.968µs -45:40.694 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/register from=127.0.0.1:59390 error="method HEAD not allowed" -45:40.694 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/register from=127.0.0.1:59390 latency=12.454µs -45:40.694 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/register from=127.0.0.1:59390 latency=1.178µs -45:40.695 [ERROR] TestAgent.http: Request error: method=GET url=/v1/coordinate/node/ from=127.0.0.1:59390 error="Permission denied" -45:40.695 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/node/ from=127.0.0.1:59390 latency=120.644µs -45:40.695 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/node/ from=127.0.0.1:59392 error="method PUT not allowed" -45:40.695 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/node/ from=127.0.0.1:59392 latency=51.705µs -45:40.695 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/node/ from=127.0.0.1:59394 error="method POST not allowed" -45:40.695 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/node/ from=127.0.0.1:59394 latency=49.579µs -45:40.696 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/node/ from=127.0.0.1:59396 error="method DELETE not allowed" -45:40.696 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/node/ from=127.0.0.1:59396 latency=51.484µs -45:40.696 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/node/ from=127.0.0.1:59398 error="method HEAD not allowed" -45:40.696 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/node/ from=127.0.0.1:59398 latency=18.132µs -45:40.696 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/node/ from=127.0.0.1:59398 latency=1.27µs -45:40.696 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-method/ from=127.0.0.1:59398 error="Bad request: Missing auth method name" -45:40.697 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/ from=127.0.0.1:59398 latency=49.966µs -45:40.697 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method/ from=127.0.0.1:59400 error="Bad request: AuthMethod decoding failed: EOF" -45:40.697 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/ from=127.0.0.1:59400 latency=59.688µs -45:40.697 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-method/ from=127.0.0.1:59402 error="method POST not allowed" -45:40.697 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-method/ from=127.0.0.1:59402 latency=48.62µs -45:40.698 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-method/ from=127.0.0.1:59404 error="Bad request: Missing auth method name" -45:40.698 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/ from=127.0.0.1:59404 latency=50.618µs -45:40.698 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-method/ from=127.0.0.1:59406 error="method HEAD not allowed" -45:40.698 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-method/ from=127.0.0.1:59406 latency=18.669µs -45:40.699 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-method/ from=127.0.0.1:59406 latency=1.938µs -45:40.699 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ from=127.0.0.1:59406 latency=113.337µs -45:40.699 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/service/ from=127.0.0.1:59408 error="method PUT not allowed" -45:40.699 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/ from=127.0.0.1:59408 latency=64.554µs -45:40.700 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/ from=127.0.0.1:59410 error="method POST not allowed" -45:40.700 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/ from=127.0.0.1:59410 latency=57.31µs -45:40.700 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/ from=127.0.0.1:59412 error="method DELETE not allowed" -45:40.700 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/ from=127.0.0.1:59412 latency=55.104µs -45:40.700 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/ from=127.0.0.1:59414 error="method HEAD not allowed" -45:40.700 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/ from=127.0.0.1:59414 latency=18.012µs -45:40.701 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/ from=127.0.0.1:59414 latency=1.663µs -45:40.701 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/force-leave/ from=127.0.0.1:59414 error="method GET not allowed" -45:40.701 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/force-leave/ from=127.0.0.1:59414 latency=64.508µs -45:40.701 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/force-leave/ from=127.0.0.1:59416 error="Permission denied" -45:40.701 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/ from=127.0.0.1:59416 latency=99.434µs -45:40.702 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/force-leave/ from=127.0.0.1:59418 error="method POST not allowed" -45:40.702 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/force-leave/ from=127.0.0.1:59418 latency=86.65µs -45:40.702 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/force-leave/ from=127.0.0.1:59420 error="method DELETE not allowed" -45:40.702 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/force-leave/ from=127.0.0.1:59420 latency=61.737µs -45:40.703 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/force-leave/ from=127.0.0.1:59422 error="method HEAD not allowed" -45:40.703 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/force-leave/ from=127.0.0.1:59422 latency=13.006µs -45:40.703 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/force-leave/ from=127.0.0.1:59422 latency=1.568µs -45:40.703 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/ca/roots from=127.0.0.1:59422 latency=180.467µs -45:40.704 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/ca/roots from=127.0.0.1:59424 error="method PUT not allowed" -45:40.704 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/ca/roots from=127.0.0.1:59424 latency=61.912µs -45:40.704 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/ca/roots from=127.0.0.1:59426 error="method POST not allowed" -45:40.704 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/ca/roots from=127.0.0.1:59426 latency=50.791µs -45:40.704 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/ca/roots from=127.0.0.1:59428 error="method DELETE not allowed" -45:40.704 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/ca/roots from=127.0.0.1:59428 latency=49.246µs -45:40.705 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/ca/roots from=127.0.0.1:59430 error="method HEAD not allowed" -45:40.705 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/ca/roots from=127.0.0.1:59430 latency=18.83µs -45:40.705 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/ca/roots from=127.0.0.1:59430 latency=1.872µs -45:40.705 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/connect/ from=127.0.0.1:59430 latency=58.465µs -45:40.706 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/connect/ from=127.0.0.1:59432 error="method PUT not allowed" -45:40.706 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/connect/ from=127.0.0.1:59432 latency=62.952µs -45:40.706 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/connect/ from=127.0.0.1:59434 error="method POST not allowed" -45:40.706 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/connect/ from=127.0.0.1:59434 latency=59.762µs -45:40.707 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/connect/ from=127.0.0.1:59436 error="method DELETE not allowed" -45:40.707 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/connect/ from=127.0.0.1:59436 latency=52.488µs -45:40.707 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/connect/ from=127.0.0.1:59438 error="method HEAD not allowed" -45:40.707 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/connect/ from=127.0.0.1:59438 latency=17.946µs -45:40.707 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/connect/ from=127.0.0.1:59438 latency=1.917µs -45:40.708 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/checks/ from=127.0.0.1:59438 latency=81.157µs -45:40.708 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/checks/ from=127.0.0.1:59440 error="method PUT not allowed" -45:40.708 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/checks/ from=127.0.0.1:59440 latency=51.237µs -45:40.708 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/checks/ from=127.0.0.1:59442 error="method POST not allowed" -45:40.708 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/checks/ from=127.0.0.1:59442 latency=50.855µs -45:40.709 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/checks/ from=127.0.0.1:59444 error="method DELETE not allowed" -45:40.709 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/checks/ from=127.0.0.1:59444 latency=58.901µs -45:40.709 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/checks/ from=127.0.0.1:59446 error="method HEAD not allowed" -45:40.709 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/checks/ from=127.0.0.1:59446 latency=15.097µs -45:40.709 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/checks/ from=127.0.0.1:59446 latency=1.227µs -45:40.709 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rule from=127.0.0.1:59446 error="method GET not allowed" -45:40.709 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule from=127.0.0.1:59446 latency=54.569µs -45:40.710 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59448 error="Bad request: BindingRule decoding failed: EOF" -45:40.710 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59448 latency=66.348µs -45:40.710 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rule from=127.0.0.1:59450 error="method POST not allowed" -45:40.710 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rule from=127.0.0.1:59450 latency=59.55µs -45:40.711 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rule from=127.0.0.1:59452 error="method DELETE not allowed" -45:40.711 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule from=127.0.0.1:59452 latency=81.044µs -45:40.711 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rule from=127.0.0.1:59454 error="method HEAD not allowed" -45:40.711 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rule from=127.0.0.1:59454 latency=14.872µs -45:40.711 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rule from=127.0.0.1:59454 latency=2.028µs -45:40.712 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:59454 latency=115.569µs -45:40.712 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/services from=127.0.0.1:59456 error="method PUT not allowed" -45:40.712 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/services from=127.0.0.1:59456 latency=50.839µs -45:40.712 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/services from=127.0.0.1:59458 error="method POST not allowed" -45:40.713 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/services from=127.0.0.1:59458 latency=59.213µs -45:40.713 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/services from=127.0.0.1:59460 error="method DELETE not allowed" -45:40.713 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/services from=127.0.0.1:59460 latency=66.111µs -45:40.713 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/services from=127.0.0.1:59462 error="method HEAD not allowed" -45:40.713 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/services from=127.0.0.1:59462 latency=12.647µs -45:40.714 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/services from=127.0.0.1:59462 latency=1.366µs -45:40.714 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/deregister/ from=127.0.0.1:59462 error="method GET not allowed" -45:40.714 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/deregister/ from=127.0.0.1:59462 latency=78.573µs -45:40.714 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/deregister/ from=127.0.0.1:59464 error="Unknown check """ -45:40.714 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/deregister/ from=127.0.0.1:59464 latency=103.543µs -45:40.715 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/deregister/ from=127.0.0.1:59466 error="method POST not allowed" -45:40.715 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/deregister/ from=127.0.0.1:59466 latency=55.28µs -45:40.715 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/deregister/ from=127.0.0.1:59468 error="method DELETE not allowed" -45:40.715 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/deregister/ from=127.0.0.1:59468 latency=49.13µs -45:40.715 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/deregister/ from=127.0.0.1:59470 error="method HEAD not allowed" -45:40.715 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/deregister/ from=127.0.0.1:59470 latency=13.768µs -45:40.715 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/deregister/ from=127.0.0.1:59470 latency=1.301µs -45:40.716 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service=consul -45:40.716 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:59470 latency=131.448µs -45:40.716 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/services from=127.0.0.1:59472 error="method PUT not allowed" -45:40.716 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/services from=127.0.0.1:59472 latency=48.99µs -45:40.716 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/services from=127.0.0.1:59474 error="method POST not allowed" -45:40.717 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/services from=127.0.0.1:59474 latency=48.724µs -45:40.717 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/services from=127.0.0.1:59476 error="method DELETE not allowed" -45:40.717 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/services from=127.0.0.1:59476 latency=64.003µs -45:40.717 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/services from=127.0.0.1:59478 error="method HEAD not allowed" -45:40.717 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/services from=127.0.0.1:59478 latency=20.231µs -45:40.718 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/services from=127.0.0.1:59478 latency=1.729µs -45:40.718 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/match from=127.0.0.1:59478 error="required query parameter 'by' not set" -45:40.718 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match from=127.0.0.1:59478 latency=81.394µs -45:40.718 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/match from=127.0.0.1:59480 error="method PUT not allowed" -45:40.718 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/match from=127.0.0.1:59480 latency=58.738µs -45:40.719 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/match from=127.0.0.1:59482 error="method POST not allowed" -45:40.719 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/match from=127.0.0.1:59482 latency=55.884µs -45:40.719 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/match from=127.0.0.1:59484 error="method DELETE not allowed" -45:40.719 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/match from=127.0.0.1:59484 latency=60.595µs -45:40.719 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/match from=127.0.0.1:59486 error="method HEAD not allowed" -45:40.719 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/match from=127.0.0.1:59486 latency=18.672µs -45:40.720 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/match from=127.0.0.1:59486 latency=1.561µs -45:40.720 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/check from=127.0.0.1:59486 error="required query parameter 'source' not set" -45:40.720 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check from=127.0.0.1:59486 latency=53.521µs -45:40.720 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/check from=127.0.0.1:59488 error="method PUT not allowed" -45:40.720 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/check from=127.0.0.1:59488 latency=49.429µs -45:40.720 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/check from=127.0.0.1:59490 error="method POST not allowed" -45:40.720 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/check from=127.0.0.1:59490 latency=51.867µs -45:40.721 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/check from=127.0.0.1:59492 error="method DELETE not allowed" -45:40.721 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/check from=127.0.0.1:59492 latency=49.405µs -45:40.721 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/check from=127.0.0.1:59494 error="method HEAD not allowed" -45:40.721 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/check from=127.0.0.1:59494 latency=13.346µs -45:40.721 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/check from=127.0.0.1:59494 latency=1.158µs -45:40.722 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/ from=127.0.0.1:59494 error="Bad request: failed intention lookup: index error: UUID must be 36 characters" -45:40.722 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/ from=127.0.0.1:59494 latency=109.443µs -45:40.722 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/ from=127.0.0.1:59496 error="Bad request: Request decode failed: EOF" -45:40.722 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/ from=127.0.0.1:59496 latency=53.741µs -45:40.722 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/ from=127.0.0.1:59498 error="method POST not allowed" -45:40.722 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/ from=127.0.0.1:59498 latency=48.344µs -45:40.723 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/ from=127.0.0.1:59500 error="Intention lookup failed: failed intention lookup: index error: UUID must be 36 characters" -45:40.723 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/ from=127.0.0.1:59500 latency=111.168µs -45:40.723 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/ from=127.0.0.1:59502 error="method HEAD not allowed" -45:40.723 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/ from=127.0.0.1:59502 latency=12.478µs -45:40.723 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/ from=127.0.0.1:59502 latency=2µs -45:40.724 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/renew/ from=127.0.0.1:59502 error="method GET not allowed" -45:40.724 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/renew/ from=127.0.0.1:59502 latency=62.864µs -45:40.724 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/renew/ from=127.0.0.1:59504 latency=42.507µs -45:40.724 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/renew/ from=127.0.0.1:59506 error="method POST not allowed" -45:40.724 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/renew/ from=127.0.0.1:59506 latency=50.304µs -45:40.725 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/renew/ from=127.0.0.1:59508 error="method DELETE not allowed" -45:40.725 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/renew/ from=127.0.0.1:59508 latency=46.054µs -45:40.725 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/renew/ from=127.0.0.1:59510 error="method HEAD not allowed" -45:40.725 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/renew/ from=127.0.0.1:59510 latency=13.266µs -45:40.725 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/renew/ from=127.0.0.1:59510 latency=1.997µs -45:40.726 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/bootstrap from=127.0.0.1:59510 error="method GET not allowed" -45:40.726 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/bootstrap from=127.0.0.1:59510 latency=62.695µs -45:40.726 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:59512 latency=116.007µs -45:40.726 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/bootstrap from=127.0.0.1:59514 error="method POST not allowed" -45:40.727 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/bootstrap from=127.0.0.1:59514 latency=61.387µs -45:40.727 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/bootstrap from=127.0.0.1:59516 error="method DELETE not allowed" -45:40.727 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/bootstrap from=127.0.0.1:59516 latency=61.665µs -45:40.727 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/bootstrap from=127.0.0.1:59518 error="method HEAD not allowed" -45:40.727 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/bootstrap from=127.0.0.1:59518 latency=13.428µs -45:40.727 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/bootstrap from=127.0.0.1:59518 latency=1.271µs -45:40.728 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token from=127.0.0.1:59518 error="method GET not allowed" -45:40.728 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token from=127.0.0.1:59518 latency=48.079µs -45:40.728 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token from=127.0.0.1:59520 error="Bad request: Token decoding failed: EOF" -45:40.728 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:59520 latency=78.083µs -45:40.728 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token from=127.0.0.1:59522 error="method POST not allowed" -45:40.729 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token from=127.0.0.1:59522 latency=69.133µs -45:40.729 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token from=127.0.0.1:59524 error="method DELETE not allowed" -45:40.729 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token from=127.0.0.1:59524 latency=56.15µs -45:40.729 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token from=127.0.0.1:59526 error="method HEAD not allowed" -45:40.729 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token from=127.0.0.1:59526 latency=14.273µs -45:40.729 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token from=127.0.0.1:59526 latency=1.3µs -45:40.730 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/join/ from=127.0.0.1:59526 error="method GET not allowed" -45:40.730 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/join/ from=127.0.0.1:59526 latency=48.279µs -45:40.730 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/join/ from=127.0.0.1:59528 error="Permission denied" -45:40.730 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/ from=127.0.0.1:59528 latency=82.71µs -45:40.730 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/join/ from=127.0.0.1:59530 error="method POST not allowed" -45:40.730 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/join/ from=127.0.0.1:59530 latency=51.03µs -45:40.731 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/join/ from=127.0.0.1:59532 error="method DELETE not allowed" -45:40.731 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/join/ from=127.0.0.1:59532 latency=48.304µs -45:40.731 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/join/ from=127.0.0.1:59534 error="method HEAD not allowed" -45:40.731 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/join/ from=127.0.0.1:59534 latency=10.323µs -45:40.731 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/join/ from=127.0.0.1:59534 latency=1.902µs -45:40.732 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/ from=127.0.0.1:59534 latency=60.508µs -45:40.732 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/service/ from=127.0.0.1:59536 error="method PUT not allowed" -45:40.732 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/service/ from=127.0.0.1:59536 latency=51.018µs -45:40.732 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/service/ from=127.0.0.1:59538 error="method POST not allowed" -45:40.732 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/service/ from=127.0.0.1:59538 latency=48.469µs -45:40.733 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/service/ from=127.0.0.1:59540 error="method DELETE not allowed" -45:40.733 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/service/ from=127.0.0.1:59540 latency=57.272µs -45:40.733 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/service/ from=127.0.0.1:59542 error="method HEAD not allowed" -45:40.733 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/service/ from=127.0.0.1:59542 latency=12.965µs -45:40.733 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/service/ from=127.0.0.1:59542 latency=1.816µs -45:40.734 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/connect/ from=127.0.0.1:59542 latency=57.094µs -45:40.734 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/connect/ from=127.0.0.1:59544 error="method PUT not allowed" -45:40.734 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/connect/ from=127.0.0.1:59544 latency=59.133µs -45:40.735 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/connect/ from=127.0.0.1:59546 error="method POST not allowed" -45:40.735 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/connect/ from=127.0.0.1:59546 latency=57.172µs -45:40.735 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/connect/ from=127.0.0.1:59548 error="method DELETE not allowed" -45:40.735 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/connect/ from=127.0.0.1:59548 latency=63.625µs -45:40.735 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/connect/ from=127.0.0.1:59550 error="method HEAD not allowed" -45:40.735 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/connect/ from=127.0.0.1:59550 latency=12.787µs -45:40.736 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/connect/ from=127.0.0.1:59550 latency=1.349µs -45:40.736 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/autopilot/health from=127.0.0.1:59550 error="Permission denied" -45:40.736 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/health from=127.0.0.1:59550 latency=115.947µs -45:40.736 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/autopilot/health from=127.0.0.1:59552 error="method PUT not allowed" -45:40.736 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/health from=127.0.0.1:59552 latency=59.814µs -45:40.737 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/autopilot/health from=127.0.0.1:59554 error="method POST not allowed" -45:40.737 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/autopilot/health from=127.0.0.1:59554 latency=52.547µs -45:40.737 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/autopilot/health from=127.0.0.1:59556 error="method DELETE not allowed" -45:40.737 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/autopilot/health from=127.0.0.1:59556 latency=61.79µs -45:40.738 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/autopilot/health from=127.0.0.1:59558 error="method HEAD not allowed" -45:40.738 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/autopilot/health from=127.0.0.1:59558 latency=17.708µs -45:40.738 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/autopilot/health from=127.0.0.1:59558 latency=2.005µs -45:40.738 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:59558 error="Permission denied" -45:40.738 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:59558 latency=104.585µs -45:40.738 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:59560 error="Bad request: Request decode failed: EOF" -45:40.738 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:59560 latency=53.583µs -45:40.739 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/ca/configuration from=127.0.0.1:59562 error="method POST not allowed" -45:40.739 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/ca/configuration from=127.0.0.1:59562 latency=54.313µs -45:40.739 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/ca/configuration from=127.0.0.1:59564 error="method DELETE not allowed" -45:40.739 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/ca/configuration from=127.0.0.1:59564 latency=47.545µs -45:40.740 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/ca/configuration from=127.0.0.1:59566 error="method HEAD not allowed" -45:40.740 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/ca/configuration from=127.0.0.1:59566 latency=14.433µs -45:40.740 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/ca/configuration from=127.0.0.1:59566 latency=1.352µs -45:40.740 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role/name/ from=127.0.0.1:59566 error="Bad request: Missing role Name" -45:40.740 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/ from=127.0.0.1:59566 latency=50.909µs -45:40.740 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role/name/ from=127.0.0.1:59568 error="method PUT not allowed" -45:40.740 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/name/ from=127.0.0.1:59568 latency=49.723µs -45:40.741 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role/name/ from=127.0.0.1:59570 error="method POST not allowed" -45:40.741 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role/name/ from=127.0.0.1:59570 latency=50.515µs -45:40.741 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role/name/ from=127.0.0.1:59572 error="method DELETE not allowed" -45:40.741 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/name/ from=127.0.0.1:59572 latency=106.937µs -45:40.742 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role/name/ from=127.0.0.1:59574 error="method HEAD not allowed" -45:40.742 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role/name/ from=127.0.0.1:59574 latency=12.328µs -45:40.742 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role/name/ from=127.0.0.1:59574 latency=2.043µs -45:40.742 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rule/ from=127.0.0.1:59574 error="Bad request: Missing binding rule ID" -45:40.742 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/ from=127.0.0.1:59574 latency=63.745µs -45:40.742 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/ from=127.0.0.1:59576 error="Bad request: BindingRule decoding failed: EOF" -45:40.742 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/ from=127.0.0.1:59576 latency=68.804µs -45:40.743 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rule/ from=127.0.0.1:59578 error="method POST not allowed" -45:40.743 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rule/ from=127.0.0.1:59578 latency=58.416µs -45:40.743 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rule/ from=127.0.0.1:59580 error="Bad request: Missing binding rule ID" -45:40.743 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/ from=127.0.0.1:59580 latency=60.179µs -45:40.743 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rule/ from=127.0.0.1:59582 error="method HEAD not allowed" -45:40.743 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rule/ from=127.0.0.1:59582 latency=12.953µs -45:40.744 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rule/ from=127.0.0.1:59582 latency=1.819µs -45:40.744 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/host from=127.0.0.1:59582 error="Permission denied" -45:40.744 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/host from=127.0.0.1:59582 latency=123.99µs -45:40.744 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/host from=127.0.0.1:59584 error="method PUT not allowed" -45:40.744 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/host from=127.0.0.1:59584 latency=52.585µs -45:40.745 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/host from=127.0.0.1:59586 error="method POST not allowed" -45:40.745 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/host from=127.0.0.1:59586 latency=64.087µs -45:40.745 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/host from=127.0.0.1:59588 error="method DELETE not allowed" -45:40.745 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/host from=127.0.0.1:59588 latency=58.367µs -45:40.746 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/host from=127.0.0.1:59590 error="method HEAD not allowed" -45:40.746 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/host from=127.0.0.1:59590 latency=18.072µs -45:40.746 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/host from=127.0.0.1:59590 latency=1.804µs -45:40.746 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/metrics from=127.0.0.1:59590 error="Permission denied" -45:40.746 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/metrics from=127.0.0.1:59590 latency=103.645µs -45:40.747 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/metrics from=127.0.0.1:59592 error="method PUT not allowed" -45:40.747 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/metrics from=127.0.0.1:59592 latency=63.305µs -45:40.747 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/metrics from=127.0.0.1:59594 error="method POST not allowed" -45:40.747 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/metrics from=127.0.0.1:59594 latency=63.725µs -45:40.747 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/metrics from=127.0.0.1:59596 error="method DELETE not allowed" -45:40.747 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/metrics from=127.0.0.1:59596 latency=51.591µs -45:40.748 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/metrics from=127.0.0.1:59598 error="method HEAD not allowed" -45:40.748 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/metrics from=127.0.0.1:59598 latency=12.28µs -45:40.748 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/metrics from=127.0.0.1:59598 latency=1.748µs -45:40.748 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/name/ from=127.0.0.1:59598 error="Bad request: Missing policy Name" -45:40.748 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/name/ from=127.0.0.1:59598 latency=50.729µs -45:40.749 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy/name/ from=127.0.0.1:59600 error="method PUT not allowed" -45:40.749 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/name/ from=127.0.0.1:59600 latency=50.101µs -45:40.749 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy/name/ from=127.0.0.1:59602 error="method POST not allowed" -45:40.749 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy/name/ from=127.0.0.1:59602 latency=51.237µs -45:40.749 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy/name/ from=127.0.0.1:59604 error="method DELETE not allowed" -45:40.749 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/name/ from=127.0.0.1:59604 latency=49.462µs -45:40.750 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy/name/ from=127.0.0.1:59606 error="method HEAD not allowed" -45:40.750 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy/name/ from=127.0.0.1:59606 latency=11.917µs -45:40.750 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy/name/ from=127.0.0.1:59606 latency=1.27µs -45:40.750 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/register from=127.0.0.1:59606 error="method GET not allowed" -45:40.750 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/register from=127.0.0.1:59606 latency=61.776µs -45:40.751 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59608 latency=46.558µs -45:40.751 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/register from=127.0.0.1:59610 error="method POST not allowed" -45:40.751 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/register from=127.0.0.1:59610 latency=49.555µs -45:40.751 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/register from=127.0.0.1:59612 error="method DELETE not allowed" -45:40.751 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/register from=127.0.0.1:59612 latency=49.484µs -45:40.752 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/register from=127.0.0.1:59614 error="method HEAD not allowed" -45:40.752 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/register from=127.0.0.1:59614 latency=17.463µs -45:40.752 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/register from=127.0.0.1:59614 latency=1.26µs -45:40.752 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:59614 latency=63.776µs -45:40.753 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/status/leader from=127.0.0.1:59616 error="method PUT not allowed" -45:40.753 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/status/leader from=127.0.0.1:59616 latency=51.292µs -45:40.753 [ERROR] TestAgent.http: Request error: method=POST url=/v1/status/leader from=127.0.0.1:59618 error="method POST not allowed" -45:40.753 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/status/leader from=127.0.0.1:59618 latency=54.748µs -45:40.753 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/status/leader from=127.0.0.1:59620 error="method DELETE not allowed" -45:40.753 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/status/leader from=127.0.0.1:59620 latency=69.704µs -45:40.754 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/status/leader from=127.0.0.1:59622 error="method HEAD not allowed" -45:40.754 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/status/leader from=127.0.0.1:59622 latency=32.524µs -45:40.754 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/status/leader from=127.0.0.1:59622 latency=1.418µs -45:40.754 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/connect/authorize from=127.0.0.1:59622 error="method GET not allowed" -45:40.754 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/authorize from=127.0.0.1:59622 latency=49.634µs -45:40.755 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/authorize from=127.0.0.1:59624 error="method PUT not allowed" -45:40.755 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/authorize from=127.0.0.1:59624 latency=56.759µs -45:40.755 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:59626 error="Bad request: Request decode failed: EOF" -45:40.755 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:59626 latency=61.948µs -45:40.755 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/authorize from=127.0.0.1:59628 error="method DELETE not allowed" -45:40.755 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/authorize from=127.0.0.1:59628 latency=56.747µs -45:40.756 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/authorize from=127.0.0.1:59630 error="method HEAD not allowed" -45:40.756 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/authorize from=127.0.0.1:59630 latency=13.332µs -45:40.756 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/authorize from=127.0.0.1:59630 latency=1.317µs -45:40.756 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59630 latency=136.893µs -45:40.757 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59632 error="method PUT not allowed" -45:40.757 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59632 latency=68.771µs -45:40.757 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59634 error="method POST not allowed" -45:40.757 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59634 latency=49.272µs -45:40.757 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59636 error="method DELETE not allowed" -45:40.757 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59636 latency=50.028µs -45:40.758 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59638 error="method HEAD not allowed" -45:40.758 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59638 latency=18.659µs -45:40.758 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:59638 latency=2.018µs -45:40.758 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/login from=127.0.0.1:59638 error="method GET not allowed" -45:40.758 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/login from=127.0.0.1:59638 latency=62.669µs -45:40.759 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/login from=127.0.0.1:59640 error="method PUT not allowed" -45:40.759 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/login from=127.0.0.1:59640 latency=50.974µs -45:40.759 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:59642 error="Bad request: Failed to decode request body: EOF" -45:40.759 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:59642 latency=56.039µs -45:40.759 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/login from=127.0.0.1:59644 error="method DELETE not allowed" -45:40.759 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/login from=127.0.0.1:59644 latency=52.206µs -45:40.760 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/login from=127.0.0.1:59646 error="method HEAD not allowed" -45:40.760 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/login from=127.0.0.1:59646 latency=14.963µs -45:40.760 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/login from=127.0.0.1:59646 latency=1.858µs -45:40.760 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy from=127.0.0.1:59646 error="method GET not allowed" -45:40.760 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy from=127.0.0.1:59646 latency=61.078µs -45:40.761 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy from=127.0.0.1:59648 error="Bad request: Policy decoding failed: EOF" -45:40.761 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:59648 latency=71.323µs -45:40.761 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy from=127.0.0.1:59650 error="method POST not allowed" -45:40.761 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy from=127.0.0.1:59650 latency=66.662µs -45:40.762 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy from=127.0.0.1:59652 error="method DELETE not allowed" -45:40.762 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy from=127.0.0.1:59652 latency=65.782µs -45:40.762 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy from=127.0.0.1:59654 error="method HEAD not allowed" -45:40.762 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy from=127.0.0.1:59654 latency=16.114µs -45:40.762 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy from=127.0.0.1:59654 latency=1.952µs -45:40.762 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/token/ from=127.0.0.1:59654 error="method GET not allowed" -45:40.762 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/token/ from=127.0.0.1:59654 latency=47.527µs -45:40.763 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/token/ from=127.0.0.1:59656 error="Permission denied" -45:40.763 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/ from=127.0.0.1:59656 latency=82.556µs -45:40.763 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/token/ from=127.0.0.1:59658 error="method POST not allowed" -45:40.763 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/token/ from=127.0.0.1:59658 latency=50.805µs -45:40.764 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/token/ from=127.0.0.1:59660 error="method DELETE not allowed" -45:40.764 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/token/ from=127.0.0.1:59660 latency=63.059µs -45:40.764 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/token/ from=127.0.0.1:59662 error="method HEAD not allowed" -45:40.764 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/token/ from=127.0.0.1:59662 latency=16.925µs -45:40.764 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/token/ from=127.0.0.1:59662 latency=1.629µs -45:40.765 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/register from=127.0.0.1:59662 error="method GET not allowed" -45:40.765 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/register from=127.0.0.1:59662 latency=63.705µs -45:40.765 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/register from=127.0.0.1:59664 latency=46.673µs -45:40.765 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/register from=127.0.0.1:59666 error="method POST not allowed" -45:40.765 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/register from=127.0.0.1:59666 latency=50.044µs -45:40.766 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/register from=127.0.0.1:59668 error="method DELETE not allowed" -45:40.766 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/register from=127.0.0.1:59668 latency=51.55µs -45:40.766 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/register from=127.0.0.1:59670 error="method HEAD not allowed" -45:40.766 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/register from=127.0.0.1:59670 latency=18.502µs -45:40.766 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/register from=127.0.0.1:59670 latency=1.832µs -45:40.767 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:59670 error="Permission denied" -45:40.767 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59670 latency=102.985µs -45:40.767 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/self from=127.0.0.1:59672 error="method PUT not allowed" -45:40.767 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/self from=127.0.0.1:59672 latency=66.389µs -45:40.767 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/self from=127.0.0.1:59674 error="method POST not allowed" -45:40.767 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/self from=127.0.0.1:59674 latency=57.401µs -45:40.768 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/self from=127.0.0.1:59676 error="method DELETE not allowed" -45:40.768 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/self from=127.0.0.1:59676 latency=60.036µs -45:40.768 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/self from=127.0.0.1:59678 error="method HEAD not allowed" -45:40.768 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/self from=127.0.0.1:59678 latency=13.637µs -45:40.769 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/self from=127.0.0.1:59678 latency=2.837µs -45:40.769 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.769 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:59678 latency=135.678µs -45:40.769 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/nodes from=127.0.0.1:59680 error="method PUT not allowed" -45:40.769 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/nodes from=127.0.0.1:59680 latency=51.476µs -45:40.770 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/nodes from=127.0.0.1:59682 error="method POST not allowed" -45:40.770 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/nodes from=127.0.0.1:59682 latency=52.69µs -45:40.770 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/nodes from=127.0.0.1:59684 error="method DELETE not allowed" -45:40.770 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/nodes from=127.0.0.1:59684 latency=62.073µs -45:40.771 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/nodes from=127.0.0.1:59686 error="method HEAD not allowed" -45:40.771 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/nodes from=127.0.0.1:59686 latency=20.581µs -45:40.771 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/nodes from=127.0.0.1:59686 latency=1.93µs -45:40.771 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/node/ from=127.0.0.1:59686 latency=65.479µs -45:40.771 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/node/ from=127.0.0.1:59688 error="method PUT not allowed" -45:40.771 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/node/ from=127.0.0.1:59688 latency=57.657µs -45:40.772 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/node/ from=127.0.0.1:59690 error="method POST not allowed" -45:40.772 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/node/ from=127.0.0.1:59690 latency=53.979µs -45:40.772 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/node/ from=127.0.0.1:59692 error="method DELETE not allowed" -45:40.772 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/node/ from=127.0.0.1:59692 latency=48.483µs -45:40.773 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/node/ from=127.0.0.1:59694 error="method HEAD not allowed" -45:40.773 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/node/ from=127.0.0.1:59694 latency=18µs -45:40.773 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/node/ from=127.0.0.1:59694 latency=1.629µs -45:40.773 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ from=127.0.0.1:59694 latency=179.209µs -45:40.773 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/config/ from=127.0.0.1:59696 error="method PUT not allowed" -45:40.774 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config/ from=127.0.0.1:59696 latency=115.463µs -45:40.774 [ERROR] TestAgent.http: Request error: method=POST url=/v1/config/ from=127.0.0.1:59698 error="method POST not allowed" -45:40.774 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/config/ from=127.0.0.1:59698 latency=56.263µs -45:40.775 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config/ from=127.0.0.1:59700 latency=42.002µs -45:40.775 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/config/ from=127.0.0.1:59702 error="method HEAD not allowed" -45:40.775 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/config/ from=127.0.0.1:59702 latency=17.731µs -45:40.775 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/config/ from=127.0.0.1:59702 latency=1.99µs -45:40.775 [ERROR] TestAgent.http: Request error: method=GET url=/v1/coordinate/update from=127.0.0.1:59702 error="method GET not allowed" -45:40.775 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/update from=127.0.0.1:59702 latency=49.172µs -45:40.776 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/update from=127.0.0.1:59704 latency=45.656µs -45:40.776 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/update from=127.0.0.1:59706 error="method POST not allowed" -45:40.776 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/update from=127.0.0.1:59706 latency=49.461µs -45:40.776 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/update from=127.0.0.1:59708 error="method DELETE not allowed" -45:40.777 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/update from=127.0.0.1:59708 latency=49.429µs -45:40.777 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/update from=127.0.0.1:59710 error="method HEAD not allowed" -45:40.777 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/update from=127.0.0.1:59710 latency=12.144µs -45:40.777 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/update from=127.0.0.1:59710 latency=1.189µs -45:40.777 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role from=127.0.0.1:59710 error="method GET not allowed" -45:40.777 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role from=127.0.0.1:59710 latency=49.049µs -45:40.778 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role from=127.0.0.1:59712 error="Bad request: Role decoding failed: EOF" -45:40.778 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59712 latency=55.791µs -45:40.778 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role from=127.0.0.1:59714 error="method POST not allowed" -45:40.778 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role from=127.0.0.1:59714 latency=48.157µs -45:40.778 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role from=127.0.0.1:59716 error="method DELETE not allowed" -45:40.778 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role from=127.0.0.1:59716 latency=49.833µs -45:40.779 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role from=127.0.0.1:59718 error="method HEAD not allowed" -45:40.779 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role from=127.0.0.1:59718 latency=18.396µs -45:40.779 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role from=127.0.0.1:59718 latency=1.281µs -45:40.779 [DEBUG] TestAgent: dropping node from result due to ACLs: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 accessorID=00000000-0000-0000-0000-000000000002 -45:40.779 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members from=127.0.0.1:59718 latency=143.762µs -45:40.780 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/members from=127.0.0.1:59720 error="method PUT not allowed" -45:40.780 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/members from=127.0.0.1:59720 latency=49.97µs -45:40.780 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/members from=127.0.0.1:59722 error="method POST not allowed" -45:40.780 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/members from=127.0.0.1:59722 latency=49.254µs -45:40.780 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/members from=127.0.0.1:59724 error="method DELETE not allowed" -45:40.780 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/members from=127.0.0.1:59724 latency=50.747µs -45:40.781 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/members from=127.0.0.1:59726 error="method HEAD not allowed" -45:40.781 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/members from=127.0.0.1:59726 latency=12.506µs -45:40.781 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/members from=127.0.0.1:59726 latency=1.213µs -45:40.781 [ERROR] TestAgent.http: Request error: method=GET url=/v1/catalog/deregister from=127.0.0.1:59726 error="method GET not allowed" -45:40.781 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/deregister from=127.0.0.1:59726 latency=48.429µs -45:40.781 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/deregister from=127.0.0.1:59728 latency=52.005µs -45:40.782 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/deregister from=127.0.0.1:59730 error="method POST not allowed" -45:40.782 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/deregister from=127.0.0.1:59730 latency=46.831µs -45:40.782 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/deregister from=127.0.0.1:59732 error="method DELETE not allowed" -45:40.782 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/deregister from=127.0.0.1:59732 latency=62.986µs -45:40.783 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/deregister from=127.0.0.1:59734 error="method HEAD not allowed" -45:40.783 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/deregister from=127.0.0.1:59734 latency=17.353µs -45:40.783 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/deregister from=127.0.0.1:59734 latency=2.257µs -45:40.783 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role/ from=127.0.0.1:59734 error="Bad request: Missing role ID" -45:40.783 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/ from=127.0.0.1:59734 latency=69.566µs -45:40.783 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role/ from=127.0.0.1:59736 error="Bad request: Role decoding failed: EOF" -45:40.783 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/ from=127.0.0.1:59736 latency=63.086µs -45:40.784 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role/ from=127.0.0.1:59738 error="method POST not allowed" -45:40.784 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role/ from=127.0.0.1:59738 latency=64.27µs -45:40.784 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role/ from=127.0.0.1:59740 error="Bad request: Missing role ID" -45:40.784 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/ from=127.0.0.1:59740 latency=47.931µs -45:40.785 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role/ from=127.0.0.1:59742 error="method HEAD not allowed" -45:40.785 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role/ from=127.0.0.1:59742 latency=22.987µs -45:40.785 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role/ from=127.0.0.1:59742 latency=1.21µs -45:40.785 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-method from=127.0.0.1:59742 error="method GET not allowed" -45:40.785 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method from=127.0.0.1:59742 latency=46.708µs -45:40.785 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59744 error="Bad request: AuthMethod decoding failed: EOF" -45:40.785 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59744 latency=57.744µs -45:40.786 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-method from=127.0.0.1:59746 error="method POST not allowed" -45:40.786 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-method from=127.0.0.1:59746 latency=70.756µs -45:40.786 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-method from=127.0.0.1:59748 error="method DELETE not allowed" -45:40.786 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method from=127.0.0.1:59748 latency=50.535µs -45:40.787 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-method from=127.0.0.1:59750 error="method HEAD not allowed" -45:40.787 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-method from=127.0.0.1:59750 latency=17.532µs -45:40.787 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-method from=127.0.0.1:59750 latency=2.012µs -45:40.787 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/federation-states from=127.0.0.1:59750 error="Permission denied" -45:40.787 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-states from=127.0.0.1:59750 latency=136.434µs -45:40.787 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-states from=127.0.0.1:59752 error="method PUT not allowed" -45:40.787 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-states from=127.0.0.1:59752 latency=62.421µs -45:40.788 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-states from=127.0.0.1:59754 error="method POST not allowed" -45:40.788 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-states from=127.0.0.1:59754 latency=53.307µs -45:40.788 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-states from=127.0.0.1:59756 error="method DELETE not allowed" -45:40.788 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-states from=127.0.0.1:59756 latency=51.479µs -45:40.788 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-states from=127.0.0.1:59758 error="method HEAD not allowed" -45:40.788 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-states from=127.0.0.1:59758 latency=11.06µs -45:40.789 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-states from=127.0.0.1:59758 latency=1.383µs -45:40.789 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59758 latency=43.513µs -45:40.789 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59760 error="method PUT not allowed" -45:40.789 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59760 latency=49.981µs -45:40.790 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59762 error="method POST not allowed" -45:40.790 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59762 latency=63.203µs -45:40.790 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59764 error="method DELETE not allowed" -45:40.790 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59764 latency=56.942µs -45:40.790 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59766 error="method HEAD not allowed" -45:40.790 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59766 latency=18.083µs -45:40.791 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:59766 latency=1.603µs -45:40.791 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa229894-f1f9-233c-3b70-2308831768f1 -45:40.791 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/nodes from=127.0.0.1:59766 latency=174.333µs -45:40.791 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/nodes from=127.0.0.1:59768 error="method PUT not allowed" -45:40.791 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/nodes from=127.0.0.1:59768 latency=48.641µs -45:40.792 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/nodes from=127.0.0.1:59770 error="method POST not allowed" -45:40.792 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/nodes from=127.0.0.1:59770 latency=65.174µs -45:40.792 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/nodes from=127.0.0.1:59772 error="method DELETE not allowed" -45:40.792 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/nodes from=127.0.0.1:59772 latency=57.842µs -45:40.793 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/nodes from=127.0.0.1:59774 error="method HEAD not allowed" -45:40.793 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/nodes from=127.0.0.1:59774 latency=13.362µs -45:40.793 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/nodes from=127.0.0.1:59774 latency=1.4µs -45:40.793 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/keyring from=127.0.0.1:59774 error="Reading keyring denied by ACLs" -45:40.793 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:59774 latency=105.61µs -45:40.793 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/keyring from=127.0.0.1:59776 error="Bad request: Request decode failed: EOF" -45:40.793 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/keyring from=127.0.0.1:59776 latency=51.83µs -45:40.794 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/keyring from=127.0.0.1:59778 error="Bad request: Request decode failed: EOF" -45:40.794 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/keyring from=127.0.0.1:59778 latency=50.159µs -45:40.794 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/keyring from=127.0.0.1:59780 error="Bad request: Request decode failed: EOF" -45:40.794 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/keyring from=127.0.0.1:59780 latency=55.891µs -45:40.795 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/keyring from=127.0.0.1:59782 error="method HEAD not allowed" -45:40.795 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/keyring from=127.0.0.1:59782 latency=19.448µs -45:40.795 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/keyring from=127.0.0.1:59782 latency=1.548µs -45:40.795 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/logout from=127.0.0.1:59782 error="method GET not allowed" -45:40.795 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/logout from=127.0.0.1:59782 latency=52.324µs -45:40.796 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/logout from=127.0.0.1:59784 error="method PUT not allowed" -45:40.796 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/logout from=127.0.0.1:59784 latency=49.729µs -45:40.796 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:59786 error="ACL not found" -45:40.796 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:59786 latency=57.091µs -45:40.796 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/logout from=127.0.0.1:59788 error="method DELETE not allowed" -45:40.796 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/logout from=127.0.0.1:59788 latency=100.858µs -45:40.797 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/logout from=127.0.0.1:59790 error="method HEAD not allowed" -45:40.797 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/logout from=127.0.0.1:59790 latency=10.99µs -45:40.797 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/logout from=127.0.0.1:59790 latency=1.448µs -45:40.797 [ERROR] TestAgent.http: Request error: method=GET url=/v1/event/fire/ from=127.0.0.1:59790 error="method GET not allowed" -45:40.797 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/event/fire/ from=127.0.0.1:59790 latency=54.062µs -45:40.798 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/ from=127.0.0.1:59792 latency=48.375µs -45:40.798 [ERROR] TestAgent.http: Request error: method=POST url=/v1/event/fire/ from=127.0.0.1:59794 error="method POST not allowed" -45:40.798 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/event/fire/ from=127.0.0.1:59794 latency=49.114µs -45:40.798 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/event/fire/ from=127.0.0.1:59796 error="method DELETE not allowed" -45:40.798 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/event/fire/ from=127.0.0.1:59796 latency=58.184µs -45:40.799 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/event/fire/ from=127.0.0.1:59798 error="method HEAD not allowed" -45:40.799 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/event/fire/ from=127.0.0.1:59798 latency=29.252µs -45:40.799 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/event/fire/ from=127.0.0.1:59798 latency=1.666µs -45:40.799 [ERROR] TestAgent.http: Request error: method=GET url=/v1/snapshot from=127.0.0.1:59798 error="Permission denied" -45:40.799 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:59798 latency=111.365µs -45:40.800 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:59800 error="Permission denied" -45:40.800 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:59800 latency=81.101µs -45:40.800 [ERROR] TestAgent.http: Request error: method=POST url=/v1/snapshot from=127.0.0.1:59802 error="method POST not allowed" -45:40.800 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/snapshot from=127.0.0.1:59802 latency=62.278µs -45:40.801 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/snapshot from=127.0.0.1:59804 error="method DELETE not allowed" -45:40.801 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/snapshot from=127.0.0.1:59804 latency=61.259µs -45:40.801 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/snapshot from=127.0.0.1:59806 error="method HEAD not allowed" -45:40.801 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/snapshot from=127.0.0.1:59806 latency=18.248µs -45:40.801 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/snapshot from=127.0.0.1:59806 latency=1.724µs -45:40.801 [ERROR] TestAgent.http: Request error: method=GET url=/v1/txn from=127.0.0.1:59806 error="method GET not allowed" -45:40.802 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/txn from=127.0.0.1:59806 latency=57.152µs -45:40.802 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/txn from=127.0.0.1:59808 latency=45.49µs -45:40.802 [ERROR] TestAgent.http: Request error: method=POST url=/v1/txn from=127.0.0.1:59810 error="method POST not allowed" -45:40.802 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/txn from=127.0.0.1:59810 latency=50.865µs -45:40.803 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/txn from=127.0.0.1:59812 error="method DELETE not allowed" -45:40.803 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/txn from=127.0.0.1:59812 latency=48.494µs -45:40.803 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/txn from=127.0.0.1:59814 error="method HEAD not allowed" -45:40.803 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/txn from=127.0.0.1:59814 latency=12.861µs -45:40.803 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/txn from=127.0.0.1:59814 latency=1.987µs -45:40.804 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/roles from=127.0.0.1:59814 error="Permission denied" -45:40.804 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:59814 latency=142.942µs -45:40.804 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/roles from=127.0.0.1:59816 error="method PUT not allowed" -45:40.804 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/roles from=127.0.0.1:59816 latency=57.733µs -45:40.804 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/roles from=127.0.0.1:59818 error="method POST not allowed" -45:40.805 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/roles from=127.0.0.1:59818 latency=50.85µs -45:40.805 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/roles from=127.0.0.1:59820 error="method DELETE not allowed" -45:40.805 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/roles from=127.0.0.1:59820 latency=63.241µs -45:40.805 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/roles from=127.0.0.1:59822 error="method HEAD not allowed" -45:40.805 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/roles from=127.0.0.1:59822 latency=18.99µs -45:40.806 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/roles from=127.0.0.1:59822 latency=1.92µs -45:40.806 [ERROR] TestAgent.http: Request error: method=GET url=/v1/discovery-chain/ from=127.0.0.1:59822 error="Bad request: Missing chain name" -45:40.806 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/discovery-chain/ from=127.0.0.1:59822 latency=63.76µs -45:40.806 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/discovery-chain/ from=127.0.0.1:59824 error="method PUT not allowed" -45:40.806 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/discovery-chain/ from=127.0.0.1:59824 latency=58.045µs -45:40.807 [ERROR] TestAgent.http: Request error: method=POST url=/v1/discovery-chain/ from=127.0.0.1:59826 error="Bad request: Missing chain name" -45:40.807 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/discovery-chain/ from=127.0.0.1:59826 latency=62.465µs -45:40.807 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/discovery-chain/ from=127.0.0.1:59828 error="method DELETE not allowed" -45:40.807 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/discovery-chain/ from=127.0.0.1:59828 latency=60.67µs -45:40.808 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/discovery-chain/ from=127.0.0.1:59830 error="method HEAD not allowed" -45:40.808 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/discovery-chain/ from=127.0.0.1:59830 latency=11.509µs -45:40.808 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/discovery-chain/ from=127.0.0.1:59830 latency=1.123µs -45:40.808 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/state/ from=127.0.0.1:59830 latency=44.15µs -45:40.808 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/state/ from=127.0.0.1:59832 error="method PUT not allowed" -45:40.808 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/state/ from=127.0.0.1:59832 latency=63.99µs -45:40.809 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/state/ from=127.0.0.1:59834 error="method POST not allowed" -45:40.809 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/state/ from=127.0.0.1:59834 latency=59.944µs -45:40.809 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/state/ from=127.0.0.1:59836 error="method DELETE not allowed" -45:40.809 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/state/ from=127.0.0.1:59836 latency=50.627µs -45:40.810 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/state/ from=127.0.0.1:59838 error="method HEAD not allowed" -45:40.810 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/state/ from=127.0.0.1:59838 latency=13.846µs -45:40.810 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/state/ from=127.0.0.1:59838 latency=1.335µs -45:40.810 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:59838 error="Permission denied" -45:40.810 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:59838 latency=125.571µs -45:40.810 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/autopilot/configuration from=127.0.0.1:59840 error="Bad request: Error parsing autopilot config: EOF" -45:40.811 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/configuration from=127.0.0.1:59840 latency=73.872µs -45:40.811 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/autopilot/configuration from=127.0.0.1:59842 error="method POST not allowed" -45:40.811 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/autopilot/configuration from=127.0.0.1:59842 latency=59.332µs -45:40.811 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/autopilot/configuration from=127.0.0.1:59844 error="method DELETE not allowed" -45:40.811 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/autopilot/configuration from=127.0.0.1:59844 latency=48.857µs -45:40.812 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/autopilot/configuration from=127.0.0.1:59846 error="method HEAD not allowed" -45:40.812 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/autopilot/configuration from=127.0.0.1:59846 latency=18.184µs -45:40.812 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/autopilot/configuration from=127.0.0.1:59846 latency=1.153µs -45:40.812 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/ from=127.0.0.1:59846 error="Bad request: Missing policy ID" -45:40.812 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/ from=127.0.0.1:59846 latency=46.994µs -45:40.813 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy/ from=127.0.0.1:59848 error="Bad request: Policy decoding failed: EOF" -45:40.813 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/ from=127.0.0.1:59848 latency=56.074µs -45:40.813 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy/ from=127.0.0.1:59850 error="method POST not allowed" -45:40.813 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy/ from=127.0.0.1:59850 latency=59.638µs -45:40.814 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy/ from=127.0.0.1:59852 error="Bad request: Missing policy ID" -45:40.814 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/ from=127.0.0.1:59852 latency=51.213µs -45:40.814 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy/ from=127.0.0.1:59854 error="method HEAD not allowed" -45:40.814 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy/ from=127.0.0.1:59854 latency=17.465µs -45:40.814 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy/ from=127.0.0.1:59854 latency=1.343µs -45:40.814 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/list from=127.0.0.1:59854 latency=159.773µs -45:40.815 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/list from=127.0.0.1:59856 error="method PUT not allowed" -45:40.815 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/list from=127.0.0.1:59856 latency=53.235µs -45:40.815 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/list from=127.0.0.1:59858 error="method POST not allowed" -45:40.815 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/list from=127.0.0.1:59858 latency=56.508µs -45:40.816 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/list from=127.0.0.1:59860 error="method DELETE not allowed" -45:40.816 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/list from=127.0.0.1:59860 latency=61.091µs -45:40.816 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/list from=127.0.0.1:59862 error="method HEAD not allowed" -45:40.816 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/list from=127.0.0.1:59862 latency=17.99µs -45:40.816 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/list from=127.0.0.1:59862 latency=2.08µs -45:40.817 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-methods from=127.0.0.1:59862 error="Permission denied" -45:40.817 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:59862 latency=139.838µs -45:40.817 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-methods from=127.0.0.1:59864 error="method PUT not allowed" -45:40.817 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-methods from=127.0.0.1:59864 latency=51.978µs -45:40.817 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-methods from=127.0.0.1:59866 error="method POST not allowed" -45:40.818 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-methods from=127.0.0.1:59866 latency=50.469µs -45:40.818 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-methods from=127.0.0.1:59868 error="method DELETE not allowed" -45:40.818 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-methods from=127.0.0.1:59868 latency=49.778µs -45:40.818 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-methods from=127.0.0.1:59870 error="method HEAD not allowed" -45:40.818 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-methods from=127.0.0.1:59870 latency=18.446µs -45:40.818 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-methods from=127.0.0.1:59870 latency=1.879µs -45:40.819 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/self from=127.0.0.1:59870 error="ACL not found" -45:40.819 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:59870 latency=131.364µs -45:40.819 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token/self from=127.0.0.1:59872 error="method PUT not allowed" -45:40.819 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/self from=127.0.0.1:59872 latency=63.189µs -45:40.820 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token/self from=127.0.0.1:59874 error="method POST not allowed" -45:40.820 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token/self from=127.0.0.1:59874 latency=57.177µs -45:40.820 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token/self from=127.0.0.1:59876 error="method DELETE not allowed" -45:40.820 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/self from=127.0.0.1:59876 latency=64.582µs -45:40.820 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token/self from=127.0.0.1:59878 error="method HEAD not allowed" -45:40.820 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token/self from=127.0.0.1:59878 latency=13.399µs -45:40.821 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token/self from=127.0.0.1:59878 latency=1.675µs -45:40.821 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/acl/authorize from=127.0.0.1:59878 error="method GET not allowed" -45:40.821 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/acl/authorize from=127.0.0.1:59878 latency=81.21µs -45:40.821 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/acl/authorize from=127.0.0.1:59880 error="method PUT not allowed" -45:40.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/acl/authorize from=127.0.0.1:59880 latency=59.174µs -45:40.822 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/acl/authorize from=127.0.0.1:59882 error="Bad request: Failed to decode request body: EOF" -45:40.822 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/acl/authorize from=127.0.0.1:59882 latency=54.944µs -45:40.822 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/acl/authorize from=127.0.0.1:59884 error="method DELETE not allowed" -45:40.822 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/acl/authorize from=127.0.0.1:59884 latency=64.885µs -45:40.822 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/acl/authorize from=127.0.0.1:59886 error="method HEAD not allowed" -45:40.823 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/acl/authorize from=127.0.0.1:59886 latency=13.31µs -45:40.823 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/acl/authorize from=127.0.0.1:59886 latency=1.389µs -45:40.823 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/fail/ from=127.0.0.1:59886 error="method GET not allowed" -45:40.823 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/fail/ from=127.0.0.1:59886 latency=64.504µs -45:40.823 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/fail/ from=127.0.0.1:59888 error="Unknown check """ -45:40.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/fail/ from=127.0.0.1:59888 latency=94.385µs -45:40.824 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/fail/ from=127.0.0.1:59890 error="method POST not allowed" -45:40.824 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/fail/ from=127.0.0.1:59890 latency=60.773µs -45:40.824 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/fail/ from=127.0.0.1:59892 error="method DELETE not allowed" -45:40.824 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/fail/ from=127.0.0.1:59892 latency=56.353µs -45:40.825 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/fail/ from=127.0.0.1:59894 error="method HEAD not allowed" -45:40.825 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/fail/ from=127.0.0.1:59894 latency=17.632µs -45:40.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/fail/ from=127.0.0.1:59894 latency=1.842µs -45:40.825 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/federation-state/ from=127.0.0.1:59894 error="Bad request: Missing datacenter name" -45:40.825 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-state/ from=127.0.0.1:59894 latency=53.804µs -45:40.826 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-state/ from=127.0.0.1:59896 error="method PUT not allowed" -45:40.826 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-state/ from=127.0.0.1:59896 latency=61.17µs -45:40.826 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-state/ from=127.0.0.1:59898 error="method POST not allowed" -45:40.826 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-state/ from=127.0.0.1:59898 latency=60.702µs -45:40.827 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-state/ from=127.0.0.1:59900 error="method DELETE not allowed" -45:40.827 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-state/ from=127.0.0.1:59900 latency=59.768µs -45:40.827 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-state/ from=127.0.0.1:59902 error="method HEAD not allowed" -45:40.827 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-state/ from=127.0.0.1:59902 latency=11.737µs -45:40.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-state/ from=127.0.0.1:59902 latency=1.385µs -45:40.827 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/raft/peer from=127.0.0.1:59902 error="method GET not allowed" -45:40.828 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/peer from=127.0.0.1:59902 latency=50.747µs -45:40.828 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/raft/peer from=127.0.0.1:59904 error="method PUT not allowed" -45:40.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/raft/peer from=127.0.0.1:59904 latency=56.78µs -45:40.828 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/raft/peer from=127.0.0.1:59906 error="method POST not allowed" -45:40.828 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/raft/peer from=127.0.0.1:59906 latency=60.507µs -45:40.829 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer from=127.0.0.1:59908 error="Bad request: Must specify either ?id with the server's ID or ?address with IP:port of peer to remove" -45:40.829 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer from=127.0.0.1:59908 latency=76.081µs -45:40.829 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/raft/peer from=127.0.0.1:59910 error="method HEAD not allowed" -45:40.829 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/raft/peer from=127.0.0.1:59910 latency=25.355µs -45:40.829 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/raft/peer from=127.0.0.1:59910 latency=1.971µs -45:40.830 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/info/ from=127.0.0.1:59910 latency=51.95µs -45:40.830 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/info/ from=127.0.0.1:59912 error="method PUT not allowed" -45:40.830 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/info/ from=127.0.0.1:59912 latency=58.402µs -45:40.830 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/info/ from=127.0.0.1:59914 error="method POST not allowed" -45:40.831 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/info/ from=127.0.0.1:59914 latency=128.685µs -45:40.831 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/info/ from=127.0.0.1:59916 error="method DELETE not allowed" -45:40.831 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/info/ from=127.0.0.1:59916 latency=51.942µs -45:40.831 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/info/ from=127.0.0.1:59918 error="method HEAD not allowed" -45:40.831 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/info/ from=127.0.0.1:59918 latency=17.699µs -45:40.832 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/info/ from=127.0.0.1:59918 latency=1.155µs -45:40.832 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/tokens from=127.0.0.1:59918 error="Permission denied" -45:40.832 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:59918 latency=112.769µs -45:40.832 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/tokens from=127.0.0.1:59920 error="method PUT not allowed" -45:40.832 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/tokens from=127.0.0.1:59920 latency=68.331µs -45:40.833 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/tokens from=127.0.0.1:59922 error="method POST not allowed" -45:40.833 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/tokens from=127.0.0.1:59922 latency=47.014µs -45:40.833 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/tokens from=127.0.0.1:59924 error="method DELETE not allowed" -45:40.833 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/tokens from=127.0.0.1:59924 latency=53.045µs -45:40.834 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/tokens from=127.0.0.1:59926 error="method HEAD not allowed" -45:40.834 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/tokens from=127.0.0.1:59926 latency=18.059µs -45:40.834 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/tokens from=127.0.0.1:59926 latency=1.876µs -45:40.834 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/health/service/id/ from=127.0.0.1:59926 error="Bad request: Missing serviceID" -45:40.834 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/health/service/id/ from=127.0.0.1:59926 latency=50.08µs -45:40.834 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/health/service/id/ from=127.0.0.1:59928 error="method PUT not allowed" -45:40.835 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/health/service/id/ from=127.0.0.1:59928 latency=78.466µs -45:40.835 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/health/service/id/ from=127.0.0.1:59930 error="method POST not allowed" -45:40.835 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/health/service/id/ from=127.0.0.1:59930 latency=70.398µs -45:40.835 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/health/service/id/ from=127.0.0.1:59932 error="method DELETE not allowed" -45:40.835 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/health/service/id/ from=127.0.0.1:59932 latency=56.054µs -45:40.836 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/health/service/id/ from=127.0.0.1:59934 error="method HEAD not allowed" -45:40.836 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/health/service/id/ from=127.0.0.1:59934 latency=13.568µs -45:40.836 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/health/service/id/ from=127.0.0.1:59934 latency=1.358µs -45:40.836 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/pass/ from=127.0.0.1:59934 error="method GET not allowed" -45:40.836 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/pass/ from=127.0.0.1:59934 latency=55.056µs -45:40.837 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/pass/ from=127.0.0.1:59936 error="Unknown check """ -45:40.837 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/pass/ from=127.0.0.1:59936 latency=105.396µs -45:40.837 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/pass/ from=127.0.0.1:59938 error="method POST not allowed" -45:40.837 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/pass/ from=127.0.0.1:59938 latency=63.878µs -45:40.838 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/pass/ from=127.0.0.1:59940 error="method DELETE not allowed" -45:40.838 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/pass/ from=127.0.0.1:59940 latency=52.573µs -45:40.838 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/pass/ from=127.0.0.1:59942 error="method HEAD not allowed" -45:40.838 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/pass/ from=127.0.0.1:59942 latency=13.095µs -45:40.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/pass/ from=127.0.0.1:59942 latency=1.825µs -45:40.838 [INFO] TestAgent: Requesting shutdown -45:40.838 [INFO] TestAgent.server: shutting down server -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:40.838 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:40.838 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:40.838 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:40.839 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:40.839 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:40.839 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:40.839 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:40.839 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:40.839 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:40.842 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:40.845Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:40.845Z [INFO] agent.router.manager: shutting down -45:40.845 [INFO] TestAgent: consul server down -45:40.845 [INFO] TestAgent: shutdown complete -45:40.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11904 network=tcp -45:40.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11904 network=udp -45:40.845 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11905 network=tcp -45:41.346 [INFO] TestAgent: Waiting for endpoints to shut down -45:41.346 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_MethodNotAllowed_OSS (0.97s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query (0.00s) +2021-10-05T16:30:47.009Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:47.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:47.066 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d Address:127.0.0.1:30287}]" +30:47.066 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30287 [Follower]" leader= +30:47.067 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d.dc1 127.0.0.1 +30:47.067 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d 127.0.0.1 +2021-10-05T16:30:47.067Z [INFO] agent.router: Initializing LAN area manager +30:47.068 [INFO] TestAgent.server: Adding LAN server: server="Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d (Addr: tcp/127.0.0.1:30287) (DC: dc1)" +30:47.068 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d.dc1 area=wan +30:47.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30282 network=udp +30:47.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30282 network=tcp +30:47.068 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30283 network=tcp +30:47.068 [INFO] TestAgent: started state syncer +30:47.068 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30288 network=tcp +30:47.110 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:47.110 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30287 [Candidate]" term=2 +30:47.119 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:47.137 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:47.137 [DEBUG] TestAgent.server.raft: vote granted: from=8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d term=2 tally=1 +30:47.137 [INFO] TestAgent.server.raft: election won: tally=1 +30:47.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30287 [Leader]" +30:47.140 [INFO] TestAgent.server: cluster leadership acquired +30:47.140 [INFO] TestAgent.server: New leader elected: payload=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.147 [INFO] TestAgent.server: initializing acls +30:47.161 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:47.161 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.170 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:47.175 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.185 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.185 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:47.185 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:47.185 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.185 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d.dc1 +30:47.187 [INFO] TestAgent.server: Updating LAN server: server="Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d (Addr: tcp/127.0.0.1:30287) (DC: dc1)" +30:47.188 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d.dc1 area=wan +30:47.197 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:47.223 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:47.223 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:47.223 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:47.223 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:47.223 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:47.223 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.224 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.238 [ERROR] TestAgent.http: Request error: method=GET url=/v1/query/ from=127.0.0.1:36400 error="failed prepared query lookup: index error: UUID must be 36 characters" +30:47.238 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/ from=127.0.0.1:36400 latency=697.296µs +30:47.239 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/ from=127.0.0.1:36402 error="Prepared Query lookup failed: failed prepared query lookup: index error: UUID must be 36 characters" +30:47.239 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/ from=127.0.0.1:36402 latency=249.086µs +30:47.240 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/ from=127.0.0.1:36404 error="method POST not allowed" +30:47.240 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/ from=127.0.0.1:36404 latency=289.706µs +30:47.246 [INFO] TestAgent.server: federation state anti-entropy synced +30:47.248 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/ from=127.0.0.1:36406 error="Prepared Query lookup failed: failed prepared query lookup: index error: UUID must be 36 characters" +30:47.248 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/ from=127.0.0.1:36406 latency=390.671µs +30:47.249 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/ from=127.0.0.1:36408 error="method HEAD not allowed" +30:47.249 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/ from=127.0.0.1:36408 latency=35.619µs +30:47.251 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/ from=127.0.0.1:36408 latency=1.247759ms +30:47.252 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/xxx/execute from=127.0.0.1:36410 latency=334.195µs +30:47.257 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/xxx/execute from=127.0.0.1:36412 error="method PUT not allowed" +30:47.257 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/xxx/execute from=127.0.0.1:36412 latency=164.041µs +30:47.257 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/xxx/execute from=127.0.0.1:36414 error="method POST not allowed" +30:47.257 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/xxx/execute from=127.0.0.1:36414 latency=107.653µs +30:47.258 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/xxx/execute from=127.0.0.1:36416 error="method DELETE not allowed" +30:47.258 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/xxx/execute from=127.0.0.1:36416 latency=108.033µs +30:47.259 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/xxx/execute from=127.0.0.1:36418 error="method HEAD not allowed" +30:47.259 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/xxx/execute from=127.0.0.1:36418 latency=21.297µs +30:47.259 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/xxx/execute from=127.0.0.1:36418 latency=90.595µs +30:47.263 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query/xxx/explain from=127.0.0.1:36420 latency=3.755165ms +30:47.268 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query/xxx/explain from=127.0.0.1:36422 error="method PUT not allowed" +30:47.268 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query/xxx/explain from=127.0.0.1:36422 latency=193.386µs +30:47.272 [ERROR] TestAgent.http: Request error: method=POST url=/v1/query/xxx/explain from=127.0.0.1:36424 error="method POST not allowed" +30:47.272 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query/xxx/explain from=127.0.0.1:36424 latency=151.739µs +30:47.273 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query/xxx/explain from=127.0.0.1:36426 error="method DELETE not allowed" +30:47.273 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query/xxx/explain from=127.0.0.1:36426 latency=495.715µs +30:47.275 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query/xxx/explain from=127.0.0.1:36428 error="method HEAD not allowed" +30:47.275 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query/xxx/explain from=127.0.0.1:36428 latency=28.849µs +30:47.276 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query/xxx/explain from=127.0.0.1:36428 latency=569.88µs +30:47.277 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/query from=127.0.0.1:36430 latency=395.374µs +30:47.277 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/query from=127.0.0.1:36432 error="method PUT not allowed" +30:47.277 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/query from=127.0.0.1:36432 latency=117.568µs +30:47.278 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/query from=127.0.0.1:36434 latency=281.927µs +30:47.280 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/query from=127.0.0.1:36436 error="method DELETE not allowed" +30:47.280 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/query from=127.0.0.1:36436 latency=157.806µs +30:47.284 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/query from=127.0.0.1:36438 error="method HEAD not allowed" +30:47.284 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/query from=127.0.0.1:36438 latency=32.144µs +30:47.285 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/query from=127.0.0.1:36438 latency=4.281µs +30:47.299 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node-services/ from=127.0.0.1:36438 latency=13.028851ms +30:47.300 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/node-services/ from=127.0.0.1:36440 error="method PUT not allowed" +30:47.300 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/node-services/ from=127.0.0.1:36440 latency=196.806µs +30:47.301 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/node-services/ from=127.0.0.1:36442 error="method POST not allowed" +30:47.310 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/node-services/ from=127.0.0.1:36442 latency=9.480555ms +30:47.314 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/node-services/ from=127.0.0.1:36444 error="method DELETE not allowed" +30:47.315 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/node-services/ from=127.0.0.1:36444 latency=1.526683ms +30:47.317 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/node-services/ from=127.0.0.1:36448 error="method HEAD not allowed" +30:47.317 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/node-services/ from=127.0.0.1:36448 latency=25.926µs +30:47.317 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/node-services/ from=127.0.0.1:36448 latency=2.554µs +30:47.317 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/leave from=127.0.0.1:36448 error="method GET not allowed" +30:47.317 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/leave from=127.0.0.1:36448 latency=131.259µs +30:47.318 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/leave from=127.0.0.1:36450 error="Permission denied" +30:47.318 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/leave from=127.0.0.1:36450 latency=183.513µs +30:47.319 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/leave from=127.0.0.1:36452 error="method POST not allowed" +30:47.319 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/leave from=127.0.0.1:36452 latency=119.176µs +30:47.320 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/leave from=127.0.0.1:36454 error="method DELETE not allowed" +30:47.324 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/leave from=127.0.0.1:36454 latency=1.445469ms +30:47.325 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/leave from=127.0.0.1:36456 error="method HEAD not allowed" +30:47.325 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/leave from=127.0.0.1:36456 latency=65.709µs +30:47.326 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/leave from=127.0.0.1:36456 latency=3.298µs +30:47.326 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/warn/ from=127.0.0.1:36456 error="method GET not allowed" +30:47.326 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/warn/ from=127.0.0.1:36456 latency=244.001µs +30:47.327 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/warn/ from=127.0.0.1:36458 error="Unknown check """ +30:47.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/warn/ from=127.0.0.1:36458 latency=203.341µs +30:47.327 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/warn/ from=127.0.0.1:36460 error="method POST not allowed" +30:47.328 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/warn/ from=127.0.0.1:36460 latency=114.293µs +30:47.332 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/warn/ from=127.0.0.1:36462 error="method DELETE not allowed" +30:47.333 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/warn/ from=127.0.0.1:36462 latency=283.014µs +30:47.333 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/warn/ from=127.0.0.1:36464 error="method HEAD not allowed" +30:47.333 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/warn/ from=127.0.0.1:36464 latency=21.518µs +30:47.334 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/warn/ from=127.0.0.1:36464 latency=2.615µs +30:47.334 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/deregister/ from=127.0.0.1:36464 error="method GET not allowed" +30:47.334 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/deregister/ from=127.0.0.1:36464 latency=108.335µs +30:47.334 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/service/deregister/ from=127.0.0.1:36466 error="Unknown service {"" {}}" +30:47.336 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/ from=127.0.0.1:36466 latency=1.186009ms +30:47.342 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/deregister/ from=127.0.0.1:36468 error="method POST not allowed" +30:47.343 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/deregister/ from=127.0.0.1:36468 latency=308.965µs +30:47.346 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/deregister/ from=127.0.0.1:36472 error="method DELETE not allowed" +30:47.346 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/deregister/ from=127.0.0.1:36472 latency=136.421µs +30:47.346 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/deregister/ from=127.0.0.1:36474 error="method HEAD not allowed" +30:47.346 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/deregister/ from=127.0.0.1:36474 latency=23.251µs +30:47.348 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/deregister/ from=127.0.0.1:36474 latency=3.497µs +30:47.348 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/state/ from=127.0.0.1:36474 latency=219.483µs +30:47.349 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/state/ from=127.0.0.1:36476 error="method PUT not allowed" +30:47.349 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/state/ from=127.0.0.1:36476 latency=204.039µs +30:47.350 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/state/ from=127.0.0.1:36478 error="method POST not allowed" +30:47.350 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/state/ from=127.0.0.1:36478 latency=126.441µs +30:47.350 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/state/ from=127.0.0.1:36480 error="method DELETE not allowed" +30:47.350 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/state/ from=127.0.0.1:36480 latency=111.675µs +30:47.351 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/state/ from=127.0.0.1:36482 error="method HEAD not allowed" +30:47.351 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/state/ from=127.0.0.1:36482 latency=19.515µs +30:47.351 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/state/ from=127.0.0.1:36482 latency=2.394µs +30:47.351 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/connect/ from=127.0.0.1:36482 latency=117.732µs +30:47.355 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/connect/ from=127.0.0.1:36484 error="method PUT not allowed" +30:47.355 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/connect/ from=127.0.0.1:36484 latency=134.312µs +30:47.364 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/connect/ from=127.0.0.1:36486 error="method POST not allowed" +30:47.364 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/connect/ from=127.0.0.1:36486 latency=226.612µs +30:47.365 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/connect/ from=127.0.0.1:36488 error="method DELETE not allowed" +30:47.365 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/connect/ from=127.0.0.1:36488 latency=114.195µs +30:47.366 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/connect/ from=127.0.0.1:36490 error="method HEAD not allowed" +30:47.366 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/connect/ from=127.0.0.1:36490 latency=20.772µs +30:47.366 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/connect/ from=127.0.0.1:36490 latency=2.591µs +30:47.366 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/list from=127.0.0.1:36490 error="Permission denied" +30:47.366 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/list from=127.0.0.1:36490 latency=218.089µs +30:47.367 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/list from=127.0.0.1:36492 error="method PUT not allowed" +30:47.367 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/list from=127.0.0.1:36492 latency=110.577µs +30:47.368 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/list from=127.0.0.1:36494 error="method POST not allowed" +30:47.368 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/list from=127.0.0.1:36494 latency=152.618µs +30:47.368 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/list from=127.0.0.1:36496 error="method DELETE not allowed" +30:47.368 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/list from=127.0.0.1:36496 latency=148.425µs +30:47.369 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/list from=127.0.0.1:36498 error="method HEAD not allowed" +30:47.369 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/list from=127.0.0.1:36498 latency=20.768µs +30:47.369 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/list from=127.0.0.1:36498 latency=2.933µs +30:47.370 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/rules/translate from=127.0.0.1:36498 error="method GET not allowed" +30:47.370 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/rules/translate from=127.0.0.1:36498 latency=113.106µs +30:47.370 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/rules/translate from=127.0.0.1:36500 error="method PUT not allowed" +30:47.370 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/rules/translate from=127.0.0.1:36500 latency=97.942µs +30:47.371 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/rules/translate from=127.0.0.1:36502 error="Permission denied" +30:47.371 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/rules/translate from=127.0.0.1:36502 latency=168.981µs +30:47.371 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/rules/translate from=127.0.0.1:36504 error="method DELETE not allowed" +30:47.372 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/rules/translate from=127.0.0.1:36504 latency=104.89µs +30:47.376 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/rules/translate from=127.0.0.1:36506 error="method HEAD not allowed" +30:47.376 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/rules/translate from=127.0.0.1:36506 latency=22.948µs +30:47.376 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/rules/translate from=127.0.0.1:36506 latency=3.185µs +30:47.376 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:36506 error="Permission denied" +30:47.376 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:36506 latency=187.016µs +30:47.377 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/self from=127.0.0.1:36508 error="method PUT not allowed" +30:47.377 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/self from=127.0.0.1:36508 latency=100.891µs +30:47.378 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/self from=127.0.0.1:36510 error="method POST not allowed" +30:47.378 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/self from=127.0.0.1:36510 latency=109.839µs +30:47.378 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/self from=127.0.0.1:36512 error="method DELETE not allowed" +30:47.378 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/self from=127.0.0.1:36512 latency=106.072µs +30:47.379 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/self from=127.0.0.1:36514 error="method HEAD not allowed" +30:47.379 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/self from=127.0.0.1:36514 latency=20.227µs +30:47.379 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/self from=127.0.0.1:36514 latency=2.558µs +30:47.379 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role from=127.0.0.1:36514 error="method GET not allowed" +30:47.379 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role from=127.0.0.1:36514 latency=108.412µs +30:47.384 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role from=127.0.0.1:36516 error="Bad request: Role decoding failed: EOF" +30:47.384 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36516 latency=167.209µs +30:47.387 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role from=127.0.0.1:36518 error="method POST not allowed" +30:47.387 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role from=127.0.0.1:36518 latency=122.526µs +30:47.388 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role from=127.0.0.1:36520 error="method DELETE not allowed" +30:47.388 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role from=127.0.0.1:36520 latency=148.709µs +30:47.392 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role from=127.0.0.1:36522 error="method HEAD not allowed" +30:47.392 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role from=127.0.0.1:36522 latency=26.723µs +30:47.392 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role from=127.0.0.1:36522 latency=3.241µs +30:47.393 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/tokens from=127.0.0.1:36522 error="Permission denied" +30:47.393 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:36522 latency=260.44µs +30:47.393 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/tokens from=127.0.0.1:36524 error="method PUT not allowed" +30:47.393 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/tokens from=127.0.0.1:36524 latency=99.091µs +30:47.394 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/tokens from=127.0.0.1:36526 error="method POST not allowed" +30:47.394 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/tokens from=127.0.0.1:36526 latency=177.922µs +30:47.395 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/tokens from=127.0.0.1:36528 error="method DELETE not allowed" +30:47.395 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/tokens from=127.0.0.1:36528 latency=170.471µs +30:47.396 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/tokens from=127.0.0.1:36530 error="method HEAD not allowed" +30:47.396 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/tokens from=127.0.0.1:36530 latency=51.674µs +30:47.396 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/tokens from=127.0.0.1:36530 latency=2.308µs +30:47.397 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36530 latency=276.9µs +30:47.397 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/checks from=127.0.0.1:36532 error="method PUT not allowed" +30:47.397 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/checks from=127.0.0.1:36532 latency=137.307µs +30:47.398 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/checks from=127.0.0.1:36534 error="method POST not allowed" +30:47.398 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/checks from=127.0.0.1:36534 latency=106.31µs +30:47.399 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/checks from=127.0.0.1:36536 error="method DELETE not allowed" +30:47.399 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/checks from=127.0.0.1:36536 latency=297.205µs +30:47.399 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/checks from=127.0.0.1:36538 error="method HEAD not allowed" +30:47.399 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/checks from=127.0.0.1:36538 latency=22.182µs +30:47.400 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/checks from=127.0.0.1:36538 latency=2.499µs +30:47.400 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/federation-state/ from=127.0.0.1:36538 error="Bad request: Missing datacenter name" +30:47.400 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-state/ from=127.0.0.1:36538 latency=129.629µs +30:47.404 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-state/ from=127.0.0.1:36540 error="method PUT not allowed" +30:47.404 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-state/ from=127.0.0.1:36540 latency=190.041µs +30:47.405 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-state/ from=127.0.0.1:36542 error="method POST not allowed" +30:47.405 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-state/ from=127.0.0.1:36542 latency=155.724µs +30:47.407 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-state/ from=127.0.0.1:36544 error="method DELETE not allowed" +30:47.407 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-state/ from=127.0.0.1:36544 latency=136.844µs +30:47.408 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-state/ from=127.0.0.1:36546 error="method HEAD not allowed" +30:47.408 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-state/ from=127.0.0.1:36546 latency=26.954µs +30:47.409 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-state/ from=127.0.0.1:36546 latency=2.28µs +30:47.412 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:36546 error="Permission denied" +30:47.412 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:36546 latency=292.793µs +30:47.413 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/autopilot/configuration from=127.0.0.1:36548 error="Bad request: Error parsing autopilot config: EOF" +30:47.413 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/configuration from=127.0.0.1:36548 latency=217.837µs +30:47.413 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/autopilot/configuration from=127.0.0.1:36550 error="method POST not allowed" +30:47.414 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/autopilot/configuration from=127.0.0.1:36550 latency=116.259µs +30:47.414 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/autopilot/configuration from=127.0.0.1:36552 error="method DELETE not allowed" +30:47.415 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/autopilot/configuration from=127.0.0.1:36552 latency=140.273µs +30:47.415 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/autopilot/configuration from=127.0.0.1:36554 error="method HEAD not allowed" +30:47.415 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/autopilot/configuration from=127.0.0.1:36554 latency=25.115µs +30:47.416 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/autopilot/configuration from=127.0.0.1:36554 latency=3.573µs +30:47.419 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36554 latency=268.069µs +30:47.419 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36556 error="method PUT not allowed" +30:47.420 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36556 latency=225.201µs +30:47.420 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36558 error="method POST not allowed" +30:47.421 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36558 latency=152.401µs +30:47.421 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36560 error="method DELETE not allowed" +30:47.421 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36560 latency=207.782µs +30:47.422 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36562 error="method HEAD not allowed" +30:47.422 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36562 latency=25.944µs +30:47.422 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-states/mesh-gateways from=127.0.0.1:36562 latency=2.362µs +30:47.422 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role/ from=127.0.0.1:36562 error="Bad request: Missing role ID" +30:47.422 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/ from=127.0.0.1:36562 latency=114.537µs +30:47.423 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role/ from=127.0.0.1:36564 error="Bad request: Role decoding failed: EOF" +30:47.423 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/ from=127.0.0.1:36564 latency=153.337µs +30:47.424 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role/ from=127.0.0.1:36566 error="method POST not allowed" +30:47.424 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role/ from=127.0.0.1:36566 latency=139.219µs +30:47.425 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role/ from=127.0.0.1:36568 error="Bad request: Missing role ID" +30:47.425 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/ from=127.0.0.1:36568 latency=135.226µs +30:47.426 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role/ from=127.0.0.1:36570 error="method HEAD not allowed" +30:47.426 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role/ from=127.0.0.1:36570 latency=25.337µs +30:47.426 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role/ from=127.0.0.1:36570 latency=2.714µs +30:47.426 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/health/service/name/ from=127.0.0.1:36570 error="Bad request: Missing service Name" +30:47.426 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/health/service/name/ from=127.0.0.1:36570 latency=115.01µs +30:47.427 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/health/service/name/ from=127.0.0.1:36572 error="method PUT not allowed" +30:47.427 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/health/service/name/ from=127.0.0.1:36572 latency=138.242µs +30:47.428 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/health/service/name/ from=127.0.0.1:36574 error="method POST not allowed" +30:47.428 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/health/service/name/ from=127.0.0.1:36574 latency=149.168µs +30:47.428 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/health/service/name/ from=127.0.0.1:36576 error="method DELETE not allowed" +30:47.428 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/health/service/name/ from=127.0.0.1:36576 latency=106.87µs +30:47.429 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/health/service/name/ from=127.0.0.1:36578 error="method HEAD not allowed" +30:47.429 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/health/service/name/ from=127.0.0.1:36578 latency=19.628µs +30:47.429 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/health/service/name/ from=127.0.0.1:36578 latency=2.591µs +30:47.430 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/datacenters from=127.0.0.1:36578 latency=325.142µs +30:47.430 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/datacenters from=127.0.0.1:36580 error="method PUT not allowed" +30:47.430 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/datacenters from=127.0.0.1:36580 latency=112.401µs +30:47.440 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/datacenters from=127.0.0.1:36582 error="method POST not allowed" +30:47.440 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/datacenters from=127.0.0.1:36582 latency=116.337µs +30:47.441 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/datacenters from=127.0.0.1:36584 error="method DELETE not allowed" +30:47.441 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/datacenters from=127.0.0.1:36584 latency=125.158µs +30:47.441 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/datacenters from=127.0.0.1:36586 error="method HEAD not allowed" +30:47.441 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/datacenters from=127.0.0.1:36586 latency=20.059µs +30:47.441 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/datacenters from=127.0.0.1:36586 latency=2.527µs +30:47.442 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/info/ from=127.0.0.1:36586 latency=131.632µs +30:47.442 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/info/ from=127.0.0.1:36588 error="method PUT not allowed" +30:47.443 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/info/ from=127.0.0.1:36588 latency=172.703µs +30:47.443 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/info/ from=127.0.0.1:36590 error="method POST not allowed" +30:47.443 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/info/ from=127.0.0.1:36590 latency=124.367µs +30:47.448 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/info/ from=127.0.0.1:36592 error="method DELETE not allowed" +30:47.448 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/info/ from=127.0.0.1:36592 latency=152.438µs +30:47.449 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/info/ from=127.0.0.1:36594 error="method HEAD not allowed" +30:47.449 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/info/ from=127.0.0.1:36594 latency=24.342µs +30:47.449 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/info/ from=127.0.0.1:36594 latency=2.59µs +30:47.449 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/update/ from=127.0.0.1:36594 error="method GET not allowed" +30:47.450 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/update/ from=127.0.0.1:36594 latency=110.769µs +30:47.450 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/update/ from=127.0.0.1:36596 latency=146.469µs +30:47.451 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/update/ from=127.0.0.1:36598 error="method POST not allowed" +30:47.451 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/update/ from=127.0.0.1:36598 latency=97.159µs +30:47.451 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/update/ from=127.0.0.1:36600 error="method DELETE not allowed" +30:47.451 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/update/ from=127.0.0.1:36600 latency=85.345µs +30:47.452 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/update/ from=127.0.0.1:36602 error="method HEAD not allowed" +30:47.452 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/update/ from=127.0.0.1:36602 latency=22.28µs +30:47.452 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/update/ from=127.0.0.1:36602 latency=3.166µs +30:47.452 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config from=127.0.0.1:36602 error="method GET not allowed" +30:47.452 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config from=127.0.0.1:36602 latency=137.152µs +30:47.453 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/config from=127.0.0.1:36604 error="Bad request: Request decoding failed: EOF" +30:47.453 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:36604 latency=135.623µs +30:47.454 [ERROR] TestAgent.http: Request error: method=POST url=/v1/config from=127.0.0.1:36606 error="method POST not allowed" +30:47.454 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/config from=127.0.0.1:36606 latency=146.289µs +30:47.455 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/config from=127.0.0.1:36608 error="method DELETE not allowed" +30:47.455 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config from=127.0.0.1:36608 latency=155.834µs +30:47.456 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/config from=127.0.0.1:36610 error="method HEAD not allowed" +30:47.456 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/config from=127.0.0.1:36610 latency=57.22µs +30:47.457 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/config from=127.0.0.1:36610 latency=2.432µs +30:47.457 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.457 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/services from=127.0.0.1:36610 latency=391.374µs +30:47.458 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/services from=127.0.0.1:36612 error="method PUT not allowed" +30:47.458 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/services from=127.0.0.1:36612 latency=162.258µs +30:47.458 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/services from=127.0.0.1:36614 error="method POST not allowed" +30:47.459 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/services from=127.0.0.1:36614 latency=154.95µs +30:47.459 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/services from=127.0.0.1:36616 error="method DELETE not allowed" +30:47.460 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/services from=127.0.0.1:36616 latency=161.176µs +30:47.460 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/services from=127.0.0.1:36618 error="method HEAD not allowed" +30:47.460 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/services from=127.0.0.1:36618 latency=20.795µs +30:47.461 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/services from=127.0.0.1:36618 latency=2.895µs +30:47.461 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/health/service/id/ from=127.0.0.1:36618 error="Bad request: Missing serviceID" +30:47.461 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/health/service/id/ from=127.0.0.1:36618 latency=134.334µs +30:47.462 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/health/service/id/ from=127.0.0.1:36620 error="method PUT not allowed" +30:47.462 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/health/service/id/ from=127.0.0.1:36620 latency=95.866µs +30:47.463 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/health/service/id/ from=127.0.0.1:36622 error="method POST not allowed" +30:47.463 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/health/service/id/ from=127.0.0.1:36622 latency=136.823µs +30:47.464 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/health/service/id/ from=127.0.0.1:36624 error="method DELETE not allowed" +30:47.464 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/health/service/id/ from=127.0.0.1:36624 latency=157µs +30:47.465 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/health/service/id/ from=127.0.0.1:36626 error="method HEAD not allowed" +30:47.465 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/health/service/id/ from=127.0.0.1:36626 latency=21.489µs +30:47.465 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/health/service/id/ from=127.0.0.1:36626 latency=2.51µs +30:47.466 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/ from=127.0.0.1:36626 latency=113.228µs +30:47.466 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/node/ from=127.0.0.1:36628 error="method PUT not allowed" +30:47.466 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/node/ from=127.0.0.1:36628 latency=129.113µs +30:47.467 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/node/ from=127.0.0.1:36630 error="method POST not allowed" +30:47.467 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/node/ from=127.0.0.1:36630 latency=138.589µs +30:47.468 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/node/ from=127.0.0.1:36632 error="method DELETE not allowed" +30:47.468 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/node/ from=127.0.0.1:36632 latency=143.712µs +30:47.468 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/node/ from=127.0.0.1:36634 error="method HEAD not allowed" +30:47.468 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/node/ from=127.0.0.1:36634 latency=21.084µs +30:47.469 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/node/ from=127.0.0.1:36634 latency=3.018µs +30:47.469 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:36634 latency=285.656µs +30:47.470 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/datacenters from=127.0.0.1:36636 error="method PUT not allowed" +30:47.470 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/datacenters from=127.0.0.1:36636 latency=169.415µs +30:47.471 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/datacenters from=127.0.0.1:36638 error="method POST not allowed" +30:47.471 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/datacenters from=127.0.0.1:36638 latency=131.219µs +30:47.472 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/datacenters from=127.0.0.1:36640 error="method DELETE not allowed" +30:47.472 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/datacenters from=127.0.0.1:36640 latency=119.148µs +30:47.476 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/datacenters from=127.0.0.1:36642 error="method HEAD not allowed" +30:47.476 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/datacenters from=127.0.0.1:36642 latency=79.049µs +30:47.477 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/datacenters from=127.0.0.1:36642 latency=3.048µs +30:47.478 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/ from=127.0.0.1:36642 latency=125.966µs +30:47.478 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/info/ from=127.0.0.1:36644 error="method PUT not allowed" +30:47.478 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/info/ from=127.0.0.1:36644 latency=113.829µs +30:47.479 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/info/ from=127.0.0.1:36646 error="method POST not allowed" +30:47.479 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/info/ from=127.0.0.1:36646 latency=122.509µs +30:47.480 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/info/ from=127.0.0.1:36648 error="method DELETE not allowed" +30:47.480 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/info/ from=127.0.0.1:36648 latency=217.858µs +30:47.485 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/info/ from=127.0.0.1:36650 error="method HEAD not allowed" +30:47.485 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/info/ from=127.0.0.1:36650 latency=29.729µs +30:47.485 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/info/ from=127.0.0.1:36650 latency=3.087µs +30:47.486 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-methods from=127.0.0.1:36650 error="Permission denied" +30:47.486 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:36650 latency=306.829µs +30:47.487 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-methods from=127.0.0.1:36652 error="method PUT not allowed" +30:47.487 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-methods from=127.0.0.1:36652 latency=134.869µs +30:47.488 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-methods from=127.0.0.1:36654 error="method POST not allowed" +30:47.488 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-methods from=127.0.0.1:36654 latency=145.471µs +30:47.488 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-methods from=127.0.0.1:36656 error="method DELETE not allowed" +30:47.488 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-methods from=127.0.0.1:36656 latency=116.201µs +30:47.489 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-methods from=127.0.0.1:36658 error="method HEAD not allowed" +30:47.489 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-methods from=127.0.0.1:36658 latency=20.849µs +30:47.489 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-methods from=127.0.0.1:36658 latency=2.837µs +30:47.490 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/deregister/ from=127.0.0.1:36658 error="method GET not allowed" +30:47.490 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/deregister/ from=127.0.0.1:36658 latency=127.783µs +30:47.490 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/deregister/ from=127.0.0.1:36660 error="Unknown check """ +30:47.490 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/deregister/ from=127.0.0.1:36660 latency=197.355µs +30:47.491 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/deregister/ from=127.0.0.1:36662 error="method POST not allowed" +30:47.491 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/deregister/ from=127.0.0.1:36662 latency=127.102µs +30:47.492 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/deregister/ from=127.0.0.1:36664 error="method DELETE not allowed" +30:47.492 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/deregister/ from=127.0.0.1:36664 latency=166.837µs +30:47.493 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/deregister/ from=127.0.0.1:36666 error="method HEAD not allowed" +30:47.493 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/deregister/ from=127.0.0.1:36666 latency=22.331µs +30:47.493 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/deregister/ from=127.0.0.1:36666 latency=2.108µs +30:47.494 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service=consul +30:47.494 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:36666 latency=298.396µs +30:47.494 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/services from=127.0.0.1:36670 error="method PUT not allowed" +30:47.494 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/services from=127.0.0.1:36670 latency=86.327µs +30:47.495 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/services from=127.0.0.1:36672 error="method POST not allowed" +30:47.495 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/services from=127.0.0.1:36672 latency=148.734µs +30:47.496 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/services from=127.0.0.1:36674 error="method DELETE not allowed" +30:47.496 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/services from=127.0.0.1:36674 latency=174.957µs +30:47.496 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/services from=127.0.0.1:36676 error="method HEAD not allowed" +30:47.496 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/services from=127.0.0.1:36676 latency=23.615µs +30:47.497 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/services from=127.0.0.1:36676 latency=3.068µs +30:47.497 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/roles from=127.0.0.1:36676 error="Permission denied" +30:47.497 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:36676 latency=237.436µs +30:47.498 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/roles from=127.0.0.1:36678 error="method PUT not allowed" +30:47.498 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/roles from=127.0.0.1:36678 latency=127.522µs +30:47.498 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/roles from=127.0.0.1:36680 error="method POST not allowed" +30:47.499 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/roles from=127.0.0.1:36680 latency=110.602µs +30:47.499 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/roles from=127.0.0.1:36682 error="method DELETE not allowed" +30:47.499 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/roles from=127.0.0.1:36682 latency=137.312µs +30:47.500 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/roles from=127.0.0.1:36684 error="method HEAD not allowed" +30:47.500 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/roles from=127.0.0.1:36684 latency=16.893µs +30:47.506 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/roles from=127.0.0.1:36684 latency=4.033µs +30:47.506 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/join/ from=127.0.0.1:36684 error="method GET not allowed" +30:47.506 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/join/ from=127.0.0.1:36684 latency=125.204µs +30:47.507 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/join/ from=127.0.0.1:36686 error="Permission denied" +30:47.507 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/ from=127.0.0.1:36686 latency=211.496µs +30:47.507 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/join/ from=127.0.0.1:36688 error="method POST not allowed" +30:47.508 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/join/ from=127.0.0.1:36688 latency=135.613µs +30:47.512 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/join/ from=127.0.0.1:36690 error="method DELETE not allowed" +30:47.512 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/join/ from=127.0.0.1:36690 latency=160.267µs +30:47.513 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/join/ from=127.0.0.1:36692 error="method HEAD not allowed" +30:47.513 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/join/ from=127.0.0.1:36692 latency=20.587µs +30:47.513 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/join/ from=127.0.0.1:36692 latency=2.997µs +30:47.513 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/fail/ from=127.0.0.1:36692 error="method GET not allowed" +30:47.514 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/fail/ from=127.0.0.1:36692 latency=106.662µs +30:47.514 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/fail/ from=127.0.0.1:36694 error="Unknown check """ +30:47.514 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/fail/ from=127.0.0.1:36694 latency=178.855µs +30:47.515 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/fail/ from=127.0.0.1:36696 error="method POST not allowed" +30:47.515 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/fail/ from=127.0.0.1:36696 latency=101.498µs +30:47.515 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/fail/ from=127.0.0.1:36698 error="method DELETE not allowed" +30:47.515 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/fail/ from=127.0.0.1:36698 latency=106.099µs +30:47.520 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/fail/ from=127.0.0.1:36700 error="method HEAD not allowed" +30:47.520 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/fail/ from=127.0.0.1:36700 latency=27.84µs +30:47.520 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/fail/ from=127.0.0.1:36700 latency=2.553µs +30:47.521 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36700 latency=129.248µs +30:47.522 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36702 error="method PUT not allowed" +30:47.522 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36702 latency=151.109µs +30:47.522 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36704 error="method POST not allowed" +30:47.522 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36704 latency=152.895µs +30:47.523 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36706 error="method DELETE not allowed" +30:47.523 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36706 latency=139.427µs +30:47.524 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36708 error="method HEAD not allowed" +30:47.524 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36708 latency=23.545µs +30:47.524 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/gateway-services-nodes/ from=127.0.0.1:36708 latency=2.73µs +30:47.524 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/connect/authorize from=127.0.0.1:36708 error="method GET not allowed" +30:47.525 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/authorize from=127.0.0.1:36708 latency=116.159µs +30:47.525 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/authorize from=127.0.0.1:36710 error="method PUT not allowed" +30:47.525 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/authorize from=127.0.0.1:36710 latency=106.905µs +30:47.526 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:36712 error="Bad request: Request decode failed: EOF" +30:47.526 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:36712 latency=165.178µs +30:47.527 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/authorize from=127.0.0.1:36714 error="method DELETE not allowed" +30:47.527 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/authorize from=127.0.0.1:36714 latency=113.133µs +30:47.531 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/authorize from=127.0.0.1:36716 error="method HEAD not allowed" +30:47.531 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/authorize from=127.0.0.1:36716 latency=26.113µs +30:47.533 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/authorize from=127.0.0.1:36716 latency=3.71µs +30:47.534 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:36716 latency=370.951µs +30:47.534 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions from=127.0.0.1:36718 error="method PUT not allowed" +30:47.534 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions from=127.0.0.1:36718 latency=104.631µs +30:47.535 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions from=127.0.0.1:36720 error="Failed to decode request body: EOF" +30:47.535 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:36720 latency=105.012µs +30:47.535 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions from=127.0.0.1:36722 error="method DELETE not allowed" +30:47.535 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions from=127.0.0.1:36722 latency=95.189µs +30:47.536 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions from=127.0.0.1:36724 error="method HEAD not allowed" +30:47.536 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions from=127.0.0.1:36724 latency=20.843µs +30:47.536 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions from=127.0.0.1:36724 latency=2.523µs +30:47.536 [ERROR] TestAgent.http: Request error: method=GET url=/v1/event/fire/ from=127.0.0.1:36724 error="method GET not allowed" +30:47.536 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/event/fire/ from=127.0.0.1:36724 latency=132.099µs +30:47.537 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/ from=127.0.0.1:36726 latency=100.463µs +30:47.538 [ERROR] TestAgent.http: Request error: method=POST url=/v1/event/fire/ from=127.0.0.1:36728 error="method POST not allowed" +30:47.538 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/event/fire/ from=127.0.0.1:36728 latency=120.835µs +30:47.538 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/event/fire/ from=127.0.0.1:36730 error="method DELETE not allowed" +30:47.539 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/event/fire/ from=127.0.0.1:36730 latency=133.234µs +30:47.539 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/event/fire/ from=127.0.0.1:36732 error="method HEAD not allowed" +30:47.539 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/event/fire/ from=127.0.0.1:36732 latency=20.912µs +30:47.539 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/event/fire/ from=127.0.0.1:36732 latency=2.496µs +30:47.540 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36732 latency=238.629µs +30:47.540 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/nodes from=127.0.0.1:36734 error="method PUT not allowed" +30:47.540 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/nodes from=127.0.0.1:36734 latency=99.166µs +30:47.541 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/nodes from=127.0.0.1:36736 error="method POST not allowed" +30:47.541 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/nodes from=127.0.0.1:36736 latency=97.548µs +30:47.541 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/nodes from=127.0.0.1:36738 error="method DELETE not allowed" +30:47.542 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/nodes from=127.0.0.1:36738 latency=97.56µs +30:47.542 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/nodes from=127.0.0.1:36740 error="method HEAD not allowed" +30:47.542 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/nodes from=127.0.0.1:36740 latency=18.219µs +30:47.542 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/nodes from=127.0.0.1:36740 latency=3.027µs +30:47.543 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/checks/ from=127.0.0.1:36740 latency=120.381µs +30:47.543 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/checks/ from=127.0.0.1:36742 error="method PUT not allowed" +30:47.543 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/checks/ from=127.0.0.1:36742 latency=154.404µs +30:47.544 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/checks/ from=127.0.0.1:36744 error="method POST not allowed" +30:47.544 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/checks/ from=127.0.0.1:36744 latency=135.449µs +30:47.545 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/checks/ from=127.0.0.1:36746 error="method DELETE not allowed" +30:47.545 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/checks/ from=127.0.0.1:36746 latency=87.68µs +30:47.545 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/checks/ from=127.0.0.1:36748 error="method HEAD not allowed" +30:47.545 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/checks/ from=127.0.0.1:36748 latency=17.725µs +30:47.546 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/checks/ from=127.0.0.1:36748 latency=2.334µs +30:47.546 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/host from=127.0.0.1:36748 error="Permission denied" +30:47.546 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/host from=127.0.0.1:36748 latency=225.399µs +30:47.547 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/host from=127.0.0.1:36750 error="method PUT not allowed" +30:47.547 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/host from=127.0.0.1:36750 latency=125.384µs +30:47.547 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/host from=127.0.0.1:36752 error="method POST not allowed" +30:47.547 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/host from=127.0.0.1:36752 latency=127.026µs +30:47.548 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/host from=127.0.0.1:36754 error="method DELETE not allowed" +30:47.548 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/host from=127.0.0.1:36754 latency=124.804µs +30:47.549 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/host from=127.0.0.1:36756 error="method HEAD not allowed" +30:47.549 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/host from=127.0.0.1:36756 latency=20.583µs +30:47.549 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/host from=127.0.0.1:36756 latency=2.86µs +30:47.549 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rule/ from=127.0.0.1:36756 error="Bad request: Missing binding rule ID" +30:47.550 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/ from=127.0.0.1:36756 latency=132.01µs +30:47.550 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/ from=127.0.0.1:36758 error="Bad request: BindingRule decoding failed: EOF" +30:47.550 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/ from=127.0.0.1:36758 latency=155.18µs +30:47.551 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rule/ from=127.0.0.1:36760 error="method POST not allowed" +30:47.551 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rule/ from=127.0.0.1:36760 latency=104.184µs +30:47.551 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rule/ from=127.0.0.1:36762 error="Bad request: Missing binding rule ID" +30:47.551 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/ from=127.0.0.1:36762 latency=97.104µs +30:47.552 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rule/ from=127.0.0.1:36764 error="method HEAD not allowed" +30:47.552 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rule/ from=127.0.0.1:36764 latency=18.21µs +30:47.552 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rule/ from=127.0.0.1:36764 latency=2.464µs +30:47.552 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/node/ from=127.0.0.1:36764 latency=107.604µs +30:47.553 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/node/ from=127.0.0.1:36766 error="method PUT not allowed" +30:47.553 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/node/ from=127.0.0.1:36766 latency=139.28µs +30:47.554 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/node/ from=127.0.0.1:36768 error="method POST not allowed" +30:47.554 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/node/ from=127.0.0.1:36768 latency=122.374µs +30:47.555 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/node/ from=127.0.0.1:36770 error="method DELETE not allowed" +30:47.555 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/node/ from=127.0.0.1:36770 latency=114.608µs +30:47.555 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/node/ from=127.0.0.1:36772 error="method HEAD not allowed" +30:47.555 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/node/ from=127.0.0.1:36772 latency=15.62µs +30:47.555 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/node/ from=127.0.0.1:36772 latency=2.091µs +30:47.556 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/destroy/ from=127.0.0.1:36772 error="method GET not allowed" +30:47.556 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/destroy/ from=127.0.0.1:36772 latency=144.231µs +30:47.564 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/destroy/ from=127.0.0.1:36774 latency=99.439µs +30:47.565 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/destroy/ from=127.0.0.1:36776 error="method POST not allowed" +30:47.565 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/destroy/ from=127.0.0.1:36776 latency=104.079µs +30:47.565 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/destroy/ from=127.0.0.1:36778 error="method DELETE not allowed" +30:47.565 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/destroy/ from=127.0.0.1:36778 latency=173.927µs +30:47.566 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/destroy/ from=127.0.0.1:36780 error="method HEAD not allowed" +30:47.566 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/destroy/ from=127.0.0.1:36780 latency=24.524µs +30:47.566 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/destroy/ from=127.0.0.1:36780 latency=3.005µs +30:47.567 [ERROR] TestAgent.http: Request error: method=GET url=/v1/coordinate/node/ from=127.0.0.1:36780 error="Permission denied" +30:47.567 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/node/ from=127.0.0.1:36780 latency=280.322µs +30:47.568 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/coordinate/node/ from=127.0.0.1:36782 error="method PUT not allowed" +30:47.568 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/node/ from=127.0.0.1:36782 latency=109.55µs +30:47.569 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/node/ from=127.0.0.1:36784 error="method POST not allowed" +30:47.569 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/node/ from=127.0.0.1:36784 latency=168.32µs +30:47.570 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/node/ from=127.0.0.1:36786 error="method DELETE not allowed" +30:47.570 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/node/ from=127.0.0.1:36786 latency=184.558µs +30:47.571 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/node/ from=127.0.0.1:36788 error="method HEAD not allowed" +30:47.571 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/node/ from=127.0.0.1:36788 latency=25.137µs +30:47.571 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/node/ from=127.0.0.1:36788 latency=3.003µs +30:47.571 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/node/ from=127.0.0.1:36788 latency=135.666µs +30:47.572 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/node/ from=127.0.0.1:36790 error="method PUT not allowed" +30:47.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/node/ from=127.0.0.1:36790 latency=123.592µs +30:47.573 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/node/ from=127.0.0.1:36792 error="method POST not allowed" +30:47.573 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/node/ from=127.0.0.1:36792 latency=123.832µs +30:47.574 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/node/ from=127.0.0.1:36794 error="method DELETE not allowed" +30:47.574 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/node/ from=127.0.0.1:36794 latency=115.392µs +30:47.574 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/node/ from=127.0.0.1:36796 error="method HEAD not allowed" +30:47.574 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/node/ from=127.0.0.1:36796 latency=20.474µs +30:47.574 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/node/ from=127.0.0.1:36796 latency=2.481µs +30:47.575 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-method from=127.0.0.1:36796 error="method GET not allowed" +30:47.575 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method from=127.0.0.1:36796 latency=97.448µs +30:47.575 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method from=127.0.0.1:36798 error="Bad request: AuthMethod decoding failed: EOF" +30:47.576 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:36798 latency=147.566µs +30:47.577 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-method from=127.0.0.1:36800 error="method POST not allowed" +30:47.577 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-method from=127.0.0.1:36800 latency=169.835µs +30:47.577 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-method from=127.0.0.1:36802 error="method DELETE not allowed" +30:47.578 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method from=127.0.0.1:36802 latency=145.521µs +30:47.578 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-method from=127.0.0.1:36804 error="method HEAD not allowed" +30:47.578 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-method from=127.0.0.1:36804 latency=19.693µs +30:47.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-method from=127.0.0.1:36804 latency=2.669µs +30:47.579 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/maintenance/ from=127.0.0.1:36804 error="method GET not allowed" +30:47.579 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/maintenance/ from=127.0.0.1:36804 latency=123.741µs +30:47.579 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/ from=127.0.0.1:36806 latency=131.575µs +30:47.580 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/maintenance/ from=127.0.0.1:36808 error="method POST not allowed" +30:47.580 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/maintenance/ from=127.0.0.1:36808 latency=130.92µs +30:47.581 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/maintenance/ from=127.0.0.1:36810 error="method DELETE not allowed" +30:47.581 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/maintenance/ from=127.0.0.1:36810 latency=126.509µs +30:47.582 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/maintenance/ from=127.0.0.1:36812 error="method HEAD not allowed" +30:47.582 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/maintenance/ from=127.0.0.1:36812 latency=23.253µs +30:47.582 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/maintenance/ from=127.0.0.1:36812 latency=2.679µs +30:47.582 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/connect/ from=127.0.0.1:36812 latency=117.167µs +30:47.583 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/connect/ from=127.0.0.1:36814 error="method PUT not allowed" +30:47.583 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/connect/ from=127.0.0.1:36814 latency=150.379µs +30:47.584 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/connect/ from=127.0.0.1:36816 error="method POST not allowed" +30:47.584 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/connect/ from=127.0.0.1:36816 latency=120.383µs +30:47.584 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/connect/ from=127.0.0.1:36818 error="method DELETE not allowed" +30:47.584 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/connect/ from=127.0.0.1:36818 latency=104.981µs +30:47.585 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/connect/ from=127.0.0.1:36820 error="method HEAD not allowed" +30:47.585 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/connect/ from=127.0.0.1:36820 latency=24.132µs +30:47.585 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/connect/ from=127.0.0.1:36820 latency=2.63µs +30:47.586 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/ from=127.0.0.1:36820 latency=96.238µs +30:47.586 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/ from=127.0.0.1:36822 latency=114.928µs +30:47.587 [ERROR] TestAgent.http: Request error: method=POST url=/v1/kv/ from=127.0.0.1:36824 error="method POST not allowed" +30:47.587 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/kv/ from=127.0.0.1:36824 latency=141.171µs +30:47.587 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/ from=127.0.0.1:36826 latency=118.282µs +30:47.588 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/kv/ from=127.0.0.1:36828 error="method HEAD not allowed" +30:47.588 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/kv/ from=127.0.0.1:36828 latency=23.418µs +30:47.588 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/kv/ from=127.0.0.1:36828 latency=2.907µs +30:47.589 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/logout from=127.0.0.1:36828 error="method GET not allowed" +30:47.589 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/logout from=127.0.0.1:36828 latency=120.225µs +30:47.589 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/logout from=127.0.0.1:36830 error="method PUT not allowed" +30:47.589 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/logout from=127.0.0.1:36830 latency=110.807µs +30:47.590 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:36832 error="ACL not found" +30:47.590 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:36832 latency=129.391µs +30:47.591 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/logout from=127.0.0.1:36834 error="method DELETE not allowed" +30:47.591 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/logout from=127.0.0.1:36834 latency=165.314µs +30:47.591 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/logout from=127.0.0.1:36836 error="method HEAD not allowed" +30:47.591 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/logout from=127.0.0.1:36836 latency=20.451µs +30:47.592 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/logout from=127.0.0.1:36836 latency=2.968µs +30:47.592 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policies from=127.0.0.1:36836 error="Permission denied" +30:47.592 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:36836 latency=286.981µs +30:47.593 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policies from=127.0.0.1:36838 error="method PUT not allowed" +30:47.593 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policies from=127.0.0.1:36838 latency=256.262µs +30:47.594 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policies from=127.0.0.1:36840 error="method POST not allowed" +30:47.594 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policies from=127.0.0.1:36840 latency=118.655µs +30:47.595 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policies from=127.0.0.1:36842 error="method DELETE not allowed" +30:47.595 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policies from=127.0.0.1:36842 latency=144.725µs +30:47.595 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policies from=127.0.0.1:36844 error="method HEAD not allowed" +30:47.595 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policies from=127.0.0.1:36844 latency=19.77µs +30:47.596 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policies from=127.0.0.1:36844 latency=2.56µs +30:47.600 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy from=127.0.0.1:36844 error="method GET not allowed" +30:47.600 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy from=127.0.0.1:36844 latency=152.74µs +30:47.601 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy from=127.0.0.1:36846 error="Bad request: Policy decoding failed: EOF" +30:47.601 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36846 latency=120.525µs +30:47.601 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy from=127.0.0.1:36848 error="method POST not allowed" +30:47.601 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy from=127.0.0.1:36848 latency=114.857µs +30:47.602 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy from=127.0.0.1:36850 error="method DELETE not allowed" +30:47.602 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy from=127.0.0.1:36850 latency=117.013µs +30:47.602 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy from=127.0.0.1:36852 error="method HEAD not allowed" +30:47.602 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy from=127.0.0.1:36852 latency=17.972µs +30:47.603 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy from=127.0.0.1:36852 latency=4.553µs +30:47.605 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/roots from=127.0.0.1:36852 latency=428.354µs +30:47.607 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/ca/roots from=127.0.0.1:36854 error="method PUT not allowed" +30:47.607 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/roots from=127.0.0.1:36854 latency=134.319µs +30:47.607 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/ca/roots from=127.0.0.1:36856 error="method POST not allowed" +30:47.607 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/ca/roots from=127.0.0.1:36856 latency=127.546µs +30:47.608 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/ca/roots from=127.0.0.1:36858 error="method DELETE not allowed" +30:47.608 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/ca/roots from=127.0.0.1:36858 latency=84.69µs +30:47.609 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/ca/roots from=127.0.0.1:36860 error="method HEAD not allowed" +30:47.609 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/ca/roots from=127.0.0.1:36860 latency=21.731µs +30:47.609 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/ca/roots from=127.0.0.1:36860 latency=2.883µs +30:47.609 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/ingress/ from=127.0.0.1:36860 latency=119.364µs +30:47.610 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/ingress/ from=127.0.0.1:36862 error="method PUT not allowed" +30:47.610 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/ingress/ from=127.0.0.1:36862 latency=149.201µs +30:47.612 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/ingress/ from=127.0.0.1:36864 error="method POST not allowed" +30:47.612 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/ingress/ from=127.0.0.1:36864 latency=175.02µs +30:47.612 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/ingress/ from=127.0.0.1:36866 error="method DELETE not allowed" +30:47.612 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/ingress/ from=127.0.0.1:36866 latency=142.157µs +30:47.613 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/ingress/ from=127.0.0.1:36868 error="method HEAD not allowed" +30:47.613 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/ingress/ from=127.0.0.1:36868 latency=23.417µs +30:47.614 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/ingress/ from=127.0.0.1:36868 latency=2.936µs +30:47.614 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:36868 error="Permission denied" +30:47.614 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:36868 latency=260.548µs +30:47.615 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/raft/configuration from=127.0.0.1:36870 error="method PUT not allowed" +30:47.615 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/raft/configuration from=127.0.0.1:36870 latency=125.596µs +30:47.616 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/raft/configuration from=127.0.0.1:36872 error="method POST not allowed" +30:47.616 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/raft/configuration from=127.0.0.1:36872 latency=207.68µs +30:47.617 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/configuration from=127.0.0.1:36874 error="method DELETE not allowed" +30:47.617 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/configuration from=127.0.0.1:36874 latency=136.317µs +30:47.617 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/raft/configuration from=127.0.0.1:36876 error="method HEAD not allowed" +30:47.617 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/raft/configuration from=127.0.0.1:36876 latency=70.686µs +30:47.618 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/raft/configuration from=127.0.0.1:36876 latency=2.632µs +30:47.618 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/role/name/ from=127.0.0.1:36876 error="Bad request: Missing role Name" +30:47.618 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/ from=127.0.0.1:36876 latency=177.508µs +30:47.619 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/role/name/ from=127.0.0.1:36878 error="method PUT not allowed" +30:47.619 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/name/ from=127.0.0.1:36878 latency=150.58µs +30:47.620 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/role/name/ from=127.0.0.1:36880 error="method POST not allowed" +30:47.620 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/role/name/ from=127.0.0.1:36880 latency=103.362µs +30:47.620 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/role/name/ from=127.0.0.1:36882 error="method DELETE not allowed" +30:47.621 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/name/ from=127.0.0.1:36882 latency=117.466µs +30:47.621 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/role/name/ from=127.0.0.1:36884 error="method HEAD not allowed" +30:47.621 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/role/name/ from=127.0.0.1:36884 latency=21.27µs +30:47.622 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/role/name/ from=127.0.0.1:36884 latency=2.523µs +30:47.622 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rules from=127.0.0.1:36884 error="Permission denied" +30:47.622 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:36884 latency=229.596µs +30:47.623 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rules from=127.0.0.1:36886 error="method PUT not allowed" +30:47.623 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rules from=127.0.0.1:36886 latency=191.825µs +30:47.623 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rules from=127.0.0.1:36888 error="method POST not allowed" +30:47.624 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rules from=127.0.0.1:36888 latency=142.235µs +30:47.624 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rules from=127.0.0.1:36890 error="method DELETE not allowed" +30:47.624 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rules from=127.0.0.1:36890 latency=117.528µs +30:47.625 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rules from=127.0.0.1:36892 error="method HEAD not allowed" +30:47.625 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rules from=127.0.0.1:36892 latency=22.114µs +30:47.625 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rules from=127.0.0.1:36892 latency=3.117µs +30:47.626 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/acl/authorize from=127.0.0.1:36892 error="method GET not allowed" +30:47.626 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/acl/authorize from=127.0.0.1:36892 latency=143.091µs +30:47.626 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/acl/authorize from=127.0.0.1:36894 error="method PUT not allowed" +30:47.626 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/acl/authorize from=127.0.0.1:36894 latency=110.303µs +30:47.627 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/acl/authorize from=127.0.0.1:36896 error="Bad request: Failed to decode request body: EOF" +30:47.627 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/acl/authorize from=127.0.0.1:36896 latency=121.156µs +30:47.628 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/acl/authorize from=127.0.0.1:36898 error="method DELETE not allowed" +30:47.628 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/acl/authorize from=127.0.0.1:36898 latency=142.475µs +30:47.628 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/acl/authorize from=127.0.0.1:36900 error="method HEAD not allowed" +30:47.629 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/acl/authorize from=127.0.0.1:36900 latency=19.534µs +30:47.629 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/acl/authorize from=127.0.0.1:36900 latency=2.613µs +30:47.629 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ from=127.0.0.1:36900 latency=293.297µs +30:47.630 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/config/ from=127.0.0.1:36902 error="method PUT not allowed" +30:47.630 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config/ from=127.0.0.1:36902 latency=129.449µs +30:47.631 [ERROR] TestAgent.http: Request error: method=POST url=/v1/config/ from=127.0.0.1:36904 error="method POST not allowed" +30:47.631 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/config/ from=127.0.0.1:36904 latency=148.431µs +30:47.632 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config/ from=127.0.0.1:36906 latency=107.18µs +30:47.632 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/config/ from=127.0.0.1:36908 error="method HEAD not allowed" +30:47.632 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/config/ from=127.0.0.1:36908 latency=16.079µs +30:47.633 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/config/ from=127.0.0.1:36908 latency=2.203µs +30:47.633 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/event/list from=127.0.0.1:36908 latency=209.881µs +30:47.634 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/event/list from=127.0.0.1:36910 error="method PUT not allowed" +30:47.634 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/list from=127.0.0.1:36910 latency=99.258µs +30:47.634 [ERROR] TestAgent.http: Request error: method=POST url=/v1/event/list from=127.0.0.1:36912 error="method POST not allowed" +30:47.634 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/event/list from=127.0.0.1:36912 latency=91.778µs +30:47.635 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/event/list from=127.0.0.1:36914 error="method DELETE not allowed" +30:47.635 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/event/list from=127.0.0.1:36914 latency=117.121µs +30:47.636 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/event/list from=127.0.0.1:36916 error="method HEAD not allowed" +30:47.636 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/event/list from=127.0.0.1:36916 latency=21.925µs +30:47.636 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/event/list from=127.0.0.1:36916 latency=3.346µs +30:47.636 [ERROR] TestAgent.http: Request error: method=GET url=/v1/snapshot from=127.0.0.1:36916 error="Permission denied" +30:47.637 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:36916 latency=210.658µs +30:47.637 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:36918 error="Permission denied" +30:47.637 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:36918 latency=215.975µs +30:47.638 [ERROR] TestAgent.http: Request error: method=POST url=/v1/snapshot from=127.0.0.1:36920 error="method POST not allowed" +30:47.638 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/snapshot from=127.0.0.1:36920 latency=119.797µs +30:47.639 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/snapshot from=127.0.0.1:36922 error="method DELETE not allowed" +30:47.639 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/snapshot from=127.0.0.1:36922 latency=114.73µs +30:47.639 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/snapshot from=127.0.0.1:36924 error="method HEAD not allowed" +30:47.639 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/snapshot from=127.0.0.1:36924 latency=21.93µs +30:47.640 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/snapshot from=127.0.0.1:36924 latency=2.568µs +30:47.640 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/metrics from=127.0.0.1:36924 error="Permission denied" +30:47.640 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/metrics from=127.0.0.1:36924 latency=185.845µs +30:47.641 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/metrics from=127.0.0.1:36926 error="method PUT not allowed" +30:47.641 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/metrics from=127.0.0.1:36926 latency=172.263µs +30:47.642 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/metrics from=127.0.0.1:36930 error="method POST not allowed" +30:47.642 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/metrics from=127.0.0.1:36930 latency=135.455µs +30:47.643 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/metrics from=127.0.0.1:36932 error="method DELETE not allowed" +30:47.643 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/metrics from=127.0.0.1:36932 latency=110.262µs +30:47.643 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/metrics from=127.0.0.1:36934 error="method HEAD not allowed" +30:47.643 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/metrics from=127.0.0.1:36934 latency=19.729µs +30:47.644 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/metrics from=127.0.0.1:36934 latency=2.674µs +30:47.644 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:36934 latency=181.002µs +30:47.645 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/services from=127.0.0.1:36936 error="method PUT not allowed" +30:47.645 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/services from=127.0.0.1:36936 latency=104.33µs +30:47.646 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/services from=127.0.0.1:36938 error="method POST not allowed" +30:47.646 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/services from=127.0.0.1:36938 latency=138.575µs +30:47.646 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/services from=127.0.0.1:36940 error="method DELETE not allowed" +30:47.646 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/services from=127.0.0.1:36940 latency=142.004µs +30:47.647 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/services from=127.0.0.1:36942 error="method HEAD not allowed" +30:47.647 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/services from=127.0.0.1:36942 latency=25.484µs +30:47.647 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/services from=127.0.0.1:36942 latency=2.701µs +30:47.648 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/keyring from=127.0.0.1:36942 error="Reading keyring denied by ACLs" +30:47.648 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:36942 latency=238.347µs +30:47.648 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/keyring from=127.0.0.1:36944 error="Bad request: Request decode failed: EOF" +30:47.649 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/keyring from=127.0.0.1:36944 latency=126.89µs +30:47.649 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/keyring from=127.0.0.1:36946 error="Bad request: Request decode failed: EOF" +30:47.649 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/keyring from=127.0.0.1:36946 latency=141.944µs +30:47.650 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/keyring from=127.0.0.1:36948 error="Bad request: Request decode failed: EOF" +30:47.650 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/keyring from=127.0.0.1:36948 latency=112.843µs +30:47.650 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/keyring from=127.0.0.1:36950 error="method HEAD not allowed" +30:47.650 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/keyring from=127.0.0.1:36950 latency=33.808µs +30:47.651 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/keyring from=127.0.0.1:36950 latency=4.08µs +30:47.651 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/create from=127.0.0.1:36950 error="method GET not allowed" +30:47.651 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/create from=127.0.0.1:36950 latency=181.077µs +30:47.652 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/create from=127.0.0.1:36952 error="Permission denied" +30:47.652 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:36952 latency=232.227µs +30:47.653 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/create from=127.0.0.1:36954 error="method POST not allowed" +30:47.653 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/create from=127.0.0.1:36954 latency=232.42µs +30:47.654 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/create from=127.0.0.1:36956 error="method DELETE not allowed" +30:47.654 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/create from=127.0.0.1:36956 latency=120.728µs +30:47.654 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/create from=127.0.0.1:36958 error="method HEAD not allowed" +30:47.654 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/create from=127.0.0.1:36958 latency=30.429µs +30:47.655 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/create from=127.0.0.1:36958 latency=3.321µs +30:47.655 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/update from=127.0.0.1:36958 error="method GET not allowed" +30:47.655 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/update from=127.0.0.1:36958 latency=120.976µs +30:47.656 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/update from=127.0.0.1:36960 latency=128.343µs +30:47.656 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/update from=127.0.0.1:36962 error="method POST not allowed" +30:47.656 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/update from=127.0.0.1:36962 latency=107.463µs +30:47.657 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/update from=127.0.0.1:36964 error="method DELETE not allowed" +30:47.657 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/update from=127.0.0.1:36964 latency=106.015µs +30:47.657 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/update from=127.0.0.1:36966 error="method HEAD not allowed" +30:47.657 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/update from=127.0.0.1:36966 latency=21.23µs +30:47.658 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/update from=127.0.0.1:36966 latency=3.075µs +30:47.658 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/clone/ from=127.0.0.1:36966 error="method GET not allowed" +30:47.658 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/clone/ from=127.0.0.1:36966 latency=133.062µs +30:47.659 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/clone/ from=127.0.0.1:36968 latency=152.164µs +30:47.659 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/clone/ from=127.0.0.1:36970 error="method POST not allowed" +30:47.660 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/clone/ from=127.0.0.1:36970 latency=114.095µs +30:47.660 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/clone/ from=127.0.0.1:36972 error="method DELETE not allowed" +30:47.660 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/clone/ from=127.0.0.1:36972 latency=86.312µs +30:47.661 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/clone/ from=127.0.0.1:36974 error="method HEAD not allowed" +30:47.661 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/clone/ from=127.0.0.1:36974 latency=19.402µs +30:47.661 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/clone/ from=127.0.0.1:36974 latency=2.164µs +30:47.661 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/token/ from=127.0.0.1:36974 error="method GET not allowed" +30:47.661 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/token/ from=127.0.0.1:36974 latency=86.885µs +30:47.662 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/token/ from=127.0.0.1:36976 error="Permission denied" +30:47.662 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/ from=127.0.0.1:36976 latency=204.617µs +30:47.663 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/token/ from=127.0.0.1:36978 error="method POST not allowed" +30:47.663 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/token/ from=127.0.0.1:36978 latency=100.098µs +30:47.663 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/token/ from=127.0.0.1:36980 error="method DELETE not allowed" +30:47.664 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/token/ from=127.0.0.1:36980 latency=131.502µs +30:47.664 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/token/ from=127.0.0.1:36982 error="method HEAD not allowed" +30:47.664 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/token/ from=127.0.0.1:36982 latency=37.395µs +30:47.664 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/token/ from=127.0.0.1:36982 latency=2.619µs +30:47.665 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/ from=127.0.0.1:36982 error="Bad request: Missing token ID" +30:47.665 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/ from=127.0.0.1:36982 latency=108.524µs +30:47.665 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token/ from=127.0.0.1:36984 error="Bad request: Token decoding failed: EOF" +30:47.665 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/ from=127.0.0.1:36984 latency=92.171µs +30:47.666 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token/ from=127.0.0.1:36986 error="method POST not allowed" +30:47.667 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token/ from=127.0.0.1:36986 latency=177.833µs +30:47.667 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token/ from=127.0.0.1:36988 error="Bad request: Missing token ID" +30:47.667 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/ from=127.0.0.1:36988 latency=90.277µs +30:47.668 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token/ from=127.0.0.1:36990 error="method HEAD not allowed" +30:47.668 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token/ from=127.0.0.1:36990 latency=19.682µs +30:47.668 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token/ from=127.0.0.1:36990 latency=5.347µs +30:47.668 [ERROR] TestAgent.http: Request error: method=GET url=/v1/coordinate/update from=127.0.0.1:36990 error="method GET not allowed" +30:47.669 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/update from=127.0.0.1:36990 latency=134.252µs +30:47.669 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/coordinate/update from=127.0.0.1:36992 latency=118.046µs +30:47.670 [ERROR] TestAgent.http: Request error: method=POST url=/v1/coordinate/update from=127.0.0.1:36994 error="method POST not allowed" +30:47.670 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/coordinate/update from=127.0.0.1:36994 latency=134.315µs +30:47.670 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/coordinate/update from=127.0.0.1:36996 error="method DELETE not allowed" +30:47.670 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/coordinate/update from=127.0.0.1:36996 latency=113.856µs +30:47.671 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/coordinate/update from=127.0.0.1:36998 error="method HEAD not allowed" +30:47.671 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/coordinate/update from=127.0.0.1:36998 latency=18.381µs +30:47.671 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/coordinate/update from=127.0.0.1:36998 latency=2.55µs +30:47.672 [ERROR] TestAgent.http: Request error: method=GET url=/v1/internal/federation-states from=127.0.0.1:36998 error="Permission denied" +30:47.673 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/federation-states from=127.0.0.1:36998 latency=309.498µs +30:47.673 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/federation-states from=127.0.0.1:37000 error="method PUT not allowed" +30:47.674 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/federation-states from=127.0.0.1:37000 latency=142.444µs +30:47.674 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/federation-states from=127.0.0.1:37002 error="method POST not allowed" +30:47.674 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/federation-states from=127.0.0.1:37002 latency=127.448µs +30:47.675 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/federation-states from=127.0.0.1:37004 error="method DELETE not allowed" +30:47.675 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/federation-states from=127.0.0.1:37004 latency=113.498µs +30:47.676 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/federation-states from=127.0.0.1:37006 error="method HEAD not allowed" +30:47.676 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/federation-states from=127.0.0.1:37006 latency=16.567µs +30:47.676 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/federation-states from=127.0.0.1:37006 latency=2.352µs +30:47.676 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/rules/translate/ from=127.0.0.1:37006 error="Bad request: Missing token ID" +30:47.676 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/rules/translate/ from=127.0.0.1:37006 latency=101.153µs +30:47.677 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/rules/translate/ from=127.0.0.1:37008 error="method PUT not allowed" +30:47.677 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/rules/translate/ from=127.0.0.1:37008 latency=112.995µs +30:47.678 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/rules/translate/ from=127.0.0.1:37010 error="method POST not allowed" +30:47.678 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/rules/translate/ from=127.0.0.1:37010 latency=101.153µs +30:47.678 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/rules/translate/ from=127.0.0.1:37012 error="method DELETE not allowed" +30:47.678 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/rules/translate/ from=127.0.0.1:37012 latency=146.43µs +30:47.679 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/rules/translate/ from=127.0.0.1:37014 error="method HEAD not allowed" +30:47.679 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/rules/translate/ from=127.0.0.1:37014 latency=29.054µs +30:47.679 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/rules/translate/ from=127.0.0.1:37014 latency=2.897µs +30:47.680 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ from=127.0.0.1:37014 latency=232.688µs +30:47.681 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/service/ from=127.0.0.1:37016 error="method PUT not allowed" +30:47.681 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/ from=127.0.0.1:37016 latency=144.508µs +30:47.683 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/ from=127.0.0.1:37018 error="method POST not allowed" +30:47.684 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/ from=127.0.0.1:37018 latency=124.961µs +30:47.693 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/ from=127.0.0.1:37022 error="method DELETE not allowed" +30:47.693 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/ from=127.0.0.1:37022 latency=162.398µs +30:47.693 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/ from=127.0.0.1:37024 error="method HEAD not allowed" +30:47.693 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/ from=127.0.0.1:37024 latency=20.866µs +30:47.694 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/ from=127.0.0.1:37024 latency=2.538µs +30:47.694 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/ from=127.0.0.1:37024 latency=122.991µs +30:47.695 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/service/ from=127.0.0.1:37026 error="method PUT not allowed" +30:47.695 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/service/ from=127.0.0.1:37026 latency=94.251µs +30:47.695 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/service/ from=127.0.0.1:37028 error="method POST not allowed" +30:47.695 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/service/ from=127.0.0.1:37028 latency=133.489µs +30:47.698 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/service/ from=127.0.0.1:37030 error="method DELETE not allowed" +30:47.698 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/service/ from=127.0.0.1:37030 latency=163.377µs +30:47.700 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/service/ from=127.0.0.1:37032 error="method HEAD not allowed" +30:47.700 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/service/ from=127.0.0.1:37032 latency=25.965µs +30:47.700 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/service/ from=127.0.0.1:37032 latency=2.825µs +30:47.701 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/ca/roots from=127.0.0.1:37032 latency=428.091µs +30:47.702 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/ca/roots from=127.0.0.1:37036 error="method PUT not allowed" +30:47.702 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/ca/roots from=127.0.0.1:37036 latency=120.394µs +30:47.703 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/ca/roots from=127.0.0.1:37038 error="method POST not allowed" +30:47.703 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/ca/roots from=127.0.0.1:37038 latency=117.669µs +30:47.704 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/ca/roots from=127.0.0.1:37040 error="method DELETE not allowed" +30:47.704 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/ca/roots from=127.0.0.1:37040 latency=141.421µs +30:47.704 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/ca/roots from=127.0.0.1:37042 error="method HEAD not allowed" +30:47.704 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/ca/roots from=127.0.0.1:37042 latency=29.355µs +30:47.705 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/ca/roots from=127.0.0.1:37042 latency=2.516µs +30:47.705 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/ from=127.0.0.1:37042 error="Bad request: failed intention lookup: index error: UUID must be 36 characters" +30:47.705 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/ from=127.0.0.1:37042 latency=268.163µs +30:47.706 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/ from=127.0.0.1:37044 error="Bad request: Request decode failed: EOF" +30:47.706 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/ from=127.0.0.1:37044 latency=135.956µs +30:47.707 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/ from=127.0.0.1:37046 error="method POST not allowed" +30:47.707 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/ from=127.0.0.1:37046 latency=145.489µs +30:47.708 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/ from=127.0.0.1:37048 error="Intention lookup failed: failed intention lookup: index error: UUID must be 36 characters" +30:47.708 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/ from=127.0.0.1:37048 latency=236.973µs +30:47.708 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/ from=127.0.0.1:37050 error="method HEAD not allowed" +30:47.708 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/ from=127.0.0.1:37050 latency=22.259µs +30:47.709 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/ from=127.0.0.1:37050 latency=3.087µs +30:47.709 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/binding-rule from=127.0.0.1:37050 error="method GET not allowed" +30:47.709 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule from=127.0.0.1:37050 latency=116.76µs +30:47.711 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37054 error="Bad request: BindingRule decoding failed: EOF" +30:47.711 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37054 latency=149.359µs +30:47.712 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/binding-rule from=127.0.0.1:37056 error="method POST not allowed" +30:47.712 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/binding-rule from=127.0.0.1:37056 latency=143.231µs +30:47.712 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/binding-rule from=127.0.0.1:37058 error="method DELETE not allowed" +30:47.712 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule from=127.0.0.1:37058 latency=145.327µs +30:47.713 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/binding-rule from=127.0.0.1:37060 error="method HEAD not allowed" +30:47.713 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/binding-rule from=127.0.0.1:37060 latency=24.78µs +30:47.713 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/binding-rule from=127.0.0.1:37060 latency=2.246µs +30:47.714 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token from=127.0.0.1:37060 error="method GET not allowed" +30:47.714 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token from=127.0.0.1:37060 latency=121.119µs +30:47.715 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token from=127.0.0.1:37062 error="Bad request: Token decoding failed: EOF" +30:47.715 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37062 latency=130.633µs +30:47.716 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token from=127.0.0.1:37064 error="method POST not allowed" +30:47.716 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token from=127.0.0.1:37064 latency=125.35µs +30:47.716 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token from=127.0.0.1:37066 error="method DELETE not allowed" +30:47.717 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token from=127.0.0.1:37066 latency=116.675µs +30:47.717 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token from=127.0.0.1:37068 error="method HEAD not allowed" +30:47.717 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token from=127.0.0.1:37068 latency=26.652µs +30:47.718 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token from=127.0.0.1:37068 latency=2.773µs +30:47.718 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/gateway-services/ from=127.0.0.1:37068 latency=98.151µs +30:47.719 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/gateway-services/ from=127.0.0.1:37070 error="method PUT not allowed" +30:47.719 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/gateway-services/ from=127.0.0.1:37070 latency=124.292µs +30:47.719 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/gateway-services/ from=127.0.0.1:37072 error="method POST not allowed" +30:47.719 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/gateway-services/ from=127.0.0.1:37072 latency=98.669µs +30:47.720 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/gateway-services/ from=127.0.0.1:37074 error="method DELETE not allowed" +30:47.720 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/gateway-services/ from=127.0.0.1:37074 latency=163.358µs +30:47.721 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/gateway-services/ from=127.0.0.1:37076 error="method HEAD not allowed" +30:47.721 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/gateway-services/ from=127.0.0.1:37076 latency=22.55µs +30:47.721 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/gateway-services/ from=127.0.0.1:37076 latency=2.667µs +30:47.721 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:37076 error="Permission denied" +30:47.722 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:37076 latency=227.367µs +30:47.722 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:37078 error="Bad request: Request decode failed: EOF" +30:47.722 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:37078 latency=92.634µs +30:47.723 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/ca/configuration from=127.0.0.1:37080 error="method POST not allowed" +30:47.723 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/ca/configuration from=127.0.0.1:37080 latency=104.327µs +30:47.723 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/ca/configuration from=127.0.0.1:37082 error="method DELETE not allowed" +30:47.723 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/ca/configuration from=127.0.0.1:37082 latency=108.037µs +30:47.724 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/ca/configuration from=127.0.0.1:37084 error="method HEAD not allowed" +30:47.724 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/ca/configuration from=127.0.0.1:37084 latency=22.764µs +30:47.724 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/ca/configuration from=127.0.0.1:37084 latency=4.15µs +30:47.724 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37084 error="URI must be either service or agent" +30:47.725 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37084 latency=220.877µs +30:47.725 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37086 error="method PUT not allowed" +30:47.725 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37086 latency=136.234µs +30:47.726 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37088 error="method POST not allowed" +30:47.726 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37088 latency=159.05µs +30:47.727 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37090 error="method DELETE not allowed" +30:47.727 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37090 latency=136.475µs +30:47.727 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37092 error="method HEAD not allowed" +30:47.727 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37092 latency=18.44µs +30:47.728 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/connect/ca/leaf/ from=127.0.0.1:37092 latency=2.119µs +30:47.728 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/ from=127.0.0.1:37092 latency=118.167µs +30:47.728 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/service/ from=127.0.0.1:37094 error="method PUT not allowed" +30:47.729 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/service/ from=127.0.0.1:37094 latency=107.507µs +30:47.729 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/service/ from=127.0.0.1:37096 error="method POST not allowed" +30:47.729 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/service/ from=127.0.0.1:37096 latency=88.822µs +30:47.730 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/service/ from=127.0.0.1:37098 error="method DELETE not allowed" +30:47.730 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/service/ from=127.0.0.1:37098 latency=133.793µs +30:47.730 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/service/ from=127.0.0.1:37100 error="method HEAD not allowed" +30:47.730 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/service/ from=127.0.0.1:37100 latency=20.472µs +30:47.730 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/service/ from=127.0.0.1:37100 latency=2.769µs +30:47.731 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/destroy/ from=127.0.0.1:37100 error="method GET not allowed" +30:47.731 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/destroy/ from=127.0.0.1:37100 latency=121.607µs +30:47.731 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/ from=127.0.0.1:37102 latency=111.872µs +30:47.732 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/destroy/ from=127.0.0.1:37104 error="method POST not allowed" +30:47.732 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/destroy/ from=127.0.0.1:37104 latency=131.138µs +30:47.733 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/destroy/ from=127.0.0.1:37106 error="method DELETE not allowed" +30:47.733 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/destroy/ from=127.0.0.1:37106 latency=137.462µs +30:47.733 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/destroy/ from=127.0.0.1:37108 error="method HEAD not allowed" +30:47.733 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/destroy/ from=127.0.0.1:37108 latency=21.671µs +30:47.734 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/destroy/ from=127.0.0.1:37108 latency=2.707µs +30:47.734 [ERROR] TestAgent.http: Request error: method=GET url=/v1/session/renew/ from=127.0.0.1:37108 error="method GET not allowed" +30:47.734 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/renew/ from=127.0.0.1:37108 latency=115.406µs +30:47.735 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/renew/ from=127.0.0.1:37110 latency=93.192µs +30:47.735 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/renew/ from=127.0.0.1:37112 error="method POST not allowed" +30:47.735 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/renew/ from=127.0.0.1:37112 latency=114.861µs +30:47.735 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/renew/ from=127.0.0.1:37114 error="method DELETE not allowed" +30:47.736 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/renew/ from=127.0.0.1:37114 latency=122.36µs +30:47.736 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/renew/ from=127.0.0.1:37116 error="method HEAD not allowed" +30:47.736 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/renew/ from=127.0.0.1:37116 latency=23.164µs +30:47.736 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/renew/ from=127.0.0.1:37116 latency=3.323µs +30:47.737 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/bootstrap from=127.0.0.1:37116 error="method GET not allowed" +30:47.737 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/bootstrap from=127.0.0.1:37116 latency=122.506µs +30:47.738 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:37118 latency=192.504µs +30:47.738 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/bootstrap from=127.0.0.1:37120 error="method POST not allowed" +30:47.738 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/bootstrap from=127.0.0.1:37120 latency=95.757µs +30:47.739 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/bootstrap from=127.0.0.1:37122 error="method DELETE not allowed" +30:47.739 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/bootstrap from=127.0.0.1:37122 latency=95.37µs +30:47.739 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/bootstrap from=127.0.0.1:37124 error="method HEAD not allowed" +30:47.739 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/bootstrap from=127.0.0.1:37124 latency=17.977µs +30:47.739 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/bootstrap from=127.0.0.1:37124 latency=2.523µs +30:47.740 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/pass/ from=127.0.0.1:37124 error="method GET not allowed" +30:47.740 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/pass/ from=127.0.0.1:37124 latency=110.08µs +30:47.740 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/check/pass/ from=127.0.0.1:37126 error="Unknown check """ +30:47.740 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/pass/ from=127.0.0.1:37126 latency=153.955µs +30:47.741 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/pass/ from=127.0.0.1:37128 error="method POST not allowed" +30:47.741 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/pass/ from=127.0.0.1:37128 latency=128.631µs +30:47.741 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/pass/ from=127.0.0.1:37130 error="method DELETE not allowed" +30:47.742 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/pass/ from=127.0.0.1:37130 latency=122.58µs +30:47.742 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/pass/ from=127.0.0.1:37132 error="method HEAD not allowed" +30:47.742 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/pass/ from=127.0.0.1:37132 latency=21.416µs +30:47.742 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/pass/ from=127.0.0.1:37132 latency=2.867µs +30:47.743 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/node/ from=127.0.0.1:37132 latency=118.12µs +30:47.743 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/health/node/ from=127.0.0.1:37134 error="method PUT not allowed" +30:47.743 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/health/node/ from=127.0.0.1:37134 latency=103.977µs +30:47.744 [ERROR] TestAgent.http: Request error: method=POST url=/v1/health/node/ from=127.0.0.1:37136 error="method POST not allowed" +30:47.744 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/health/node/ from=127.0.0.1:37136 latency=101.006µs +30:47.744 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/health/node/ from=127.0.0.1:37138 error="method DELETE not allowed" +30:47.744 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/health/node/ from=127.0.0.1:37138 latency=113.52µs +30:47.745 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/health/node/ from=127.0.0.1:37140 error="method HEAD not allowed" +30:47.745 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/health/node/ from=127.0.0.1:37140 latency=24.134µs +30:47.745 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/health/node/ from=127.0.0.1:37140 latency=2.412µs +30:47.745 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:37140 latency=131.73µs +30:47.746 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/status/leader from=127.0.0.1:37142 error="method PUT not allowed" +30:47.746 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/status/leader from=127.0.0.1:37142 latency=148.578µs +30:47.747 [ERROR] TestAgent.http: Request error: method=POST url=/v1/status/leader from=127.0.0.1:37144 error="method POST not allowed" +30:47.747 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/status/leader from=127.0.0.1:37144 latency=116.483µs +30:47.747 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/status/leader from=127.0.0.1:37146 error="method DELETE not allowed" +30:47.747 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/status/leader from=127.0.0.1:37146 latency=126.462µs +30:47.748 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/status/leader from=127.0.0.1:37148 error="method HEAD not allowed" +30:47.748 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/status/leader from=127.0.0.1:37148 latency=21.116µs +30:47.748 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/status/leader from=127.0.0.1:37148 latency=2.956µs +30:47.749 [DEBUG] TestAgent: dropping node from result due to ACLs: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d accessorID=00000000-0000-0000-0000-000000000002 +30:47.749 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members from=127.0.0.1:37148 latency=284.376µs +30:47.749 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/members from=127.0.0.1:37150 error="method PUT not allowed" +30:47.749 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/members from=127.0.0.1:37150 latency=112.412µs +30:47.750 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/members from=127.0.0.1:37152 error="method POST not allowed" +30:47.750 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/members from=127.0.0.1:37152 latency=94.616µs +30:47.750 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/members from=127.0.0.1:37154 error="method DELETE not allowed" +30:47.750 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/members from=127.0.0.1:37154 latency=116.439µs +30:47.751 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/members from=127.0.0.1:37156 error="method HEAD not allowed" +30:47.751 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/members from=127.0.0.1:37156 latency=19.04µs +30:47.751 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/members from=127.0.0.1:37156 latency=2.771µs +30:47.752 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.752 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/internal/ui/nodes from=127.0.0.1:37156 latency=348.939µs +30:47.752 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/internal/ui/nodes from=127.0.0.1:37158 error="method PUT not allowed" +30:47.752 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/internal/ui/nodes from=127.0.0.1:37158 latency=102.328µs +30:47.753 [ERROR] TestAgent.http: Request error: method=POST url=/v1/internal/ui/nodes from=127.0.0.1:37160 error="method POST not allowed" +30:47.753 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/internal/ui/nodes from=127.0.0.1:37160 latency=127.184µs +30:47.754 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/internal/ui/nodes from=127.0.0.1:37162 error="method DELETE not allowed" +30:47.754 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/internal/ui/nodes from=127.0.0.1:37162 latency=133.846µs +30:47.754 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/internal/ui/nodes from=127.0.0.1:37164 error="method HEAD not allowed" +30:47.754 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/internal/ui/nodes from=127.0.0.1:37164 latency=18.922µs +30:47.754 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/internal/ui/nodes from=127.0.0.1:37164 latency=2.309µs +30:47.755 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/list from=127.0.0.1:37164 latency=261.56µs +30:47.755 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/session/list from=127.0.0.1:37166 error="method PUT not allowed" +30:47.756 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/list from=127.0.0.1:37166 latency=97.86µs +30:47.756 [ERROR] TestAgent.http: Request error: method=POST url=/v1/session/list from=127.0.0.1:37168 error="method POST not allowed" +30:47.756 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/session/list from=127.0.0.1:37168 latency=96.508µs +30:47.757 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/session/list from=127.0.0.1:37170 error="method DELETE not allowed" +30:47.757 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/session/list from=127.0.0.1:37170 latency=98.173µs +30:47.757 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/session/list from=127.0.0.1:37172 error="method HEAD not allowed" +30:47.757 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/session/list from=127.0.0.1:37172 latency=20.386µs +30:47.757 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/session/list from=127.0.0.1:37172 latency=2.794µs +30:47.758 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/login from=127.0.0.1:37172 error="method GET not allowed" +30:47.758 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/login from=127.0.0.1:37172 latency=101.417µs +30:47.758 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/login from=127.0.0.1:37174 error="method PUT not allowed" +30:47.758 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/login from=127.0.0.1:37174 latency=109.007µs +30:47.759 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:37176 error="Bad request: Failed to decode request body: EOF" +30:47.759 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:37176 latency=115.203µs +30:47.760 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/login from=127.0.0.1:37178 error="method DELETE not allowed" +30:47.760 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/login from=127.0.0.1:37178 latency=94.723µs +30:47.760 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/login from=127.0.0.1:37180 error="method HEAD not allowed" +30:47.760 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/login from=127.0.0.1:37180 latency=19.238µs +30:47.760 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/login from=127.0.0.1:37180 latency=2.58µs +30:47.761 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8e7e7fa1-24c8-b84f-b504-0a6d0fb54e6d +30:47.761 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:37180 latency=204.329µs +30:47.761 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/catalog/nodes from=127.0.0.1:37182 error="method PUT not allowed" +30:47.761 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/nodes from=127.0.0.1:37182 latency=110.77µs +30:47.762 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/nodes from=127.0.0.1:37184 error="method POST not allowed" +30:47.762 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/nodes from=127.0.0.1:37184 latency=96.63µs +30:47.762 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/nodes from=127.0.0.1:37186 error="method DELETE not allowed" +30:47.762 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/nodes from=127.0.0.1:37186 latency=133.495µs +30:47.763 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/nodes from=127.0.0.1:37188 error="method HEAD not allowed" +30:47.763 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/nodes from=127.0.0.1:37188 latency=22.246µs +30:47.763 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/nodes from=127.0.0.1:37188 latency=2.735µs +30:47.764 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/replication from=127.0.0.1:37188 latency=262.496µs +30:47.764 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/replication from=127.0.0.1:37190 error="method PUT not allowed" +30:47.764 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/replication from=127.0.0.1:37190 latency=99.792µs +30:47.765 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/replication from=127.0.0.1:37192 error="method POST not allowed" +30:47.765 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/replication from=127.0.0.1:37192 latency=124.999µs +30:47.766 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/replication from=127.0.0.1:37194 error="method DELETE not allowed" +30:47.766 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/replication from=127.0.0.1:37194 latency=118.14µs +30:47.766 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/replication from=127.0.0.1:37196 error="method HEAD not allowed" +30:47.766 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/replication from=127.0.0.1:37196 latency=19.45µs +30:47.766 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/replication from=127.0.0.1:37196 latency=2.362µs +30:47.767 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/name/ from=127.0.0.1:37196 error="Bad request: Missing policy Name" +30:47.767 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/name/ from=127.0.0.1:37196 latency=104.128µs +30:47.767 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy/name/ from=127.0.0.1:37198 error="method PUT not allowed" +30:47.767 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/name/ from=127.0.0.1:37198 latency=117.718µs +30:47.768 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy/name/ from=127.0.0.1:37200 error="method POST not allowed" +30:47.768 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy/name/ from=127.0.0.1:37200 latency=107.587µs +30:47.768 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy/name/ from=127.0.0.1:37202 error="method DELETE not allowed" +30:47.769 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/name/ from=127.0.0.1:37202 latency=96.784µs +30:47.769 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy/name/ from=127.0.0.1:37204 error="method HEAD not allowed" +30:47.769 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy/name/ from=127.0.0.1:37204 latency=21.017µs +30:47.769 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy/name/ from=127.0.0.1:37204 latency=2.744µs +30:47.770 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/ from=127.0.0.1:37204 error="Bad request: Missing policy ID" +30:47.770 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/ from=127.0.0.1:37204 latency=105.659µs +30:47.770 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/policy/ from=127.0.0.1:37206 error="Bad request: Policy decoding failed: EOF" +30:47.770 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/ from=127.0.0.1:37206 latency=108.328µs +30:47.771 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/policy/ from=127.0.0.1:37208 error="method POST not allowed" +30:47.771 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/policy/ from=127.0.0.1:37208 latency=98.016µs +30:47.771 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/policy/ from=127.0.0.1:37210 error="Bad request: Missing policy ID" +30:47.771 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/ from=127.0.0.1:37210 latency=97.069µs +30:47.772 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/policy/ from=127.0.0.1:37212 error="method HEAD not allowed" +30:47.772 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/policy/ from=127.0.0.1:37212 latency=19.226µs +30:47.772 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/policy/ from=127.0.0.1:37212 latency=2.471µs +30:47.772 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/force-leave/ from=127.0.0.1:37212 error="method GET not allowed" +30:47.773 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/force-leave/ from=127.0.0.1:37212 latency=120.927µs +30:47.773 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/force-leave/ from=127.0.0.1:37214 error="Permission denied" +30:47.773 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/ from=127.0.0.1:37214 latency=168.75µs +30:47.774 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/force-leave/ from=127.0.0.1:37216 error="method POST not allowed" +30:47.774 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/force-leave/ from=127.0.0.1:37216 latency=131.316µs +30:47.774 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/force-leave/ from=127.0.0.1:37218 error="method DELETE not allowed" +30:47.775 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/force-leave/ from=127.0.0.1:37218 latency=128.177µs +30:47.775 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/force-leave/ from=127.0.0.1:37220 error="method HEAD not allowed" +30:47.775 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/force-leave/ from=127.0.0.1:37220 latency=21.192µs +30:47.775 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/force-leave/ from=127.0.0.1:37220 latency=2.627µs +30:47.776 [ERROR] TestAgent.http: Request error: method=GET url=/v1/catalog/deregister from=127.0.0.1:37220 error="method GET not allowed" +30:47.776 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/deregister from=127.0.0.1:37220 latency=131.846µs +30:47.776 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/deregister from=127.0.0.1:37222 latency=111.302µs +30:47.777 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/deregister from=127.0.0.1:37224 error="method POST not allowed" +30:47.777 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/deregister from=127.0.0.1:37224 latency=136.162µs +30:47.777 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/deregister from=127.0.0.1:37226 error="method DELETE not allowed" +30:47.778 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/deregister from=127.0.0.1:37226 latency=111.512µs +30:47.778 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/deregister from=127.0.0.1:37228 error="method HEAD not allowed" +30:47.778 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/deregister from=127.0.0.1:37228 latency=17.322µs +30:47.778 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/deregister from=127.0.0.1:37228 latency=2.378µs +30:47.779 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/auth-method/ from=127.0.0.1:37228 error="Bad request: Missing auth method name" +30:47.779 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/ from=127.0.0.1:37228 latency=133.257µs +30:47.779 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method/ from=127.0.0.1:37230 error="Bad request: AuthMethod decoding failed: EOF" +30:47.779 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/ from=127.0.0.1:37230 latency=124.204µs +30:47.780 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/auth-method/ from=127.0.0.1:37232 error="method POST not allowed" +30:47.780 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/auth-method/ from=127.0.0.1:37232 latency=124.606µs +30:47.781 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/auth-method/ from=127.0.0.1:37234 error="Bad request: Missing auth method name" +30:47.781 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/ from=127.0.0.1:37234 latency=96.176µs +30:47.781 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/auth-method/ from=127.0.0.1:37236 error="method HEAD not allowed" +30:47.781 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/auth-method/ from=127.0.0.1:37236 latency=17.708µs +30:47.781 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/auth-method/ from=127.0.0.1:37236 latency=2.595µs +30:47.782 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/self from=127.0.0.1:37236 error="ACL not found" +30:47.782 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:37236 latency=174.841µs +30:47.782 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/token/self from=127.0.0.1:37238 error="method PUT not allowed" +30:47.782 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/self from=127.0.0.1:37238 latency=116.05µs +30:47.783 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/token/self from=127.0.0.1:37240 error="method POST not allowed" +30:47.783 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/token/self from=127.0.0.1:37240 latency=119.837µs +30:47.784 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/token/self from=127.0.0.1:37242 error="method DELETE not allowed" +30:47.784 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/self from=127.0.0.1:37242 latency=118.822µs +30:47.789 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/token/self from=127.0.0.1:37244 error="method HEAD not allowed" +30:47.789 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/token/self from=127.0.0.1:37244 latency=29.406µs +30:47.790 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/token/self from=127.0.0.1:37244 latency=3.125µs +30:47.790 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/service/register from=127.0.0.1:37244 error="method GET not allowed" +30:47.790 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/register from=127.0.0.1:37244 latency=126.667µs +30:47.791 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37246 latency=115.978µs +30:47.791 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/service/register from=127.0.0.1:37248 error="method POST not allowed" +30:47.791 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/service/register from=127.0.0.1:37248 latency=117.581µs +30:47.793 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/service/register from=127.0.0.1:37250 error="method DELETE not allowed" +30:47.793 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/service/register from=127.0.0.1:37250 latency=137.652µs +30:47.794 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/service/register from=127.0.0.1:37252 error="method HEAD not allowed" +30:47.794 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/service/register from=127.0.0.1:37252 latency=20.752µs +30:47.794 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/service/register from=127.0.0.1:37252 latency=2.575µs +30:47.795 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/create from=127.0.0.1:37252 error="method GET not allowed" +30:47.795 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/create from=127.0.0.1:37252 latency=150.325µs +30:47.796 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/create from=127.0.0.1:37254 error="Permission denied" +30:47.796 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/create from=127.0.0.1:37254 latency=336.014µs +30:47.796 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/create from=127.0.0.1:37256 error="method POST not allowed" +30:47.797 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/create from=127.0.0.1:37256 latency=96.842µs +30:47.797 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/acl/create from=127.0.0.1:37258 error="method DELETE not allowed" +30:47.797 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/create from=127.0.0.1:37258 latency=113.804µs +30:47.798 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/acl/create from=127.0.0.1:37260 error="method HEAD not allowed" +30:47.798 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/acl/create from=127.0.0.1:37260 latency=13.964µs +30:47.798 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/acl/create from=127.0.0.1:37260 latency=2.055µs +30:47.798 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/check/register from=127.0.0.1:37260 error="method GET not allowed" +30:47.798 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/check/register from=127.0.0.1:37260 latency=108.228µs +30:47.799 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/register from=127.0.0.1:37262 latency=106.247µs +30:47.799 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/check/register from=127.0.0.1:37264 error="method POST not allowed" +30:47.800 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/check/register from=127.0.0.1:37264 latency=201.549µs +30:47.804 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/check/register from=127.0.0.1:37266 error="method DELETE not allowed" +30:47.804 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/check/register from=127.0.0.1:37266 latency=136.182µs +30:47.805 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/check/register from=127.0.0.1:37268 error="method HEAD not allowed" +30:47.805 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/check/register from=127.0.0.1:37268 latency=25.145µs +30:47.805 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/check/register from=127.0.0.1:37268 latency=2.938µs +30:47.806 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/check from=127.0.0.1:37268 error="required query parameter 'source' not set" +30:47.806 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check from=127.0.0.1:37268 latency=142.496µs +30:47.806 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/check from=127.0.0.1:37270 error="method PUT not allowed" +30:47.807 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/check from=127.0.0.1:37270 latency=132.862µs +30:47.807 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/check from=127.0.0.1:37272 error="method POST not allowed" +30:47.807 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/check from=127.0.0.1:37272 latency=93.014µs +30:47.808 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/check from=127.0.0.1:37274 error="method DELETE not allowed" +30:47.808 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/check from=127.0.0.1:37274 latency=119.829µs +30:47.808 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/check from=127.0.0.1:37276 error="method HEAD not allowed" +30:47.808 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/check from=127.0.0.1:37276 latency=22.293µs +30:47.809 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/check from=127.0.0.1:37276 latency=2.475µs +30:47.809 [ERROR] TestAgent.http: Request error: method=GET url=/v1/discovery-chain/ from=127.0.0.1:37276 error="Bad request: Missing chain name" +30:47.809 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/discovery-chain/ from=127.0.0.1:37276 latency=115.585µs +30:47.810 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/discovery-chain/ from=127.0.0.1:37278 error="method PUT not allowed" +30:47.810 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/discovery-chain/ from=127.0.0.1:37278 latency=135.431µs +30:47.810 [ERROR] TestAgent.http: Request error: method=POST url=/v1/discovery-chain/ from=127.0.0.1:37280 error="Bad request: Missing chain name" +30:47.810 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/discovery-chain/ from=127.0.0.1:37280 latency=123.373µs +30:47.811 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/discovery-chain/ from=127.0.0.1:37282 error="method DELETE not allowed" +30:47.811 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/discovery-chain/ from=127.0.0.1:37282 latency=129.005µs +30:47.812 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/discovery-chain/ from=127.0.0.1:37284 error="method HEAD not allowed" +30:47.812 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/discovery-chain/ from=127.0.0.1:37284 latency=22.492µs +30:47.812 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/discovery-chain/ from=127.0.0.1:37284 latency=4.841µs +30:47.812 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/raft/peer from=127.0.0.1:37284 error="method GET not allowed" +30:47.812 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/peer from=127.0.0.1:37284 latency=113.462µs +30:47.813 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/raft/peer from=127.0.0.1:37286 error="method PUT not allowed" +30:47.813 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/raft/peer from=127.0.0.1:37286 latency=110.723µs +30:47.813 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/raft/peer from=127.0.0.1:37288 error="method POST not allowed" +30:47.814 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/raft/peer from=127.0.0.1:37288 latency=110.435µs +30:47.814 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer from=127.0.0.1:37290 error="Bad request: Must specify either ?id with the server's ID or ?address with IP:port of peer to remove" +30:47.814 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer from=127.0.0.1:37290 latency=163.496µs +30:47.815 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/raft/peer from=127.0.0.1:37292 error="method HEAD not allowed" +30:47.815 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/raft/peer from=127.0.0.1:37292 latency=22.198µs +30:47.815 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/raft/peer from=127.0.0.1:37292 latency=2.746µs +30:47.815 [ERROR] TestAgent.http: Request error: method=GET url=/v1/operator/autopilot/health from=127.0.0.1:37292 error="Permission denied" +30:47.815 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/health from=127.0.0.1:37292 latency=227.524µs +30:47.816 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/operator/autopilot/health from=127.0.0.1:37294 error="method PUT not allowed" +30:47.816 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/health from=127.0.0.1:37294 latency=125.291µs +30:47.817 [ERROR] TestAgent.http: Request error: method=POST url=/v1/operator/autopilot/health from=127.0.0.1:37296 error="method POST not allowed" +30:47.817 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/autopilot/health from=127.0.0.1:37296 latency=106.838µs +30:47.817 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/autopilot/health from=127.0.0.1:37298 error="method DELETE not allowed" +30:47.818 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/autopilot/health from=127.0.0.1:37298 latency=111.593µs +30:47.818 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/operator/autopilot/health from=127.0.0.1:37300 error="method HEAD not allowed" +30:47.818 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/operator/autopilot/health from=127.0.0.1:37300 latency=24.596µs +30:47.818 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/operator/autopilot/health from=127.0.0.1:37300 latency=2.874µs +30:47.819 [ERROR] TestAgent.http: Request error: method=GET url=/v1/txn from=127.0.0.1:37300 error="method GET not allowed" +30:47.819 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/txn from=127.0.0.1:37300 latency=122.355µs +30:47.819 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/txn from=127.0.0.1:37302 latency=100.607µs +30:47.820 [ERROR] TestAgent.http: Request error: method=POST url=/v1/txn from=127.0.0.1:37304 error="method POST not allowed" +30:47.820 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/txn from=127.0.0.1:37304 latency=119.914µs +30:47.820 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/txn from=127.0.0.1:37306 error="method DELETE not allowed" +30:47.820 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/txn from=127.0.0.1:37306 latency=110.466µs +30:47.821 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/txn from=127.0.0.1:37308 error="method HEAD not allowed" +30:47.821 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/txn from=127.0.0.1:37308 latency=19.714µs +30:47.821 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/txn from=127.0.0.1:37308 latency=2.858µs +30:47.821 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/maintenance from=127.0.0.1:37308 error="method GET not allowed" +30:47.822 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/maintenance from=127.0.0.1:37308 latency=129.718µs +30:47.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance from=127.0.0.1:37310 latency=103.652µs +30:47.823 [ERROR] TestAgent.http: Request error: method=POST url=/v1/agent/maintenance from=127.0.0.1:37312 error="method POST not allowed" +30:47.823 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/agent/maintenance from=127.0.0.1:37312 latency=105.952µs +30:47.823 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/agent/maintenance from=127.0.0.1:37314 error="method DELETE not allowed" +30:47.823 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/agent/maintenance from=127.0.0.1:37314 latency=94.277µs +30:47.824 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/agent/maintenance from=127.0.0.1:37316 error="method HEAD not allowed" +30:47.824 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/agent/maintenance from=127.0.0.1:37316 latency=23.655µs +30:47.824 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/agent/maintenance from=127.0.0.1:37316 latency=2.495µs +30:47.825 [ERROR] TestAgent.http: Request error: method=GET url=/v1/catalog/register from=127.0.0.1:37316 error="method GET not allowed" +30:47.825 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/register from=127.0.0.1:37316 latency=101.28µs +30:47.825 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/catalog/register from=127.0.0.1:37318 latency=118.163µs +30:47.826 [ERROR] TestAgent.http: Request error: method=POST url=/v1/catalog/register from=127.0.0.1:37320 error="method POST not allowed" +30:47.826 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/catalog/register from=127.0.0.1:37320 latency=129.484µs +30:47.826 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/catalog/register from=127.0.0.1:37322 error="method DELETE not allowed" +30:47.827 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/catalog/register from=127.0.0.1:37322 latency=111.229µs +30:47.827 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/catalog/register from=127.0.0.1:37324 error="method HEAD not allowed" +30:47.827 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/catalog/register from=127.0.0.1:37324 latency=23.652µs +30:47.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/catalog/register from=127.0.0.1:37324 latency=3.831µs +30:47.828 [ERROR] TestAgent.http: Request error: method=GET url=/v1/connect/intentions/match from=127.0.0.1:37324 error="required query parameter 'by' not set" +30:47.828 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match from=127.0.0.1:37324 latency=127.863µs +30:47.828 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/connect/intentions/match from=127.0.0.1:37326 error="method PUT not allowed" +30:47.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/match from=127.0.0.1:37326 latency=113.825µs +30:47.829 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions/match from=127.0.0.1:37328 error="method POST not allowed" +30:47.829 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions/match from=127.0.0.1:37328 latency=101.123µs +30:47.830 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/connect/intentions/match from=127.0.0.1:37330 error="method DELETE not allowed" +30:47.830 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/match from=127.0.0.1:37330 latency=111.92µs +30:47.830 [ERROR] TestAgent.http: Request error: method=HEAD url=/v1/connect/intentions/match from=127.0.0.1:37332 error="method HEAD not allowed" +30:47.830 [DEBUG] TestAgent.http: Request finished: method=HEAD url=/v1/connect/intentions/match from=127.0.0.1:37332 latency=26.586µs +30:47.830 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=/v1/connect/intentions/match from=127.0.0.1:37332 latency=3.151µs +30:47.831 [INFO] TestAgent: Requesting shutdown +30:47.831 [INFO] TestAgent.server: shutting down server +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:47.831 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:47.831 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:47.831 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:47.831 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:47.842 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:47.848Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:47.848Z [INFO] agent.router.manager: shutting down +30:47.848 [INFO] TestAgent: consul server down +30:47.848 [INFO] TestAgent: shutdown complete +30:47.848 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30282 network=tcp +30:47.848 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30282 network=udp +30:47.848 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30283 network=tcp +30:48.352 [INFO] TestAgent: Waiting for endpoints to shut down +30:48.352 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_MethodNotAllowed_OSS (1.38s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query/ (0.00s) @@ -16991,1410 +17141,1423 @@ --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query/xxx/explain (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query/xxx/explain (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query/xxx/explain (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/query (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/query (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/query (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/query (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/query (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/query (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node-services/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node-services/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/leave (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/warn/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/deregister/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/connect/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/tokens (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/datacenters (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/info/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/update/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/update/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/update/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/update/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/update/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/update/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/id/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/info/ (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/roles (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/authorize (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/checks/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/node/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/destroy/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/destroy/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/destroy/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/destroy/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/destroy/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/deregister/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/maintenance/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/name/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rules (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rules (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rules (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rules (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rules (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rules (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/checks (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/update (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/clone/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/datacenters (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/rules/translate/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/ (0.01s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/service/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/roots (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/roots (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/roots (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/roots (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/roots (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/checks/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/check (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/gateway-services/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/ca/configuration (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/ca/configuration (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/ca/configuration (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/ca/configuration (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/ca/configuration (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/ca/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/host (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/metrics (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/node/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/status/leader (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/list (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/list (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/login (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/login (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/login (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/login (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/login (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/login (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/self (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/nodes (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/nodes (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/nodes (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/nodes (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/nodes (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/name/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/force-leave/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/deregister (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/deregister (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/deregister (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/deregister (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/deregister (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/deregister (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/txn (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/roles (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token/self (0.01s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token/self (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/check (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/discovery-chain/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/discovery-chain/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/discovery-chain/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/discovery-chain/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/discovery-chain/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/discovery-chain/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/auth-methods (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/internal/federation-state/ (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/raft/peer (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/raft/peer (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/raft/peer (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/raft/peer (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/raft/peer (0.00s) --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/raft/peer (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/GET_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/PUT_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/POST_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/DELETE_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/HEAD_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_MethodNotAllowed_OSS/OPTIONS_/v1/connect/intentions/match (0.00s) === RUN TestHTTPAPI_OptionMethod_OSS +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/explain === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/execute -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/explain -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/leave -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/replication -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/gateway-services/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/maintenance +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/update/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/services +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/id/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/node/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/configuration -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/roots -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/register -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/node/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/services +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/info/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-methods === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/deregister/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/connect/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/checks/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/services -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/match -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/check -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/bootstrap -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/datacenters +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/roles === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/join/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/service/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/renew/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/connect/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/health -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/name/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/authorize +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/fire/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/checks/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/host +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/nodes +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/node/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/destroy/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/node/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/node/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/maintenance/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/connect/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/kv/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/logout +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policies +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/roots +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/ingress/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/name/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rules +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/configuration +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/list +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/snapshot +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/services +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/keyring +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/create +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/update +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/clone/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/token/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/metrics +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/service/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/update +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/roots +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/gateway-services/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/configuration -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/name/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/register +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/destroy/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/renew/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/bootstrap +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/node/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/service/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/members +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/nodes +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/list === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/status/leader === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/login -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/token/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/register -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/self === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/nodes -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/node/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/members -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/deregister -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/update -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/nodes -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/keyring -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/logout -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/fire/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/snapshot -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/roles -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/discovery-chain/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/state/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/txn +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/replication +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/name/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/list -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-methods +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/deregister +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/self -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/info/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/tokens -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/id/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/fail/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-state/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/peer +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/register === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/create -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/datacenters -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/node/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/create -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/services -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/update/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/register +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/discovery-chain/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/peer +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/health +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/txn +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/maintenance +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/register +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/match +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/check +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/leave +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/deregister/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/node-services/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/node/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policies -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/ingress/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/destroy/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/roots -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/kv/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/destroy/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/update -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rules +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/connect/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/list +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/self +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/state/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/tokens === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/checks +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-state/ +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/name/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/deregister/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/service/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/list -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/nodes -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/info/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/list -=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/clone/ === RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/datacenters -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways +=== RUN TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/info/ [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:41.362Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:41.362Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:41.377Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:41.377Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:41.401 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55c6b1f1-2719-7c2b-d0d5-868b58c83973 Address:127.0.0.1:11916}]" -45:41.401 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11916 [Follower]" leader= -45:41.401 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973.dc1 127.0.0.1 -45:41.402 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 127.0.0.1 -2022-11-07T22:45:41.402Z [INFO] agent.router: Initializing LAN area manager -45:41.402 [INFO] TestAgent.server: Adding LAN server: server="Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 (Addr: tcp/127.0.0.1:11916) (DC: dc1)" -45:41.402 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973.dc1 area=wan -45:41.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11911 network=udp -45:41.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11911 network=tcp -45:41.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11912 network=tcp -45:41.403 [INFO] TestAgent: started state syncer -45:41.403 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11917 network=tcp -45:41.450 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:41.450 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11916 [Candidate]" term=2 -45:41.452 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:41.458 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:41.458 [DEBUG] TestAgent.server.raft: vote granted: from=55c6b1f1-2719-7c2b-d0d5-868b58c83973 term=2 tally=1 -45:41.458 [INFO] TestAgent.server.raft: election won: tally=1 -45:41.458 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11916 [Leader]" -45:41.458 [INFO] TestAgent.server: cluster leadership acquired -45:41.459 [INFO] TestAgent.server: New leader elected: payload=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 -45:41.464 [INFO] TestAgent.server: initializing acls -45:41.466 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:41.472 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:41.472 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:41.472 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:41.472 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 -45:41.472 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973.dc1 -45:41.472 [INFO] TestAgent.server: Updating LAN server: server="Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 (Addr: tcp/127.0.0.1:11916) (DC: dc1)" -45:41.472 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973.dc1 area=wan -45:41.484 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:41.496 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:41.496 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:41.496 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:41.496 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:41.496 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:41.496 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 -45:41.496 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 -45:41.501 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55c6b1f1-2719-7c2b-d0d5-868b58c83973 -45:41.505 [INFO] TestAgent.server: federation state anti-entropy synced -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/query from= latency=2.868µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/query/ from= latency=100.653µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/query/xxx/execute from= latency=13.097µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/query/xxx/explain from= latency=11.176µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/leave from= latency=1.649µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/replication from= latency=1.386µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/token/ from= latency=1.34µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/gateway-services/ from= latency=1.192µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/maintenance from= latency=3.142µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/node/ from= latency=1.213µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/operator/raft/configuration from= latency=1.222µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/auth-method/ from= latency=1.414µs -45:41.575 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/service/ from= latency=1.188µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/force-leave/ from= latency=1.286µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/connect/ca/roots from= latency=1.416µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/register from= latency=1.302µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/coordinate/node/ from= latency=1.521µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/binding-rule from= latency=1.265µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/services from= latency=1.181µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/deregister/ from= latency=1.138µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/connect/ from= latency=1.083µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/checks/ from= latency=1.446µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/services from= latency=1.228µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/intentions/match from= latency=1.114µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/intentions/check from= latency=1.115µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/intentions/ from= latency=1.28µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/bootstrap from= latency=1.083µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/token from= latency=1.163µs -45:41.576 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/join/ from= latency=1.131µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/service/ from= latency=1.217µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/renew/ from= latency=1.298µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/connect/ from= latency=1.213µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/operator/autopilot/health from= latency=1.292µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/role/name/ from= latency=1.223µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/binding-rule/ from= latency=1.448µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/host from= latency=1.133µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/metrics from= latency=1.27µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/ca/configuration from= latency=1.557µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/policy/name/ from= latency=1.264µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/service/register from= latency=1.175µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/status/leader from= latency=1.394µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/login from= latency=1.143µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/policy from= latency=1.175µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/token/ from= latency=1.238µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/register from= latency=1.387µs -45:41.577 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/connect/authorize from= latency=1.128µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/federation-states/mesh-gateways from= latency=1.193µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/self from= latency=1.219µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/nodes from= latency=1.246µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/ui/node/ from= latency=1.338µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/role from= latency=1.253µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/members from= latency=1.178µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/deregister from= latency=1.174µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/config/ from= latency=1.316µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/coordinate/update from= latency=2.403µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/role/ from= latency=1.321µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/auth-method from= latency=2.371µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/federation-states from= latency=1.23µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/ui/gateway-services-nodes/ from= latency=1.063µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/ui/nodes from= latency=1.333µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/operator/keyring from= latency=1.444µs -45:41.578 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/logout from= latency=1.273µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/event/fire/ from= latency=1.546µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/snapshot from= latency=1.318µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/roles from= latency=1.12µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/discovery-chain/ from= latency=1.361µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/state/ from= latency=1.506µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/operator/autopilot/configuration from= latency=1.463µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/txn from= latency=1.333µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/policy/ from= latency=1.506µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/list from= latency=1.309µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/auth-methods from= latency=1.228µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/token/self from= latency=1.137µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/acl/authorize from= latency=1.186µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/info/ from= latency=1.202µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/tokens from= latency=1.3µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/health/service/id/ from= latency=1.173µs -45:41.579 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/pass/ from= latency=1.16µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/fail/ from= latency=1.235µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/federation-state/ from= latency=1.169µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/operator/raft/peer from= latency=1.115µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/create from= latency=1.101µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/rules/translate from= latency=1.212µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/rules/translate/ from= latency=1.097µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/coordinate/datacenters from= latency=1.242µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/node/ from= latency=1.195µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/create from= latency=1.267µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/internal/ui/services from= latency=1.363µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/update/ from= latency=1.306µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/connect/ca/leaf/ from= latency=1.177µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/node-services/ from= latency=1.46µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/node/ from= latency=1.197µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/policies from= latency=1.384µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/intentions from= latency=1.408µs -45:41.580 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/health/ingress/ from= latency=1.154µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/destroy/ from= latency=1.262µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/connect/ca/roots from= latency=1.232µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/kv/ from= latency=1.422µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/destroy/ from= latency=1.347µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/update from= latency=1.413µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/binding-rules from= latency=1.197µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/checks from= latency=1.218µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/health/service/name/ from= latency=1.146µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/service/deregister/ from= latency=1.281µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/service/maintenance/ from= latency=1.229µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/service/ from= latency=1.212µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/list from= latency=1.178µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/config from= latency=1.223µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/coordinate/nodes from= latency=1.185µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/session/info/ from= latency=1.197µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/agent/check/warn/ from= latency=1.529µs -45:41.581 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/event/list from= latency=1.355µs -45:41.582 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/acl/clone/ from= latency=1.126µs -45:41.582 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:11912/v1/catalog/datacenters from= latency=1.438µs -45:41.582 [INFO] TestAgent: Requesting shutdown -45:41.582 [INFO] TestAgent.server: shutting down server -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:41.582 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:41.582 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:41.582 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:41.582 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:41.582 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:41.586 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:41.589Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:41.589Z [INFO] agent.router.manager: shutting down -45:41.589 [INFO] TestAgent: consul server down -45:41.589 [INFO] TestAgent: shutdown complete -45:41.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11911 network=tcp -45:41.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11911 network=udp -45:41.589 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11912 network=tcp -45:42.089 [INFO] TestAgent: Waiting for endpoints to shut down -45:42.089 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_OptionMethod_OSS (0.74s) +2021-10-05T16:30:48.392Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:30:48.392Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:48.444Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:30:48.444Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:48.529 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 Address:127.0.0.1:30294}]" +30:48.530 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715.dc1 127.0.0.1 +30:48.530 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 127.0.0.1 +2021-10-05T16:30:48.530Z [INFO] agent.router: Initializing LAN area manager +30:48.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30289 network=udp +30:48.530 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30294 [Follower]" leader= +30:48.531 [INFO] TestAgent.server: Adding LAN server: server="Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 (Addr: tcp/127.0.0.1:30294) (DC: dc1)" +30:48.531 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715.dc1 area=wan +30:48.531 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30289 network=tcp +30:48.531 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30290 network=tcp +30:48.532 [INFO] TestAgent: started state syncer +30:48.532 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30295 network=tcp +30:48.581 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:48.596 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:48.596 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30294 [Candidate]" term=2 +30:48.614 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:48.614 [DEBUG] TestAgent.server.raft: vote granted: from=6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 term=2 tally=1 +30:48.614 [INFO] TestAgent.server.raft: election won: tally=1 +30:48.614 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30294 [Leader]" +30:48.616 [INFO] TestAgent.server: cluster leadership acquired +30:48.616 [INFO] TestAgent.server: New leader elected: payload=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 +30:48.620 [INFO] TestAgent.server: initializing acls +30:48.624 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:48.633 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:48.633 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:48.633 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:48.633 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 +30:48.633 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715.dc1 +30:48.634 [INFO] TestAgent.server: Updating LAN server: server="Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 (Addr: tcp/127.0.0.1:30294) (DC: dc1)" +30:48.634 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715.dc1 area=wan +30:48.672 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:48.704 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:48.704 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:48.704 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:48.704 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:48.704 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:48.704 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 +30:48.704 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 +30:48.719 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a1f0a5a-a272-1030-2c2b-8f01ad9a0715 +30:48.724 [INFO] TestAgent.server: federation state anti-entropy synced +30:48.824 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/query/xxx/explain from= latency=127.557µs +30:48.824 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/query from= latency=3.365µs +30:48.824 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/query/ from= latency=14.803µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/query/xxx/execute from= latency=15.6µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/update/ from= latency=2.564µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/config from= latency=2.537µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/ui/services from= latency=2.336µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/health/service/id/ from= latency=2.292µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/node/ from= latency=4.436µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/info/ from= latency=2.215µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/auth-methods from= latency=2.218µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/deregister/ from= latency=2.526µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/services from= latency=2.124µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/coordinate/datacenters from= latency=2.629µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/roles from= latency=2.434µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/join/ from= latency=2.232µs +30:48.825 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/fail/ from= latency=2.18µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/connect/authorize from= latency=2.298µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/intentions from= latency=3.245µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/event/fire/ from= latency=2.333µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/ui/gateway-services-nodes/ from= latency=2.25µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/checks/ from= latency=2.265µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/host from= latency=2.221µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/coordinate/nodes from= latency=2.38µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/binding-rule/ from= latency=2.579µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/ui/node/ from= latency=2.222µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/destroy/ from= latency=2.449µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/coordinate/node/ from= latency=2.132µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/node/ from= latency=2.082µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/service/maintenance/ from= latency=2.346µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/connect/ from= latency=1.985µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/kv/ from= latency=2.115µs +30:48.826 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/logout from= latency=2.187µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/policies from= latency=1.803µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/policy from= latency=1.72µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/auth-method from= latency=1.747µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/ca/roots from= latency=1.826µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/ingress/ from= latency=1.809µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/role/name/ from= latency=1.754µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/binding-rules from= latency=1.961µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/acl/authorize from= latency=1.713µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/operator/raft/configuration from= latency=1.624µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/config/ from= latency=1.907µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/event/list from= latency=1.932µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/snapshot from= latency=1.825µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/services from= latency=1.767µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/operator/keyring from= latency=2.086µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/create from= latency=1.774µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/update from= latency=1.857µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/clone/ from= latency=1.761µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/token/ from= latency=1.739µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/metrics from= latency=2.219µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/token/ from= latency=2.084µs +30:48.827 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/federation-states from= latency=2.372µs +30:48.828 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/rules/translate/ from= latency=1.784µs +30:48.828 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/service/ from= latency=1.673µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/service/ from= latency=2.931µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/coordinate/update from= latency=2.139µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/connect/ca/roots from= latency=2.182µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/intentions/ from= latency=2.422µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/binding-rule from= latency=2.099µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/token from= latency=2.107µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/gateway-services/ from= latency=2.037µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/connect/ca/leaf/ from= latency=2.047µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/ca/configuration from= latency=2.304µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/destroy/ from= latency=2.136µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/renew/ from= latency=1.963µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/bootstrap from= latency=2.128µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/pass/ from= latency=2.125µs +30:48.837 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/node/ from= latency=2.063µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/service/ from= latency=2.028µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/members from= latency=1.92µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/ui/nodes from= latency=2.155µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/session/list from= latency=1.99µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/status/leader from= latency=1.946µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/login from= latency=1.974µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/nodes from= latency=1.928µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/replication from= latency=2.025µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/policy/name/ from= latency=1.934µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/policy/ from= latency=2.146µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/force-leave/ from= latency=1.963µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/deregister from= latency=2.032µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/auth-method/ from= latency=2.208µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/token/self from= latency=2.015µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/service/register from= latency=2.114µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/create from= latency=2.033µs +30:48.838 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/register from= latency=1.914µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/discovery-chain/ from= latency=1.974µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/operator/raft/peer from= latency=2.505µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/operator/autopilot/health from= latency=2.296µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/txn from= latency=2.204µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/maintenance from= latency=2.344µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/register from= latency=2.293µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/intentions/match from= latency=2.332µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/connect/intentions/check from= latency=2.191µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/leave from= latency=2.238µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/check/warn/ from= latency=2.406µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/service/deregister/ from= latency=2.184µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/node-services/ from= latency=2.255µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/connect/ from= latency=2.329µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/list from= latency=2.246µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/rules/translate from= latency=2.34µs +30:48.839 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/self from= latency=2.23µs +30:48.840 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/health/state/ from= latency=2.274µs +30:48.840 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/role from= latency=2.321µs +30:48.845 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/tokens from= latency=3.211µs +30:48.845 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/checks from= latency=2.429µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/federation-state/ from= latency=2.37µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/operator/autopilot/configuration from= latency=2.49µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/role/ from= latency=2.688µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/agent/health/service/name/ from= latency=2.239µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/catalog/datacenters from= latency=2.399µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/internal/federation-states/mesh-gateways from= latency=2.4µs +30:48.846 [DEBUG] TestAgent.http: Request finished: method=OPTIONS url=http://127.0.0.1:30290/v1/acl/info/ from= latency=2.299µs +30:48.846 [INFO] TestAgent: Requesting shutdown +30:48.846 [INFO] TestAgent.server: shutting down server +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:48.846 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:48.846 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:48.846 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:48.846 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:48.860 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:48.868Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:48.868Z [INFO] agent.router.manager: shutting down +30:48.868 [INFO] TestAgent: consul server down +30:48.868 [INFO] TestAgent: shutdown complete +30:48.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30289 network=tcp +30:48.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30289 network=udp +30:48.868 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30290 network=tcp +30:49.369 [INFO] TestAgent: Waiting for endpoints to shut down +30:49.369 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_OptionMethod_OSS (0.99s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/explain (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/execute (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/query/xxx/explain (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/leave (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/replication (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/gateway-services/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/update/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/services (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/id/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/node/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/configuration (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/node/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/info/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-methods (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/deregister/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/connect/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/checks/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/services (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/match (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/check (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/bootstrap (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/datacenters (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/roles (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/service/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/connect/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/health (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/name/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/authorize (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/checks/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/host (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/nodes (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/node/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/destroy/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/node/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/node/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/maintenance/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/connect/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/kv/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policies (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/ingress/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/name/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rules (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/configuration (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/list (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/services (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/clone/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/token/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/metrics (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/ (0.01s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/service/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/update (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/roots (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rule (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/gateway-services/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/configuration (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/name/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/renew/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/node/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/service/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/members (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/nodes (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/list (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/status/leader (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/login (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/self (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/nodes (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/node/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/members (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/deregister (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/gateway-services-nodes/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/nodes (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/roles (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/discovery-chain/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/state/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/replication (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/name/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/list (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-methods (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/deregister (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/auth-method/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/token/self (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/info/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/tokens (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/id/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/fail/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-state/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/peer (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/register (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/create (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/datacenters (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/node/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/create (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/ui/services (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/update/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/connect/ca/leaf/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/discovery-chain/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/raft/peer (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/health (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/match (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions/check (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/deregister/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/node-services/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/node/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/policies (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/intentions (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/ingress/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/destroy/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/connect/ca/roots (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/binding-rules (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/connect/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/list (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/rules/translate (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/self (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/health/state/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/tokens (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/checks (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-state/ (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/role/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/health/service/name/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/service/maintenance/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/service/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/list (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/config (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/coordinate/nodes (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/session/info/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/event/list (0.00s) - --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/clone/ (0.00s) --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/catalog/datacenters (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/internal/federation-states/mesh-gateways (0.00s) + --- PASS: TestHTTPAPI_OptionMethod_OSS/OPTIONS_/v1/acl/info/ (0.00s) === RUN TestHTTPAPI_AllowedNets_OSS [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/internal/acl/authorize +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/config/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/snapshot +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/create +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/update +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/clone/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/token/ +=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/operator/keyring +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/keyring +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/keyring === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/token/ === RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/token/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/maintenance -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/auth-method/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/force-leave/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/register -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/coordinate/update === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/connect/intentions/ === RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/connect/intentions/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/bootstrap +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/token -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/join/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/renew/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule/ -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/binding-rule/ === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/connect/ca/configuration -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/register +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/bootstrap +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/pass/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/destroy/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/renew/ === RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/login -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/token/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/register -=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/agent/connect/authorize -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/coordinate/update -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/deregister -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/config/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role/ -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/role/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method -=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/operator/keyring -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/keyring -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/keyring -=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/logout -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/event/fire/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/snapshot -=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/discovery-chain/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/autopilot/configuration -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/txn === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy/ === RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/policy/ -=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/internal/acl/authorize -=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/raft/peer -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/pass/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/force-leave/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/deregister +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/auth-method/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/register === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/create +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/register +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/txn +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/maintenance +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/register +=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/discovery-chain/ +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/raft/peer +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/leave +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/warn/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/deregister/ === RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/rules/translate -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/create +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/autopilot/configuration +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role/ +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/role/ === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/update/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/config +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/deregister/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/join/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/fail/ +=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/agent/connect/authorize === RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/connect/intentions +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/event/fire/ +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule/ +=== RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/binding-rule/ === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/destroy/ +=== RUN TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/logout +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method +=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/maintenance/ === RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/kv/ === RUN TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/kv/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/destroy/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/maintenance/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/update -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/deregister/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/config -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/warn/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/clone/ -=== RUN TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/leave [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:42.106Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:42.106Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:42.122Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:42.122Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:42.144 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06ca5c04-6037-36b8-0cd7-a302e6ffd891 Address:127.0.0.1:11923}]" -45:42.144 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11923 [Follower]" leader= -45:42.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891.dc1 127.0.0.1 -45:42.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 127.0.0.1 -2022-11-07T22:45:42.145Z [INFO] agent.router: Initializing LAN area manager -45:42.145 [INFO] TestAgent.server: Adding LAN server: server="Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 (Addr: tcp/127.0.0.1:11923) (DC: dc1)" -45:42.145 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891.dc1 area=wan -45:42.145 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11918 network=tcp -45:42.145 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11918 network=udp -45:42.146 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11919 network=tcp -45:42.146 [INFO] TestAgent: started state syncer -45:42.146 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11924 network=tcp -45:42.189 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:42.189 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11923 [Candidate]" term=2 -45:42.195 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:42.198 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:42.198 [DEBUG] TestAgent.server.raft: vote granted: from=06ca5c04-6037-36b8-0cd7-a302e6ffd891 term=2 tally=1 -45:42.198 [INFO] TestAgent.server.raft: election won: tally=1 -45:42.198 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11923 [Leader]" -45:42.198 [INFO] TestAgent.server: cluster leadership acquired -45:42.198 [INFO] TestAgent.server: New leader elected: payload=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 -45:42.203 [INFO] TestAgent.server: initializing acls -45:42.206 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:42.211 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:42.211 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:42.211 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:42.211 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 -45:42.211 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891.dc1 -45:42.211 [INFO] TestAgent.server: Updating LAN server: server="Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 (Addr: tcp/127.0.0.1:11923) (DC: dc1)" -45:42.211 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891.dc1 area=wan -45:42.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:42.237 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:42.237 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:42.237 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:42.237 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:42.237 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:42.237 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 -45:42.237 [INFO] TestAgent.server: member joined, marking health alive: member=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 -45:42.241 [INFO] TestAgent.server: federation state anti-entropy synced -45:42.247 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06ca5c04-6037-36b8-0cd7-a302e6ffd891 -45:42.451 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:42.455 [INFO] TestAgent: Synced node info -45:42.455 [DEBUG] TestAgent: Node info in sync -45:42.569 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/token/ from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/token/ from=192.168.1.2:5555 latency=45.192µs -45:42.569 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/acl/token/ from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/acl/token/ from=192.168.1.2:5555 latency=18.729µs -45:42.569 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/maintenance from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/maintenance from=192.168.1.2:5555 latency=10.471µs -45:42.569 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/auth-method/ from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/auth-method/ from=192.168.1.2:5555 latency=10.135µs -45:42.569 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/acl/auth-method/ from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/acl/auth-method/ from=192.168.1.2:5555 latency=8.96µs -45:42.569 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/force-leave/ from=192.168.1.2:5555 error="Access is restricted" -45:42.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/force-leave/ from=192.168.1.2:5555 latency=14.248µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/catalog/register from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/catalog/register from=192.168.1.2:5555 latency=9.085µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/binding-rule from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/binding-rule from=192.168.1.2:5555 latency=8.614µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/deregister/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/deregister/ from=192.168.1.2:5555 latency=8.976µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/connect/intentions/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/connect/intentions/ from=192.168.1.2:5555 latency=9.548µs -45:42.570 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/connect/intentions/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/connect/intentions/ from=192.168.1.2:5555 latency=8.742µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/bootstrap from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/bootstrap from=192.168.1.2:5555 latency=9.629µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/token from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/token from=192.168.1.2:5555 latency=9.272µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/join/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/join/ from=192.168.1.2:5555 latency=8.913µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/session/renew/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/session/renew/ from=192.168.1.2:5555 latency=12.46µs -45:42.570 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/binding-rule/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/binding-rule/ from=192.168.1.2:5555 latency=8.792µs -45:42.570 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/acl/binding-rule/ from=192.168.1.2:5555 error="Access is restricted" -45:42.570 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/acl/binding-rule/ from=192.168.1.2:5555 latency=8.771µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/connect/ca/configuration from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/connect/ca/configuration from=192.168.1.2:5555 latency=8.725µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/service/register from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/service/register from=192.168.1.2:5555 latency=8.879µs -45:42.571 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/acl/login from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/acl/login from=192.168.1.2:5555 latency=25.489µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/policy from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/policy from=192.168.1.2:5555 latency=9.053µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/token/ from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/token/ from=192.168.1.2:5555 latency=8.486µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/register from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/register from=192.168.1.2:5555 latency=8.674µs -45:42.571 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/agent/connect/authorize from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/agent/connect/authorize from=192.168.1.2:5555 latency=8.758µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/coordinate/update from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/coordinate/update from=192.168.1.2:5555 latency=12.133µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/role from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/role from=192.168.1.2:5555 latency=8.387µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/catalog/deregister from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/catalog/deregister from=192.168.1.2:5555 latency=8.457µs -45:42.571 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/config/ from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/config/ from=192.168.1.2:5555 latency=9.427µs -45:42.571 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/role/ from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/role/ from=192.168.1.2:5555 latency=8.62µs -45:42.571 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/acl/role/ from=192.168.1.2:5555 error="Access is restricted" -45:42.571 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/acl/role/ from=192.168.1.2:5555 latency=9.124µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/auth-method from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/auth-method from=192.168.1.2:5555 latency=8.834µs -45:42.572 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 latency=8.915µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 latency=23.905µs -45:42.572 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/operator/keyring from=192.168.1.2:5555 latency=21.466µs -45:42.572 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/acl/logout from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/acl/logout from=192.168.1.2:5555 latency=8.996µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/event/fire/ from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/event/fire/ from=192.168.1.2:5555 latency=8.414µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/snapshot from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/snapshot from=192.168.1.2:5555 latency=8.388µs -45:42.572 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/discovery-chain/ from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/discovery-chain/ from=192.168.1.2:5555 latency=8.808µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/operator/autopilot/configuration from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/operator/autopilot/configuration from=192.168.1.2:5555 latency=8.41µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/txn from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/txn from=192.168.1.2:5555 latency=8.76µs -45:42.572 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/policy/ from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/policy/ from=192.168.1.2:5555 latency=9.311µs -45:42.572 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/acl/policy/ from=192.168.1.2:5555 error="Access is restricted" -45:42.572 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/acl/policy/ from=192.168.1.2:5555 latency=9.259µs -45:42.573 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/internal/acl/authorize from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/internal/acl/authorize from=192.168.1.2:5555 latency=9.036µs -45:42.573 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/operator/raft/peer from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/operator/raft/peer from=192.168.1.2:5555 latency=9.314µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/pass/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/pass/ from=192.168.1.2:5555 latency=8.386µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/fail/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/fail/ from=192.168.1.2:5555 latency=8.348µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/create from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/create from=192.168.1.2:5555 latency=8.75µs -45:42.573 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/acl/rules/translate from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/acl/rules/translate from=192.168.1.2:5555 latency=13.194µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/session/create from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/session/create from=192.168.1.2:5555 latency=8.681µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/update/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/update/ from=192.168.1.2:5555 latency=9.324µs -45:42.573 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:11919/v1/connect/intentions from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:11919/v1/connect/intentions from=192.168.1.2:5555 latency=8.502µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/destroy/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/destroy/ from=192.168.1.2:5555 latency=9.018µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/kv/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/kv/ from=192.168.1.2:5555 latency=8.582µs -45:42.573 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:11919/v1/kv/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:11919/v1/kv/ from=192.168.1.2:5555 latency=8.774µs -45:42.573 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/session/destroy/ from=192.168.1.2:5555 error="Access is restricted" -45:42.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/session/destroy/ from=192.168.1.2:5555 latency=8.834µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/service/maintenance/ from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/service/maintenance/ from=192.168.1.2:5555 latency=13.521µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/update from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/update from=192.168.1.2:5555 latency=8.511µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/service/deregister/ from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/service/deregister/ from=192.168.1.2:5555 latency=8.756µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/config from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/config from=192.168.1.2:5555 latency=9.819µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/check/warn/ from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/check/warn/ from=192.168.1.2:5555 latency=8.89µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/acl/clone/ from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/acl/clone/ from=192.168.1.2:5555 latency=8.824µs -45:42.574 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:11919/v1/agent/leave from=192.168.1.2:5555 error="Access is restricted" -45:42.574 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:11919/v1/agent/leave from=192.168.1.2:5555 latency=8.261µs -45:42.574 [INFO] TestAgent: Requesting shutdown -45:42.574 [INFO] TestAgent.server: shutting down server -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:42.574 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:42.574 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:42.574 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:42.574 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:42.578 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:42.581Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:42.581Z [INFO] agent.router.manager: shutting down -45:42.581 [INFO] TestAgent: consul server down -45:42.581 [INFO] TestAgent: shutdown complete -45:42.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11918 network=tcp -45:42.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11918 network=udp -45:42.581 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11919 network=tcp -45:43.081 [INFO] TestAgent: Waiting for endpoints to shut down -45:43.081 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_AllowedNets_OSS (0.99s) +2021-10-05T16:30:49.405Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:30:49.405Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:49.449Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:30:49.449Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:49.552 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae534b84-0e36-9919-8fd7-cd065de13933 Address:127.0.0.1:30301}]" +30:49.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 127.0.0.1 +30:49.553 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ae534b84-0e36-9919-8fd7-cd065de13933 127.0.0.1 +2021-10-05T16:30:49.553Z [INFO] agent.router: Initializing LAN area manager +30:49.553 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30301 [Follower]" leader= +30:49.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30296 network=udp +30:49.554 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 area=wan +30:49.554 [INFO] TestAgent.server: Adding LAN server: server="Node-ae534b84-0e36-9919-8fd7-cd065de13933 (Addr: tcp/127.0.0.1:30301) (DC: dc1)" +30:49.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30296 network=tcp +30:49.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30297 network=tcp +30:49.554 [INFO] TestAgent: started state syncer +30:49.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30302 network=tcp +30:49.604 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:49.612 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:49.612 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30301 [Candidate]" term=2 +30:49.660 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:49.660 [DEBUG] TestAgent.server.raft: vote granted: from=ae534b84-0e36-9919-8fd7-cd065de13933 term=2 tally=1 +30:49.660 [INFO] TestAgent.server.raft: election won: tally=1 +30:49.660 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30301 [Leader]" +30:49.664 [INFO] TestAgent.server: cluster leadership acquired +30:49.664 [INFO] TestAgent.server: New leader elected: payload=Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.682 [INFO] TestAgent.server: initializing acls +30:49.696 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:49.704 [INFO] TestAgent.server: initializing acls +30:49.716 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:49.736 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:49.736 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:49.736 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:49.736 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:49.736 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.736 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 +30:49.736 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 area=wan +30:49.736 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:49.737 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.737 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 +30:49.737 [INFO] TestAgent.server: Updating LAN server: server="Node-ae534b84-0e36-9919-8fd7-cd065de13933 (Addr: tcp/127.0.0.1:30301) (DC: dc1)" +30:49.737 [INFO] TestAgent.server: Updating LAN server: server="Node-ae534b84-0e36-9919-8fd7-cd065de13933 (Addr: tcp/127.0.0.1:30301) (DC: dc1)" +30:49.737 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ae534b84-0e36-9919-8fd7-cd065de13933.dc1 area=wan +30:49.780 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:49.802 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:49.802 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:49.802 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:49.802 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:49.802 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:49.803 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.803 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.816 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.816 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae534b84-0e36-9919-8fd7-cd065de13933 +30:49.820 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/internal/acl/authorize from=192.168.1.2:5555 error="Access is restricted" +30:49.820 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/internal/acl/authorize from=192.168.1.2:5555 latency=33.394µs +30:49.820 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/config/ from=192.168.1.2:5555 error="Access is restricted" +30:49.820 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/config/ from=192.168.1.2:5555 latency=14.627µs +30:49.820 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/snapshot from=192.168.1.2:5555 error="Access is restricted" +30:49.820 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/snapshot from=192.168.1.2:5555 latency=14.733µs +30:49.820 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/session/create from=192.168.1.2:5555 error="Access is restricted" +30:49.820 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/session/create from=192.168.1.2:5555 latency=14.377µs +30:49.820 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/update from=192.168.1.2:5555 error="Access is restricted" +30:49.820 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/update from=192.168.1.2:5555 latency=13.886µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/clone/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/clone/ from=192.168.1.2:5555 latency=14.522µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/token/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/token/ from=192.168.1.2:5555 latency=14.371µs +30:49.821 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 latency=13.915µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 latency=14.753µs +30:49.821 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/operator/keyring from=192.168.1.2:5555 latency=13.986µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/token/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/token/ from=192.168.1.2:5555 latency=14.148µs +30:49.821 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/acl/token/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/acl/token/ from=192.168.1.2:5555 latency=13.861µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/coordinate/update from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/coordinate/update from=192.168.1.2:5555 latency=13.874µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/connect/intentions/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/connect/intentions/ from=192.168.1.2:5555 latency=14.007µs +30:49.821 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/connect/intentions/ from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/connect/intentions/ from=192.168.1.2:5555 latency=13.359µs +30:49.821 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/binding-rule from=192.168.1.2:5555 error="Access is restricted" +30:49.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/binding-rule from=192.168.1.2:5555 latency=14.213µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/token from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/token from=192.168.1.2:5555 latency=14.129µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/connect/ca/configuration from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/connect/ca/configuration from=192.168.1.2:5555 latency=12.489µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/bootstrap from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/bootstrap from=192.168.1.2:5555 latency=12.343µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/pass/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/pass/ from=192.168.1.2:5555 latency=12.373µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/session/destroy/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/session/destroy/ from=192.168.1.2:5555 latency=13.355µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/session/renew/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/session/renew/ from=192.168.1.2:5555 latency=12.613µs +30:49.822 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/acl/login from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/acl/login from=192.168.1.2:5555 latency=11.834µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/policy/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/policy/ from=192.168.1.2:5555 latency=14.772µs +30:49.822 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/acl/policy/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/acl/policy/ from=192.168.1.2:5555 latency=12.79µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/force-leave/ from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/force-leave/ from=192.168.1.2:5555 latency=12.388µs +30:49.822 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/catalog/deregister from=192.168.1.2:5555 error="Access is restricted" +30:49.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/catalog/deregister from=192.168.1.2:5555 latency=12.637µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/auth-method/ from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/auth-method/ from=192.168.1.2:5555 latency=12.262µs +30:49.823 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/acl/auth-method/ from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/acl/auth-method/ from=192.168.1.2:5555 latency=12.274µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/service/register from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/service/register from=192.168.1.2:5555 latency=12.409µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/create from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/create from=192.168.1.2:5555 latency=32.268µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/register from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/register from=192.168.1.2:5555 latency=13.903µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/txn from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/txn from=192.168.1.2:5555 latency=12.334µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/maintenance from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/maintenance from=192.168.1.2:5555 latency=12.23µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/catalog/register from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/catalog/register from=192.168.1.2:5555 latency=12.419µs +30:49.823 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/discovery-chain/ from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/discovery-chain/ from=192.168.1.2:5555 latency=12.171µs +30:49.823 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/operator/raft/peer from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/operator/raft/peer from=192.168.1.2:5555 latency=12.737µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/leave from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/leave from=192.168.1.2:5555 latency=12.129µs +30:49.823 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/warn/ from=192.168.1.2:5555 error="Access is restricted" +30:49.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/warn/ from=192.168.1.2:5555 latency=12.156µs +30:49.824 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/service/deregister/ from=192.168.1.2:5555 error="Access is restricted" +30:49.824 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/service/deregister/ from=192.168.1.2:5555 latency=12.109µs +30:49.828 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/acl/rules/translate from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/acl/rules/translate from=192.168.1.2:5555 latency=28.26µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/role from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/role from=192.168.1.2:5555 latency=15.794µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/operator/autopilot/configuration from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/operator/autopilot/configuration from=192.168.1.2:5555 latency=13.04µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/role/ from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/role/ from=192.168.1.2:5555 latency=12.359µs +30:49.828 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/acl/role/ from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/acl/role/ from=192.168.1.2:5555 latency=13.254µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/update/ from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/update/ from=192.168.1.2:5555 latency=12.494µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/config from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/config from=192.168.1.2:5555 latency=12.756µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/deregister/ from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/deregister/ from=192.168.1.2:5555 latency=12.804µs +30:49.828 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/join/ from=192.168.1.2:5555 error="Access is restricted" +30:49.828 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/join/ from=192.168.1.2:5555 latency=12.979µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/check/fail/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/check/fail/ from=192.168.1.2:5555 latency=12.538µs +30:49.829 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/agent/connect/authorize from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/agent/connect/authorize from=192.168.1.2:5555 latency=12.095µs +30:49.829 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/connect/intentions from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/connect/intentions from=192.168.1.2:5555 latency=12.54µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/event/fire/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/event/fire/ from=192.168.1.2:5555 latency=14.467µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/binding-rule/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/binding-rule/ from=192.168.1.2:5555 latency=13.332µs +30:49.829 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/acl/binding-rule/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/acl/binding-rule/ from=192.168.1.2:5555 latency=14.309µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/destroy/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/destroy/ from=192.168.1.2:5555 latency=14.514µs +30:49.829 [ERROR] TestAgent.http: Request error: method=POST url=http://127.0.0.1:30297/v1/acl/logout from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=POST url=http://127.0.0.1:30297/v1/acl/logout from=192.168.1.2:5555 latency=15.083µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/policy from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/policy from=192.168.1.2:5555 latency=14.719µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/acl/auth-method from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/acl/auth-method from=192.168.1.2:5555 latency=13.526µs +30:49.829 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/agent/service/maintenance/ from=192.168.1.2:5555 error="Access is restricted" +30:49.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/agent/service/maintenance/ from=192.168.1.2:5555 latency=14.259µs +30:49.830 [ERROR] TestAgent.http: Request error: method=PUT url=http://127.0.0.1:30297/v1/kv/ from=192.168.1.2:5555 error="Access is restricted" +30:49.830 [DEBUG] TestAgent.http: Request finished: method=PUT url=http://127.0.0.1:30297/v1/kv/ from=192.168.1.2:5555 latency=13.74µs +30:49.830 [ERROR] TestAgent.http: Request error: method=DELETE url=http://127.0.0.1:30297/v1/kv/ from=192.168.1.2:5555 error="Access is restricted" +30:49.830 [DEBUG] TestAgent.http: Request finished: method=DELETE url=http://127.0.0.1:30297/v1/kv/ from=192.168.1.2:5555 latency=13.797µs +30:49.830 [INFO] TestAgent: Requesting shutdown +30:49.830 [INFO] TestAgent.server: shutting down server +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:49.830 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:49.830 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:49.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:49.830 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:49.830 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:49.830 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:49.830 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:49.834 [INFO] TestAgent.server: federation state anti-entropy synced +30:49.834 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:49.844 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:49.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:49.852Z [INFO] agent.router.manager: shutting down +30:49.852 [INFO] TestAgent: consul server down +30:49.852 [INFO] TestAgent: shutdown complete +30:49.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30296 network=tcp +30:49.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30296 network=udp +30:49.852 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30297 network=tcp +30:50.353 [INFO] TestAgent: Waiting for endpoints to shut down +30:50.353 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_AllowedNets_OSS (0.98s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/internal/acl/authorize (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/config/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/snapshot (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/update (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/clone/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/token/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/keyring (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/keyring (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/token/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/token/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/maintenance (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/auth-method/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/force-leave/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/register (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/coordinate/update (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/connect/intentions/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/connect/intentions/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/token (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/join/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/renew/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/binding-rule/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/connect/ca/configuration (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/register (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/bootstrap (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/pass/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/destroy/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/renew/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/login (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/token/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/register (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/agent/connect/authorize (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/coordinate/update (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/deregister (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/config/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/role/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/keyring (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/logout (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/event/fire/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/snapshot (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/discovery-chain/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/autopilot/configuration (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/txn (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/policy/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/internal/acl/authorize (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/raft/peer (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/pass/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/force-leave/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/deregister (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/auth-method/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/register (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/create (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/register (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/txn (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/maintenance (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/catalog/register (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/discovery-chain/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/operator/raft/peer (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/leave (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/warn/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/deregister/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/rules/translate (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/create (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/operator/autopilot/configuration (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/role/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/role/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/update/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/config (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/deregister/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/join/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/fail/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/agent/connect/authorize (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/connect/intentions (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/event/fire/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/binding-rule/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/acl/binding-rule/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/destroy/ (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/POST_/v1/acl/logout (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/policy (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/auth-method (0.00s) + --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/maintenance/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/kv/ (0.00s) --- PASS: TestHTTPAPI_AllowedNets_OSS/DELETE_/v1/kv/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/session/destroy/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/maintenance/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/update (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/service/deregister/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/config (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/check/warn/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/acl/clone/ (0.00s) - --- PASS: TestHTTPAPI_AllowedNets_OSS/PUT_/v1/agent/leave (0.00s) === RUN TestHTTPServer_UnixSocket === PAUSE TestHTTPServer_UnixSocket === RUN TestHTTPServer_UnixSocket_FileExists @@ -18416,127 +18579,93 @@ [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:43.102Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:43.111Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:43.135 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9fbb1724-d846-cef9-af58-1f00efc6ae3c Address:127.0.0.1:11930}]" -45:43.135 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11930 [Follower]" leader= -45:43.135 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c.dc1 127.0.0.1 -45:43.136 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c 127.0.0.1 -2022-11-07T22:45:43.136Z [INFO] agent.router: Initializing LAN area manager -45:43.136 [INFO] TestAgent.server: Adding LAN server: server="Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c (Addr: tcp/127.0.0.1:11930) (DC: dc1)" -45:43.136 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c.dc1 area=wan -45:43.136 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11925 network=tcp -45:43.136 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11925 network=udp -45:43.137 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11926 network=tcp -45:43.137 [INFO] TestAgent: started state syncer -45:43.137 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11931 network=tcp -45:43.190 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:43.190 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11930 [Candidate]" term=2 -45:43.198 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:43.198 [DEBUG] TestAgent.server.raft: vote granted: from=9fbb1724-d846-cef9-af58-1f00efc6ae3c term=2 tally=1 -45:43.198 [INFO] TestAgent.server.raft: election won: tally=1 -45:43.198 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11930 [Leader]" -45:43.198 [INFO] TestAgent.server: cluster leadership acquired -45:43.198 [INFO] TestAgent.server: New leader elected: payload=Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c -45:43.204 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11930 -45:43.215 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:43.234 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:43.234 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:43.234 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:43.234 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:43.234 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:43.234 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c -45:43.234 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9fbb1724-d846-cef9-af58-1f00efc6ae3c -45:43.238 [INFO] TestAgent.server: federation state anti-entropy synced -45:43.295 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:43.298 [INFO] TestAgent: Synced node info -45:43.298 [DEBUG] TestAgent: Node info in sync -45:43.424 [INFO] TestAgent: Requesting shutdown -45:43.424 [INFO] TestAgent.server: shutting down server -45:43.424 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:43.424 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:43.424 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:43.424 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:43.424 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:43.424 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:43.424 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:43.424 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:43.424 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:43.424 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:43.428 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:43.431Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:43.431Z [INFO] agent.router.manager: shutting down -45:43.431 [INFO] TestAgent: consul server down -45:43.431 [INFO] TestAgent: shutdown complete -45:43.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11925 network=tcp -45:43.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11925 network=udp -45:43.431 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11926 network=tcp -45:43.938 [INFO] TestAgent: Waiting for endpoints to shut down -45:43.938 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_Ban_Nonprintable_Characters (0.86s) +2021-10-05T16:30:50.388Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:50.414Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:50.471 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b077c4db-924d-32ec-0aea-be4aeb683434 Address:127.0.0.1:30308}]" +30:50.471 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30308 [Follower]" leader= +30:50.472 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b077c4db-924d-32ec-0aea-be4aeb683434.dc1 127.0.0.1 +30:50.473 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b077c4db-924d-32ec-0aea-be4aeb683434 127.0.0.1 +2021-10-05T16:30:50.473Z [INFO] agent.router: Initializing LAN area manager +30:50.473 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30303 network=udp +30:50.473 [INFO] TestAgent.server: Adding LAN server: server="Node-b077c4db-924d-32ec-0aea-be4aeb683434 (Addr: tcp/127.0.0.1:30308) (DC: dc1)" +30:50.473 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b077c4db-924d-32ec-0aea-be4aeb683434.dc1 area=wan +30:50.473 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30303 network=tcp +30:50.474 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30304 network=tcp +30:50.474 [INFO] TestAgent: started state syncer +30:50.474 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30309 network=tcp +30:50.528 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:50.528 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30308 [Candidate]" term=2 +30:50.543 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:50.543 [DEBUG] TestAgent.server.raft: vote granted: from=b077c4db-924d-32ec-0aea-be4aeb683434 term=2 tally=1 +30:50.543 [INFO] TestAgent.server.raft: election won: tally=1 +30:50.543 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30308 [Leader]" +30:50.544 [INFO] TestAgent.server: cluster leadership acquired +30:50.544 [INFO] TestAgent.server: New leader elected: payload=Node-b077c4db-924d-32ec-0aea-be4aeb683434 +30:50.549 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30308 +30:50.588 [INFO] TestAgent: Synced node info +30:50.588 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:50.609 [INFO] TestAgent: Requesting shutdown +30:50.609 [INFO] TestAgent.server: shutting down server +30:50.609 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:50.615 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:50.624Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:50.624Z [INFO] agent.router.manager: shutting down +30:50.624 [INFO] TestAgent: consul server down +30:50.624 [INFO] TestAgent: shutdown complete +30:50.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30303 network=tcp +30:50.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30303 network=udp +30:50.624 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30304 network=tcp +30:50.624 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +30:50.624 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +30:51.128 [INFO] TestAgent: Waiting for endpoints to shut down +30:51.128 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_Ban_Nonprintable_Characters (0.77s) === RUN TestHTTPAPI_Allow_Nonprintable_Characters_With_Flag [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:43.954Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:43.968Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:43.994 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ac12339-a6fe-8bdd-068f-2d13fa06f7dd Address:127.0.0.1:11937}]" -45:43.994 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11937 [Follower]" leader= -45:43.994 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd.dc1 127.0.0.1 -45:43.995 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd 127.0.0.1 -2022-11-07T22:45:43.995Z [INFO] agent.router: Initializing LAN area manager -45:43.995 [INFO] TestAgent.server: Adding LAN server: server="Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd (Addr: tcp/127.0.0.1:11937) (DC: dc1)" -45:43.995 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd.dc1 area=wan -45:43.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11932 network=udp -45:43.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11932 network=tcp -45:43.996 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11933 network=tcp -45:43.996 [INFO] TestAgent: started state syncer -45:43.996 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11938 network=tcp -45:44.036 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:44.036 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11937 [Candidate]" term=2 -45:44.045 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:44.045 [DEBUG] TestAgent.server.raft: vote granted: from=0ac12339-a6fe-8bdd-068f-2d13fa06f7dd term=2 tally=1 -45:44.045 [INFO] TestAgent.server.raft: election won: tally=1 -45:44.045 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11937 [Leader]" -45:44.045 [INFO] TestAgent.server: cluster leadership acquired -45:44.045 [INFO] TestAgent.server: New leader elected: payload=Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd -45:44.050 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11937 -45:44.061 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:44.077 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:44.077 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:44.077 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:44.077 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:44.077 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:44.077 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd -45:44.077 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0ac12339-a6fe-8bdd-068f-2d13fa06f7dd -45:44.083 [INFO] TestAgent.server: federation state anti-entropy synced -45:44.209 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:44.213 [INFO] TestAgent: Synced node info -45:44.288 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/bad%00ness from= latency=42.597µs -45:44.288 [INFO] TestAgent: Requesting shutdown -45:44.288 [INFO] TestAgent.server: shutting down server -45:44.288 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:44.288 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:44.288 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:44.288 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:44.288 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:44.288 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:44.288 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:44.288 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:44.288 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:44.288 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:44.292 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:44.296Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:44.296Z [INFO] agent.router.manager: shutting down -45:44.296 [INFO] TestAgent: consul server down -45:44.296 [INFO] TestAgent: shutdown complete -45:44.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11932 network=tcp -45:44.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11932 network=udp -45:44.296 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11933 network=tcp -45:44.796 [INFO] TestAgent: Waiting for endpoints to shut down -45:44.796 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_Allow_Nonprintable_Characters_With_Flag (0.86s) +2021-10-05T16:30:51.156Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:51.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:51.258 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4dc96da2-5144-9f8a-0324-414a018b1cf3 Address:127.0.0.1:30315}]" +30:51.258 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30315 [Follower]" leader= +30:51.258 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4dc96da2-5144-9f8a-0324-414a018b1cf3.dc1 127.0.0.1 +30:51.259 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4dc96da2-5144-9f8a-0324-414a018b1cf3 127.0.0.1 +2021-10-05T16:30:51.259Z [INFO] agent.router: Initializing LAN area manager +30:51.259 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30310 network=udp +30:51.259 [INFO] TestAgent.server: Adding LAN server: server="Node-4dc96da2-5144-9f8a-0324-414a018b1cf3 (Addr: tcp/127.0.0.1:30315) (DC: dc1)" +30:51.259 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4dc96da2-5144-9f8a-0324-414a018b1cf3.dc1 area=wan +30:51.260 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30310 network=tcp +30:51.260 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30311 network=tcp +30:51.260 [INFO] TestAgent: started state syncer +30:51.260 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30316 network=tcp +30:51.316 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:51.316 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30315 [Candidate]" term=2 +30:51.356 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:51.356 [DEBUG] TestAgent.server.raft: vote granted: from=4dc96da2-5144-9f8a-0324-414a018b1cf3 term=2 tally=1 +30:51.356 [INFO] TestAgent.server.raft: election won: tally=1 +30:51.356 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30315 [Leader]" +30:51.356 [INFO] TestAgent.server: cluster leadership acquired +30:51.356 [INFO] TestAgent.server: New leader elected: payload=Node-4dc96da2-5144-9f8a-0324-414a018b1cf3 +30:51.376 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30315 +30:51.388 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:51.413 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/bad%00ness from= latency=52.232µs +30:51.413 [INFO] TestAgent: Requesting shutdown +30:51.413 [INFO] TestAgent.server: shutting down server +30:51.413 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:51.414 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:51.417 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:51.426Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:51.426Z [INFO] agent.router.manager: shutting down +30:51.432 [INFO] TestAgent: consul server down +30:51.432 [INFO] TestAgent: shutdown complete +30:51.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30310 network=tcp +30:51.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30310 network=udp +30:51.432 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30311 network=tcp +30:51.432 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +30:51.432 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +30:51.936 [INFO] TestAgent: Waiting for endpoints to shut down +30:51.936 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_Allow_Nonprintable_Characters_With_Flag (0.81s) === RUN TestHTTPAPI_TranslateAddrHeader === PAUSE TestHTTPAPI_TranslateAddrHeader === RUN TestHTTPAPIResponseHeaders @@ -18601,62 +18730,62 @@ [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:44.808Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:44.816Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:44.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c5ce7156-b4f4-da49-af80-e5858da25a89 Address:127.0.0.1:11944}]" -45:44.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11944 [Follower]" leader= -45:44.838 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c5ce7156-b4f4-da49-af80-e5858da25a89.dc1 127.0.0.1 -45:44.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c5ce7156-b4f4-da49-af80-e5858da25a89 127.0.0.1 -2022-11-07T22:45:44.839Z [INFO] agent.router: Initializing LAN area manager -45:44.839 [INFO] TestAgent.server: Adding LAN server: server="Node-c5ce7156-b4f4-da49-af80-e5858da25a89 (Addr: tcp/127.0.0.1:11944) (DC: dc1)" -45:44.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c5ce7156-b4f4-da49-af80-e5858da25a89.dc1 area=wan -45:44.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11939 network=tcp -45:44.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11939 network=udp -45:44.839 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11940 network=tcp -45:44.839 [INFO] TestAgent: started state syncer -45:44.839 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11945 network=tcp -45:44.903 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:44.903 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11944 [Candidate]" term=2 -45:44.911 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:44.911 [DEBUG] TestAgent.server.raft: vote granted: from=c5ce7156-b4f4-da49-af80-e5858da25a89 term=2 tally=1 -45:44.911 [INFO] TestAgent.server.raft: election won: tally=1 -45:44.911 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11944 [Leader]" -45:44.911 [INFO] TestAgent.server: cluster leadership acquired -45:44.911 [INFO] TestAgent.server: New leader elected: payload=Node-c5ce7156-b4f4-da49-af80-e5858da25a89 -45:44.916 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11944 -45:44.926 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:44.944 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:44.944 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:44.944 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:44.944 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:44.944 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:44.944 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c5ce7156-b4f4-da49-af80-e5858da25a89 -45:44.944 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c5ce7156-b4f4-da49-af80-e5858da25a89 -45:44.953 [INFO] TestAgent.server: federation state anti-entropy synced -45:45.012 [INFO] TestAgent: Requesting shutdown -45:45.012 [INFO] TestAgent.server: shutting down server -45:45.012 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:45.012 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:45.012 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:45.012 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:45.012 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:45.012 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:45.012 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:45.012 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:45.012 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:45.012 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:45.012 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:45.015 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:45.019Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:45.019Z [INFO] agent.router.manager: shutting down -45:45.019 [INFO] TestAgent: consul server down -45:45.019 [INFO] TestAgent: shutdown complete -45:45.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11939 network=tcp -45:45.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11939 network=udp -45:45.019 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11940 network=tcp -45:45.519 [INFO] TestAgent: Waiting for endpoints to shut down -45:45.519 [INFO] TestAgent: Endpoints down ---- PASS: TestParseConsistencyAndMaxStale (0.72s) +2021-10-05T16:30:51.973Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:52.013Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:52.131 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 Address:127.0.0.1:30322}]" +30:52.133 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2.dc1 127.0.0.1 +30:52.133 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 127.0.0.1 +2021-10-05T16:30:52.133Z [INFO] agent.router: Initializing LAN area manager +30:52.134 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30317 network=udp +30:52.134 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30322 [Follower]" leader= +30:52.134 [INFO] TestAgent.server: Adding LAN server: server="Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 (Addr: tcp/127.0.0.1:30322) (DC: dc1)" +30:52.134 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2.dc1 area=wan +30:52.134 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30317 network=tcp +30:52.135 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30318 network=tcp +30:52.135 [INFO] TestAgent: started state syncer +30:52.135 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30323 network=tcp +30:52.188 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:52.188 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30322 [Candidate]" term=2 +30:52.206 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:52.206 [DEBUG] TestAgent.server.raft: vote granted: from=b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 term=2 tally=1 +30:52.206 [INFO] TestAgent.server.raft: election won: tally=1 +30:52.206 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30322 [Leader]" +30:52.206 [INFO] TestAgent.server: cluster leadership acquired +30:52.206 [INFO] TestAgent.server: New leader elected: payload=Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 +30:52.217 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30322 +30:52.230 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:52.255 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:52.255 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:52.255 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:52.255 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:52.255 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:52.256 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 +30:52.256 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b2e0db46-9f9c-fbbb-38f8-3e3becd7c3a2 +30:52.278 [INFO] TestAgent.server: federation state anti-entropy synced +30:52.280 [INFO] TestAgent: Requesting shutdown +30:52.280 [INFO] TestAgent.server: shutting down server +30:52.280 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:52.280 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:52.280 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:52.280 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:52.280 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:52.280 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:52.280 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:52.280 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:52.280 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:52.280 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:52.280 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:52.288 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:52.288Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:52.288Z [INFO] agent.router.manager: shutting down +30:52.288 [INFO] TestAgent: consul server down +30:52.288 [INFO] TestAgent: shutdown complete +30:52.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30317 network=tcp +30:52.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30317 network=udp +30:52.288 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30318 network=tcp +30:52.792 [INFO] TestAgent: Waiting for endpoints to shut down +30:52.792 [INFO] TestAgent: Endpoints down +--- PASS: TestParseConsistencyAndMaxStale (0.85s) === RUN TestParseConsistency_Invalid === PAUSE TestParseConsistency_Invalid === RUN TestACLResolution @@ -18748,8 +18877,8 @@ === RUN TestNewNodeIDFromConfig/valid_NodeID_in_config === RUN TestNewNodeIDFromConfig/invalid_NodeID_in_file === RUN TestNewNodeIDFromConfig/valid_NodeID_in_file ---- PASS: TestNewNodeIDFromConfig (0.01s) - --- PASS: TestNewNodeIDFromConfig/a_new_ID_is_generated_when_none_is_specified (0.00s) +--- PASS: TestNewNodeIDFromConfig (0.02s) + --- PASS: TestNewNodeIDFromConfig/a_new_ID_is_generated_when_none_is_specified (0.01s) --- PASS: TestNewNodeIDFromConfig/running_again_should_get_the_NodeID_that_was_persisted_to_disk (0.00s) --- PASS: TestNewNodeIDFromConfig/invalid_NodeID_in_config (0.00s) --- PASS: TestNewNodeIDFromConfig/valid_NodeID_in_config (0.00s) @@ -18844,256 +18973,240 @@ [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:45.537Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:45.546Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:45.568 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:719a74b5-eb60-b1bb-955d-63a6269021da Address:127.0.0.1:11951}]" -45:45.568 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11951 [Follower]" leader= -45:45.568 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-719a74b5-eb60-b1bb-955d-63a6269021da.dc1 127.0.0.1 -45:45.569 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-719a74b5-eb60-b1bb-955d-63a6269021da 127.0.0.1 -2022-11-07T22:45:45.569Z [INFO] agent.router: Initializing LAN area manager -45:45.569 [INFO] TestAgent.server: Adding LAN server: server="Node-719a74b5-eb60-b1bb-955d-63a6269021da (Addr: tcp/127.0.0.1:11951) (DC: dc1)" -45:45.569 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-719a74b5-eb60-b1bb-955d-63a6269021da.dc1 area=wan -45:45.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11946 network=tcp -45:45.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11946 network=udp -45:45.570 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11947 network=tcp -45:45.570 [INFO] TestAgent: started state syncer -45:45.570 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11952 network=tcp -45:45.614 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:45.614 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11951 [Candidate]" term=2 -45:45.622 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:45.622 [DEBUG] TestAgent.server.raft: vote granted: from=719a74b5-eb60-b1bb-955d-63a6269021da term=2 tally=1 -45:45.622 [INFO] TestAgent.server.raft: election won: tally=1 -45:45.622 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11951 [Leader]" -45:45.623 [INFO] TestAgent.server: cluster leadership acquired -45:45.623 [INFO] TestAgent.server: New leader elected: payload=Node-719a74b5-eb60-b1bb-955d-63a6269021da -45:45.628 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11951 -45:45.639 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:45.658 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:45.658 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:45.658 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:45.658 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:45.658 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:45.658 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-719a74b5-eb60-b1bb-955d-63a6269021da -45:45.658 [INFO] TestAgent.server: member joined, marking health alive: member=Node-719a74b5-eb60-b1bb-955d-63a6269021da -45:45.669 [INFO] TestAgent.server: federation state anti-entropy synced -45:45.752 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:45.755 [INFO] TestAgent: Synced node info -45:45.850 [INFO] TestAgent: Requesting shutdown -45:45.850 [INFO] TestAgent.server: shutting down server -45:45.850 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:45.850 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:45.850 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:45.850 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:45.850 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:45.850 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:45.850 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:45.850 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:45.850 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:45.850 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:45.854 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:45.857Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:45.857Z [INFO] agent.router.manager: shutting down -45:45.857 [INFO] TestAgent: consul server down -45:45.857 [INFO] TestAgent: shutdown complete -45:45.857 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11946 network=tcp -45:45.857 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11946 network=udp -45:45.857 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11947 network=tcp -45:46.358 [INFO] TestAgent: Waiting for endpoints to shut down -45:46.358 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_RegisterService (0.83s) +2021-10-05T16:30:52.856Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:52.890Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:52.964 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2 Address:127.0.0.1:30329}]" +30:52.964 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30329 [Follower]" leader= +30:52.965 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2.dc1 127.0.0.1 +30:52.965 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2 127.0.0.1 +2021-10-05T16:30:52.965Z [INFO] agent.router: Initializing LAN area manager +30:52.966 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30324 network=udp +30:52.966 [INFO] TestAgent.server: Adding LAN server: server="Node-cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2 (Addr: tcp/127.0.0.1:30329) (DC: dc1)" +30:52.966 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2.dc1 area=wan +30:52.966 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30324 network=tcp +30:52.967 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30325 network=tcp +30:52.967 [INFO] TestAgent: started state syncer +30:52.967 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30330 network=tcp +30:53.012 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:53.012 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30329 [Candidate]" term=2 +30:53.038 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:53.038 [DEBUG] TestAgent.server.raft: vote granted: from=cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2 term=2 tally=1 +30:53.038 [INFO] TestAgent.server.raft: election won: tally=1 +30:53.038 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30329 [Leader]" +30:53.038 [INFO] TestAgent.server: cluster leadership acquired +30:53.038 [INFO] TestAgent.server: New leader elected: payload=Node-cfd92bb8-dcbd-d5ac-9a50-2f4ddbf321d2 +30:53.072 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30329 +30:53.108 [INFO] TestAgent: Synced node info +30:53.108 [DEBUG] TestAgent: Node info in sync +30:53.142 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:53.191 [INFO] TestAgent: Requesting shutdown +30:53.191 [INFO] TestAgent.server: shutting down server +30:53.191 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:53.196 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:53.204Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:53.204Z [INFO] agent.router.manager: shutting down +30:53.209 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +30:53.209 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +30:53.210 [INFO] TestAgent: consul server down +30:53.210 [INFO] TestAgent: shutdown complete +30:53.210 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30324 network=tcp +30:53.210 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30324 network=udp +30:53.210 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30325 network=tcp +30:53.712 [INFO] TestAgent: Waiting for endpoints to shut down +30:53.712 [INFO] TestAgent: Endpoints down +--- PASS: TestServiceManager_RegisterService (0.89s) === RUN TestServiceManager_RegisterSidecar [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:46.367Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:46.376Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:46.402 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7e4fff3a-be87-878f-3276-47105bf70f62 Address:127.0.0.1:11958}]" -45:46.402 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11958 [Follower]" leader= -45:46.402 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7e4fff3a-be87-878f-3276-47105bf70f62.dc1 127.0.0.1 -45:46.403 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7e4fff3a-be87-878f-3276-47105bf70f62 127.0.0.1 -2022-11-07T22:45:46.403Z [INFO] agent.router: Initializing LAN area manager -45:46.403 [INFO] TestAgent.server: Adding LAN server: server="Node-7e4fff3a-be87-878f-3276-47105bf70f62 (Addr: tcp/127.0.0.1:11958) (DC: dc1)" -45:46.403 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7e4fff3a-be87-878f-3276-47105bf70f62.dc1 area=wan -45:46.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11953 network=udp -45:46.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11953 network=tcp -45:46.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11954 network=tcp -45:46.404 [INFO] TestAgent: started state syncer -45:46.404 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11959 network=tcp -45:46.444 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:46.444 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11958 [Candidate]" term=2 -45:46.452 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:46.452 [DEBUG] TestAgent.server.raft: vote granted: from=7e4fff3a-be87-878f-3276-47105bf70f62 term=2 tally=1 -45:46.452 [INFO] TestAgent.server.raft: election won: tally=1 -45:46.452 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11958 [Leader]" -45:46.452 [INFO] TestAgent.server: cluster leadership acquired -45:46.453 [INFO] TestAgent.server: New leader elected: payload=Node-7e4fff3a-be87-878f-3276-47105bf70f62 -45:46.457 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11958 -45:46.467 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:46.485 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:46.485 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:46.485 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:46.485 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:46.485 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:46.485 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7e4fff3a-be87-878f-3276-47105bf70f62 -45:46.485 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7e4fff3a-be87-878f-3276-47105bf70f62 -45:46.489 [INFO] TestAgent.server: federation state anti-entropy synced -45:46.539 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:46.542 [INFO] TestAgent: Synced node info -45:46.754 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -45:46.754 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -45:46.754 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -45:46.754 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -45:46.758 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -45:46.758 [INFO] TestAgent: Requesting shutdown -45:46.758 [INFO] TestAgent.server: shutting down server -45:46.758 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:46.758 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:46.758 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:46.758 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:46.758 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:46.758 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:46.758 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:46.758 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:46.758 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:46.758 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:46.762 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:46.768Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:46.768Z [INFO] agent.router.manager: shutting down -45:46.768 [INFO] TestAgent: consul server down -45:46.768 [INFO] TestAgent: shutdown complete -45:46.768 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11953 network=tcp -45:46.768 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11953 network=udp -45:46.768 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11954 network=tcp -45:47.269 [INFO] TestAgent: Waiting for endpoints to shut down -45:47.269 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_RegisterSidecar (0.91s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:53.751Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:53.791Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:53.884 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3dc83da0-3204-088f-5003-70d69903a4de Address:127.0.0.1:30336}]" +30:53.885 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3dc83da0-3204-088f-5003-70d69903a4de.dc1 127.0.0.1 +30:53.885 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3dc83da0-3204-088f-5003-70d69903a4de 127.0.0.1 +2021-10-05T16:30:53.885Z [INFO] agent.router: Initializing LAN area manager +30:53.886 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30331 network=udp +30:53.886 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30336 [Follower]" leader= +30:53.886 [INFO] TestAgent.server: Adding LAN server: server="Node-3dc83da0-3204-088f-5003-70d69903a4de (Addr: tcp/127.0.0.1:30336) (DC: dc1)" +30:53.886 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3dc83da0-3204-088f-5003-70d69903a4de.dc1 area=wan +30:53.886 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30331 network=tcp +30:53.887 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30332 network=tcp +30:53.887 [INFO] TestAgent: started state syncer +30:53.887 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30337 network=tcp +30:53.935 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:53.935 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30336 [Candidate]" term=2 +30:53.945 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:53.945 [DEBUG] TestAgent.server.raft: vote granted: from=3dc83da0-3204-088f-5003-70d69903a4de term=2 tally=1 +30:53.945 [INFO] TestAgent.server.raft: election won: tally=1 +30:53.945 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30336 [Leader]" +30:53.945 [INFO] TestAgent.server: cluster leadership acquired +30:53.945 [INFO] TestAgent.server: New leader elected: payload=Node-3dc83da0-3204-088f-5003-70d69903a4de +30:53.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30336 +30:53.975 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:54.009 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:54.009 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:54.009 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:54.009 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:54.009 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:54.009 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3dc83da0-3204-088f-5003-70d69903a4de +30:54.009 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3dc83da0-3204-088f-5003-70d69903a4de +30:54.033 [INFO] TestAgent.server: federation state anti-entropy synced +30:54.041 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +30:54.041 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +30:54.041 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +30:54.041 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +30:54.051 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +30:54.051 [INFO] TestAgent: Requesting shutdown +30:54.051 [INFO] TestAgent.server: shutting down server +30:54.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:54.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:54.051 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:54.051 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:54.051 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:54.051 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:54.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:54.051 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:54.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:54.051 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:54.051 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:54.059 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:54.088Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:54.088Z [INFO] agent.router.manager: shutting down +30:54.088 [INFO] TestAgent: consul server down +30:54.088 [INFO] TestAgent: shutdown complete +30:54.088 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30331 network=tcp +30:54.088 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30331 network=udp +30:54.088 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30332 network=tcp +30:54.596 [INFO] TestAgent: Waiting for endpoints to shut down +30:54.596 [INFO] TestAgent: Endpoints down +--- PASS: TestServiceManager_RegisterSidecar (0.88s) === RUN TestServiceManager_RegisterMeshGateway [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:47.289Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:47.298Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:47.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc5b6fed-4f95-d74e-0f24-5dd866d2553e Address:127.0.0.1:11965}]" -45:47.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11965 [Follower]" leader= -45:47.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e.dc1 127.0.0.1 -45:47.321 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e 127.0.0.1 -2022-11-07T22:45:47.321Z [INFO] agent.router: Initializing LAN area manager -45:47.321 [INFO] TestAgent.server: Adding LAN server: server="Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e (Addr: tcp/127.0.0.1:11965) (DC: dc1)" -45:47.321 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e.dc1 area=wan -45:47.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11960 network=tcp -45:47.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11960 network=udp -45:47.322 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11961 network=tcp -45:47.322 [INFO] TestAgent: started state syncer -45:47.322 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11966 network=tcp -45:47.390 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:47.390 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11965 [Candidate]" term=2 -45:47.398 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:47.398 [DEBUG] TestAgent.server.raft: vote granted: from=bc5b6fed-4f95-d74e-0f24-5dd866d2553e term=2 tally=1 -45:47.398 [INFO] TestAgent.server.raft: election won: tally=1 -45:47.398 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11965 [Leader]" -45:47.398 [INFO] TestAgent.server: cluster leadership acquired -45:47.398 [INFO] TestAgent.server: New leader elected: payload=Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e -45:47.403 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11965 -45:47.413 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:47.430 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:47.430 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:47.430 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:47.430 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:47.430 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:47.430 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e -45:47.430 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc5b6fed-4f95-d74e-0f24-5dd866d2553e -45:47.442 [INFO] TestAgent.server: federation state anti-entropy synced -45:47.530 [DEBUG] TestAgent: added local registration for service: service=mesh-gateway -45:47.530 [INFO] TestAgent: Requesting shutdown -45:47.530 [INFO] TestAgent.server: shutting down server -45:47.530 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:47.530 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:47.530 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:47.530 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:47.530 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:47.530 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:47.530 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:47.530 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:47.530 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:47.530 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:47.530 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:47.534 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:47.538Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:47.538Z [INFO] agent.router.manager: shutting down -45:47.538 [INFO] TestAgent: consul server down -45:47.538 [INFO] TestAgent: shutdown complete -45:47.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11960 network=tcp -45:47.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11960 network=udp -45:47.538 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11961 network=tcp -45:48.038 [INFO] TestAgent: Waiting for endpoints to shut down -45:48.038 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_RegisterMeshGateway (0.77s) +2021-10-05T16:30:54.642Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:54.658Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:54.690 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 Address:127.0.0.1:30343}]" +30:54.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2.dc1 127.0.0.1 +30:54.691 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 127.0.0.1 +2021-10-05T16:30:54.691Z [INFO] agent.router: Initializing LAN area manager +30:54.692 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30343 [Follower]" leader= +30:54.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30338 network=udp +30:54.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30338 network=tcp +30:54.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2.dc1 area=wan +30:54.692 [INFO] TestAgent.server: Adding LAN server: server="Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +30:54.692 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30339 network=tcp +30:54.692 [INFO] TestAgent: started state syncer +30:54.692 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30344 network=tcp +30:54.760 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:54.760 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30343 [Candidate]" term=2 +30:54.778 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:54.778 [DEBUG] TestAgent.server.raft: vote granted: from=71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 term=2 tally=1 +30:54.778 [INFO] TestAgent.server.raft: election won: tally=1 +30:54.778 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30343 [Leader]" +30:54.778 [INFO] TestAgent.server: cluster leadership acquired +30:54.778 [INFO] TestAgent.server: New leader elected: payload=Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 +30:54.784 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30343 +30:54.808 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:54.834 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:54.834 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:54.834 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:54.834 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:54.834 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:54.834 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 +30:54.834 [INFO] TestAgent.server: member joined, marking health alive: member=Node-71e9cfb8-f69e-cdd2-7184-3523f3c26eb2 +30:54.835 [INFO] TestAgent: Synced node info +30:54.835 [DEBUG] TestAgent: Node info in sync +30:54.855 [INFO] TestAgent.server: federation state anti-entropy synced +30:55.023 [DEBUG] TestAgent: added local registration for service: service=mesh-gateway +30:55.023 [INFO] TestAgent: Requesting shutdown +30:55.024 [INFO] TestAgent.server: shutting down server +30:55.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:55.024 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:55.024 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:55.024 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:55.024 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:55.024 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:55.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:55.024 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:55.024 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:55.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:55.029 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:55.029Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:55.029Z [INFO] agent.router.manager: shutting down +30:55.029 [INFO] TestAgent: consul server down +30:55.029 [INFO] TestAgent: shutdown complete +30:55.029 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30338 network=tcp +30:55.029 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30338 network=udp +30:55.029 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30339 network=tcp +30:55.532 [INFO] TestAgent: Waiting for endpoints to shut down +30:55.532 [INFO] TestAgent: Endpoints down +--- PASS: TestServiceManager_RegisterMeshGateway (0.94s) === RUN TestServiceManager_RegisterTerminatingGateway [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:48.050Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:48.059Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:48.141 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63a1a002-d641-548c-758f-a8e9efdb68c5 Address:127.0.0.1:11972}]" -45:48.141 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11972 [Follower]" leader= -45:48.141 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63a1a002-d641-548c-758f-a8e9efdb68c5.dc1 127.0.0.1 -45:48.142 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63a1a002-d641-548c-758f-a8e9efdb68c5 127.0.0.1 -2022-11-07T22:45:48.142Z [INFO] agent.router: Initializing LAN area manager -45:48.142 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11967 network=udp -45:48.142 [INFO] TestAgent.server: Adding LAN server: server="Node-63a1a002-d641-548c-758f-a8e9efdb68c5 (Addr: tcp/127.0.0.1:11972) (DC: dc1)" -45:48.142 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63a1a002-d641-548c-758f-a8e9efdb68c5.dc1 area=wan -45:48.142 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11967 network=tcp -45:48.143 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11968 network=tcp -45:48.143 [INFO] TestAgent: started state syncer -45:48.143 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11973 network=tcp -45:48.197 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:48.197 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11972 [Candidate]" term=2 -45:48.204 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:48.204 [DEBUG] TestAgent.server.raft: vote granted: from=63a1a002-d641-548c-758f-a8e9efdb68c5 term=2 tally=1 -45:48.204 [INFO] TestAgent.server.raft: election won: tally=1 -45:48.204 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11972 [Leader]" -45:48.205 [INFO] TestAgent.server: cluster leadership acquired -45:48.205 [INFO] TestAgent.server: New leader elected: payload=Node-63a1a002-d641-548c-758f-a8e9efdb68c5 -45:48.209 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11972 -45:48.220 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:48.240 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:48.240 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:48.240 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:48.240 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:48.240 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:48.240 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63a1a002-d641-548c-758f-a8e9efdb68c5 -45:48.240 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63a1a002-d641-548c-758f-a8e9efdb68c5 -45:48.249 [INFO] TestAgent.server: federation state anti-entropy synced -45:48.281 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:48.285 [INFO] TestAgent: Synced node info -45:48.583 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=terminating-gateway -45:48.583 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=terminating-gateway -45:48.587 [DEBUG] TestAgent: added local registration for service: service=terminating-gateway -45:48.587 [INFO] TestAgent: Requesting shutdown -45:48.587 [INFO] TestAgent.server: shutting down server -45:48.587 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:48.587 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:48.587 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:48.587 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:48.587 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:48.587 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:48.587 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:48.587 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:48.587 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:48.587 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:48.590 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:48.594Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:48.594Z [INFO] agent.router.manager: shutting down -45:48.594 [INFO] TestAgent: consul server down -45:48.594 [INFO] TestAgent: shutdown complete -45:48.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11967 network=tcp -45:48.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11967 network=udp -45:48.594 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11968 network=tcp -45:49.094 [INFO] TestAgent: Waiting for endpoints to shut down -45:49.094 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_RegisterTerminatingGateway (1.06s) +2021-10-05T16:30:55.559Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:55.593Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:55.646 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c0b5397-fbe2-4fa0-6579-1617b6bcc437 Address:127.0.0.1:30350}]" +30:55.646 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30350 [Follower]" leader= +30:55.647 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437.dc1 127.0.0.1 +30:55.647 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437 127.0.0.1 +2021-10-05T16:30:55.647Z [INFO] agent.router: Initializing LAN area manager +30:55.647 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30345 network=udp +30:55.648 [INFO] TestAgent.server: Adding LAN server: server="Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437 (Addr: tcp/127.0.0.1:30350) (DC: dc1)" +30:55.648 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437.dc1 area=wan +30:55.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30345 network=tcp +30:55.648 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30346 network=tcp +30:55.649 [INFO] TestAgent: started state syncer +30:55.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30351 network=tcp +30:55.708 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:55.708 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30350 [Candidate]" term=2 +30:55.721 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:55.721 [DEBUG] TestAgent.server.raft: vote granted: from=6c0b5397-fbe2-4fa0-6579-1617b6bcc437 term=2 tally=1 +30:55.721 [INFO] TestAgent.server.raft: election won: tally=1 +30:55.721 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30350 [Leader]" +30:55.721 [INFO] TestAgent.server: cluster leadership acquired +30:55.721 [INFO] TestAgent.server: New leader elected: payload=Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437 +30:55.750 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30350 +30:55.790 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:55.825 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:55.825 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:55.825 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:55.825 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:55.825 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:55.826 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437 +30:55.826 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6c0b5397-fbe2-4fa0-6579-1617b6bcc437 +30:55.858 [INFO] TestAgent.server: federation state anti-entropy synced +30:55.863 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=terminating-gateway +30:55.864 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=terminating-gateway +30:55.886 [DEBUG] TestAgent: added local registration for service: service=terminating-gateway +30:55.886 [INFO] TestAgent: Requesting shutdown +30:55.886 [INFO] TestAgent.server: shutting down server +30:55.886 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:55.886 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:55.886 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:55.886 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:55.886 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:55.886 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:55.886 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:55.887 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:55.887 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:55.887 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:55.887 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:55.891 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:55.891Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:55.891Z [INFO] agent.router.manager: shutting down +30:55.891 [INFO] TestAgent: consul server down +30:55.891 [INFO] TestAgent: shutdown complete +30:55.891 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30345 network=tcp +30:55.891 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30345 network=udp +30:55.891 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30346 network=tcp +30:56.392 [INFO] TestAgent: Waiting for endpoints to shut down +30:56.392 [INFO] TestAgent: Endpoints down +--- PASS: TestServiceManager_RegisterTerminatingGateway (0.86s) === RUN TestServiceManager_PersistService_API service_manager_test.go:264: DM-disabled --- SKIP: TestServiceManager_PersistService_API (0.00s) @@ -19103,64 +19216,64 @@ [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:49.106Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:49.115Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:49.142 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83485d53-86b4-b0df-8a69-f375316b1c4c Address:127.0.0.1:11979}]" -45:49.142 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11979 [Follower]" leader= -45:49.143 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83485d53-86b4-b0df-8a69-f375316b1c4c.dc1 127.0.0.1 -45:49.143 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83485d53-86b4-b0df-8a69-f375316b1c4c 127.0.0.1 -2022-11-07T22:45:49.143Z [INFO] agent.router: Initializing LAN area manager -45:49.143 [INFO] TestAgent.server: Adding LAN server: server="Node-83485d53-86b4-b0df-8a69-f375316b1c4c (Addr: tcp/127.0.0.1:11979) (DC: dc1)" -45:49.143 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83485d53-86b4-b0df-8a69-f375316b1c4c.dc1 area=wan -45:49.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11974 network=udp -45:49.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11974 network=tcp -45:49.144 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11975 network=tcp -45:49.144 [INFO] TestAgent: started state syncer -45:49.144 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11980 network=tcp -45:49.209 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:49.209 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11979 [Candidate]" term=2 -45:49.217 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:49.217 [DEBUG] TestAgent.server.raft: vote granted: from=83485d53-86b4-b0df-8a69-f375316b1c4c term=2 tally=1 -45:49.217 [INFO] TestAgent.server.raft: election won: tally=1 -45:49.217 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11979 [Leader]" -45:49.217 [INFO] TestAgent.server: cluster leadership acquired -45:49.217 [INFO] TestAgent.server: New leader elected: payload=Node-83485d53-86b4-b0df-8a69-f375316b1c4c -45:49.223 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11979 -45:49.234 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:49.252 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:49.252 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:49.252 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:49.252 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:49.252 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:49.252 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83485d53-86b4-b0df-8a69-f375316b1c4c -45:49.252 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83485d53-86b4-b0df-8a69-f375316b1c4c -45:49.257 [INFO] TestAgent.server: federation state anti-entropy synced -45:49.273 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:49.276 [INFO] TestAgent: Synced node info -45:49.276 [DEBUG] TestAgent: Node info in sync -45:49.495 [INFO] TestAgent: Requesting shutdown -45:49.496 [INFO] TestAgent.server: shutting down server -45:49.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:49.496 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:49.496 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:49.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:49.496 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:49.496 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:49.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:49.496 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:49.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:49.496 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:49.499 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:49.503Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:49.503Z [INFO] agent.router.manager: shutting down -45:49.503 [INFO] TestAgent: consul server down -45:49.503 [INFO] TestAgent: shutdown complete -45:49.503 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11974 network=tcp -45:49.503 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11974 network=udp -45:49.503 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11975 network=tcp -45:50.003 [INFO] TestAgent: Waiting for endpoints to shut down -45:50.003 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_Disabled (0.91s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:56.411Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:56.426Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:56.518 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 Address:127.0.0.1:30357}]" +30:56.518 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30357 [Follower]" leader= +30:56.519 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6.dc1 127.0.0.1 +30:56.520 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 127.0.0.1 +2021-10-05T16:30:56.520Z [INFO] agent.router: Initializing LAN area manager +30:56.520 [INFO] TestAgent.server: Adding LAN server: server="Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 (Addr: tcp/127.0.0.1:30357) (DC: dc1)" +30:56.520 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6.dc1 area=wan +30:56.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30352 network=tcp +30:56.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30352 network=udp +30:56.521 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30353 network=tcp +30:56.521 [INFO] TestAgent: started state syncer +30:56.521 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30358 network=tcp +30:56.565 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:56.565 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30357 [Candidate]" term=2 +30:56.573 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:56.573 [DEBUG] TestAgent.server.raft: vote granted: from=2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 term=2 tally=1 +30:56.573 [INFO] TestAgent.server.raft: election won: tally=1 +30:56.573 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30357 [Leader]" +30:56.576 [INFO] TestAgent.server: cluster leadership acquired +30:56.576 [INFO] TestAgent.server: New leader elected: payload=Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 +30:56.579 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30357 +30:56.591 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:56.644 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:56.644 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:56.644 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:56.644 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:56.644 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:56.644 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 +30:56.644 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2acc50cf-1c5f-0877-bae3-1e0a47a8adf6 +30:56.676 [INFO] TestAgent.server: federation state anti-entropy synced +30:56.852 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:56.872 [INFO] TestAgent: Synced node info +30:56.956 [INFO] TestAgent: Requesting shutdown +30:56.956 [INFO] TestAgent.server: shutting down server +30:56.956 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:56.956 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:56.956 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:56.956 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:56.956 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:56.957 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:56.957 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:56.957 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:56.957 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:56.957 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:56.968 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:56.980Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:56.980Z [INFO] agent.router.manager: shutting down +30:56.980 [INFO] TestAgent: consul server down +30:56.980 [INFO] TestAgent: shutdown complete +30:56.980 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30352 network=tcp +30:56.980 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30352 network=udp +30:56.980 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30353 network=tcp +30:57.481 [INFO] TestAgent: Waiting for endpoints to shut down +30:57.481 [INFO] TestAgent: Endpoints down +--- PASS: TestServiceManager_Disabled (1.09s) === RUN TestSessionCreate === PAUSE TestSessionCreate === RUN TestSessionCreate_NodeChecks @@ -19185,126 +19298,127 @@ [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:50.013Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:50.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:50.043 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f2039472-53d7-c54f-01c2-a71730712a00 Address:127.0.0.1:11986}]" -45:50.043 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11986 [Follower]" leader= -45:50.044 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f2039472-53d7-c54f-01c2-a71730712a00.dc1 127.0.0.1 -45:50.044 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f2039472-53d7-c54f-01c2-a71730712a00 127.0.0.1 -2022-11-07T22:45:50.044Z [INFO] agent.router: Initializing LAN area manager -45:50.044 [INFO] TestAgent.server: Adding LAN server: server="Node-f2039472-53d7-c54f-01c2-a71730712a00 (Addr: tcp/127.0.0.1:11986) (DC: dc1)" -45:50.044 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f2039472-53d7-c54f-01c2-a71730712a00.dc1 area=wan -45:50.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11981 network=udp -45:50.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11981 network=tcp -45:50.045 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11982 network=tcp -45:50.045 [INFO] TestAgent: started state syncer -45:50.045 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11987 network=tcp -45:50.086 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:50.086 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11986 [Candidate]" term=2 -45:50.095 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:50.095 [DEBUG] TestAgent.server.raft: vote granted: from=f2039472-53d7-c54f-01c2-a71730712a00 term=2 tally=1 -45:50.095 [INFO] TestAgent.server.raft: election won: tally=1 -45:50.095 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11986 [Leader]" -45:50.095 [INFO] TestAgent.server: cluster leadership acquired -45:50.095 [INFO] TestAgent.server: New leader elected: payload=Node-f2039472-53d7-c54f-01c2-a71730712a00 -45:50.101 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11986 -45:50.111 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:50.127 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:50.127 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:50.127 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:50.127 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:50.127 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:50.127 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f2039472-53d7-c54f-01c2-a71730712a00 -45:50.127 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f2039472-53d7-c54f-01c2-a71730712a00 -45:50.131 [INFO] TestAgent.server: federation state anti-entropy synced -45:50.423 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:50.435 [INFO] TestAgent: Requesting shutdown -45:50.435 [INFO] TestAgent.server: shutting down server -45:50.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:50.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:50.435 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:50.435 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:50.435 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:50.435 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:50.435 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:50.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:50.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:50.435 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:50.442 [INFO] TestAgent: Synced node info -45:50.442 [DEBUG] TestAgent: Node info in sync -45:50.443 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:50.446Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:50.446Z [INFO] agent.router.manager: shutting down -45:50.446 [INFO] TestAgent: consul server down -45:50.446 [INFO] TestAgent: shutdown complete -45:50.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11981 network=tcp -45:50.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11981 network=udp -45:50.446 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11982 network=tcp -45:50.946 [INFO] TestAgent: Waiting for endpoints to shut down -45:50.946 [INFO] TestAgent: Endpoints down +2021-10-05T16:30:57.495Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:57.509Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:57.549 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 Address:127.0.0.1:30364}]" +30:57.550 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47.dc1 127.0.0.1 +30:57.550 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 127.0.0.1 +2021-10-05T16:30:57.550Z [INFO] agent.router: Initializing LAN area manager +30:57.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30359 network=udp +30:57.550 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30364 [Follower]" leader= +30:57.551 [INFO] TestAgent.server: Adding LAN server: server="Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 (Addr: tcp/127.0.0.1:30364) (DC: dc1)" +30:57.551 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47.dc1 area=wan +30:57.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30359 network=tcp +30:57.551 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30360 network=tcp +30:57.551 [INFO] TestAgent: started state syncer +30:57.551 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30365 network=tcp +30:57.600 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:57.600 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30364 [Candidate]" term=2 +30:57.642 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:57.643 [DEBUG] TestAgent.server.raft: vote granted: from=ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 term=2 tally=1 +30:57.643 [INFO] TestAgent.server.raft: election won: tally=1 +30:57.643 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30364 [Leader]" +30:57.643 [INFO] TestAgent.server: cluster leadership acquired +30:57.643 [INFO] TestAgent.server: New leader elected: payload=Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 +30:57.659 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30364 +30:57.696 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:57.742 [INFO] TestAgent: Synced node info +30:57.742 [DEBUG] TestAgent: Node info in sync +30:57.748 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:57.748 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:57.748 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:57.748 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:57.748 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:57.748 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 +30:57.748 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ea6e528b-6bc5-e9e7-57ea-6bb66a7b5b47 +30:57.767 [INFO] TestAgent.server: federation state anti-entropy synced +30:57.780 [INFO] TestAgent: Requesting shutdown +30:57.780 [INFO] TestAgent.server: shutting down server +30:57.780 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:57.780 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:57.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:57.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:57.780 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:57.780 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:57.780 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:57.780 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:57.780 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:57.780 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:57.788 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:57.800Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:57.800Z [INFO] agent.router.manager: shutting down +30:57.800 [INFO] TestAgent: consul server down +30:57.800 [INFO] TestAgent: shutdown complete +30:57.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30359 network=tcp +30:57.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30359 network=udp +30:57.800 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30360 network=tcp +30:58.304 [INFO] TestAgent: Waiting for endpoints to shut down +30:58.304 [INFO] TestAgent: Endpoints down === RUN TestSessionList/#01 [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:50.957Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:50.966Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:50.988 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 Address:127.0.0.1:11993}]" -45:50.988 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11993 [Follower]" leader= -45:50.988 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544.dc1 127.0.0.1 -45:50.989 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 127.0.0.1 -2022-11-07T22:45:50.989Z [INFO] agent.router: Initializing LAN area manager -45:50.989 [INFO] TestAgent.server: Adding LAN server: server="Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 (Addr: tcp/127.0.0.1:11993) (DC: dc1)" -45:50.989 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544.dc1 area=wan -45:50.989 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11988 network=udp -45:50.989 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11988 network=tcp -45:50.990 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11989 network=tcp -45:50.990 [INFO] TestAgent: started state syncer -45:50.990 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11994 network=tcp -45:51.031 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:51.031 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11993 [Candidate]" term=2 -45:51.040 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:51.040 [DEBUG] TestAgent.server.raft: vote granted: from=d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 term=2 tally=1 -45:51.040 [INFO] TestAgent.server.raft: election won: tally=1 -45:51.040 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11993 [Leader]" -45:51.040 [INFO] TestAgent.server: cluster leadership acquired -45:51.040 [INFO] TestAgent.server: New leader elected: payload=Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 -45:51.046 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11993 -45:51.056 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:51.074 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:51.074 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:51.074 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:51.074 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:51.074 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:51.074 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 -45:51.074 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d2a5da8c-6b0b-2faa-8eca-77dfc7fd5544 -45:51.082 [INFO] TestAgent.server: federation state anti-entropy synced -45:51.145 [INFO] TestAgent: Requesting shutdown -45:51.146 [INFO] TestAgent.server: shutting down server -45:51.146 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:51.146 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:51.146 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:51.146 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:51.146 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:51.146 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:51.146 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:51.146 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:51.146 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:51.146 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:51.146 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:51.149 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:51.152Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:51.152Z [INFO] agent.router.manager: shutting down -45:51.152 [INFO] TestAgent: consul server down -45:51.152 [INFO] TestAgent: shutdown complete -45:51.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11988 network=tcp -45:51.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11988 network=udp -45:51.152 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11989 network=tcp -45:51.652 [INFO] TestAgent: Waiting for endpoints to shut down -45:51.652 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionList (1.65s) - --- PASS: TestSessionList/#00 (0.94s) - --- PASS: TestSessionList/#01 (0.71s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:58.337Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:58.363Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:58.448 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d8365a2c-20eb-d7ca-4e96-fb53cfd50638 Address:127.0.0.1:30371}]" +30:58.448 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30371 [Follower]" leader= +30:58.449 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638.dc1 127.0.0.1 +30:58.449 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638 127.0.0.1 +2021-10-05T16:30:58.449Z [INFO] agent.router: Initializing LAN area manager +30:58.450 [INFO] TestAgent.server: Adding LAN server: server="Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638 (Addr: tcp/127.0.0.1:30371) (DC: dc1)" +30:58.450 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638.dc1 area=wan +30:58.450 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30366 network=udp +30:58.450 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30366 network=tcp +30:58.450 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30367 network=tcp +30:58.451 [INFO] TestAgent: started state syncer +30:58.451 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30372 network=tcp +30:58.517 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:58.517 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30371 [Candidate]" term=2 +30:58.564 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:58.564 [DEBUG] TestAgent.server.raft: vote granted: from=d8365a2c-20eb-d7ca-4e96-fb53cfd50638 term=2 tally=1 +30:58.564 [INFO] TestAgent.server.raft: election won: tally=1 +30:58.564 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30371 [Leader]" +30:58.564 [INFO] TestAgent.server: cluster leadership acquired +30:58.564 [INFO] TestAgent.server: New leader elected: payload=Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638 +30:58.596 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30371 +30:58.636 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:58.683 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:58.683 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:58.683 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:58.683 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:58.683 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:58.683 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638 +30:58.683 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d8365a2c-20eb-d7ca-4e96-fb53cfd50638 +30:58.726 [INFO] TestAgent.server: federation state anti-entropy synced +30:58.860 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:58.878 [INFO] TestAgent: Synced node info +30:58.912 [INFO] TestAgent: Requesting shutdown +30:58.912 [INFO] TestAgent.server: shutting down server +30:58.912 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:58.912 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:58.912 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:58.912 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:58.912 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:58.912 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:58.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:58.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:58.912 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:58.912 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:58.918 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:58.923Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:58.923Z [INFO] agent.router.manager: shutting down +30:58.923 [INFO] TestAgent: consul server down +30:58.923 [INFO] TestAgent: shutdown complete +30:58.923 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30366 network=tcp +30:58.923 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30366 network=udp +30:58.924 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30367 network=tcp +30:59.427 [INFO] TestAgent: Waiting for endpoints to shut down +30:59.427 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionList (1.95s) + --- PASS: TestSessionList/#00 (0.82s) + --- PASS: TestSessionList/#01 (1.12s) === RUN TestSessionsForNode session_endpoint_test.go:677: DM-skipped --- SKIP: TestSessionsForNode (0.00s) @@ -19314,556 +19428,562 @@ === RUN TestAgent_sidecarServiceFromNodeService/no_sidecar [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:51.665Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:51.674Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:51.699 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cca0637c-92a7-6f2a-6bfc-62b9cec13982 Address:127.0.0.1:12000}]" -45:51.699 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12000 [Follower]" leader= -45:51.700 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982.dc1 127.0.0.1 -45:51.700 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 127.0.0.1 -2022-11-07T22:45:51.700Z [INFO] agent.router: Initializing LAN area manager -45:51.701 [INFO] jones.server: Adding LAN server: server="Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 (Addr: tcp/127.0.0.1:12000) (DC: dc1)" -45:51.701 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982.dc1 area=wan -45:51.701 [INFO] jones: Started DNS server: address=127.0.0.1:11995 network=tcp -45:51.701 [INFO] jones: Started DNS server: address=127.0.0.1:11995 network=udp -45:51.701 [INFO] jones: Started HTTP server: address=127.0.0.1:11996 network=tcp -45:51.701 [INFO] jones: started state syncer -45:51.701 [INFO] jones: Started gRPC server: address=127.0.0.1:12001 network=tcp -45:51.755 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:51.755 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12000 [Candidate]" term=2 -45:51.763 [DEBUG] jones.server.raft: votes: needed=1 -45:51.763 [DEBUG] jones.server.raft: vote granted: from=cca0637c-92a7-6f2a-6bfc-62b9cec13982 term=2 tally=1 -45:51.763 [INFO] jones.server.raft: election won: tally=1 -45:51.763 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12000 [Leader]" -45:51.763 [INFO] jones.server: cluster leadership acquired -45:51.763 [INFO] jones.server: New leader elected: payload=Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 -45:51.768 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12000 -45:51.781 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:51.798 [INFO] jones: Requesting shutdown -45:51.798 [INFO] jones.server: shutting down server -45:51.798 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:51.798 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" -45:51.799 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:51.799 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:51.799 [INFO] jones.leader: started routine: routine="federation state pruning" -45:51.799 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:51.799 [INFO] jones.leader: started routine: routine="CA root pruning" -45:51.799 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 -45:51.799 [INFO] jones.server: member joined, marking health alive: member=Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 -45:51.802 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:51.809Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:51.809Z [INFO] agent.router.manager: shutting down -45:51.815 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -45:51.815 [ERROR] jones.server: failed to reconcile member: member="{Node-cca0637c-92a7-6f2a-6bfc-62b9cec13982 127.0.0.1 11998 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:cca0637c-92a7-6f2a-6bfc-62b9cec13982 port:12000 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11999] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -45:51.815 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:51.815 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:51.815 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:51.815 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:51.815 [INFO] jones: consul server down -45:51.815 [INFO] jones: shutdown complete -45:51.815 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11995 network=tcp -45:51.815 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11995 network=udp -45:51.815 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:51.815 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:51.815 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:51.815 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:51.815 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:11996 network=tcp -45:52.315 [INFO] jones: Waiting for endpoints to shut down -45:52.315 [INFO] jones: Endpoints down +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:30:59.451Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:59.482Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:59.538 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:200a4cd4-6b73-3e1e-611b-faee6771e207 Address:127.0.0.1:30378}]" +30:59.539 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-200a4cd4-6b73-3e1e-611b-faee6771e207.dc1 127.0.0.1 +30:59.540 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30378 [Follower]" leader= +30:59.540 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-200a4cd4-6b73-3e1e-611b-faee6771e207 127.0.0.1 +2021-10-05T16:30:59.540Z [INFO] agent.router: Initializing LAN area manager +30:59.540 [INFO] jones.server: Adding LAN server: server="Node-200a4cd4-6b73-3e1e-611b-faee6771e207 (Addr: tcp/127.0.0.1:30378) (DC: dc1)" +30:59.540 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-200a4cd4-6b73-3e1e-611b-faee6771e207.dc1 area=wan +30:59.540 [INFO] jones: Started DNS server: address=127.0.0.1:30373 network=udp +30:59.540 [INFO] jones: Started DNS server: address=127.0.0.1:30373 network=tcp +30:59.541 [INFO] jones: Started HTTP server: address=127.0.0.1:30374 network=tcp +30:59.541 [INFO] jones: started state syncer +30:59.541 [INFO] jones: Started gRPC server: address=127.0.0.1:30379 network=tcp +30:59.584 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +30:59.584 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30378 [Candidate]" term=2 +30:59.632 [DEBUG] jones.server.raft: votes: needed=1 +30:59.632 [DEBUG] jones.server.raft: vote granted: from=200a4cd4-6b73-3e1e-611b-faee6771e207 term=2 tally=1 +30:59.632 [INFO] jones.server.raft: election won: tally=1 +30:59.632 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30378 [Leader]" +30:59.632 [INFO] jones.server: cluster leadership acquired +30:59.632 [INFO] jones.server: New leader elected: payload=Node-200a4cd4-6b73-3e1e-611b-faee6771e207 +30:59.653 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30378 +30:59.670 [INFO] jones: Synced node info +30:59.670 [DEBUG] jones: Node info in sync +30:59.702 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:59.728 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +30:59.728 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +30:59.728 [INFO] jones.leader: started routine: routine="federation state pruning" +30:59.728 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +30:59.728 [INFO] jones.leader: started routine: routine="CA root pruning" +30:59.728 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-200a4cd4-6b73-3e1e-611b-faee6771e207 +30:59.728 [INFO] jones.server: member joined, marking health alive: member=Node-200a4cd4-6b73-3e1e-611b-faee6771e207 +30:59.767 [INFO] jones.server: federation state anti-entropy synced +30:59.908 [INFO] jones: Requesting shutdown +30:59.908 [INFO] jones.server: shutting down server +30:59.908 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +30:59.908 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +30:59.908 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +30:59.908 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +30:59.908 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +30:59.908 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +30:59.908 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +30:59.908 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +30:59.908 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +30:59.908 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +30:59.916 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:59.923Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:59.923Z [INFO] agent.router.manager: shutting down +30:59.923 [INFO] jones: consul server down +30:59.923 [INFO] jones: shutdown complete +30:59.923 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30373 network=tcp +30:59.924 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30373 network=udp +30:59.924 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30374 network=tcp +31:00.424 [INFO] jones: Waiting for endpoints to shut down +31:00.424 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/all_the_defaults [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:52.327Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:52.336Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:52.359 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8250dd00-16be-4572-aec4-b8c1a144e3c0 Address:127.0.0.1:12007}]" -45:52.359 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12007 [Follower]" leader= -45:52.359 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-8250dd00-16be-4572-aec4-b8c1a144e3c0.dc1 127.0.0.1 -45:52.359 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-8250dd00-16be-4572-aec4-b8c1a144e3c0 127.0.0.1 -2022-11-07T22:45:52.360Z [INFO] agent.router: Initializing LAN area manager -45:52.360 [INFO] jones.server: Adding LAN server: server="Node-8250dd00-16be-4572-aec4-b8c1a144e3c0 (Addr: tcp/127.0.0.1:12007) (DC: dc1)" -45:52.360 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-8250dd00-16be-4572-aec4-b8c1a144e3c0.dc1 area=wan -45:52.360 [INFO] jones: Started DNS server: address=127.0.0.1:12002 network=udp -45:52.360 [INFO] jones: Started DNS server: address=127.0.0.1:12002 network=tcp -45:52.360 [INFO] jones: Started HTTP server: address=127.0.0.1:12003 network=tcp -45:52.360 [INFO] jones: started state syncer -45:52.360 [INFO] jones: Started gRPC server: address=127.0.0.1:12008 network=tcp -45:52.394 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:52.394 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12007 [Candidate]" term=2 -45:52.402 [DEBUG] jones.server.raft: votes: needed=1 -45:52.402 [DEBUG] jones.server.raft: vote granted: from=8250dd00-16be-4572-aec4-b8c1a144e3c0 term=2 tally=1 -45:52.402 [INFO] jones.server.raft: election won: tally=1 -45:52.402 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12007 [Leader]" -45:52.403 [INFO] jones.server: cluster leadership acquired -45:52.403 [INFO] jones.server: New leader elected: payload=Node-8250dd00-16be-4572-aec4-b8c1a144e3c0 -45:52.408 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12007 -45:52.419 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:52.437 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:52.437 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:52.437 [INFO] jones.leader: started routine: routine="federation state pruning" -45:52.438 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:52.438 [INFO] jones.leader: started routine: routine="CA root pruning" -45:52.438 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-8250dd00-16be-4572-aec4-b8c1a144e3c0 -45:52.438 [INFO] jones.server: member joined, marking health alive: member=Node-8250dd00-16be-4572-aec4-b8c1a144e3c0 -45:52.448 [INFO] jones.server: federation state anti-entropy synced -45:52.492 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth -45:52.495 [INFO] jones: Synced node info -45:52.784 [INFO] jones: Requesting shutdown -45:52.784 [INFO] jones.server: shutting down server -45:52.784 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:52.785 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:52.785 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:52.785 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:52.785 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:52.785 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:52.785 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:52.785 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:52.785 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:52.785 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:52.788 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:52.792Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:52.792Z [INFO] agent.router.manager: shutting down -45:52.792 [INFO] jones: consul server down -45:52.792 [INFO] jones: shutdown complete -45:52.792 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12002 network=tcp -45:52.792 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12002 network=udp -45:52.792 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12003 network=tcp -45:53.292 [INFO] jones: Waiting for endpoints to shut down -45:53.292 [INFO] jones: Endpoints down +2021-10-05T16:31:00.460Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:00.471Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:00.500 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2304aed1-3198-ca22-0a89-641d448fe985 Address:127.0.0.1:30385}]" +31:00.501 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-2304aed1-3198-ca22-0a89-641d448fe985.dc1 127.0.0.1 +31:00.502 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-2304aed1-3198-ca22-0a89-641d448fe985 127.0.0.1 +2021-10-05T16:31:00.502Z [INFO] agent.router: Initializing LAN area manager +31:00.502 [INFO] jones: Started DNS server: address=127.0.0.1:30380 network=udp +31:00.502 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30385 [Follower]" leader= +31:00.503 [INFO] jones.server: Adding LAN server: server="Node-2304aed1-3198-ca22-0a89-641d448fe985 (Addr: tcp/127.0.0.1:30385) (DC: dc1)" +31:00.503 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-2304aed1-3198-ca22-0a89-641d448fe985.dc1 area=wan +31:00.503 [INFO] jones: Started DNS server: address=127.0.0.1:30380 network=tcp +31:00.503 [INFO] jones: Started HTTP server: address=127.0.0.1:30381 network=tcp +31:00.503 [INFO] jones: started state syncer +31:00.504 [INFO] jones: Started gRPC server: address=127.0.0.1:30386 network=tcp +31:00.556 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:00.556 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30385 [Candidate]" term=2 +31:00.600 [DEBUG] jones.server.raft: votes: needed=1 +31:00.600 [DEBUG] jones.server.raft: vote granted: from=2304aed1-3198-ca22-0a89-641d448fe985 term=2 tally=1 +31:00.600 [INFO] jones.server.raft: election won: tally=1 +31:00.600 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30385 [Leader]" +31:00.600 [INFO] jones.server: cluster leadership acquired +31:00.600 [INFO] jones.server: New leader elected: payload=Node-2304aed1-3198-ca22-0a89-641d448fe985 +31:00.606 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30385 +31:00.652 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:00.711 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:00.711 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:00.711 [INFO] jones.leader: started routine: routine="federation state pruning" +31:00.711 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:00.711 [INFO] jones.leader: started routine: routine="CA root pruning" +31:00.711 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-2304aed1-3198-ca22-0a89-641d448fe985 +31:00.711 [INFO] jones.server: member joined, marking health alive: member=Node-2304aed1-3198-ca22-0a89-641d448fe985 +31:00.747 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth +31:00.772 [INFO] jones: Synced node info +31:00.772 [INFO] jones: Requesting shutdown +31:00.772 [INFO] jones.server: shutting down server +31:00.772 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:00.772 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:00.772 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:00.772 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:00.772 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:00.772 [INFO] jones.server: federation state anti-entropy synced +31:00.772 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:00.772 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:00.772 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:00.772 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:00.777 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:00.792Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:00.792Z [INFO] agent.router.manager: shutting down +31:00.792 [INFO] jones: consul server down +31:00.792 [INFO] jones: shutdown complete +31:00.792 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30380 network=tcp +31:00.792 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30380 network=udp +31:00.792 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30381 network=tcp +31:01.292 [INFO] jones: Waiting for endpoints to shut down +31:01.292 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/all_the_allowed_overrides [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:53.303Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:53.312Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:53.459 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bcc2dd77-c8e5-0dd6-9907-707733a057d9 Address:127.0.0.1:12014}]" -45:53.459 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12014 [Follower]" leader= -45:53.459 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9.dc1 127.0.0.1 -45:53.460 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9 127.0.0.1 -2022-11-07T22:45:53.460Z [INFO] agent.router: Initializing LAN area manager -45:53.460 [INFO] jones.server: Adding LAN server: server="Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9 (Addr: tcp/127.0.0.1:12014) (DC: dc1)" -45:53.460 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9.dc1 area=wan -45:53.460 [INFO] jones: Started DNS server: address=127.0.0.1:12009 network=udp -45:53.460 [INFO] jones: Started DNS server: address=127.0.0.1:12009 network=tcp -45:53.460 [INFO] jones: Started HTTP server: address=127.0.0.1:12010 network=tcp -45:53.461 [INFO] jones: started state syncer -45:53.461 [INFO] jones: Started gRPC server: address=127.0.0.1:12015 network=tcp -45:53.497 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:53.497 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12014 [Candidate]" term=2 -45:53.505 [DEBUG] jones.server.raft: votes: needed=1 -45:53.505 [DEBUG] jones.server.raft: vote granted: from=bcc2dd77-c8e5-0dd6-9907-707733a057d9 term=2 tally=1 -45:53.505 [INFO] jones.server.raft: election won: tally=1 -45:53.505 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12014 [Leader]" -45:53.505 [INFO] jones.server: cluster leadership acquired -45:53.505 [INFO] jones.server: New leader elected: payload=Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9 -45:53.513 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12014 -45:53.526 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:53.545 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:53.545 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:53.545 [INFO] jones.leader: started routine: routine="federation state pruning" -45:53.545 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:53.545 [INFO] jones.leader: started routine: routine="CA root pruning" -45:53.545 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9 -45:53.545 [INFO] jones.server: member joined, marking health alive: member=Node-bcc2dd77-c8e5-0dd6-9907-707733a057d9 -45:53.557 [INFO] jones.server: federation state anti-entropy synced -45:53.557 [INFO] jones: Synced node info -45:53.557 [DEBUG] jones: Node info in sync -45:53.767 [INFO] jones: Requesting shutdown -45:53.767 [INFO] jones.server: shutting down server -45:53.767 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:53.767 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:53.767 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:53.767 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:53.767 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:53.767 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:53.767 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:53.767 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:53.767 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:53.767 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:53.771 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:53.774Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:53.774Z [INFO] agent.router.manager: shutting down -45:53.774 [INFO] jones: consul server down -45:53.774 [INFO] jones: shutdown complete -45:53.774 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12009 network=tcp -45:53.774 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12009 network=udp -45:53.774 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12010 network=tcp -45:54.274 [INFO] jones: Waiting for endpoints to shut down -45:54.274 [INFO] jones: Endpoints down +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:01.310Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:01.327Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:01.440 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:00033d72-6b94-d4d9-524c-26ffb82367e2 Address:127.0.0.1:30392}]" +31:01.441 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30392 [Follower]" leader= +31:01.441 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-00033d72-6b94-d4d9-524c-26ffb82367e2.dc1 127.0.0.1 +31:01.441 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-00033d72-6b94-d4d9-524c-26ffb82367e2 127.0.0.1 +2021-10-05T16:31:01.441Z [INFO] agent.router: Initializing LAN area manager +31:01.442 [INFO] jones: Started DNS server: address=127.0.0.1:30387 network=udp +31:01.442 [INFO] jones.server: Adding LAN server: server="Node-00033d72-6b94-d4d9-524c-26ffb82367e2 (Addr: tcp/127.0.0.1:30392) (DC: dc1)" +31:01.442 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-00033d72-6b94-d4d9-524c-26ffb82367e2.dc1 area=wan +31:01.442 [INFO] jones: Started DNS server: address=127.0.0.1:30387 network=tcp +31:01.443 [INFO] jones: Started HTTP server: address=127.0.0.1:30388 network=tcp +31:01.443 [INFO] jones: started state syncer +31:01.443 [INFO] jones: Started gRPC server: address=127.0.0.1:30393 network=tcp +31:01.482 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:01.482 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30392 [Candidate]" term=2 +31:01.496 [DEBUG] jones.server.raft: votes: needed=1 +31:01.496 [DEBUG] jones.server.raft: vote granted: from=00033d72-6b94-d4d9-524c-26ffb82367e2 term=2 tally=1 +31:01.496 [INFO] jones.server.raft: election won: tally=1 +31:01.496 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30392 [Leader]" +31:01.496 [INFO] jones.server: cluster leadership acquired +31:01.496 [INFO] jones.server: New leader elected: payload=Node-00033d72-6b94-d4d9-524c-26ffb82367e2 +31:01.529 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30392 +31:01.552 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:01.591 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:01.591 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:01.591 [INFO] jones.leader: started routine: routine="federation state pruning" +31:01.591 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:01.591 [INFO] jones.leader: started routine: routine="CA root pruning" +31:01.591 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-00033d72-6b94-d4d9-524c-26ffb82367e2 +31:01.592 [INFO] jones.server: member joined, marking health alive: member=Node-00033d72-6b94-d4d9-524c-26ffb82367e2 +31:01.611 [INFO] jones.server: federation state anti-entropy synced +31:01.614 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth +31:01.628 [INFO] jones: Synced node info +31:01.628 [DEBUG] jones: Node info in sync +31:01.848 [INFO] jones: Requesting shutdown +31:01.848 [INFO] jones.server: shutting down server +31:01.848 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:01.848 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:01.848 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:01.848 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:01.848 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:01.848 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:01.848 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:01.848 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:01.848 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:01.848 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:01.860 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:01.872Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:01.872Z [INFO] agent.router.manager: shutting down +31:01.872 [INFO] jones: consul server down +31:01.872 [INFO] jones: shutdown complete +31:01.872 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30387 network=tcp +31:01.872 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30387 network=udp +31:01.872 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30388 network=tcp +31:02.372 [INFO] jones: Waiting for endpoints to shut down +31:02.372 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/no_auto_ports_available [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:54.286Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.296Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.326 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 Address:127.0.0.1:12021}]" -45:54.326 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12021 [Follower]" leader= -45:54.327 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7.dc1 127.0.0.1 -45:54.327 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 127.0.0.1 -2022-11-07T22:45:54.327Z [INFO] agent.router: Initializing LAN area manager -45:54.327 [INFO] jones.server: Adding LAN server: server="Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 (Addr: tcp/127.0.0.1:12021) (DC: dc1)" -45:54.328 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7.dc1 area=wan -45:54.328 [INFO] jones: Started DNS server: address=127.0.0.1:12016 network=udp -45:54.328 [INFO] jones: Started DNS server: address=127.0.0.1:12016 network=tcp -45:54.328 [INFO] jones: Started HTTP server: address=127.0.0.1:12017 network=tcp -45:54.328 [INFO] jones: started state syncer -45:54.328 [INFO] jones: Started gRPC server: address=127.0.0.1:12022 network=tcp -45:54.383 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:54.383 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12021 [Candidate]" term=2 -45:54.393 [DEBUG] jones.server.raft: votes: needed=1 -45:54.393 [DEBUG] jones.server.raft: vote granted: from=c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 term=2 tally=1 -45:54.393 [INFO] jones.server.raft: election won: tally=1 -45:54.393 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12021 [Leader]" -45:54.393 [INFO] jones.server: cluster leadership acquired -45:54.393 [INFO] jones.server: New leader elected: payload=Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 -45:54.398 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12021 -45:54.404 [INFO] jones: Synced node info -45:54.404 [DEBUG] jones: Node info in sync -45:54.417 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:54.424 [INFO] jones: Requesting shutdown -45:54.424 [INFO] jones.server: shutting down server -45:54.424 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:54.431 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -45:54.434 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:54.434 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:54.434 [INFO] jones.leader: started routine: routine="federation state pruning" -45:54.434 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:54.434 [INFO] jones.leader: started routine: routine="CA root pruning" -45:54.434 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 -45:54.435 [INFO] jones.server: member joined, marking health alive: member=Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 -2022-11-07T22:45:54.436Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:54.436Z [INFO] agent.router.manager: shutting down -45:54.436 [ERROR] jones.server: failed to reconcile member: member="{Node-c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 127.0.0.1 12019 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:c1ce2fd2-46d8-ae5b-0ed2-09f8846a86a7 port:12021 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:12020] alive 1 5 2 2 5 4}" error="raft is already shutdown" -45:54.436 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:54.436 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:54.436 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:54.436 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:54.436 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:54.436 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:54.436 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:54.439 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -45:54.439 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:54.439 [INFO] jones: consul server down -45:54.439 [INFO] jones: shutdown complete -45:54.439 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12016 network=tcp -45:54.439 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12016 network=udp -45:54.439 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12017 network=tcp -45:54.940 [INFO] jones: Waiting for endpoints to shut down -45:54.940 [INFO] jones: Endpoints down +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:02.389Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:02.422Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:02.463 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3a155d90-9aaf-1594-4db0-637f04efdb77 Address:127.0.0.1:30399}]" +31:02.463 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30399 [Follower]" leader= +31:02.464 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-3a155d90-9aaf-1594-4db0-637f04efdb77.dc1 127.0.0.1 +31:02.464 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-3a155d90-9aaf-1594-4db0-637f04efdb77 127.0.0.1 +2021-10-05T16:31:02.464Z [INFO] agent.router: Initializing LAN area manager +31:02.465 [INFO] jones: Started DNS server: address=127.0.0.1:30394 network=udp +31:02.465 [INFO] jones.server: Adding LAN server: server="Node-3a155d90-9aaf-1594-4db0-637f04efdb77 (Addr: tcp/127.0.0.1:30399) (DC: dc1)" +31:02.465 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-3a155d90-9aaf-1594-4db0-637f04efdb77.dc1 area=wan +31:02.465 [INFO] jones: Started DNS server: address=127.0.0.1:30394 network=tcp +31:02.466 [INFO] jones: Started HTTP server: address=127.0.0.1:30395 network=tcp +31:02.466 [INFO] jones: started state syncer +31:02.466 [INFO] jones: Started gRPC server: address=127.0.0.1:30400 network=tcp +31:02.533 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:02.533 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30399 [Candidate]" term=2 +31:02.544 [DEBUG] jones.server.raft: votes: needed=1 +31:02.544 [DEBUG] jones.server.raft: vote granted: from=3a155d90-9aaf-1594-4db0-637f04efdb77 term=2 tally=1 +31:02.544 [INFO] jones.server.raft: election won: tally=1 +31:02.544 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30399 [Leader]" +31:02.544 [INFO] jones.server: cluster leadership acquired +31:02.544 [INFO] jones.server: New leader elected: payload=Node-3a155d90-9aaf-1594-4db0-637f04efdb77 +31:02.552 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30399 +31:02.592 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:02.688 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:02.688 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:02.688 [INFO] jones.leader: started routine: routine="federation state pruning" +31:02.688 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:02.688 [INFO] jones.leader: started routine: routine="CA root pruning" +31:02.688 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-3a155d90-9aaf-1594-4db0-637f04efdb77 +31:02.688 [INFO] jones.server: member joined, marking health alive: member=Node-3a155d90-9aaf-1594-4db0-637f04efdb77 +31:02.769 [INFO] jones: Synced node info +31:02.784 [INFO] jones.server: federation state anti-entropy synced +31:02.891 [INFO] jones: Requesting shutdown +31:02.891 [INFO] jones.server: shutting down server +31:02.892 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:02.892 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:02.892 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:02.892 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:02.892 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:02.892 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:02.892 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:02.892 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:02.892 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:02.892 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:02.897 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:02.898Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:02.898Z [INFO] agent.router.manager: shutting down +31:02.898 [INFO] jones: consul server down +31:02.898 [INFO] jones: shutdown complete +31:02.898 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30394 network=tcp +31:02.898 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30394 network=udp +31:02.898 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30395 network=tcp +31:03.400 [INFO] jones: Waiting for endpoints to shut down +31:03.400 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/auto_ports_disabled [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:54.950Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.959Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.979 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55295f59-c6a3-7f79-3746-d63b4d2ffacc Address:127.0.0.1:12028}]" -45:54.979 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12028 [Follower]" leader= -45:54.980 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc.dc1 127.0.0.1 -45:54.980 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc 127.0.0.1 -2022-11-07T22:45:54.980Z [INFO] agent.router: Initializing LAN area manager -45:54.980 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc.dc1 area=wan -45:54.980 [INFO] jones.server: Adding LAN server: server="Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc (Addr: tcp/127.0.0.1:12028) (DC: dc1)" -45:54.980 [INFO] jones: Started DNS server: address=127.0.0.1:12023 network=udp -45:54.980 [INFO] jones: Started DNS server: address=127.0.0.1:12023 network=tcp -45:54.981 [INFO] jones: Started HTTP server: address=127.0.0.1:12024 network=tcp -45:54.981 [INFO] jones: started state syncer -45:54.981 [INFO] jones: Started gRPC server: address=127.0.0.1:12029 network=tcp -45:55.045 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:55.045 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12028 [Candidate]" term=2 -45:55.054 [DEBUG] jones.server.raft: votes: needed=1 -45:55.054 [DEBUG] jones.server.raft: vote granted: from=55295f59-c6a3-7f79-3746-d63b4d2ffacc term=2 tally=1 -45:55.054 [INFO] jones.server.raft: election won: tally=1 -45:55.054 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12028 [Leader]" -45:55.054 [INFO] jones.server: cluster leadership acquired -45:55.054 [INFO] jones.server: New leader elected: payload=Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc -45:55.060 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12028 -45:55.072 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:55.086 [INFO] jones: Requesting shutdown -45:55.086 [INFO] jones.server: shutting down server -45:55.086 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:55.087 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" -45:55.089 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:55.094Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:55.094Z [INFO] agent.router.manager: shutting down -45:55.094 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:55.094 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:55.094 [INFO] jones.leader: started routine: routine="federation state pruning" -45:55.094 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:55.094 [INFO] jones.leader: started routine: routine="CA root pruning" -45:55.094 [INFO] jones: consul server down -45:55.094 [INFO] jones: shutdown complete -45:55.094 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12023 network=tcp -45:55.095 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12023 network=udp -45:55.095 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc -45:55.095 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12024 network=tcp -45:55.095 [INFO] jones.server: member joined, marking health alive: member=Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc -45:55.095 [DEBUG] jones.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:55.095 [DEBUG] jones.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:55.095 [DEBUG] jones.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:55.095 [DEBUG] jones.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:55.095 [DEBUG] jones.server: federation states are not enabled in this datacenter: datacenter=dc1 -45:55.095 [ERROR] jones.server: failed to reconcile member: member="{Node-55295f59-c6a3-7f79-3746-d63b4d2ffacc 127.0.0.1 12026 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:55295f59-c6a3-7f79-3746-d63b4d2ffacc port:12028 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:12027] alive 1 5 2 2 5 4}" error="raft is already shutdown" -45:55.095 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:55.095 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:55.095 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:55.095 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:55.095 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:55.095 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:55.095 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:55.095 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:55.595 [INFO] jones: Waiting for endpoints to shut down -45:55.595 [INFO] jones: Endpoints down +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:03.450Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:03.497Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:03.556 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:25d1edb7-a18c-25d3-7994-6474993f404e Address:127.0.0.1:30406}]" +31:03.557 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-25d1edb7-a18c-25d3-7994-6474993f404e.dc1 127.0.0.1 +31:03.558 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-25d1edb7-a18c-25d3-7994-6474993f404e 127.0.0.1 +2021-10-05T16:31:03.558Z [INFO] agent.router: Initializing LAN area manager +31:03.558 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30406 [Follower]" leader= +31:03.559 [INFO] jones: Started DNS server: address=127.0.0.1:30401 network=udp +31:03.559 [INFO] jones.server: Adding LAN server: server="Node-25d1edb7-a18c-25d3-7994-6474993f404e (Addr: tcp/127.0.0.1:30406) (DC: dc1)" +31:03.559 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-25d1edb7-a18c-25d3-7994-6474993f404e.dc1 area=wan +31:03.559 [INFO] jones: Started DNS server: address=127.0.0.1:30401 network=tcp +31:03.559 [INFO] jones: Started HTTP server: address=127.0.0.1:30402 network=tcp +31:03.560 [INFO] jones: started state syncer +31:03.560 [INFO] jones: Started gRPC server: address=127.0.0.1:30407 network=tcp +31:03.599 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:03.599 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30406 [Candidate]" term=2 +31:03.618 [DEBUG] jones.server.raft: votes: needed=1 +31:03.618 [DEBUG] jones.server.raft: vote granted: from=25d1edb7-a18c-25d3-7994-6474993f404e term=2 tally=1 +31:03.618 [INFO] jones.server.raft: election won: tally=1 +31:03.618 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30406 [Leader]" +31:03.618 [INFO] jones.server: cluster leadership acquired +31:03.618 [INFO] jones.server: New leader elected: payload=Node-25d1edb7-a18c-25d3-7994-6474993f404e +31:03.626 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30406 +31:03.638 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:03.672 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:03.672 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:03.672 [INFO] jones.leader: started routine: routine="federation state pruning" +31:03.672 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:03.672 [INFO] jones.leader: started routine: routine="CA root pruning" +31:03.672 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-25d1edb7-a18c-25d3-7994-6474993f404e +31:03.672 [INFO] jones.server: member joined, marking health alive: member=Node-25d1edb7-a18c-25d3-7994-6474993f404e +31:03.706 [INFO] jones.server: federation state anti-entropy synced +31:03.836 [INFO] jones: Requesting shutdown +31:03.836 [INFO] jones.server: shutting down server +31:03.836 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:03.836 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:03.836 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:03.836 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:03.836 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:03.836 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:03.836 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:03.836 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" +31:03.836 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:03.836 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:03.836 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:03.848 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:03.860Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:03.860Z [INFO] agent.router.manager: shutting down +31:03.860 [INFO] jones: consul server down +31:03.860 [INFO] jones: shutdown complete +31:03.860 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30401 network=tcp +31:03.860 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30401 network=udp +31:03.860 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30402 network=tcp +31:04.363 [INFO] jones: Waiting for endpoints to shut down +31:04.363 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/inherit_tags_and_meta [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:55.607Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:55.617Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:55.641 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3f60095c-aa40-6590-06a1-7eea39fd7576 Address:127.0.0.1:12035}]" -45:55.641 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:12035 [Follower]" leader= -45:55.641 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-3f60095c-aa40-6590-06a1-7eea39fd7576.dc1 127.0.0.1 -45:55.642 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-3f60095c-aa40-6590-06a1-7eea39fd7576 127.0.0.1 -2022-11-07T22:45:55.642Z [INFO] agent.router: Initializing LAN area manager -45:55.642 [INFO] jones.server: Adding LAN server: server="Node-3f60095c-aa40-6590-06a1-7eea39fd7576 (Addr: tcp/127.0.0.1:12035) (DC: dc1)" -45:55.642 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-3f60095c-aa40-6590-06a1-7eea39fd7576.dc1 area=wan -45:55.642 [INFO] jones: Started DNS server: address=127.0.0.1:12030 network=udp -45:55.642 [INFO] jones: Started DNS server: address=127.0.0.1:12030 network=tcp -45:55.643 [INFO] jones: Started HTTP server: address=127.0.0.1:12031 network=tcp -45:55.643 [INFO] jones: started state syncer -45:55.643 [INFO] jones: Started gRPC server: address=127.0.0.1:12036 network=tcp -45:55.680 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:55.680 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:12035 [Candidate]" term=2 -45:55.688 [DEBUG] jones.server.raft: votes: needed=1 -45:55.688 [DEBUG] jones.server.raft: vote granted: from=3f60095c-aa40-6590-06a1-7eea39fd7576 term=2 tally=1 -45:55.688 [INFO] jones.server.raft: election won: tally=1 -45:55.688 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:12035 [Leader]" -45:55.688 [INFO] jones.server: cluster leadership acquired -45:55.688 [INFO] jones.server: New leader elected: payload=Node-3f60095c-aa40-6590-06a1-7eea39fd7576 -45:55.694 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12035 -45:55.705 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:55.723 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:55.723 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:55.723 [INFO] jones.leader: started routine: routine="federation state pruning" -45:55.723 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:55.723 [INFO] jones.leader: started routine: routine="CA root pruning" -45:55.723 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-3f60095c-aa40-6590-06a1-7eea39fd7576 -45:55.723 [INFO] jones.server: member joined, marking health alive: member=Node-3f60095c-aa40-6590-06a1-7eea39fd7576 -45:55.734 [INFO] jones.server: federation state anti-entropy synced -45:55.985 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth -45:55.989 [INFO] jones: Synced node info -45:55.989 [DEBUG] jones: Node info in sync -45:56.001 [INFO] jones: Requesting shutdown -45:56.001 [INFO] jones.server: shutting down server -45:56.001 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:56.001 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:56.001 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:56.001 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:56.001 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:56.001 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:56.001 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:56.001 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:56.001 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:56.001 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:56.005 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:56.008Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:56.008Z [INFO] agent.router.manager: shutting down -45:56.008 [INFO] jones: consul server down -45:56.008 [INFO] jones: shutdown complete -45:56.008 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12030 network=tcp -45:56.008 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12030 network=udp -45:56.008 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12031 network=tcp -45:56.509 [INFO] jones: Waiting for endpoints to shut down -45:56.509 [INFO] jones: Endpoints down +2021-10-05T16:31:04.404Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:04.421Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:04.468 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 Address:127.0.0.1:30413}]" +31:04.469 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2.dc1 127.0.0.1 +31:04.469 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 127.0.0.1 +2021-10-05T16:31:04.469Z [INFO] agent.router: Initializing LAN area manager +31:04.469 [INFO] jones: Started DNS server: address=127.0.0.1:30408 network=udp +31:04.470 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:30413 [Follower]" leader= +31:04.470 [INFO] jones.server: Adding LAN server: server="Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 (Addr: tcp/127.0.0.1:30413) (DC: dc1)" +31:04.470 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2.dc1 area=wan +31:04.470 [INFO] jones: Started DNS server: address=127.0.0.1:30408 network=tcp +31:04.471 [INFO] jones: Started HTTP server: address=127.0.0.1:30409 network=tcp +31:04.471 [INFO] jones: started state syncer +31:04.471 [INFO] jones: Started gRPC server: address=127.0.0.1:30414 network=tcp +31:04.540 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:04.540 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:30413 [Candidate]" term=2 +31:04.584 [DEBUG] jones.server.raft: votes: needed=1 +31:04.584 [DEBUG] jones.server.raft: vote granted: from=6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 term=2 tally=1 +31:04.584 [INFO] jones.server.raft: election won: tally=1 +31:04.584 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:30413 [Leader]" +31:04.584 [INFO] jones.server: cluster leadership acquired +31:04.584 [INFO] jones.server: New leader elected: payload=Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 +31:04.616 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30413 +31:04.668 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:04.728 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:04.728 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:04.728 [INFO] jones.leader: started routine: routine="federation state pruning" +31:04.728 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:04.728 [INFO] jones.leader: started routine: routine="CA root pruning" +31:04.728 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 +31:04.728 [INFO] jones.server: member joined, marking health alive: member=Node-6e9a6dd0-9f43-b7c3-c164-b5dbe729e6f2 +31:04.736 [INFO] jones.server: federation state anti-entropy synced +31:04.780 [INFO] jones: Requesting shutdown +31:04.780 [INFO] jones.server: shutting down server +31:04.780 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:04.780 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:04.780 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:04.780 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:04.780 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:04.780 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:04.780 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:04.781 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" +31:04.781 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:04.781 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:04.781 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:04.786 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:04.804Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:04.804Z [INFO] agent.router.manager: shutting down +31:04.804 [INFO] jones: consul server down +31:04.804 [INFO] jones: shutdown complete +31:04.804 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30408 network=tcp +31:04.804 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30408 network=udp +31:04.804 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30409 network=tcp +31:05.308 [INFO] jones: Waiting for endpoints to shut down +31:05.308 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/invalid_check_type [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:56.520Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:56.529Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:56.555 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3984445c-d363-9100-c07b-fc05f8bdc822 Address:127.0.0.1:11022}]" -45:56.555 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:11022 [Follower]" leader= -45:56.556 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-3984445c-d363-9100-c07b-fc05f8bdc822.dc1 127.0.0.1 -45:56.556 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-3984445c-d363-9100-c07b-fc05f8bdc822 127.0.0.1 -2022-11-07T22:45:56.556Z [INFO] agent.router: Initializing LAN area manager -45:56.557 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-3984445c-d363-9100-c07b-fc05f8bdc822.dc1 area=wan -45:56.557 [INFO] jones.server: Adding LAN server: server="Node-3984445c-d363-9100-c07b-fc05f8bdc822 (Addr: tcp/127.0.0.1:11022) (DC: dc1)" -45:56.557 [INFO] jones: Started DNS server: address=127.0.0.1:12037 network=tcp -45:56.557 [INFO] jones: Started DNS server: address=127.0.0.1:12037 network=udp -45:56.557 [INFO] jones: Started HTTP server: address=127.0.0.1:12038 network=tcp -45:56.557 [INFO] jones: started state syncer -45:56.557 [INFO] jones: Started gRPC server: address=127.0.0.1:11023 network=tcp -45:56.605 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:56.605 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:11022 [Candidate]" term=2 -45:56.613 [DEBUG] jones.server.raft: votes: needed=1 -45:56.613 [DEBUG] jones.server.raft: vote granted: from=3984445c-d363-9100-c07b-fc05f8bdc822 term=2 tally=1 -45:56.613 [INFO] jones.server.raft: election won: tally=1 -45:56.613 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:11022 [Leader]" -45:56.613 [INFO] jones.server: cluster leadership acquired -45:56.613 [INFO] jones.server: New leader elected: payload=Node-3984445c-d363-9100-c07b-fc05f8bdc822 -45:56.619 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11022 -45:56.629 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:56.646 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:56.646 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:56.646 [INFO] jones.leader: started routine: routine="federation state pruning" -45:56.646 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:56.646 [INFO] jones.leader: started routine: routine="CA root pruning" -45:56.646 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-3984445c-d363-9100-c07b-fc05f8bdc822 -45:56.646 [INFO] jones.server: member joined, marking health alive: member=Node-3984445c-d363-9100-c07b-fc05f8bdc822 -45:56.659 [INFO] jones: Synced node info -45:56.659 [INFO] jones.server: federation state anti-entropy synced -45:56.982 [INFO] jones: Requesting shutdown -45:56.982 [INFO] jones.server: shutting down server -45:56.982 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:56.982 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:56.982 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:56.982 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:56.982 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:56.982 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:56.982 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:56.982 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:56.982 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:56.982 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:56.987 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:56.990Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:56.990Z [INFO] agent.router.manager: shutting down -45:56.990 [INFO] jones: consul server down -45:56.990 [INFO] jones: shutdown complete -45:56.990 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12037 network=tcp -45:56.990 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:12037 network=udp -45:56.990 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:12038 network=tcp -45:57.490 [INFO] jones: Waiting for endpoints to shut down -45:57.490 [INFO] jones: Endpoints down +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:05.331Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:05.366Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:05.444 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c5d526fd-c62f-424c-e72a-9dec31314d57 Address:127.0.0.1:29400}]" +31:05.444 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:29400 [Follower]" leader= +31:05.445 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-c5d526fd-c62f-424c-e72a-9dec31314d57.dc1 127.0.0.1 +31:05.445 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-c5d526fd-c62f-424c-e72a-9dec31314d57 127.0.0.1 +2021-10-05T16:31:05.445Z [INFO] agent.router: Initializing LAN area manager +31:05.445 [INFO] jones.server: Adding LAN server: server="Node-c5d526fd-c62f-424c-e72a-9dec31314d57 (Addr: tcp/127.0.0.1:29400) (DC: dc1)" +31:05.445 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-c5d526fd-c62f-424c-e72a-9dec31314d57.dc1 area=wan +31:05.446 [INFO] jones: Started DNS server: address=127.0.0.1:30415 network=udp +31:05.446 [INFO] jones: Started DNS server: address=127.0.0.1:30415 network=tcp +31:05.446 [INFO] jones: Started HTTP server: address=127.0.0.1:30416 network=tcp +31:05.446 [INFO] jones: started state syncer +31:05.446 [INFO] jones: Started gRPC server: address=127.0.0.1:29401 network=tcp +31:05.490 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:05.490 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:29400 [Candidate]" term=2 +31:05.511 [DEBUG] jones.server.raft: votes: needed=1 +31:05.511 [DEBUG] jones.server.raft: vote granted: from=c5d526fd-c62f-424c-e72a-9dec31314d57 term=2 tally=1 +31:05.511 [INFO] jones.server.raft: election won: tally=1 +31:05.511 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:29400 [Leader]" +31:05.512 [INFO] jones.server: cluster leadership acquired +31:05.512 [INFO] jones.server: New leader elected: payload=Node-c5d526fd-c62f-424c-e72a-9dec31314d57 +31:05.528 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29400 +31:05.564 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:05.625 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:05.625 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:05.625 [INFO] jones.leader: started routine: routine="federation state pruning" +31:05.625 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:05.625 [INFO] jones.leader: started routine: routine="CA root pruning" +31:05.625 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-c5d526fd-c62f-424c-e72a-9dec31314d57 +31:05.625 [INFO] jones.server: member joined, marking health alive: member=Node-c5d526fd-c62f-424c-e72a-9dec31314d57 +31:05.638 [INFO] jones.server: federation state anti-entropy synced +31:05.775 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth +31:05.779 [INFO] jones: Synced node info +31:05.786 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth +31:05.786 [DEBUG] jones: Node info in sync +31:05.786 [DEBUG] jones: Node info in sync +31:05.868 [INFO] jones: Requesting shutdown +31:05.868 [INFO] jones.server: shutting down server +31:05.868 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:05.868 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:05.868 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:05.868 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:05.868 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:05.868 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:05.868 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:05.868 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:05.868 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:05.868 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:05.876 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:05.882Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:05.882Z [INFO] agent.router.manager: shutting down +31:05.882 [INFO] jones: consul server down +31:05.882 [INFO] jones: shutdown complete +31:05.882 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30415 network=tcp +31:05.882 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:30415 network=udp +31:05.882 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:30416 network=tcp +31:06.384 [INFO] jones: Waiting for endpoints to shut down +31:06.384 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/invalid_meta [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:57.502Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.512Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.559 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba3179ea-e398-0be8-def9-e2bce34126ba Address:127.0.0.1:11029}]" -45:57.559 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:11029 [Follower]" leader= -45:57.559 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-ba3179ea-e398-0be8-def9-e2bce34126ba.dc1 127.0.0.1 -45:57.560 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-ba3179ea-e398-0be8-def9-e2bce34126ba 127.0.0.1 -2022-11-07T22:45:57.560Z [INFO] agent.router: Initializing LAN area manager -45:57.560 [INFO] jones.server: Adding LAN server: server="Node-ba3179ea-e398-0be8-def9-e2bce34126ba (Addr: tcp/127.0.0.1:11029) (DC: dc1)" -45:57.560 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-ba3179ea-e398-0be8-def9-e2bce34126ba.dc1 area=wan -45:57.560 [INFO] jones: Started DNS server: address=127.0.0.1:11024 network=udp -45:57.560 [INFO] jones: Started DNS server: address=127.0.0.1:11024 network=tcp -45:57.561 [INFO] jones: Started HTTP server: address=127.0.0.1:11025 network=tcp -45:57.561 [INFO] jones: started state syncer -45:57.561 [INFO] jones: Started gRPC server: address=127.0.0.1:11030 network=tcp -45:57.601 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.601 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:11029 [Candidate]" term=2 -45:57.614 [DEBUG] jones.server.raft: votes: needed=1 -45:57.614 [DEBUG] jones.server.raft: vote granted: from=ba3179ea-e398-0be8-def9-e2bce34126ba term=2 tally=1 -45:57.614 [INFO] jones.server.raft: election won: tally=1 -45:57.614 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:11029 [Leader]" -45:57.615 [INFO] jones.server: cluster leadership acquired -45:57.615 [INFO] jones.server: New leader elected: payload=Node-ba3179ea-e398-0be8-def9-e2bce34126ba -45:57.621 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11029 -45:57.639 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.656 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.656 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:57.656 [INFO] jones.leader: started routine: routine="federation state pruning" -45:57.656 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:57.656 [INFO] jones.leader: started routine: routine="CA root pruning" -45:57.656 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-ba3179ea-e398-0be8-def9-e2bce34126ba -45:57.656 [INFO] jones.server: member joined, marking health alive: member=Node-ba3179ea-e398-0be8-def9-e2bce34126ba -45:57.666 [INFO] jones.server: federation state anti-entropy synced -45:57.685 [INFO] jones: Requesting shutdown -45:57.685 [INFO] jones.server: shutting down server -45:57.685 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:57.685 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:57.685 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:57.685 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:57.685 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:57.685 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.685 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:57.686 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" -45:57.686 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:57.686 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:57.686 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:57.689 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.692Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.693Z [INFO] agent.router.manager: shutting down -45:57.693 [INFO] jones: consul server down -45:57.693 [INFO] jones: shutdown complete -45:57.693 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11024 network=tcp -45:57.693 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11024 network=udp -45:57.693 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:11025 network=tcp -45:58.193 [INFO] jones: Waiting for endpoints to shut down -45:58.193 [INFO] jones: Endpoints down +2021-10-05T16:31:06.399Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:06.414Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:06.441 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5809bcda-57a1-fa6e-41a1-e2447f9f4954 Address:127.0.0.1:29407}]" +31:06.442 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954.dc1 127.0.0.1 +31:06.443 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954 127.0.0.1 +2021-10-05T16:31:06.443Z [INFO] agent.router: Initializing LAN area manager +31:06.443 [INFO] jones: Started DNS server: address=127.0.0.1:29402 network=udp +31:06.443 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:29407 [Follower]" leader= +31:06.443 [INFO] jones.server: Adding LAN server: server="Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954 (Addr: tcp/127.0.0.1:29407) (DC: dc1)" +31:06.443 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954.dc1 area=wan +31:06.443 [INFO] jones: Started DNS server: address=127.0.0.1:29402 network=tcp +31:06.444 [INFO] jones: Started HTTP server: address=127.0.0.1:29403 network=tcp +31:06.444 [INFO] jones: started state syncer +31:06.444 [INFO] jones: Started gRPC server: address=127.0.0.1:29408 network=tcp +31:06.508 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:06.508 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:29407 [Candidate]" term=2 +31:06.548 [DEBUG] jones.server.raft: votes: needed=1 +31:06.548 [DEBUG] jones.server.raft: vote granted: from=5809bcda-57a1-fa6e-41a1-e2447f9f4954 term=2 tally=1 +31:06.548 [INFO] jones.server.raft: election won: tally=1 +31:06.548 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:29407 [Leader]" +31:06.552 [INFO] jones.server: cluster leadership acquired +31:06.552 [INFO] jones.server: New leader elected: payload=Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954 +31:06.567 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29407 +31:06.582 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:06.640 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:06.640 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:06.640 [INFO] jones.leader: started routine: routine="federation state pruning" +31:06.640 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:06.640 [INFO] jones.leader: started routine: routine="CA root pruning" +31:06.640 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954 +31:06.640 [INFO] jones.server: member joined, marking health alive: member=Node-5809bcda-57a1-fa6e-41a1-e2447f9f4954 +31:06.672 [INFO] jones.server: federation state anti-entropy synced +31:06.719 [INFO] jones: Requesting shutdown +31:06.719 [INFO] jones.server: shutting down server +31:06.719 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:06.719 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:06.719 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:06.719 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:06.719 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:06.719 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:06.719 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:06.719 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" +31:06.719 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:06.719 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:06.719 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:06.730 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:06.736Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:06.736Z [INFO] agent.router.manager: shutting down +31:06.736 [INFO] jones: consul server down +31:06.736 [INFO] jones: shutdown complete +31:06.736 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:29402 network=tcp +31:06.736 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:29402 network=udp +31:06.736 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:29403 network=tcp +31:07.240 [INFO] jones: Waiting for endpoints to shut down +31:07.240 [INFO] jones: Endpoints down === RUN TestAgent_sidecarServiceFromNodeService/re-registering_same_sidecar_with_no_port_should_pick_same_one [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:58.203Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:58.212Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:58.233 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:79604a72-425b-fb46-3b0f-61f31ca7f76d Address:127.0.0.1:11036}]" -45:58.233 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:11036 [Follower]" leader= -45:58.233 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-79604a72-425b-fb46-3b0f-61f31ca7f76d.dc1 127.0.0.1 -45:58.234 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-79604a72-425b-fb46-3b0f-61f31ca7f76d 127.0.0.1 -2022-11-07T22:45:58.234Z [INFO] agent.router: Initializing LAN area manager -45:58.234 [INFO] jones.server: Adding LAN server: server="Node-79604a72-425b-fb46-3b0f-61f31ca7f76d (Addr: tcp/127.0.0.1:11036) (DC: dc1)" -45:58.234 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-79604a72-425b-fb46-3b0f-61f31ca7f76d.dc1 area=wan -45:58.234 [INFO] jones: Started DNS server: address=127.0.0.1:11031 network=udp -45:58.234 [INFO] jones: Started DNS server: address=127.0.0.1:11031 network=tcp -45:58.234 [INFO] jones: Started HTTP server: address=127.0.0.1:11032 network=tcp -45:58.235 [INFO] jones: started state syncer -45:58.235 [INFO] jones: Started gRPC server: address=127.0.0.1:11037 network=tcp -45:58.298 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= -45:58.298 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:11036 [Candidate]" term=2 -45:58.308 [DEBUG] jones.server.raft: votes: needed=1 -45:58.308 [DEBUG] jones.server.raft: vote granted: from=79604a72-425b-fb46-3b0f-61f31ca7f76d term=2 tally=1 -45:58.308 [INFO] jones.server.raft: election won: tally=1 -45:58.308 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:11036 [Leader]" -45:58.308 [INFO] jones.server: cluster leadership acquired -45:58.308 [INFO] jones.server: New leader elected: payload=Node-79604a72-425b-fb46-3b0f-61f31ca7f76d -45:58.314 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11036 -45:58.325 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:58.343 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul -45:58.343 [INFO] jones.leader: started routine: routine="federation state anti-entropy" -45:58.343 [INFO] jones.leader: started routine: routine="federation state pruning" -45:58.343 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" -45:58.343 [INFO] jones.leader: started routine: routine="CA root pruning" -45:58.343 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-79604a72-425b-fb46-3b0f-61f31ca7f76d -45:58.343 [INFO] jones.server: member joined, marking health alive: member=Node-79604a72-425b-fb46-3b0f-61f31ca7f76d -45:58.352 [INFO] jones.server: federation state anti-entropy synced -45:58.608 [INFO] jones: Requesting shutdown -45:58.608 [INFO] jones.server: shutting down server -45:58.608 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" -45:58.608 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" -45:58.608 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" -45:58.608 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" -45:58.608 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave -45:58.608 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" -45:58.608 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" -45:58.608 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" -45:58.608 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" -45:58.608 [ERROR] jones.anti_entropy: failed to sync remote state: error="No cluster leader" -45:58.608 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" -45:58.613 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:58.616Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.616Z [INFO] agent.router.manager: shutting down -45:58.616 [INFO] jones: consul server down -45:58.616 [INFO] jones: shutdown complete -45:58.616 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11031 network=tcp -45:58.616 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:11031 network=udp -45:58.616 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:11032 network=tcp -45:59.116 [INFO] jones: Waiting for endpoints to shut down -45:59.116 [INFO] jones: Endpoints down ---- PASS: TestAgent_sidecarServiceFromNodeService (7.46s) - --- PASS: TestAgent_sidecarServiceFromNodeService/no_sidecar (0.66s) - --- PASS: TestAgent_sidecarServiceFromNodeService/all_the_defaults (0.98s) - --- PASS: TestAgent_sidecarServiceFromNodeService/all_the_allowed_overrides (0.98s) - --- PASS: TestAgent_sidecarServiceFromNodeService/no_auto_ports_available (0.66s) - --- PASS: TestAgent_sidecarServiceFromNodeService/auto_ports_disabled (0.66s) - --- PASS: TestAgent_sidecarServiceFromNodeService/inherit_tags_and_meta (0.91s) - --- PASS: TestAgent_sidecarServiceFromNodeService/invalid_check_type (0.98s) - --- PASS: TestAgent_sidecarServiceFromNodeService/invalid_meta (0.70s) - --- PASS: TestAgent_sidecarServiceFromNodeService/re-registering_same_sidecar_with_no_port_should_pick_same_one (0.92s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:07.255Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:07.270Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:07.364 [INFO] jones.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a83c5b5d-865a-1aa5-809b-507dbe0c43c2 Address:127.0.0.1:29414}]" +31:07.365 [INFO] jones.server.serf.wan: serf: EventMemberJoin: Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2.dc1 127.0.0.1 +31:07.365 [INFO] jones.server.serf.lan: serf: EventMemberJoin: Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2 127.0.0.1 +2021-10-05T16:31:07.365Z [INFO] agent.router: Initializing LAN area manager +31:07.366 [INFO] jones: Started DNS server: address=127.0.0.1:29409 network=udp +31:07.366 [INFO] jones.server.raft: entering follower state: follower="Node at 127.0.0.1:29414 [Follower]" leader= +31:07.366 [INFO] jones.server: Adding LAN server: server="Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2 (Addr: tcp/127.0.0.1:29414) (DC: dc1)" +31:07.366 [INFO] jones.server: Handled event for server in area: event=member-join server=Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2.dc1 area=wan +31:07.366 [INFO] jones: Started DNS server: address=127.0.0.1:29409 network=tcp +31:07.367 [INFO] jones: Started HTTP server: address=127.0.0.1:29410 network=tcp +31:07.367 [INFO] jones: started state syncer +31:07.367 [INFO] jones: Started gRPC server: address=127.0.0.1:29415 network=tcp +31:07.436 [WARN] jones.server.raft: heartbeat timeout reached, starting election: last-leader= +31:07.436 [INFO] jones.server.raft: entering candidate state: node="Node at 127.0.0.1:29414 [Candidate]" term=2 +31:07.465 [DEBUG] jones.server.raft: votes: needed=1 +31:07.465 [DEBUG] jones.server.raft: vote granted: from=a83c5b5d-865a-1aa5-809b-507dbe0c43c2 term=2 tally=1 +31:07.465 [INFO] jones.server.raft: election won: tally=1 +31:07.465 [INFO] jones.server.raft: entering leader state: leader="Node at 127.0.0.1:29414 [Leader]" +31:07.468 [INFO] jones.server: cluster leadership acquired +31:07.468 [INFO] jones.server: New leader elected: payload=Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2 +31:07.481 [DEBUG] jones.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29414 +31:07.497 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:07.537 [INFO] jones.server.connect: initialized primary datacenter CA with provider: provider=consul +31:07.537 [INFO] jones.leader: started routine: routine="federation state anti-entropy" +31:07.537 [INFO] jones.leader: started routine: routine="federation state pruning" +31:07.537 [INFO] jones.leader: started routine: routine="intermediate cert renew watch" +31:07.537 [INFO] jones.leader: started routine: routine="CA root pruning" +31:07.538 [DEBUG] jones.server: Skipping self join check for node since the cluster is too small: node=Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2 +31:07.538 [INFO] jones.server: member joined, marking health alive: member=Node-a83c5b5d-865a-1aa5-809b-507dbe0c43c2 +31:07.550 [INFO] jones.server: federation state anti-entropy synced +31:07.751 [DEBUG] jones: Skipping remote check since it is managed automatically: check=serfHealth +31:07.760 [INFO] jones: Synced node info +31:07.760 [DEBUG] jones: Node info in sync +31:07.809 [INFO] jones: Requesting shutdown +31:07.809 [INFO] jones.server: shutting down server +31:07.809 [DEBUG] jones.leader: stopping routine: routine="federation state pruning" +31:07.809 [DEBUG] jones.leader: stopping routine: routine="intermediate cert renew watch" +31:07.809 [DEBUG] jones.leader: stopping routine: routine="CA root pruning" +31:07.809 [DEBUG] jones.leader: stopping routine: routine="federation state anti-entropy" +31:07.809 [WARN] jones.server.serf.lan: serf: Shutdown without a Leave +31:07.810 [ERROR] jones.server: error performing anti-entropy sync of federation state: error="context canceled" +31:07.810 [DEBUG] jones.leader: stopped routine: routine="federation state anti-entropy" +31:07.810 [DEBUG] jones.leader: stopped routine: routine="federation state pruning" +31:07.810 [DEBUG] jones.leader: stopped routine: routine="intermediate cert renew watch" +31:07.810 [DEBUG] jones.leader: stopped routine: routine="CA root pruning" +31:07.820 [WARN] jones.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:07.824Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:07.824Z [INFO] agent.router.manager: shutting down +31:07.824 [INFO] jones: consul server down +31:07.824 [INFO] jones: shutdown complete +31:07.824 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:29409 network=tcp +31:07.824 [INFO] jones: Stopping server: protocol=DNS address=127.0.0.1:29409 network=udp +31:07.824 [INFO] jones: Stopping server: protocol=HTTP address=127.0.0.1:29410 network=tcp +31:08.324 [INFO] jones: Waiting for endpoints to shut down +31:08.324 [INFO] jones: Endpoints down +--- PASS: TestAgent_sidecarServiceFromNodeService (8.90s) + --- PASS: TestAgent_sidecarServiceFromNodeService/no_sidecar (1.00s) + --- PASS: TestAgent_sidecarServiceFromNodeService/all_the_defaults (0.87s) + --- PASS: TestAgent_sidecarServiceFromNodeService/all_the_allowed_overrides (1.08s) + --- PASS: TestAgent_sidecarServiceFromNodeService/no_auto_ports_available (1.03s) + --- PASS: TestAgent_sidecarServiceFromNodeService/auto_ports_disabled (0.96s) + --- PASS: TestAgent_sidecarServiceFromNodeService/inherit_tags_and_meta (0.94s) + --- PASS: TestAgent_sidecarServiceFromNodeService/invalid_check_type (1.08s) + --- PASS: TestAgent_sidecarServiceFromNodeService/invalid_meta (0.86s) + --- PASS: TestAgent_sidecarServiceFromNodeService/re-registering_same_sidecar_with_no_port_should_pick_same_one (1.08s) === RUN TestSnapshot snapshot_endpoint_test.go:16: DM-skipped --- SKIP: TestSnapshot (0.00s) @@ -20878,276 +20998,109 @@ === PAUSE TestDefaultConfig/#498 === RUN TestDefaultConfig/#499 === PAUSE TestDefaultConfig/#499 +=== CONT TestDefaultConfig/#450 +=== CONT TestDefaultConfig/#346 === CONT TestDefaultConfig/#00 -=== CONT TestDefaultConfig/#122 -=== CONT TestDefaultConfig/#499 -=== CONT TestDefaultConfig/#484 -=== CONT TestDefaultConfig/#470 -=== CONT TestDefaultConfig/#457 -=== CONT TestDefaultConfig/#444 -=== CONT TestDefaultConfig/#432 -=== CONT TestDefaultConfig/#421 -=== CONT TestDefaultConfig/#410 -=== CONT TestDefaultConfig/#400 -=== CONT TestDefaultConfig/#121 -=== CONT TestDefaultConfig/#431 -=== CONT TestDefaultConfig/#430 -=== CONT TestDefaultConfig/#429 -=== CONT TestDefaultConfig/#428 -=== CONT TestDefaultConfig/#427 -=== CONT TestDefaultConfig/#391 -=== CONT TestDefaultConfig/#426 -=== CONT TestDefaultConfig/#425 -=== CONT TestDefaultConfig/#424 -=== CONT TestDefaultConfig/#423 -=== CONT TestDefaultConfig/#422 -=== CONT TestDefaultConfig/#443 -=== CONT TestDefaultConfig/#390 -=== CONT TestDefaultConfig/#442 -=== CONT TestDefaultConfig/#389 -=== CONT TestDefaultConfig/#388 -=== CONT TestDefaultConfig/#441 -=== CONT TestDefaultConfig/#387 -=== CONT TestDefaultConfig/#386 -=== CONT TestDefaultConfig/#440 -=== CONT TestDefaultConfig/#399 -=== CONT TestDefaultConfig/#385 -=== CONT TestDefaultConfig/#439 -=== CONT TestDefaultConfig/#384 -=== CONT TestDefaultConfig/#398 -=== CONT TestDefaultConfig/#438 -=== CONT TestDefaultConfig/#383 -=== CONT TestDefaultConfig/#397 -=== CONT TestDefaultConfig/#382 -=== CONT TestDefaultConfig/#437 -=== CONT TestDefaultConfig/#396 -=== CONT TestDefaultConfig/#381 -=== CONT TestDefaultConfig/#436 -=== CONT TestDefaultConfig/#380 -=== CONT TestDefaultConfig/#395 -=== CONT TestDefaultConfig/#435 -=== CONT TestDefaultConfig/#379 -=== CONT TestDefaultConfig/#394 -=== CONT TestDefaultConfig/#434 -=== CONT TestDefaultConfig/#393 -=== CONT TestDefaultConfig/#378 -=== CONT TestDefaultConfig/#392 -=== CONT TestDefaultConfig/#377 -=== CONT TestDefaultConfig/#433 -=== CONT TestDefaultConfig/#376 -=== CONT TestDefaultConfig/#373 -=== CONT TestDefaultConfig/#375 -=== CONT TestDefaultConfig/#372 -=== CONT TestDefaultConfig/#374 -=== CONT TestDefaultConfig/#366 -=== CONT TestDefaultConfig/#243 -=== CONT TestDefaultConfig/#359 -=== CONT TestDefaultConfig/#371 -=== CONT TestDefaultConfig/#365 -=== CONT TestDefaultConfig/#358 -=== CONT TestDefaultConfig/#370 -=== CONT TestDefaultConfig/#357 -=== CONT TestDefaultConfig/#364 -=== CONT TestDefaultConfig/#369 -=== CONT TestDefaultConfig/#363 -=== CONT TestDefaultConfig/#368 -=== CONT TestDefaultConfig/#355 -=== CONT TestDefaultConfig/#367 -=== CONT TestDefaultConfig/#354 -=== CONT TestDefaultConfig/#353 -=== CONT TestDefaultConfig/#356 -=== CONT TestDefaultConfig/#352 -=== CONT TestDefaultConfig/#469 === CONT TestDefaultConfig/#345 -=== CONT TestDefaultConfig/#468 -=== CONT TestDefaultConfig/#351 -=== CONT TestDefaultConfig/#344 -=== CONT TestDefaultConfig/#350 -=== CONT TestDefaultConfig/#467 -=== CONT TestDefaultConfig/#362 -=== CONT TestDefaultConfig/#466 -=== CONT TestDefaultConfig/#343 -=== CONT TestDefaultConfig/#349 -=== CONT TestDefaultConfig/#361 -=== CONT TestDefaultConfig/#360 -=== CONT TestDefaultConfig/#339 -=== CONT TestDefaultConfig/#342 -=== CONT TestDefaultConfig/#338 -=== CONT TestDefaultConfig/#341 -=== CONT TestDefaultConfig/#337 -=== CONT TestDefaultConfig/#340 -=== CONT TestDefaultConfig/#336 -=== CONT TestDefaultConfig/#335 -=== CONT TestDefaultConfig/#333 -=== CONT TestDefaultConfig/#334 +=== CONT TestDefaultConfig/#217 +=== CONT TestDefaultConfig/#495 +=== CONT TestDefaultConfig/#499 +=== CONT TestDefaultConfig/#498 +=== CONT TestDefaultConfig/#497 +=== CONT TestDefaultConfig/#496 === CONT TestDefaultConfig/#332 +=== CONT TestDefaultConfig/#330 +=== CONT TestDefaultConfig/#329 +=== CONT TestDefaultConfig/#328 === CONT TestDefaultConfig/#327 === CONT TestDefaultConfig/#331 -=== CONT TestDefaultConfig/#409 -=== CONT TestDefaultConfig/#330 -=== CONT TestDefaultConfig/#408 === CONT TestDefaultConfig/#326 -=== CONT TestDefaultConfig/#329 === CONT TestDefaultConfig/#325 -=== CONT TestDefaultConfig/#328 === CONT TestDefaultConfig/#324 -=== CONT TestDefaultConfig/#407 +=== CONT TestDefaultConfig/#323 +=== CONT TestDefaultConfig/#322 +=== CONT TestDefaultConfig/#321 +=== CONT TestDefaultConfig/#320 +=== CONT TestDefaultConfig/#319 +=== CONT TestDefaultConfig/#318 +=== CONT TestDefaultConfig/#317 === CONT TestDefaultConfig/#316 -=== CONT TestDefaultConfig/#348 -=== CONT TestDefaultConfig/#406 -=== CONT TestDefaultConfig/#347 -=== CONT TestDefaultConfig/#405 === CONT TestDefaultConfig/#315 -=== CONT TestDefaultConfig/#404 === CONT TestDefaultConfig/#314 -=== CONT TestDefaultConfig/#403 -=== CONT TestDefaultConfig/#346 -=== CONT TestDefaultConfig/#402 +=== CONT TestDefaultConfig/#313 +=== CONT TestDefaultConfig/#312 === CONT TestDefaultConfig/#311 -=== CONT TestDefaultConfig/#401 === CONT TestDefaultConfig/#310 -=== CONT TestDefaultConfig/#321 === CONT TestDefaultConfig/#309 -=== CONT TestDefaultConfig/#306 === CONT TestDefaultConfig/#308 -=== CONT TestDefaultConfig/#323 -=== CONT TestDefaultConfig/#305 === CONT TestDefaultConfig/#307 -=== CONT TestDefaultConfig/#322 -=== CONT TestDefaultConfig/#320 +=== CONT TestDefaultConfig/#306 +=== CONT TestDefaultConfig/#305 === CONT TestDefaultConfig/#304 +=== CONT TestDefaultConfig/#303 === CONT TestDefaultConfig/#302 -=== CONT TestDefaultConfig/#298 === CONT TestDefaultConfig/#301 -=== CONT TestDefaultConfig/#313 +=== CONT TestDefaultConfig/#300 +=== CONT TestDefaultConfig/#299 +=== CONT TestDefaultConfig/#298 === CONT TestDefaultConfig/#297 +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! === CONT TestDefaultConfig/#296 +=== CONT TestDefaultConfig/#267 === CONT TestDefaultConfig/#295 -=== CONT TestDefaultConfig/#312 === CONT TestDefaultConfig/#294 -=== CONT TestDefaultConfig/#290 === CONT TestDefaultConfig/#293 -=== CONT TestDefaultConfig/#300 -=== CONT TestDefaultConfig/#289 === CONT TestDefaultConfig/#292 -=== CONT TestDefaultConfig/#299 +=== CONT TestDefaultConfig/#291 +=== CONT TestDefaultConfig/#290 +=== CONT TestDefaultConfig/#289 === CONT TestDefaultConfig/#288 +=== CONT TestDefaultConfig/#287 +=== CONT TestDefaultConfig/#286 +=== CONT TestDefaultConfig/#285 === CONT TestDefaultConfig/#284 -=== CONT TestDefaultConfig/#291 === CONT TestDefaultConfig/#283 -=== CONT TestDefaultConfig/#303 -=== CONT TestDefaultConfig/#280 === CONT TestDefaultConfig/#282 -=== CONT TestDefaultConfig/#277 -=== CONT TestDefaultConfig/#279 === CONT TestDefaultConfig/#281 +=== CONT TestDefaultConfig/#280 +=== CONT TestDefaultConfig/#278 +=== CONT TestDefaultConfig/#279 +=== CONT TestDefaultConfig/#277 +=== CONT TestDefaultConfig/#276 +=== CONT TestDefaultConfig/#275 === CONT TestDefaultConfig/#274 === CONT TestDefaultConfig/#273 === CONT TestDefaultConfig/#272 -=== CONT TestDefaultConfig/#276 === CONT TestDefaultConfig/#271 === CONT TestDefaultConfig/#270 === CONT TestDefaultConfig/#269 === CONT TestDefaultConfig/#268 -=== CONT TestDefaultConfig/#267 === CONT TestDefaultConfig/#266 === CONT TestDefaultConfig/#265 -=== CONT TestDefaultConfig/#275 === CONT TestDefaultConfig/#264 === CONT TestDefaultConfig/#263 +=== CONT TestDefaultConfig/#262 === CONT TestDefaultConfig/#261 === CONT TestDefaultConfig/#260 -=== CONT TestDefaultConfig/#259 +=== CONT TestDefaultConfig/#258 === CONT TestDefaultConfig/#257 === CONT TestDefaultConfig/#256 -=== CONT TestDefaultConfig/#258 +=== CONT TestDefaultConfig/#259 === CONT TestDefaultConfig/#255 -=== CONT TestDefaultConfig/#253 === CONT TestDefaultConfig/#254 +=== CONT TestDefaultConfig/#253 === CONT TestDefaultConfig/#252 -=== CONT TestDefaultConfig/#278 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#251 === CONT TestDefaultConfig/#250 === CONT TestDefaultConfig/#249 === CONT TestDefaultConfig/#248 -=== CONT TestDefaultConfig/#251 -=== CONT TestDefaultConfig/#246 === CONT TestDefaultConfig/#247 +=== CONT TestDefaultConfig/#246 === CONT TestDefaultConfig/#245 === CONT TestDefaultConfig/#244 -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== CONT TestDefaultConfig/#114 -=== CONT TestDefaultConfig/#287 -=== CONT TestDefaultConfig/#286 -=== CONT TestDefaultConfig/#285 -=== CONT TestDefaultConfig/#60 -=== CONT TestDefaultConfig/#120 -=== CONT TestDefaultConfig/#119 -=== CONT TestDefaultConfig/#118 -=== CONT TestDefaultConfig/#117 -=== CONT TestDefaultConfig/#116 -=== CONT TestDefaultConfig/#115 -=== CONT TestDefaultConfig/#113 -=== CONT TestDefaultConfig/#112 -=== CONT TestDefaultConfig/#111 -=== CONT TestDefaultConfig/#110 -=== CONT TestDefaultConfig/#109 -=== CONT TestDefaultConfig/#108 -=== CONT TestDefaultConfig/#107 -=== CONT TestDefaultConfig/#106 -=== CONT TestDefaultConfig/#105 -=== CONT TestDefaultConfig/#104 -=== CONT TestDefaultConfig/#103 -=== CONT TestDefaultConfig/#102 -=== CONT TestDefaultConfig/#101 -=== CONT TestDefaultConfig/#100 -=== CONT TestDefaultConfig/#99 -=== CONT TestDefaultConfig/#30 -=== CONT TestDefaultConfig/#98 -=== CONT TestDefaultConfig/#97 -=== CONT TestDefaultConfig/#59 -=== CONT TestDefaultConfig/#96 -=== CONT TestDefaultConfig/#95 -=== CONT TestDefaultConfig/#94 -=== CONT TestDefaultConfig/#93 -=== CONT TestDefaultConfig/#92 -=== CONT TestDefaultConfig/#91 -=== CONT TestDefaultConfig/#90 -=== CONT TestDefaultConfig/#89 -=== CONT TestDefaultConfig/#88 -=== CONT TestDefaultConfig/#87 -=== CONT TestDefaultConfig/#86 -=== CONT TestDefaultConfig/#85 -=== CONT TestDefaultConfig/#84 -=== CONT TestDefaultConfig/#82 -=== CONT TestDefaultConfig/#81 -=== CONT TestDefaultConfig/#80 -=== CONT TestDefaultConfig/#83 -=== CONT TestDefaultConfig/#79 -=== CONT TestDefaultConfig/#78 -=== CONT TestDefaultConfig/#77 -=== CONT TestDefaultConfig/#76 -=== CONT TestDefaultConfig/#75 -=== CONT TestDefaultConfig/#74 -=== CONT TestDefaultConfig/#72 -=== CONT TestDefaultConfig/#71 -=== CONT TestDefaultConfig/#70 -=== CONT TestDefaultConfig/#69 -=== CONT TestDefaultConfig/#73 -=== CONT TestDefaultConfig/#68 -=== CONT TestDefaultConfig/#67 -=== CONT TestDefaultConfig/#66 -=== CONT TestDefaultConfig/#65 -=== CONT TestDefaultConfig/#64 -=== CONT TestDefaultConfig/#63 -=== CONT TestDefaultConfig/#62 -=== CONT TestDefaultConfig/#61 -=== CONT TestDefaultConfig/#415 -=== CONT TestDefaultConfig/#420 -=== CONT TestDefaultConfig/#419 -=== CONT TestDefaultConfig/#418 -=== CONT TestDefaultConfig/#417 -=== CONT TestDefaultConfig/#416 -=== CONT TestDefaultConfig/#182 +=== CONT TestDefaultConfig/#243 +=== CONT TestDefaultConfig/#242 === CONT TestDefaultConfig/#241 === CONT TestDefaultConfig/#240 === CONT TestDefaultConfig/#239 @@ -21172,63 +21125,94 @@ === CONT TestDefaultConfig/#220 === CONT TestDefaultConfig/#219 === CONT TestDefaultConfig/#218 -=== CONT TestDefaultConfig/#217 -=== CONT TestDefaultConfig/#216 -=== CONT TestDefaultConfig/#215 -=== CONT TestDefaultConfig/#214 -=== CONT TestDefaultConfig/#213 -=== CONT TestDefaultConfig/#212 -=== CONT TestDefaultConfig/#211 -=== CONT TestDefaultConfig/#210 -=== CONT TestDefaultConfig/#209 -=== CONT TestDefaultConfig/#208 -=== CONT TestDefaultConfig/#207 -=== CONT TestDefaultConfig/#206 -=== CONT TestDefaultConfig/#205 -=== CONT TestDefaultConfig/#204 -=== CONT TestDefaultConfig/#203 -=== CONT TestDefaultConfig/#202 -=== CONT TestDefaultConfig/#201 -=== CONT TestDefaultConfig/#200 -=== CONT TestDefaultConfig/#199 -=== CONT TestDefaultConfig/#198 -=== CONT TestDefaultConfig/#197 -=== CONT TestDefaultConfig/#196 +=== CONT TestDefaultConfig/#89 +=== CONT TestDefaultConfig/#449 +=== CONT TestDefaultConfig/#448 +=== CONT TestDefaultConfig/#447 === CONT TestDefaultConfig/#195 +=== CONT TestDefaultConfig/#446 +=== CONT TestDefaultConfig/#445 +=== CONT TestDefaultConfig/#444 +=== CONT TestDefaultConfig/#443 +=== CONT TestDefaultConfig/#442 +=== CONT TestDefaultConfig/#441 +=== CONT TestDefaultConfig/#440 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#439 +=== CONT TestDefaultConfig/#438 +=== CONT TestDefaultConfig/#437 +=== CONT TestDefaultConfig/#436 +=== CONT TestDefaultConfig/#435 +=== CONT TestDefaultConfig/#434 +=== CONT TestDefaultConfig/#433 +=== CONT TestDefaultConfig/#432 +=== CONT TestDefaultConfig/#431 +=== CONT TestDefaultConfig/#430 +=== CONT TestDefaultConfig/#429 +=== CONT TestDefaultConfig/#428 +=== CONT TestDefaultConfig/#427 +=== CONT TestDefaultConfig/#426 +=== CONT TestDefaultConfig/#425 === CONT TestDefaultConfig/#194 +=== CONT TestDefaultConfig/#424 +=== CONT TestDefaultConfig/#423 === CONT TestDefaultConfig/#193 +=== CONT TestDefaultConfig/#422 === CONT TestDefaultConfig/#192 +=== CONT TestDefaultConfig/#421 === CONT TestDefaultConfig/#191 +=== CONT TestDefaultConfig/#420 === CONT TestDefaultConfig/#190 +=== CONT TestDefaultConfig/#419 === CONT TestDefaultConfig/#189 +=== CONT TestDefaultConfig/#418 === CONT TestDefaultConfig/#188 +=== CONT TestDefaultConfig/#417 === CONT TestDefaultConfig/#187 +=== CONT TestDefaultConfig/#416 === CONT TestDefaultConfig/#186 +=== CONT TestDefaultConfig/#415 === CONT TestDefaultConfig/#185 +=== CONT TestDefaultConfig/#414 === CONT TestDefaultConfig/#184 +=== CONT TestDefaultConfig/#413 === CONT TestDefaultConfig/#183 === CONT TestDefaultConfig/#412 -=== CONT TestDefaultConfig/#414 -=== CONT TestDefaultConfig/#413 === CONT TestDefaultConfig/#411 -=== CONT TestDefaultConfig/#462 -=== CONT TestDefaultConfig/#465 -=== CONT TestDefaultConfig/#464 -=== CONT TestDefaultConfig/#463 -=== CONT TestDefaultConfig/#152 -=== CONT TestDefaultConfig/#181 +=== CONT TestDefaultConfig/#410 +=== CONT TestDefaultConfig/#409 +=== CONT TestDefaultConfig/#408 +=== CONT TestDefaultConfig/#407 +=== CONT TestDefaultConfig/#406 +=== CONT TestDefaultConfig/#182 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#404 === CONT TestDefaultConfig/#180 +=== CONT TestDefaultConfig/#403 +=== CONT TestDefaultConfig/#405 === CONT TestDefaultConfig/#179 +=== CONT TestDefaultConfig/#181 +=== CONT TestDefaultConfig/#402 === CONT TestDefaultConfig/#178 +=== CONT TestDefaultConfig/#401 === CONT TestDefaultConfig/#177 +=== CONT TestDefaultConfig/#400 === CONT TestDefaultConfig/#176 +=== CONT TestDefaultConfig/#399 === CONT TestDefaultConfig/#175 +=== CONT TestDefaultConfig/#398 +=== CONT TestDefaultConfig/#397 === CONT TestDefaultConfig/#174 +=== CONT TestDefaultConfig/#396 === CONT TestDefaultConfig/#173 +=== CONT TestDefaultConfig/#395 === CONT TestDefaultConfig/#172 +=== CONT TestDefaultConfig/#394 === CONT TestDefaultConfig/#171 +=== CONT TestDefaultConfig/#393 === CONT TestDefaultConfig/#170 === CONT TestDefaultConfig/#169 +=== CONT TestDefaultConfig/#156 === CONT TestDefaultConfig/#168 === CONT TestDefaultConfig/#167 === CONT TestDefaultConfig/#166 @@ -21239,648 +21223,793 @@ === CONT TestDefaultConfig/#161 === CONT TestDefaultConfig/#160 === CONT TestDefaultConfig/#159 -=== CONT TestDefaultConfig/#157 === CONT TestDefaultConfig/#158 -=== CONT TestDefaultConfig/#153 -=== CONT TestDefaultConfig/#451 -=== CONT TestDefaultConfig/#456 -=== CONT TestDefaultConfig/#455 -=== CONT TestDefaultConfig/#454 -=== CONT TestDefaultConfig/#453 -=== CONT TestDefaultConfig/#452 -=== CONT TestDefaultConfig/#492 -=== CONT TestDefaultConfig/#498 -=== CONT TestDefaultConfig/#156 -=== CONT TestDefaultConfig/#497 -=== CONT TestDefaultConfig/#496 -=== CONT TestDefaultConfig/#495 +=== CONT TestDefaultConfig/#157 === CONT TestDefaultConfig/#155 -=== CONT TestDefaultConfig/#494 === CONT TestDefaultConfig/#154 -=== CONT TestDefaultConfig/#493 -=== CONT TestDefaultConfig/#488 -=== CONT TestDefaultConfig/#491 -=== CONT TestDefaultConfig/#490 -=== CONT TestDefaultConfig/#489 -=== CONT TestDefaultConfig/#137 +=== CONT TestDefaultConfig/#153 +=== CONT TestDefaultConfig/#152 === CONT TestDefaultConfig/#151 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === CONT TestDefaultConfig/#150 === CONT TestDefaultConfig/#149 === CONT TestDefaultConfig/#148 === CONT TestDefaultConfig/#147 === CONT TestDefaultConfig/#146 -=== CONT TestDefaultConfig/#145 === CONT TestDefaultConfig/#144 === CONT TestDefaultConfig/#143 === CONT TestDefaultConfig/#142 +=== CONT TestDefaultConfig/#145 === CONT TestDefaultConfig/#141 === CONT TestDefaultConfig/#140 === CONT TestDefaultConfig/#139 -=== CONT TestDefaultConfig/#318 -=== CONT TestDefaultConfig/#477 -=== CONT TestDefaultConfig/#483 -=== CONT TestDefaultConfig/#482 === CONT TestDefaultConfig/#138 -=== CONT TestDefaultConfig/#479 -=== CONT TestDefaultConfig/#478 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== CONT TestDefaultConfig/#480 -=== CONT TestDefaultConfig/#130 -=== CONT TestDefaultConfig/#481 +=== CONT TestDefaultConfig/#137 === CONT TestDefaultConfig/#136 === CONT TestDefaultConfig/#135 === CONT TestDefaultConfig/#134 === CONT TestDefaultConfig/#133 === CONT TestDefaultConfig/#132 === CONT TestDefaultConfig/#131 -=== CONT TestDefaultConfig/#126 +=== CONT TestDefaultConfig/#130 === CONT TestDefaultConfig/#129 -=== CONT TestDefaultConfig/#128 === CONT TestDefaultConfig/#127 -=== CONT TestDefaultConfig/#15 -=== CONT TestDefaultConfig/#29 -=== CONT TestDefaultConfig/#27 -=== CONT TestDefaultConfig/#26 -=== CONT TestDefaultConfig/#25 -=== CONT TestDefaultConfig/#24 -=== CONT TestDefaultConfig/#23 -=== CONT TestDefaultConfig/#22 -=== CONT TestDefaultConfig/#21 -=== CONT TestDefaultConfig/#20 -=== CONT TestDefaultConfig/#28 -=== CONT TestDefaultConfig/#19 -=== CONT TestDefaultConfig/#18 -=== CONT TestDefaultConfig/#17 -=== CONT TestDefaultConfig/#16 -=== CONT TestDefaultConfig/#317 -=== CONT TestDefaultConfig/#460 -=== CONT TestDefaultConfig/#461 -=== CONT TestDefaultConfig/#459 -=== CONT TestDefaultConfig/#474 -=== CONT TestDefaultConfig/#475 -=== CONT TestDefaultConfig/#476 -=== CONT TestDefaultConfig/#44 +=== CONT TestDefaultConfig/#126 +=== CONT TestDefaultConfig/#125 +=== CONT TestDefaultConfig/#128 +=== CONT TestDefaultConfig/#124 +=== CONT TestDefaultConfig/#123 +=== CONT TestDefaultConfig/#122 +=== CONT TestDefaultConfig/#121 +=== CONT TestDefaultConfig/#120 +=== CONT TestDefaultConfig/#119 +=== CONT TestDefaultConfig/#118 +=== CONT TestDefaultConfig/#117 +=== CONT TestDefaultConfig/#116 +=== CONT TestDefaultConfig/#115 +=== CONT TestDefaultConfig/#114 +=== CONT TestDefaultConfig/#113 +=== CONT TestDefaultConfig/#112 +=== CONT TestDefaultConfig/#111 +=== CONT TestDefaultConfig/#110 +=== CONT TestDefaultConfig/#109 +=== CONT TestDefaultConfig/#108 +=== CONT TestDefaultConfig/#100 +=== CONT TestDefaultConfig/#107 +=== CONT TestDefaultConfig/#106 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#104 +=== CONT TestDefaultConfig/#103 +=== CONT TestDefaultConfig/#105 +=== CONT TestDefaultConfig/#102 +=== CONT TestDefaultConfig/#101 +=== CONT TestDefaultConfig/#99 +=== CONT TestDefaultConfig/#98 +=== CONT TestDefaultConfig/#97 +=== CONT TestDefaultConfig/#96 +=== CONT TestDefaultConfig/#95 +=== CONT TestDefaultConfig/#94 +=== CONT TestDefaultConfig/#93 +=== CONT TestDefaultConfig/#92 +=== CONT TestDefaultConfig/#91 +=== CONT TestDefaultConfig/#90 +=== CONT TestDefaultConfig/#88 +=== CONT TestDefaultConfig/#87 +=== CONT TestDefaultConfig/#86 +=== CONT TestDefaultConfig/#84 +=== CONT TestDefaultConfig/#83 +=== CONT TestDefaultConfig/#82 +=== CONT TestDefaultConfig/#85 +=== CONT TestDefaultConfig/#81 +=== CONT TestDefaultConfig/#80 +=== CONT TestDefaultConfig/#79 +=== CONT TestDefaultConfig/#78 +=== CONT TestDefaultConfig/#392 +=== CONT TestDefaultConfig/#76 +=== CONT TestDefaultConfig/#390 +=== CONT TestDefaultConfig/#75 +=== CONT TestDefaultConfig/#77 +=== CONT TestDefaultConfig/#391 +=== CONT TestDefaultConfig/#74 +=== CONT TestDefaultConfig/#73 +=== CONT TestDefaultConfig/#72 +=== CONT TestDefaultConfig/#389 +=== CONT TestDefaultConfig/#71 +=== CONT TestDefaultConfig/#388 +=== CONT TestDefaultConfig/#387 +=== CONT TestDefaultConfig/#70 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#62 +=== CONT TestDefaultConfig/#69 +=== CONT TestDefaultConfig/#68 +=== CONT TestDefaultConfig/#385 +=== CONT TestDefaultConfig/#67 +=== CONT TestDefaultConfig/#384 +=== CONT TestDefaultConfig/#66 +=== CONT TestDefaultConfig/#383 +=== CONT TestDefaultConfig/#65 +=== CONT TestDefaultConfig/#382 +=== CONT TestDefaultConfig/#386 +=== CONT TestDefaultConfig/#64 +=== CONT TestDefaultConfig/#381 +=== CONT TestDefaultConfig/#63 +=== CONT TestDefaultConfig/#380 +=== CONT TestDefaultConfig/#379 +=== CONT TestDefaultConfig/#61 +=== CONT TestDefaultConfig/#60 +=== CONT TestDefaultConfig/#59 +=== CONT TestDefaultConfig/#378 +=== CONT TestDefaultConfig/#376 +=== CONT TestDefaultConfig/#377 +=== CONT TestDefaultConfig/#375 +=== CONT TestDefaultConfig/#58 +=== CONT TestDefaultConfig/#374 +=== CONT TestDefaultConfig/#373 === CONT TestDefaultConfig/#57 +=== CONT TestDefaultConfig/#372 === CONT TestDefaultConfig/#56 === CONT TestDefaultConfig/#55 === CONT TestDefaultConfig/#54 === CONT TestDefaultConfig/#53 === CONT TestDefaultConfig/#52 === CONT TestDefaultConfig/#51 +=== CONT TestDefaultConfig/#371 === CONT TestDefaultConfig/#50 -=== CONT TestDefaultConfig/#48 -=== CONT TestDefaultConfig/#47 -=== CONT TestDefaultConfig/#46 +=== CONT TestDefaultConfig/#370 === CONT TestDefaultConfig/#49 -=== CONT TestDefaultConfig/#45 -=== CONT TestDefaultConfig/#37 +=== CONT TestDefaultConfig/#369 +=== CONT TestDefaultConfig/#368 +=== CONT TestDefaultConfig/#367 +=== CONT TestDefaultConfig/#48 +=== CONT TestDefaultConfig/#366 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#365 +=== CONT TestDefaultConfig/#364 +=== CONT TestDefaultConfig/#361 +=== CONT TestDefaultConfig/#360 +=== CONT TestDefaultConfig/#362 +=== CONT TestDefaultConfig/#359 +=== CONT TestDefaultConfig/#363 +=== CONT TestDefaultConfig/#358 +=== CONT TestDefaultConfig/#357 +=== CONT TestDefaultConfig/#356 +=== CONT TestDefaultConfig/#355 +=== CONT TestDefaultConfig/#354 +=== CONT TestDefaultConfig/#353 +=== CONT TestDefaultConfig/#352 +=== CONT TestDefaultConfig/#351 +=== CONT TestDefaultConfig/#349 +=== CONT TestDefaultConfig/#348 +=== CONT TestDefaultConfig/#350 +=== CONT TestDefaultConfig/#347 === CONT TestDefaultConfig/#43 === CONT TestDefaultConfig/#42 === CONT TestDefaultConfig/#41 === CONT TestDefaultConfig/#40 === CONT TestDefaultConfig/#39 === CONT TestDefaultConfig/#38 -=== CONT TestDefaultConfig/#458 -=== CONT TestDefaultConfig/#34 +=== CONT TestDefaultConfig/#37 === CONT TestDefaultConfig/#36 === CONT TestDefaultConfig/#35 -=== CONT TestDefaultConfig/#448 -=== CONT TestDefaultConfig/#450 -=== CONT TestDefaultConfig/#449 -=== CONT TestDefaultConfig/#32 +=== CONT TestDefaultConfig/#34 === CONT TestDefaultConfig/#33 +=== CONT TestDefaultConfig/#32 === CONT TestDefaultConfig/#31 -=== CONT TestDefaultConfig/#472 -=== CONT TestDefaultConfig/#473 -=== CONT TestDefaultConfig/#471 -=== CONT TestDefaultConfig/#446 -=== CONT TestDefaultConfig/#447 -=== CONT TestDefaultConfig/#486 -=== CONT TestDefaultConfig/#487 -=== CONT TestDefaultConfig/#124 -=== CONT TestDefaultConfig/#125 -=== CONT TestDefaultConfig/#08 +=== CONT TestDefaultConfig/#30 +=== CONT TestDefaultConfig/#29 +=== CONT TestDefaultConfig/#28 +=== CONT TestDefaultConfig/#27 +=== CONT TestDefaultConfig/#26 +=== CONT TestDefaultConfig/#25 +=== CONT TestDefaultConfig/#24 +=== CONT TestDefaultConfig/#23 +=== CONT TestDefaultConfig/#22 +=== CONT TestDefaultConfig/#21 +=== CONT TestDefaultConfig/#19 +=== CONT TestDefaultConfig/#20 +=== CONT TestDefaultConfig/#18 +=== CONT TestDefaultConfig/#17 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#16 +=== CONT TestDefaultConfig/#15 === CONT TestDefaultConfig/#14 === CONT TestDefaultConfig/#13 === CONT TestDefaultConfig/#12 === CONT TestDefaultConfig/#11 === CONT TestDefaultConfig/#10 === CONT TestDefaultConfig/#09 -=== CONT TestDefaultConfig/#123 -=== CONT TestDefaultConfig/#485 -=== CONT TestDefaultConfig/#04 +=== CONT TestDefaultConfig/#08 === CONT TestDefaultConfig/#07 === CONT TestDefaultConfig/#06 === CONT TestDefaultConfig/#05 -=== CONT TestDefaultConfig/#445 +=== CONT TestDefaultConfig/#04 === CONT TestDefaultConfig/#02 -=== CONT TestDefaultConfig/#03 === CONT TestDefaultConfig/#01 -=== CONT TestDefaultConfig/#58 -=== CONT TestDefaultConfig/#262 -=== CONT TestDefaultConfig/#242 -=== CONT TestDefaultConfig/#319 ---- PASS: TestDefaultConfig (0.01s) - --- PASS: TestDefaultConfig/#421 (0.01s) - --- PASS: TestDefaultConfig/#410 (0.01s) - --- PASS: TestDefaultConfig/#457 (0.01s) - --- PASS: TestDefaultConfig/#432 (0.01s) - --- PASS: TestDefaultConfig/#499 (0.01s) - --- PASS: TestDefaultConfig/#484 (0.02s) - --- PASS: TestDefaultConfig/#426 (0.01s) - --- PASS: TestDefaultConfig/#425 (0.01s) - --- PASS: TestDefaultConfig/#427 (0.01s) - --- PASS: TestDefaultConfig/#444 (0.02s) - --- PASS: TestDefaultConfig/#391 (0.01s) - --- PASS: TestDefaultConfig/#400 (0.02s) - --- PASS: TestDefaultConfig/#429 (0.02s) - --- PASS: TestDefaultConfig/#423 (0.01s) - --- PASS: TestDefaultConfig/#424 (0.01s) - --- PASS: TestDefaultConfig/#431 (0.02s) - --- PASS: TestDefaultConfig/#428 (0.02s) - --- PASS: TestDefaultConfig/#387 (0.01s) - --- PASS: TestDefaultConfig/#390 (0.02s) - --- PASS: TestDefaultConfig/#442 (0.02s) - --- PASS: TestDefaultConfig/#389 (0.02s) - --- PASS: TestDefaultConfig/#441 (0.01s) - --- PASS: TestDefaultConfig/#443 (0.02s) - --- PASS: TestDefaultConfig/#470 (0.04s) - --- PASS: TestDefaultConfig/#430 (0.03s) - --- PASS: TestDefaultConfig/#386 (0.01s) - --- PASS: TestDefaultConfig/#122 (0.04s) - --- PASS: TestDefaultConfig/#440 (0.01s) - --- PASS: TestDefaultConfig/#399 (0.01s) - --- PASS: TestDefaultConfig/#121 (0.04s) - --- PASS: TestDefaultConfig/#438 (0.01s) - --- PASS: TestDefaultConfig/#384 (0.01s) - --- PASS: TestDefaultConfig/#00 (0.05s) - --- PASS: TestDefaultConfig/#388 (0.03s) - --- PASS: TestDefaultConfig/#396 (0.01s) - --- PASS: TestDefaultConfig/#437 (0.01s) - --- PASS: TestDefaultConfig/#439 (0.02s) - --- PASS: TestDefaultConfig/#397 (0.01s) - --- PASS: TestDefaultConfig/#435 (0.01s) - --- PASS: TestDefaultConfig/#422 (0.04s) - --- PASS: TestDefaultConfig/#436 (0.02s) - --- PASS: TestDefaultConfig/#379 (0.01s) - --- PASS: TestDefaultConfig/#395 (0.01s) - --- PASS: TestDefaultConfig/#385 (0.03s) - --- PASS: TestDefaultConfig/#394 (0.01s) - --- PASS: TestDefaultConfig/#378 (0.01s) - --- PASS: TestDefaultConfig/#392 (0.01s) - --- PASS: TestDefaultConfig/#398 (0.03s) - --- PASS: TestDefaultConfig/#393 (0.02s) - --- PASS: TestDefaultConfig/#373 (0.01s) - --- PASS: TestDefaultConfig/#366 (0.01s) - --- PASS: TestDefaultConfig/#372 (0.01s) - --- PASS: TestDefaultConfig/#380 (0.03s) - --- PASS: TestDefaultConfig/#382 (0.03s) - --- PASS: TestDefaultConfig/#376 (0.02s) - --- PASS: TestDefaultConfig/#383 (0.03s) - --- PASS: TestDefaultConfig/#434 (0.02s) - --- PASS: TestDefaultConfig/#377 (0.02s) - --- PASS: TestDefaultConfig/#433 (0.02s) - --- PASS: TestDefaultConfig/#375 (0.02s) - --- PASS: TestDefaultConfig/#381 (0.03s) - --- PASS: TestDefaultConfig/#243 (0.01s) - --- PASS: TestDefaultConfig/#371 (0.01s) - --- PASS: TestDefaultConfig/#365 (0.02s) - --- PASS: TestDefaultConfig/#374 (0.03s) - --- PASS: TestDefaultConfig/#357 (0.02s) - --- PASS: TestDefaultConfig/#358 (0.02s) - --- PASS: TestDefaultConfig/#370 (0.02s) - --- PASS: TestDefaultConfig/#364 (0.02s) - --- PASS: TestDefaultConfig/#359 (0.02s) - --- PASS: TestDefaultConfig/#363 (0.01s) - --- PASS: TestDefaultConfig/#356 (0.01s) - --- PASS: TestDefaultConfig/#354 (0.01s) - --- PASS: TestDefaultConfig/#352 (0.01s) - --- PASS: TestDefaultConfig/#469 (0.01s) - --- PASS: TestDefaultConfig/#367 (0.02s) - --- PASS: TestDefaultConfig/#355 (0.02s) - --- PASS: TestDefaultConfig/#368 (0.02s) - --- PASS: TestDefaultConfig/#345 (0.02s) - --- PASS: TestDefaultConfig/#369 (0.03s) - --- PASS: TestDefaultConfig/#362 (0.02s) - --- PASS: TestDefaultConfig/#343 (0.02s) - --- PASS: TestDefaultConfig/#350 (0.02s) - --- PASS: TestDefaultConfig/#468 (0.02s) - --- PASS: TestDefaultConfig/#344 (0.02s) - --- PASS: TestDefaultConfig/#467 (0.02s) - --- PASS: TestDefaultConfig/#466 (0.02s) - --- PASS: TestDefaultConfig/#353 (0.03s) - --- PASS: TestDefaultConfig/#351 (0.02s) - --- PASS: TestDefaultConfig/#361 (0.02s) - --- PASS: TestDefaultConfig/#360 (0.02s) - --- PASS: TestDefaultConfig/#342 (0.02s) - --- PASS: TestDefaultConfig/#339 (0.02s) - --- PASS: TestDefaultConfig/#338 (0.02s) - --- PASS: TestDefaultConfig/#341 (0.02s) - --- PASS: TestDefaultConfig/#337 (0.01s) - --- PASS: TestDefaultConfig/#336 (0.02s) - --- PASS: TestDefaultConfig/#332 (0.01s) - --- PASS: TestDefaultConfig/#349 (0.03s) - --- PASS: TestDefaultConfig/#340 (0.02s) - --- PASS: TestDefaultConfig/#333 (0.02s) +=== CONT TestDefaultConfig/#473 +=== CONT TestDefaultConfig/#494 +=== CONT TestDefaultConfig/#493 +=== CONT TestDefaultConfig/#492 +=== CONT TestDefaultConfig/#03 +=== CONT TestDefaultConfig/#491 +=== CONT TestDefaultConfig/#490 +=== CONT TestDefaultConfig/#489 +=== CONT TestDefaultConfig/#488 +=== CONT TestDefaultConfig/#487 +=== CONT TestDefaultConfig/#486 +=== CONT TestDefaultConfig/#485 +=== CONT TestDefaultConfig/#484 +=== CONT TestDefaultConfig/#483 +=== CONT TestDefaultConfig/#482 +=== CONT TestDefaultConfig/#481 +=== CONT TestDefaultConfig/#480 +=== CONT TestDefaultConfig/#479 +=== CONT TestDefaultConfig/#478 +=== CONT TestDefaultConfig/#477 +=== CONT TestDefaultConfig/#476 +=== CONT TestDefaultConfig/#475 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#474 +=== CONT TestDefaultConfig/#45 +=== CONT TestDefaultConfig/#46 +=== CONT TestDefaultConfig/#206 +=== CONT TestDefaultConfig/#216 +=== CONT TestDefaultConfig/#215 +=== CONT TestDefaultConfig/#214 +=== CONT TestDefaultConfig/#44 +=== CONT TestDefaultConfig/#212 +=== CONT TestDefaultConfig/#211 +=== CONT TestDefaultConfig/#201 +=== CONT TestDefaultConfig/#210 +=== CONT TestDefaultConfig/#205 +=== CONT TestDefaultConfig/#213 +=== CONT TestDefaultConfig/#209 +=== CONT TestDefaultConfig/#204 +=== CONT TestDefaultConfig/#208 +=== CONT TestDefaultConfig/#203 +=== CONT TestDefaultConfig/#202 +=== CONT TestDefaultConfig/#336 +=== CONT TestDefaultConfig/#207 +=== CONT TestDefaultConfig/#344 +=== CONT TestDefaultConfig/#343 +=== CONT TestDefaultConfig/#342 +=== CONT TestDefaultConfig/#462 +=== CONT TestDefaultConfig/#341 +=== CONT TestDefaultConfig/#472 +=== CONT TestDefaultConfig/#340 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestDefaultConfig/#471 +=== CONT TestDefaultConfig/#469 +=== CONT TestDefaultConfig/#468 +=== CONT TestDefaultConfig/#470 +=== CONT TestDefaultConfig/#467 +=== CONT TestDefaultConfig/#466 +=== CONT TestDefaultConfig/#465 +=== CONT TestDefaultConfig/#464 +=== CONT TestDefaultConfig/#463 +=== CONT TestDefaultConfig/#339 +=== CONT TestDefaultConfig/#338 +=== CONT TestDefaultConfig/#337 +=== CONT TestDefaultConfig/#198 +=== CONT TestDefaultConfig/#200 +=== CONT TestDefaultConfig/#199 +=== CONT TestDefaultConfig/#335 +=== CONT TestDefaultConfig/#334 +=== CONT TestDefaultConfig/#197 +=== CONT TestDefaultConfig/#333 +=== CONT TestDefaultConfig/#456 +=== CONT TestDefaultConfig/#461 +=== CONT TestDefaultConfig/#460 +=== CONT TestDefaultConfig/#47 +=== CONT TestDefaultConfig/#459 +=== CONT TestDefaultConfig/#458 +=== CONT TestDefaultConfig/#457 +=== CONT TestDefaultConfig/#196 +=== CONT TestDefaultConfig/#453 +=== CONT TestDefaultConfig/#455 +=== CONT TestDefaultConfig/#454 +=== CONT TestDefaultConfig/#452 +=== CONT TestDefaultConfig/#451 +--- PASS: TestDefaultConfig (0.02s) + --- PASS: TestDefaultConfig/#346 (0.06s) + --- PASS: TestDefaultConfig/#345 (0.06s) + --- PASS: TestDefaultConfig/#450 (0.07s) + --- PASS: TestDefaultConfig/#00 (0.07s) + --- PASS: TestDefaultConfig/#217 (0.07s) + --- PASS: TestDefaultConfig/#496 (0.04s) + --- PASS: TestDefaultConfig/#498 (0.06s) + --- PASS: TestDefaultConfig/#332 (0.03s) + --- PASS: TestDefaultConfig/#499 (0.07s) + --- PASS: TestDefaultConfig/#495 (0.08s) + --- PASS: TestDefaultConfig/#497 (0.06s) + --- PASS: TestDefaultConfig/#330 (0.03s) + --- PASS: TestDefaultConfig/#327 (0.03s) + --- PASS: TestDefaultConfig/#328 (0.03s) --- PASS: TestDefaultConfig/#331 (0.02s) - --- PASS: TestDefaultConfig/#335 (0.02s) - --- PASS: TestDefaultConfig/#334 (0.02s) - --- PASS: TestDefaultConfig/#327 (0.02s) - --- PASS: TestDefaultConfig/#409 (0.02s) - --- PASS: TestDefaultConfig/#408 (0.02s) - --- PASS: TestDefaultConfig/#329 (0.02s) - --- PASS: TestDefaultConfig/#330 (0.02s) - --- PASS: TestDefaultConfig/#325 (0.02s) + --- PASS: TestDefaultConfig/#329 (0.04s) --- PASS: TestDefaultConfig/#326 (0.02s) - --- PASS: TestDefaultConfig/#324 (0.02s) - --- PASS: TestDefaultConfig/#348 (0.01s) - --- PASS: TestDefaultConfig/#316 (0.02s) - --- PASS: TestDefaultConfig/#347 (0.01s) - --- PASS: TestDefaultConfig/#406 (0.02s) - --- PASS: TestDefaultConfig/#403 (0.01s) - --- PASS: TestDefaultConfig/#321 (0.02s) + --- PASS: TestDefaultConfig/#319 (0.01s) + --- PASS: TestDefaultConfig/#322 (0.06s) + --- PASS: TestDefaultConfig/#318 (0.06s) + --- PASS: TestDefaultConfig/#325 (0.07s) + --- PASS: TestDefaultConfig/#324 (0.08s) + --- PASS: TestDefaultConfig/#314 (0.01s) + --- PASS: TestDefaultConfig/#313 (0.01s) + --- PASS: TestDefaultConfig/#316 (0.06s) + --- PASS: TestDefaultConfig/#320 (0.07s) --- PASS: TestDefaultConfig/#310 (0.02s) - --- PASS: TestDefaultConfig/#405 (0.02s) - --- PASS: TestDefaultConfig/#404 (0.02s) - --- PASS: TestDefaultConfig/#314 (0.02s) - --- PASS: TestDefaultConfig/#346 (0.02s) - --- PASS: TestDefaultConfig/#311 (0.02s) - --- PASS: TestDefaultConfig/#328 (0.04s) - --- PASS: TestDefaultConfig/#401 (0.02s) - --- PASS: TestDefaultConfig/#308 (0.02s) - --- PASS: TestDefaultConfig/#402 (0.02s) - --- PASS: TestDefaultConfig/#309 (0.02s) - --- PASS: TestDefaultConfig/#407 (0.04s) - --- PASS: TestDefaultConfig/#306 (0.02s) - --- PASS: TestDefaultConfig/#323 (0.02s) + --- PASS: TestDefaultConfig/#323 (0.10s) --- PASS: TestDefaultConfig/#307 (0.02s) - --- PASS: TestDefaultConfig/#313 (0.01s) - --- PASS: TestDefaultConfig/#296 (0.01s) - --- PASS: TestDefaultConfig/#320 (0.01s) - --- PASS: TestDefaultConfig/#301 (0.01s) - --- PASS: TestDefaultConfig/#305 (0.02s) - --- PASS: TestDefaultConfig/#315 (0.04s) - --- PASS: TestDefaultConfig/#304 (0.02s) - --- PASS: TestDefaultConfig/#302 (0.02s) - --- PASS: TestDefaultConfig/#297 (0.02s) - --- PASS: TestDefaultConfig/#300 (0.02s) - --- PASS: TestDefaultConfig/#294 (0.02s) - --- PASS: TestDefaultConfig/#322 (0.03s) - --- PASS: TestDefaultConfig/#290 (0.02s) - --- PASS: TestDefaultConfig/#312 (0.02s) + --- PASS: TestDefaultConfig/#315 (0.09s) + --- PASS: TestDefaultConfig/#309 (0.03s) + --- PASS: TestDefaultConfig/#321 (0.10s) + --- PASS: TestDefaultConfig/#306 (0.01s) + --- PASS: TestDefaultConfig/#308 (0.03s) + --- PASS: TestDefaultConfig/#305 (0.01s) + --- PASS: TestDefaultConfig/#304 (0.01s) + --- PASS: TestDefaultConfig/#312 (0.08s) + --- PASS: TestDefaultConfig/#317 (0.14s) + --- PASS: TestDefaultConfig/#311 (0.09s) + --- PASS: TestDefaultConfig/#298 (0.07s) + --- PASS: TestDefaultConfig/#300 (0.07s) + --- PASS: TestDefaultConfig/#303 (0.09s) + --- PASS: TestDefaultConfig/#296 (0.05s) + --- PASS: TestDefaultConfig/#301 (0.09s) --- PASS: TestDefaultConfig/#293 (0.02s) - --- PASS: TestDefaultConfig/#295 (0.02s) - --- PASS: TestDefaultConfig/#289 (0.02s) - --- PASS: TestDefaultConfig/#298 (0.03s) - --- PASS: TestDefaultConfig/#292 (0.02s) - --- PASS: TestDefaultConfig/#299 (0.02s) + --- PASS: TestDefaultConfig/#294 (0.02s) + --- PASS: TestDefaultConfig/#299 (0.09s) + --- PASS: TestDefaultConfig/#267 (0.04s) + --- PASS: TestDefaultConfig/#302 (0.10s) + --- PASS: TestDefaultConfig/#295 (0.05s) --- PASS: TestDefaultConfig/#291 (0.02s) - --- PASS: TestDefaultConfig/#280 (0.01s) - --- PASS: TestDefaultConfig/#277 (0.01s) --- PASS: TestDefaultConfig/#288 (0.02s) - --- PASS: TestDefaultConfig/#283 (0.02s) - --- PASS: TestDefaultConfig/#273 (0.01s) - --- PASS: TestDefaultConfig/#282 (0.02s) - --- PASS: TestDefaultConfig/#274 (0.02s) - --- PASS: TestDefaultConfig/#270 (0.02s) - --- PASS: TestDefaultConfig/#284 (0.03s) - --- PASS: TestDefaultConfig/#271 (0.02s) - --- PASS: TestDefaultConfig/#264 (0.02s) - --- PASS: TestDefaultConfig/#267 (0.02s) - --- PASS: TestDefaultConfig/#279 (0.03s) - --- PASS: TestDefaultConfig/#265 (0.02s) - --- PASS: TestDefaultConfig/#268 (0.03s) - --- PASS: TestDefaultConfig/#276 (0.03s) - --- PASS: TestDefaultConfig/#263 (0.02s) - --- PASS: TestDefaultConfig/#303 (0.04s) - --- PASS: TestDefaultConfig/#261 (0.02s) - --- PASS: TestDefaultConfig/#260 (0.01s) - --- PASS: TestDefaultConfig/#258 (0.01s) - --- PASS: TestDefaultConfig/#266 (0.03s) - --- PASS: TestDefaultConfig/#259 (0.01s) - --- PASS: TestDefaultConfig/#257 (0.02s) - --- PASS: TestDefaultConfig/#272 (0.06s) - --- PASS: TestDefaultConfig/#253 (0.03s) - --- PASS: TestDefaultConfig/#256 (0.04s) - --- PASS: TestDefaultConfig/#249 (0.03s) - --- PASS: TestDefaultConfig/#269 (0.06s) - --- PASS: TestDefaultConfig/#246 (0.03s) - --- PASS: TestDefaultConfig/#255 (0.04s) - --- PASS: TestDefaultConfig/#250 (0.04s) - --- PASS: TestDefaultConfig/#278 (0.04s) - --- PASS: TestDefaultConfig/#254 (0.04s) - --- PASS: TestDefaultConfig/#252 (0.04s) - --- PASS: TestDefaultConfig/#248 (0.04s) + --- PASS: TestDefaultConfig/#292 (0.03s) + --- PASS: TestDefaultConfig/#286 (0.02s) + --- PASS: TestDefaultConfig/#290 (0.04s) + --- PASS: TestDefaultConfig/#297 (0.14s) + --- PASS: TestDefaultConfig/#285 (0.05s) + --- PASS: TestDefaultConfig/#284 (0.07s) + --- PASS: TestDefaultConfig/#283 (0.07s) + --- PASS: TestDefaultConfig/#282 (0.06s) + --- PASS: TestDefaultConfig/#279 (0.04s) + --- PASS: TestDefaultConfig/#274 (0.01s) + --- PASS: TestDefaultConfig/#277 (0.04s) + --- PASS: TestDefaultConfig/#280 (0.08s) + --- PASS: TestDefaultConfig/#275 (0.02s) + --- PASS: TestDefaultConfig/#278 (0.05s) + --- PASS: TestDefaultConfig/#276 (0.02s) --- PASS: TestDefaultConfig/#281 (0.08s) - --- PASS: TestDefaultConfig/#245 (0.04s) - --- PASS: TestDefaultConfig/#247 (0.04s) - --- PASS: TestDefaultConfig/#275 (0.07s) - --- PASS: TestDefaultConfig/#286 (0.01s) - --- PASS: TestDefaultConfig/#285 (0.01s) - --- PASS: TestDefaultConfig/#114 (0.02s) - --- PASS: TestDefaultConfig/#287 (0.02s) - --- PASS: TestDefaultConfig/#118 (0.01s) - --- PASS: TestDefaultConfig/#119 (0.01s) - --- PASS: TestDefaultConfig/#115 (0.01s) - --- PASS: TestDefaultConfig/#116 (0.02s) - --- PASS: TestDefaultConfig/#244 (0.03s) - --- PASS: TestDefaultConfig/#109 (0.02s) - --- PASS: TestDefaultConfig/#103 (0.01s) - --- PASS: TestDefaultConfig/#112 (0.02s) - --- PASS: TestDefaultConfig/#104 (0.01s) - --- PASS: TestDefaultConfig/#108 (0.02s) - --- PASS: TestDefaultConfig/#105 (0.02s) - --- PASS: TestDefaultConfig/#113 (0.02s) - --- PASS: TestDefaultConfig/#101 (0.01s) - --- PASS: TestDefaultConfig/#106 (0.02s) - --- PASS: TestDefaultConfig/#60 (0.03s) - --- PASS: TestDefaultConfig/#102 (0.02s) - --- PASS: TestDefaultConfig/#251 (0.07s) - --- PASS: TestDefaultConfig/#107 (0.02s) - --- PASS: TestDefaultConfig/#30 (0.01s) - --- PASS: TestDefaultConfig/#100 (0.02s) - --- PASS: TestDefaultConfig/#99 (0.01s) - --- PASS: TestDefaultConfig/#93 (0.01s) - --- PASS: TestDefaultConfig/#111 (0.04s) - --- PASS: TestDefaultConfig/#110 (0.04s) - --- PASS: TestDefaultConfig/#120 (0.04s) - --- PASS: TestDefaultConfig/#117 (0.04s) - --- PASS: TestDefaultConfig/#98 (0.02s) - --- PASS: TestDefaultConfig/#92 (0.02s) - --- PASS: TestDefaultConfig/#90 (0.02s) - --- PASS: TestDefaultConfig/#86 (0.01s) - --- PASS: TestDefaultConfig/#96 (0.02s) - --- PASS: TestDefaultConfig/#84 (0.01s) - --- PASS: TestDefaultConfig/#95 (0.03s) - --- PASS: TestDefaultConfig/#88 (0.02s) - --- PASS: TestDefaultConfig/#59 (0.03s) - --- PASS: TestDefaultConfig/#91 (0.03s) - --- PASS: TestDefaultConfig/#80 (0.01s) - --- PASS: TestDefaultConfig/#78 (0.01s) - --- PASS: TestDefaultConfig/#81 (0.01s) - --- PASS: TestDefaultConfig/#89 (0.03s) - --- PASS: TestDefaultConfig/#83 (0.01s) - --- PASS: TestDefaultConfig/#79 (0.01s) - --- PASS: TestDefaultConfig/#77 (0.02s) - --- PASS: TestDefaultConfig/#76 (0.02s) - --- PASS: TestDefaultConfig/#75 (0.01s) - --- PASS: TestDefaultConfig/#94 (0.04s) - --- PASS: TestDefaultConfig/#87 (0.04s) - --- PASS: TestDefaultConfig/#70 (0.02s) - --- PASS: TestDefaultConfig/#74 (0.02s) - --- PASS: TestDefaultConfig/#64 (0.01s) - --- PASS: TestDefaultConfig/#85 (0.04s) - --- PASS: TestDefaultConfig/#62 (0.01s) - --- PASS: TestDefaultConfig/#72 (0.02s) - --- PASS: TestDefaultConfig/#63 (0.01s) - --- PASS: TestDefaultConfig/#66 (0.02s) - --- PASS: TestDefaultConfig/#68 (0.02s) - --- PASS: TestDefaultConfig/#82 (0.04s) - --- PASS: TestDefaultConfig/#71 (0.03s) - --- PASS: TestDefaultConfig/#418 (0.01s) - --- PASS: TestDefaultConfig/#67 (0.03s) - --- PASS: TestDefaultConfig/#420 (0.01s) - --- PASS: TestDefaultConfig/#97 (0.06s) - --- PASS: TestDefaultConfig/#73 (0.03s) - --- PASS: TestDefaultConfig/#65 (0.03s) - --- PASS: TestDefaultConfig/#419 (0.02s) - --- PASS: TestDefaultConfig/#417 (0.01s) - --- PASS: TestDefaultConfig/#182 (0.02s) - --- PASS: TestDefaultConfig/#415 (0.03s) - --- PASS: TestDefaultConfig/#416 (0.02s) - --- PASS: TestDefaultConfig/#241 (0.02s) - --- PASS: TestDefaultConfig/#240 (0.02s) - --- PASS: TestDefaultConfig/#232 (0.01s) - --- PASS: TestDefaultConfig/#239 (0.02s) + --- PASS: TestDefaultConfig/#272 (0.01s) + --- PASS: TestDefaultConfig/#287 (0.10s) + --- PASS: TestDefaultConfig/#271 (0.01s) + --- PASS: TestDefaultConfig/#289 (0.11s) + --- PASS: TestDefaultConfig/#264 (0.01s) + --- PASS: TestDefaultConfig/#273 (0.02s) + --- PASS: TestDefaultConfig/#269 (0.04s) + --- PASS: TestDefaultConfig/#266 (0.05s) + --- PASS: TestDefaultConfig/#270 (0.05s) + --- PASS: TestDefaultConfig/#265 (0.06s) + --- PASS: TestDefaultConfig/#268 (0.09s) + --- PASS: TestDefaultConfig/#262 (0.08s) + --- PASS: TestDefaultConfig/#261 (0.08s) + --- PASS: TestDefaultConfig/#253 (0.01s) + --- PASS: TestDefaultConfig/#254 (0.02s) + --- PASS: TestDefaultConfig/#263 (0.10s) + --- PASS: TestDefaultConfig/#257 (0.07s) + --- PASS: TestDefaultConfig/#259 (0.06s) + --- PASS: TestDefaultConfig/#256 (0.06s) + --- PASS: TestDefaultConfig/#250 (0.02s) + --- PASS: TestDefaultConfig/#255 (0.06s) + --- PASS: TestDefaultConfig/#249 (0.02s) + --- PASS: TestDefaultConfig/#258 (0.09s) + --- PASS: TestDefaultConfig/#247 (0.01s) + --- PASS: TestDefaultConfig/#251 (0.04s) + --- PASS: TestDefaultConfig/#252 (0.04s) + --- PASS: TestDefaultConfig/#260 (0.11s) + --- PASS: TestDefaultConfig/#245 (0.03s) + --- PASS: TestDefaultConfig/#241 (0.03s) + --- PASS: TestDefaultConfig/#242 (0.05s) + --- PASS: TestDefaultConfig/#239 (0.06s) + --- PASS: TestDefaultConfig/#248 (0.08s) --- PASS: TestDefaultConfig/#235 (0.02s) - --- PASS: TestDefaultConfig/#230 (0.01s) - --- PASS: TestDefaultConfig/#231 (0.01s) - --- PASS: TestDefaultConfig/#237 (0.02s) - --- PASS: TestDefaultConfig/#229 (0.01s) - --- PASS: TestDefaultConfig/#228 (0.01s) - --- PASS: TestDefaultConfig/#61 (0.04s) - --- PASS: TestDefaultConfig/#69 (0.05s) - --- PASS: TestDefaultConfig/#224 (0.01s) - --- PASS: TestDefaultConfig/#234 (0.03s) - --- PASS: TestDefaultConfig/#227 (0.02s) - --- PASS: TestDefaultConfig/#236 (0.03s) - --- PASS: TestDefaultConfig/#223 (0.02s) - --- PASS: TestDefaultConfig/#226 (0.02s) - --- PASS: TestDefaultConfig/#238 (0.04s) + --- PASS: TestDefaultConfig/#243 (0.08s) + --- PASS: TestDefaultConfig/#240 (0.08s) + --- PASS: TestDefaultConfig/#238 (0.07s) + --- PASS: TestDefaultConfig/#237 (0.06s) + --- PASS: TestDefaultConfig/#246 (0.10s) + --- PASS: TestDefaultConfig/#244 (0.10s) + --- PASS: TestDefaultConfig/#232 (0.02s) + --- PASS: TestDefaultConfig/#230 (0.02s) --- PASS: TestDefaultConfig/#233 (0.03s) - --- PASS: TestDefaultConfig/#217 (0.02s) - --- PASS: TestDefaultConfig/#221 (0.02s) - --- PASS: TestDefaultConfig/#222 (0.02s) - --- PASS: TestDefaultConfig/#220 (0.02s) - --- PASS: TestDefaultConfig/#214 (0.01s) - --- PASS: TestDefaultConfig/#218 (0.02s) - --- PASS: TestDefaultConfig/#212 (0.01s) - --- PASS: TestDefaultConfig/#213 (0.01s) - --- PASS: TestDefaultConfig/#215 (0.02s) - --- PASS: TestDefaultConfig/#210 (0.01s) - --- PASS: TestDefaultConfig/#209 (0.01s) - --- PASS: TestDefaultConfig/#206 (0.01s) - --- PASS: TestDefaultConfig/#205 (0.01s) - --- PASS: TestDefaultConfig/#201 (0.01s) - --- PASS: TestDefaultConfig/#211 (0.02s) - --- PASS: TestDefaultConfig/#225 (0.04s) - --- PASS: TestDefaultConfig/#208 (0.02s) - --- PASS: TestDefaultConfig/#203 (0.02s) - --- PASS: TestDefaultConfig/#216 (0.04s) - --- PASS: TestDefaultConfig/#207 (0.02s) - --- PASS: TestDefaultConfig/#202 (0.02s) - --- PASS: TestDefaultConfig/#196 (0.02s) - --- PASS: TestDefaultConfig/#187 (0.01s) - --- PASS: TestDefaultConfig/#186 (0.01s) - --- PASS: TestDefaultConfig/#197 (0.02s) - --- PASS: TestDefaultConfig/#219 (0.04s) - --- PASS: TestDefaultConfig/#198 (0.02s) - --- PASS: TestDefaultConfig/#189 (0.01s) - --- PASS: TestDefaultConfig/#191 (0.01s) - --- PASS: TestDefaultConfig/#195 (0.02s) - --- PASS: TestDefaultConfig/#185 (0.01s) - --- PASS: TestDefaultConfig/#204 (0.03s) - --- PASS: TestDefaultConfig/#200 (0.03s) - --- PASS: TestDefaultConfig/#190 (0.02s) - --- PASS: TestDefaultConfig/#199 (0.03s) - --- PASS: TestDefaultConfig/#194 (0.02s) - --- PASS: TestDefaultConfig/#193 (0.02s) + --- PASS: TestDefaultConfig/#229 (0.02s) + --- PASS: TestDefaultConfig/#231 (0.03s) + --- PASS: TestDefaultConfig/#234 (0.04s) + --- PASS: TestDefaultConfig/#228 (0.03s) + --- PASS: TestDefaultConfig/#221 (0.01s) + --- PASS: TestDefaultConfig/#227 (0.03s) + --- PASS: TestDefaultConfig/#223 (0.08s) + --- PASS: TestDefaultConfig/#226 (0.09s) + --- PASS: TestDefaultConfig/#449 (0.01s) + --- PASS: TestDefaultConfig/#236 (0.16s) + --- PASS: TestDefaultConfig/#224 (0.10s) + --- PASS: TestDefaultConfig/#447 (0.01s) + --- PASS: TestDefaultConfig/#218 (0.09s) + --- PASS: TestDefaultConfig/#222 (0.10s) + --- PASS: TestDefaultConfig/#195 (0.01s) + --- PASS: TestDefaultConfig/#448 (0.03s) + --- PASS: TestDefaultConfig/#219 (0.10s) + --- PASS: TestDefaultConfig/#225 (0.12s) + --- PASS: TestDefaultConfig/#89 (0.10s) + --- PASS: TestDefaultConfig/#220 (0.11s) + --- PASS: TestDefaultConfig/#443 (0.01s) + --- PASS: TestDefaultConfig/#440 (0.02s) + --- PASS: TestDefaultConfig/#439 (0.02s) + --- PASS: TestDefaultConfig/#444 (0.05s) + --- PASS: TestDefaultConfig/#445 (0.06s) + --- PASS: TestDefaultConfig/#441 (0.05s) + --- PASS: TestDefaultConfig/#434 (0.05s) + --- PASS: TestDefaultConfig/#433 (0.06s) + --- PASS: TestDefaultConfig/#436 (0.09s) + --- PASS: TestDefaultConfig/#437 (0.10s) + --- PASS: TestDefaultConfig/#446 (0.12s) + --- PASS: TestDefaultConfig/#429 (0.01s) + --- PASS: TestDefaultConfig/#438 (0.11s) + --- PASS: TestDefaultConfig/#435 (0.10s) + --- PASS: TestDefaultConfig/#425 (0.01s) + --- PASS: TestDefaultConfig/#442 (0.12s) + --- PASS: TestDefaultConfig/#431 (0.08s) + --- PASS: TestDefaultConfig/#423 (0.01s) + --- PASS: TestDefaultConfig/#432 (0.08s) + --- PASS: TestDefaultConfig/#427 (0.02s) + --- PASS: TestDefaultConfig/#192 (0.01s) + --- PASS: TestDefaultConfig/#430 (0.05s) + --- PASS: TestDefaultConfig/#424 (0.02s) + --- PASS: TestDefaultConfig/#428 (0.04s) + --- PASS: TestDefaultConfig/#420 (0.01s) + --- PASS: TestDefaultConfig/#193 (0.04s) + --- PASS: TestDefaultConfig/#194 (0.05s) + --- PASS: TestDefaultConfig/#190 (0.05s) + --- PASS: TestDefaultConfig/#191 (0.05s) + --- PASS: TestDefaultConfig/#188 (0.04s) + --- PASS: TestDefaultConfig/#189 (0.06s) + --- PASS: TestDefaultConfig/#416 (0.02s) + --- PASS: TestDefaultConfig/#422 (0.09s) + --- PASS: TestDefaultConfig/#418 (0.07s) + --- PASS: TestDefaultConfig/#426 (0.11s) + --- PASS: TestDefaultConfig/#421 (0.09s) + --- PASS: TestDefaultConfig/#419 (0.09s) + --- PASS: TestDefaultConfig/#186 (0.02s) --- PASS: TestDefaultConfig/#184 (0.02s) - --- PASS: TestDefaultConfig/#414 (0.02s) - --- PASS: TestDefaultConfig/#188 (0.02s) - --- PASS: TestDefaultConfig/#412 (0.02s) --- PASS: TestDefaultConfig/#183 (0.02s) - --- PASS: TestDefaultConfig/#413 (0.02s) - --- PASS: TestDefaultConfig/#192 (0.03s) - --- PASS: TestDefaultConfig/#152 (0.02s) - --- PASS: TestDefaultConfig/#177 (0.01s) - --- PASS: TestDefaultConfig/#463 (0.02s) - --- PASS: TestDefaultConfig/#462 (0.02s) - --- PASS: TestDefaultConfig/#465 (0.02s) - --- PASS: TestDefaultConfig/#464 (0.02s) - --- PASS: TestDefaultConfig/#181 (0.02s) - --- PASS: TestDefaultConfig/#178 (0.02s) - --- PASS: TestDefaultConfig/#175 (0.01s) - --- PASS: TestDefaultConfig/#169 (0.01s) - --- PASS: TestDefaultConfig/#173 (0.01s) - --- PASS: TestDefaultConfig/#179 (0.02s) - --- PASS: TestDefaultConfig/#180 (0.03s) - --- PASS: TestDefaultConfig/#174 (0.02s) - --- PASS: TestDefaultConfig/#172 (0.02s) - --- PASS: TestDefaultConfig/#170 (0.01s) - --- PASS: TestDefaultConfig/#167 (0.02s) - --- PASS: TestDefaultConfig/#166 (0.02s) - --- PASS: TestDefaultConfig/#165 (0.02s) - --- PASS: TestDefaultConfig/#159 (0.01s) - --- PASS: TestDefaultConfig/#171 (0.02s) - --- PASS: TestDefaultConfig/#164 (0.02s) + --- PASS: TestDefaultConfig/#415 (0.03s) + --- PASS: TestDefaultConfig/#185 (0.03s) + --- PASS: TestDefaultConfig/#187 (0.05s) + --- PASS: TestDefaultConfig/#417 (0.07s) --- PASS: TestDefaultConfig/#411 (0.04s) - --- PASS: TestDefaultConfig/#176 (0.03s) - --- PASS: TestDefaultConfig/#158 (0.01s) - --- PASS: TestDefaultConfig/#162 (0.01s) - --- PASS: TestDefaultConfig/#161 (0.01s) - --- PASS: TestDefaultConfig/#160 (0.01s) - --- PASS: TestDefaultConfig/#157 (0.01s) + --- PASS: TestDefaultConfig/#412 (0.05s) + --- PASS: TestDefaultConfig/#414 (0.06s) + --- PASS: TestDefaultConfig/#407 (0.04s) + --- PASS: TestDefaultConfig/#413 (0.07s) + --- PASS: TestDefaultConfig/#409 (0.06s) + --- PASS: TestDefaultConfig/#182 (0.06s) + --- PASS: TestDefaultConfig/#410 (0.08s) + --- PASS: TestDefaultConfig/#402 (0.01s) + --- PASS: TestDefaultConfig/#408 (0.08s) + --- PASS: TestDefaultConfig/#403 (0.04s) + --- PASS: TestDefaultConfig/#180 (0.06s) + --- PASS: TestDefaultConfig/#406 (0.10s) + --- PASS: TestDefaultConfig/#401 (0.03s) + --- PASS: TestDefaultConfig/#404 (0.07s) + --- PASS: TestDefaultConfig/#179 (0.05s) + --- PASS: TestDefaultConfig/#178 (0.04s) + --- PASS: TestDefaultConfig/#399 (0.02s) + --- PASS: TestDefaultConfig/#400 (0.03s) + --- PASS: TestDefaultConfig/#405 (0.07s) + --- PASS: TestDefaultConfig/#181 (0.07s) + --- PASS: TestDefaultConfig/#175 (0.02s) + --- PASS: TestDefaultConfig/#396 (0.03s) + --- PASS: TestDefaultConfig/#398 (0.04s) + --- PASS: TestDefaultConfig/#177 (0.07s) + --- PASS: TestDefaultConfig/#173 (0.04s) + --- PASS: TestDefaultConfig/#176 (0.06s) + --- PASS: TestDefaultConfig/#397 (0.06s) + --- PASS: TestDefaultConfig/#394 (0.07s) + --- PASS: TestDefaultConfig/#170 (0.05s) --- PASS: TestDefaultConfig/#168 (0.03s) - --- PASS: TestDefaultConfig/#451 (0.02s) - --- PASS: TestDefaultConfig/#454 (0.02s) - --- PASS: TestDefaultConfig/#155 (0.02s) - --- PASS: TestDefaultConfig/#497 (0.02s) - --- PASS: TestDefaultConfig/#492 (0.02s) - --- PASS: TestDefaultConfig/#488 (0.01s) - --- PASS: TestDefaultConfig/#496 (0.02s) - --- PASS: TestDefaultConfig/#495 (0.02s) - --- PASS: TestDefaultConfig/#493 (0.02s) - --- PASS: TestDefaultConfig/#490 (0.01s) - --- PASS: TestDefaultConfig/#455 (0.03s) - --- PASS: TestDefaultConfig/#452 (0.02s) - --- PASS: TestDefaultConfig/#453 (0.03s) + --- PASS: TestDefaultConfig/#393 (0.06s) + --- PASS: TestDefaultConfig/#165 (0.01s) + --- PASS: TestDefaultConfig/#156 (0.06s) + --- PASS: TestDefaultConfig/#166 (0.02s) + --- PASS: TestDefaultConfig/#172 (0.10s) + --- PASS: TestDefaultConfig/#169 (0.06s) + --- PASS: TestDefaultConfig/#395 (0.11s) + --- PASS: TestDefaultConfig/#171 (0.08s) + --- PASS: TestDefaultConfig/#174 (0.12s) + --- PASS: TestDefaultConfig/#164 (0.03s) + --- PASS: TestDefaultConfig/#163 (0.02s) + --- PASS: TestDefaultConfig/#167 (0.04s) + --- PASS: TestDefaultConfig/#155 (0.03s) --- PASS: TestDefaultConfig/#153 (0.03s) - --- PASS: TestDefaultConfig/#137 (0.01s) - --- PASS: TestDefaultConfig/#456 (0.04s) - --- PASS: TestDefaultConfig/#163 (0.05s) - --- PASS: TestDefaultConfig/#491 (0.02s) - --- PASS: TestDefaultConfig/#151 (0.02s) - --- PASS: TestDefaultConfig/#156 (0.04s) - --- PASS: TestDefaultConfig/#149 (0.02s) - --- PASS: TestDefaultConfig/#150 (0.02s) - --- PASS: TestDefaultConfig/#489 (0.02s) - --- PASS: TestDefaultConfig/#498 (0.04s) - --- PASS: TestDefaultConfig/#146 (0.01s) - --- PASS: TestDefaultConfig/#494 (0.04s) - --- PASS: TestDefaultConfig/#147 (0.02s) - --- PASS: TestDefaultConfig/#154 (0.04s) - --- PASS: TestDefaultConfig/#142 (0.01s) - --- PASS: TestDefaultConfig/#143 (0.02s) - --- PASS: TestDefaultConfig/#140 (0.01s) + --- PASS: TestDefaultConfig/#162 (0.05s) + --- PASS: TestDefaultConfig/#157 (0.04s) + --- PASS: TestDefaultConfig/#159 (0.05s) + --- PASS: TestDefaultConfig/#161 (0.05s) + --- PASS: TestDefaultConfig/#149 (0.04s) + --- PASS: TestDefaultConfig/#160 (0.09s) + --- PASS: TestDefaultConfig/#158 (0.08s) + --- PASS: TestDefaultConfig/#154 (0.08s) + --- PASS: TestDefaultConfig/#148 (0.05s) + --- PASS: TestDefaultConfig/#151 (0.07s) + --- PASS: TestDefaultConfig/#152 (0.09s) --- PASS: TestDefaultConfig/#145 (0.02s) - --- PASS: TestDefaultConfig/#144 (0.02s) - --- PASS: TestDefaultConfig/#139 (0.01s) - --- PASS: TestDefaultConfig/#318 (0.01s) - --- PASS: TestDefaultConfig/#483 (0.01s) - --- PASS: TestDefaultConfig/#479 (0.01s) - --- PASS: TestDefaultConfig/#478 (0.01s) - --- PASS: TestDefaultConfig/#477 (0.02s) - --- PASS: TestDefaultConfig/#141 (0.02s) - --- PASS: TestDefaultConfig/#480 (0.02s) + --- PASS: TestDefaultConfig/#142 (0.02s) + --- PASS: TestDefaultConfig/#147 (0.07s) + --- PASS: TestDefaultConfig/#144 (0.04s) + --- PASS: TestDefaultConfig/#143 (0.04s) + --- PASS: TestDefaultConfig/#146 (0.07s) --- PASS: TestDefaultConfig/#138 (0.02s) - --- PASS: TestDefaultConfig/#130 (0.02s) - --- PASS: TestDefaultConfig/#482 (0.03s) - --- PASS: TestDefaultConfig/#148 (0.04s) - --- PASS: TestDefaultConfig/#135 (0.02s) - --- PASS: TestDefaultConfig/#136 (0.02s) - --- PASS: TestDefaultConfig/#128 (0.01s) - --- PASS: TestDefaultConfig/#132 (0.02s) - --- PASS: TestDefaultConfig/#129 (0.02s) - --- PASS: TestDefaultConfig/#127 (0.01s) - --- PASS: TestDefaultConfig/#133 (0.02s) - --- PASS: TestDefaultConfig/#29 (0.01s) - --- PASS: TestDefaultConfig/#15 (0.01s) - --- PASS: TestDefaultConfig/#131 (0.02s) - --- PASS: TestDefaultConfig/#134 (0.03s) - --- PASS: TestDefaultConfig/#481 (0.03s) - --- PASS: TestDefaultConfig/#22 (0.01s) - --- PASS: TestDefaultConfig/#126 (0.03s) - --- PASS: TestDefaultConfig/#21 (0.02s) - --- PASS: TestDefaultConfig/#20 (0.02s) - --- PASS: TestDefaultConfig/#25 (0.02s) - --- PASS: TestDefaultConfig/#28 (0.01s) - --- PASS: TestDefaultConfig/#26 (0.02s) - --- PASS: TestDefaultConfig/#27 (0.02s) - --- PASS: TestDefaultConfig/#317 (0.02s) - --- PASS: TestDefaultConfig/#19 (0.02s) - --- PASS: TestDefaultConfig/#461 (0.01s) - --- PASS: TestDefaultConfig/#16 (0.02s) - --- PASS: TestDefaultConfig/#18 (0.02s) - --- PASS: TestDefaultConfig/#23 (0.02s) - --- PASS: TestDefaultConfig/#24 (0.02s) - --- PASS: TestDefaultConfig/#460 (0.02s) - --- PASS: TestDefaultConfig/#475 (0.01s) - --- PASS: TestDefaultConfig/#57 (0.01s) + --- PASS: TestDefaultConfig/#140 (0.02s) + --- PASS: TestDefaultConfig/#141 (0.03s) + --- PASS: TestDefaultConfig/#136 (0.03s) + --- PASS: TestDefaultConfig/#150 (0.12s) + --- PASS: TestDefaultConfig/#139 (0.05s) + --- PASS: TestDefaultConfig/#134 (0.04s) + --- PASS: TestDefaultConfig/#135 (0.04s) + --- PASS: TestDefaultConfig/#137 (0.06s) + --- PASS: TestDefaultConfig/#129 (0.06s) + --- PASS: TestDefaultConfig/#133 (0.09s) + --- PASS: TestDefaultConfig/#125 (0.05s) + --- PASS: TestDefaultConfig/#130 (0.09s) + --- PASS: TestDefaultConfig/#132 (0.10s) + --- PASS: TestDefaultConfig/#128 (0.06s) + --- PASS: TestDefaultConfig/#123 (0.02s) + --- PASS: TestDefaultConfig/#120 (0.02s) + --- PASS: TestDefaultConfig/#122 (0.02s) + --- PASS: TestDefaultConfig/#131 (0.11s) + --- PASS: TestDefaultConfig/#124 (0.07s) + --- PASS: TestDefaultConfig/#127 (0.08s) + --- PASS: TestDefaultConfig/#121 (0.03s) + --- PASS: TestDefaultConfig/#119 (0.03s) + --- PASS: TestDefaultConfig/#118 (0.03s) + --- PASS: TestDefaultConfig/#116 (0.02s) + --- PASS: TestDefaultConfig/#126 (0.14s) + --- PASS: TestDefaultConfig/#117 (0.07s) + --- PASS: TestDefaultConfig/#115 (0.07s) + --- PASS: TestDefaultConfig/#111 (0.10s) + --- PASS: TestDefaultConfig/#112 (0.11s) + --- PASS: TestDefaultConfig/#109 (0.08s) + --- PASS: TestDefaultConfig/#114 (0.12s) + --- PASS: TestDefaultConfig/#106 (0.05s) + --- PASS: TestDefaultConfig/#113 (0.12s) + --- PASS: TestDefaultConfig/#104 (0.01s) + --- PASS: TestDefaultConfig/#108 (0.11s) + --- PASS: TestDefaultConfig/#102 (0.02s) + --- PASS: TestDefaultConfig/#110 (0.12s) + --- PASS: TestDefaultConfig/#101 (0.02s) + --- PASS: TestDefaultConfig/#99 (0.02s) + --- PASS: TestDefaultConfig/#96 (0.01s) + --- PASS: TestDefaultConfig/#97 (0.02s) + --- PASS: TestDefaultConfig/#100 (0.08s) + --- PASS: TestDefaultConfig/#98 (0.03s) + --- PASS: TestDefaultConfig/#103 (0.04s) + --- PASS: TestDefaultConfig/#95 (0.02s) + --- PASS: TestDefaultConfig/#107 (0.12s) + --- PASS: TestDefaultConfig/#105 (0.08s) + --- PASS: TestDefaultConfig/#94 (0.05s) + --- PASS: TestDefaultConfig/#91 (0.06s) + --- PASS: TestDefaultConfig/#86 (0.07s) + --- PASS: TestDefaultConfig/#88 (0.09s) + --- PASS: TestDefaultConfig/#93 (0.10s) + --- PASS: TestDefaultConfig/#82 (0.04s) + --- PASS: TestDefaultConfig/#90 (0.10s) + --- PASS: TestDefaultConfig/#85 (0.04s) + --- PASS: TestDefaultConfig/#92 (0.10s) + --- PASS: TestDefaultConfig/#79 (0.01s) + --- PASS: TestDefaultConfig/#81 (0.03s) + --- PASS: TestDefaultConfig/#87 (0.10s) + --- PASS: TestDefaultConfig/#83 (0.06s) + --- PASS: TestDefaultConfig/#76 (0.01s) + --- PASS: TestDefaultConfig/#84 (0.07s) + --- PASS: TestDefaultConfig/#80 (0.03s) + --- PASS: TestDefaultConfig/#390 (0.02s) + --- PASS: TestDefaultConfig/#78 (0.03s) + --- PASS: TestDefaultConfig/#392 (0.06s) + --- PASS: TestDefaultConfig/#72 (0.04s) + --- PASS: TestDefaultConfig/#391 (0.05s) + --- PASS: TestDefaultConfig/#77 (0.06s) + --- PASS: TestDefaultConfig/#75 (0.11s) + --- PASS: TestDefaultConfig/#62 (0.07s) + --- PASS: TestDefaultConfig/#71 (0.11s) + --- PASS: TestDefaultConfig/#388 (0.11s) + --- PASS: TestDefaultConfig/#73 (0.13s) + --- PASS: TestDefaultConfig/#389 (0.12s) + --- PASS: TestDefaultConfig/#74 (0.14s) + --- PASS: TestDefaultConfig/#385 (0.02s) + --- PASS: TestDefaultConfig/#68 (0.02s) + --- PASS: TestDefaultConfig/#387 (0.10s) + --- PASS: TestDefaultConfig/#384 (0.02s) + --- PASS: TestDefaultConfig/#70 (0.11s) + --- PASS: TestDefaultConfig/#66 (0.02s) + --- PASS: TestDefaultConfig/#382 (0.02s) + --- PASS: TestDefaultConfig/#69 (0.11s) + --- PASS: TestDefaultConfig/#65 (0.02s) + --- PASS: TestDefaultConfig/#383 (0.03s) + --- PASS: TestDefaultConfig/#381 (0.02s) + --- PASS: TestDefaultConfig/#67 (0.05s) + --- PASS: TestDefaultConfig/#61 (0.03s) + --- PASS: TestDefaultConfig/#380 (0.05s) + --- PASS: TestDefaultConfig/#386 (0.06s) + --- PASS: TestDefaultConfig/#63 (0.07s) + --- PASS: TestDefaultConfig/#379 (0.07s) + --- PASS: TestDefaultConfig/#378 (0.07s) + --- PASS: TestDefaultConfig/#60 (0.08s) + --- PASS: TestDefaultConfig/#372 (0.01s) + --- PASS: TestDefaultConfig/#58 (0.05s) + --- PASS: TestDefaultConfig/#375 (0.05s) + --- PASS: TestDefaultConfig/#373 (0.03s) --- PASS: TestDefaultConfig/#55 (0.01s) - --- PASS: TestDefaultConfig/#48 (0.01s) - --- PASS: TestDefaultConfig/#476 (0.01s) - --- PASS: TestDefaultConfig/#44 (0.01s) - --- PASS: TestDefaultConfig/#474 (0.02s) - --- PASS: TestDefaultConfig/#17 (0.03s) - --- PASS: TestDefaultConfig/#51 (0.01s) - --- PASS: TestDefaultConfig/#56 (0.02s) + --- PASS: TestDefaultConfig/#64 (0.12s) + --- PASS: TestDefaultConfig/#59 (0.11s) + --- PASS: TestDefaultConfig/#374 (0.05s) + --- PASS: TestDefaultConfig/#377 (0.07s) + --- PASS: TestDefaultConfig/#56 (0.03s) --- PASS: TestDefaultConfig/#52 (0.02s) - --- PASS: TestDefaultConfig/#54 (0.02s) - --- PASS: TestDefaultConfig/#53 (0.02s) - --- PASS: TestDefaultConfig/#50 (0.02s) - --- PASS: TestDefaultConfig/#47 (0.02s) - --- PASS: TestDefaultConfig/#45 (0.02s) - --- PASS: TestDefaultConfig/#46 (0.02s) - --- PASS: TestDefaultConfig/#459 (0.03s) + --- PASS: TestDefaultConfig/#376 (0.09s) + --- PASS: TestDefaultConfig/#54 (0.03s) + --- PASS: TestDefaultConfig/#57 (0.05s) + --- PASS: TestDefaultConfig/#50 (0.03s) + --- PASS: TestDefaultConfig/#51 (0.05s) + --- PASS: TestDefaultConfig/#53 (0.05s) + --- PASS: TestDefaultConfig/#48 (0.05s) + --- PASS: TestDefaultConfig/#367 (0.07s) + --- PASS: TestDefaultConfig/#366 (0.05s) + --- PASS: TestDefaultConfig/#368 (0.07s) + --- PASS: TestDefaultConfig/#371 (0.09s) + --- PASS: TestDefaultConfig/#365 (0.05s) + --- PASS: TestDefaultConfig/#360 (0.01s) + --- PASS: TestDefaultConfig/#370 (0.09s) + --- PASS: TestDefaultConfig/#49 (0.09s) + --- PASS: TestDefaultConfig/#361 (0.02s) + --- PASS: TestDefaultConfig/#363 (0.01s) + --- PASS: TestDefaultConfig/#364 (0.06s) + --- PASS: TestDefaultConfig/#362 (0.02s) + --- PASS: TestDefaultConfig/#369 (0.10s) + --- PASS: TestDefaultConfig/#356 (0.02s) + --- PASS: TestDefaultConfig/#357 (0.02s) + --- PASS: TestDefaultConfig/#355 (0.02s) + --- PASS: TestDefaultConfig/#350 (0.03s) + --- PASS: TestDefaultConfig/#359 (0.06s) + --- PASS: TestDefaultConfig/#353 (0.05s) + --- PASS: TestDefaultConfig/#354 (0.03s) + --- PASS: TestDefaultConfig/#358 (0.06s) + --- PASS: TestDefaultConfig/#348 (0.05s) + --- PASS: TestDefaultConfig/#352 (0.07s) + --- PASS: TestDefaultConfig/#349 (0.07s) --- PASS: TestDefaultConfig/#37 (0.01s) - --- PASS: TestDefaultConfig/#42 (0.01s) - --- PASS: TestDefaultConfig/#43 (0.02s) - --- PASS: TestDefaultConfig/#49 (0.02s) - --- PASS: TestDefaultConfig/#40 (0.02s) - --- PASS: TestDefaultConfig/#458 (0.01s) - --- PASS: TestDefaultConfig/#38 (0.01s) - --- PASS: TestDefaultConfig/#34 (0.01s) + --- PASS: TestDefaultConfig/#42 (0.05s) + --- PASS: TestDefaultConfig/#351 (0.10s) --- PASS: TestDefaultConfig/#35 (0.01s) - --- PASS: TestDefaultConfig/#448 (0.01s) - --- PASS: TestDefaultConfig/#41 (0.02s) - --- PASS: TestDefaultConfig/#449 (0.02s) - --- PASS: TestDefaultConfig/#39 (0.03s) - --- PASS: TestDefaultConfig/#31 (0.02s) + --- PASS: TestDefaultConfig/#43 (0.06s) + --- PASS: TestDefaultConfig/#41 (0.06s) + --- PASS: TestDefaultConfig/#40 (0.06s) --- PASS: TestDefaultConfig/#33 (0.02s) - --- PASS: TestDefaultConfig/#32 (0.02s) - --- PASS: TestDefaultConfig/#473 (0.02s) - --- PASS: TestDefaultConfig/#472 (0.02s) - --- PASS: TestDefaultConfig/#450 (0.02s) - --- PASS: TestDefaultConfig/#487 (0.02s) - --- PASS: TestDefaultConfig/#447 (0.02s) - --- PASS: TestDefaultConfig/#36 (0.03s) - --- PASS: TestDefaultConfig/#486 (0.02s) - --- PASS: TestDefaultConfig/#14 (0.01s) - --- PASS: TestDefaultConfig/#124 (0.02s) - --- PASS: TestDefaultConfig/#125 (0.02s) - --- PASS: TestDefaultConfig/#08 (0.02s) - --- PASS: TestDefaultConfig/#13 (0.02s) - --- PASS: TestDefaultConfig/#446 (0.04s) - --- PASS: TestDefaultConfig/#05 (0.02s) - --- PASS: TestDefaultConfig/#471 (0.04s) - --- PASS: TestDefaultConfig/#07 (0.03s) - --- PASS: TestDefaultConfig/#485 (0.03s) - --- PASS: TestDefaultConfig/#04 (0.03s) - --- PASS: TestDefaultConfig/#123 (0.03s) - --- PASS: TestDefaultConfig/#09 (0.03s) - --- PASS: TestDefaultConfig/#242 (0.01s) - --- PASS: TestDefaultConfig/#445 (0.03s) - --- PASS: TestDefaultConfig/#06 (0.03s) + --- PASS: TestDefaultConfig/#31 (0.01s) + --- PASS: TestDefaultConfig/#347 (0.11s) + --- PASS: TestDefaultConfig/#36 (0.04s) + --- PASS: TestDefaultConfig/#38 (0.06s) + --- PASS: TestDefaultConfig/#32 (0.03s) + --- PASS: TestDefaultConfig/#28 (0.02s) + --- PASS: TestDefaultConfig/#39 (0.08s) + --- PASS: TestDefaultConfig/#30 (0.04s) + --- PASS: TestDefaultConfig/#27 (0.04s) + --- PASS: TestDefaultConfig/#23 (0.05s) + --- PASS: TestDefaultConfig/#34 (0.09s) + --- PASS: TestDefaultConfig/#29 (0.09s) + --- PASS: TestDefaultConfig/#25 (0.09s) + --- PASS: TestDefaultConfig/#24 (0.12s) + --- PASS: TestDefaultConfig/#22 (0.11s) + --- PASS: TestDefaultConfig/#19 (0.09s) + --- PASS: TestDefaultConfig/#21 (0.13s) + --- PASS: TestDefaultConfig/#17 (0.10s) + --- PASS: TestDefaultConfig/#14 (0.04s) + --- PASS: TestDefaultConfig/#12 (0.03s) + --- PASS: TestDefaultConfig/#26 (0.17s) + --- PASS: TestDefaultConfig/#13 (0.04s) + --- PASS: TestDefaultConfig/#18 (0.11s) --- PASS: TestDefaultConfig/#11 (0.03s) - --- PASS: TestDefaultConfig/#03 (0.03s) - --- PASS: TestDefaultConfig/#12 (0.04s) - --- PASS: TestDefaultConfig/#10 (0.04s) - --- PASS: TestDefaultConfig/#01 (0.03s) - --- PASS: TestDefaultConfig/#02 (0.03s) - --- PASS: TestDefaultConfig/#58 (0.03s) - --- PASS: TestDefaultConfig/#262 (0.02s) - --- PASS: TestDefaultConfig/#319 (0.02s) + --- PASS: TestDefaultConfig/#16 (0.10s) + --- PASS: TestDefaultConfig/#15 (0.09s) + --- PASS: TestDefaultConfig/#05 (0.02s) + --- PASS: TestDefaultConfig/#20 (0.14s) + --- PASS: TestDefaultConfig/#10 (0.05s) + --- PASS: TestDefaultConfig/#08 (0.05s) + --- PASS: TestDefaultConfig/#06 (0.05s) + --- PASS: TestDefaultConfig/#04 (0.07s) + --- PASS: TestDefaultConfig/#07 (0.08s) + --- PASS: TestDefaultConfig/#01 (0.08s) + --- PASS: TestDefaultConfig/#473 (0.08s) + --- PASS: TestDefaultConfig/#492 (0.05s) + --- PASS: TestDefaultConfig/#09 (0.12s) + --- PASS: TestDefaultConfig/#489 (0.02s) + --- PASS: TestDefaultConfig/#02 (0.10s) + --- PASS: TestDefaultConfig/#03 (0.05s) + --- PASS: TestDefaultConfig/#494 (0.08s) + --- PASS: TestDefaultConfig/#488 (0.02s) + --- PASS: TestDefaultConfig/#493 (0.08s) + --- PASS: TestDefaultConfig/#486 (0.02s) + --- PASS: TestDefaultConfig/#484 (0.02s) + --- PASS: TestDefaultConfig/#487 (0.03s) + --- PASS: TestDefaultConfig/#490 (0.06s) + --- PASS: TestDefaultConfig/#482 (0.02s) + --- PASS: TestDefaultConfig/#483 (0.03s) + --- PASS: TestDefaultConfig/#491 (0.08s) + --- PASS: TestDefaultConfig/#485 (0.04s) + --- PASS: TestDefaultConfig/#480 (0.06s) + --- PASS: TestDefaultConfig/#479 (0.07s) + --- PASS: TestDefaultConfig/#478 (0.07s) + --- PASS: TestDefaultConfig/#481 (0.09s) + --- PASS: TestDefaultConfig/#477 (0.09s) + --- PASS: TestDefaultConfig/#214 (0.05s) + --- PASS: TestDefaultConfig/#216 (0.07s) + --- PASS: TestDefaultConfig/#475 (0.13s) + --- PASS: TestDefaultConfig/#44 (0.05s) + --- PASS: TestDefaultConfig/#476 (0.14s) + --- PASS: TestDefaultConfig/#206 (0.10s) + --- PASS: TestDefaultConfig/#211 (0.02s) + --- PASS: TestDefaultConfig/#215 (0.09s) + --- PASS: TestDefaultConfig/#212 (0.03s) + --- PASS: TestDefaultConfig/#210 (0.03s) + --- PASS: TestDefaultConfig/#45 (0.16s) + --- PASS: TestDefaultConfig/#46 (0.16s) + --- PASS: TestDefaultConfig/#204 (0.02s) + --- PASS: TestDefaultConfig/#474 (0.17s) + --- PASS: TestDefaultConfig/#213 (0.03s) + --- PASS: TestDefaultConfig/#205 (0.04s) + --- PASS: TestDefaultConfig/#203 (0.02s) + --- PASS: TestDefaultConfig/#209 (0.06s) + --- PASS: TestDefaultConfig/#201 (0.08s) + --- PASS: TestDefaultConfig/#208 (0.08s) + --- PASS: TestDefaultConfig/#344 (0.07s) + --- PASS: TestDefaultConfig/#336 (0.08s) + --- PASS: TestDefaultConfig/#343 (0.11s) + --- PASS: TestDefaultConfig/#472 (0.08s) + --- PASS: TestDefaultConfig/#342 (0.12s) + --- PASS: TestDefaultConfig/#462 (0.12s) + --- PASS: TestDefaultConfig/#469 (0.02s) + --- PASS: TestDefaultConfig/#202 (0.14s) + --- PASS: TestDefaultConfig/#207 (0.14s) + --- PASS: TestDefaultConfig/#470 (0.03s) + --- PASS: TestDefaultConfig/#341 (0.12s) + --- PASS: TestDefaultConfig/#471 (0.08s) + --- PASS: TestDefaultConfig/#340 (0.09s) + --- PASS: TestDefaultConfig/#464 (0.02s) + --- PASS: TestDefaultConfig/#467 (0.03s) + --- PASS: TestDefaultConfig/#465 (0.03s) + --- PASS: TestDefaultConfig/#466 (0.03s) + --- PASS: TestDefaultConfig/#468 (0.05s) + --- PASS: TestDefaultConfig/#335 (0.02s) + --- PASS: TestDefaultConfig/#337 (0.04s) + --- PASS: TestDefaultConfig/#200 (0.04s) + --- PASS: TestDefaultConfig/#339 (0.05s) + --- PASS: TestDefaultConfig/#463 (0.07s) + --- PASS: TestDefaultConfig/#198 (0.06s) + --- PASS: TestDefaultConfig/#334 (0.06s) + --- PASS: TestDefaultConfig/#199 (0.07s) + --- PASS: TestDefaultConfig/#461 (0.05s) + --- PASS: TestDefaultConfig/#333 (0.07s) + --- PASS: TestDefaultConfig/#338 (0.11s) + --- PASS: TestDefaultConfig/#47 (0.06s) + --- PASS: TestDefaultConfig/#196 (0.02s) + --- PASS: TestDefaultConfig/#456 (0.08s) + --- PASS: TestDefaultConfig/#457 (0.05s) + --- PASS: TestDefaultConfig/#459 (0.07s) + --- PASS: TestDefaultConfig/#453 (0.03s) + --- PASS: TestDefaultConfig/#460 (0.08s) + --- PASS: TestDefaultConfig/#454 (0.02s) + --- PASS: TestDefaultConfig/#455 (0.03s) + --- PASS: TestDefaultConfig/#458 (0.07s) + --- PASS: TestDefaultConfig/#197 (0.13s) + --- PASS: TestDefaultConfig/#452 (0.03s) + --- PASS: TestDefaultConfig/#451 (0.03s) === RUN TestTxnEndpoint_Bad_JSON === PAUSE TestTxnEndpoint_Bad_JSON === RUN TestTxnEndpoint_Bad_Size_Item @@ -21931,20 +22060,21 @@ util_test.go:230: DM-skipped --- SKIP: TestForwardSignals (0.00s) === RUN TestMakeWatchHandler -2022-11-07T22:45:59.721Z [DEBUG] TestMakeWatchHandler: watch handler output: watch_handler="bash -c 'echo $CONSUL_INDEX >> handler_index_out && cat >> handler_out'" output= +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:11.328Z [DEBUG] TestMakeWatchHandler: watch handler output: watch_handler="bash -c 'echo $CONSUL_INDEX >> handler_index_out && cat >> handler_out'" output= --- PASS: TestMakeWatchHandler (0.01s) === RUN TestMakeHTTPWatchHandler -2022-11-07T22:45:59.722Z [TRACE] TestMakeHTTPWatchHandler: http watch handler output: watch=http://127.0.0.1:37253 output="Ok, i see" +2021-10-05T16:31:11.329Z [TRACE] TestMakeHTTPWatchHandler: http watch handler output: watch=http://127.0.0.1:42311 output="Ok, i see" --- PASS: TestMakeHTTPWatchHandler (0.00s) === RUN TestMakeWatchPlan === RUN TestMakeWatchPlan/handler_type_script,_with_deprecated_handler_field -2022-11-07T22:45:59.722Z [WARN] The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html +2021-10-05T16:31:11.329Z [WARN] The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html === RUN TestMakeWatchPlan/handler_type_script,_with_single_arg === RUN TestMakeWatchPlan/handler_type_script,_with_multiple_args_from_slice_of_interface === RUN TestMakeWatchPlan/handler_type_script,_with_multiple_args_from_slice_of_strings === RUN TestMakeWatchPlan/handler_type_script,_with_not_string_args === RUN TestMakeWatchPlan/conflicting_handler -2022-11-07T22:45:59.723Z [WARN] The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html +2021-10-05T16:31:11.330Z [WARN] The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html === RUN TestMakeWatchPlan/no_handler_type --- PASS: TestMakeWatchPlan (0.00s) --- PASS: TestMakeWatchPlan/handler_type_script,_with_deprecated_handler_field (0.00s) @@ -21954,28 +22084,24 @@ --- PASS: TestMakeWatchPlan/handler_type_script,_with_not_string_args (0.00s) --- PASS: TestMakeWatchPlan/conflicting_handler (0.00s) --- PASS: TestMakeWatchPlan/no_handler_type (0.00s) -=== CONT TestOperator_Keyring_LocalOnly -=== CONT TestDNS_ServiceLookup_FilterCritical -=== CONT TestOperator_KeyringListLocalOnly -=== CONT TestKVSEndpoint_DELETE_ConflictingFlags -=== CONT TestKVSEndpoint_Recurse -=== CONT TestIntentionsSpecificUpdate_good -=== CONT TestStringHash -=== CONT TestIntentionsCheck_noSource -=== CONT TestIngestUserEvent -=== CONT TestSetFilePermissions === CONT TestACL_Legacy_Disabled_Response -=== CONT TestDNS_ServiceLookup_Randomize -=== CONT TestDNS_ServiceLookup_OnlyPassing -=== CONT TestOperator_Keyring_InvalidRelayFactor -=== CONT TestOperator_KeyringUse -=== CONT TestDNS_ServiceLookup_OnlyFailing -=== CONT TestOperator_KeyringRemove -=== CONT TestUserEventToken -=== CONT TestFireReceiveEvent ---- PASS: TestStringHash (0.00s) +=== CONT TestSetFilePermissions +=== CONT TestSessionDeleteDestroy --- PASS: TestSetFilePermissions (0.00s) +=== CONT TestStringHash +--- PASS: TestStringHash (0.00s) +=== CONT TestUserEventToken +=== CONT TestPreparedQuery_Get === CONT TestKVSEndpoint_PUT_ConflictingFlags +=== CONT TestPreparedQuery_Explain +=== RUN TestPreparedQuery_Explain/#00 +=== CONT TestPreparedQuery_ExecuteCached +=== CONT TestPreparedQuery_Execute +=== RUN TestPreparedQuery_Execute/#00 +=== CONT TestPreparedQuery_List +=== RUN TestPreparedQuery_List/#00 +=== CONT TestIntentionsCheck_noSource +=== RUN TestPreparedQuery_Get/#00 [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestACL_Legacy_Disabled_Response/0 @@ -21985,1662 +22111,612 @@ === RUN TestACL_Legacy_Disabled_Response/4 === RUN TestACL_Legacy_Disabled_Response/5 [WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:59.746Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.768Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.820 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bca56907-0f50-d1a8-944a-abb6013ac572 Address:127.0.0.1:11092}]" -45:59.820 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11092 [Follower]" leader= -45:59.821 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bca56907-0f50-d1a8-944a-abb6013ac572.dc1 127.0.0.1 -45:59.838 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bca56907-0f50-d1a8-944a-abb6013ac572 127.0.0.1 -2022-11-07T22:45:59.838Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-bca56907-0f50-d1a8-944a-abb6013ac572 (Addr: tcp/127.0.0.1:11092) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bca56907-0f50-d1a8-944a-abb6013ac572.dc1 area=wan -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11087 network=udp -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11087 network=tcp -45:59.839 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11088 network=tcp -45:59.839 [INFO] TestAgent: started state syncer -45:59.839 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11093 network=tcp -45:59.858 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.858 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11092 [Candidate]" term=2 -45:59.879 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.879 [DEBUG] TestAgent.server.raft: vote granted: from=bca56907-0f50-d1a8-944a-abb6013ac572 term=2 tally=1 -45:59.879 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.879 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11092 [Leader]" -45:59.879 [INFO] TestAgent.server: cluster leadership acquired -45:59.879 [INFO] TestAgent.server: New leader elected: payload=Node-bca56907-0f50-d1a8-944a-abb6013ac572 -45:59.912 [INFO] TestAgent: Synced node info -45:59.912 [DEBUG] TestAgent: Node info in sync -45:59.912 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11092 -45:59.976 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.005 [INFO] TestAgent: Requesting shutdown -46:00.005 [INFO] TestAgent.server: shutting down server -46:00.005 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.005 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.005 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.005 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.005 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.005 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.005 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bca56907-0f50-d1a8-944a-abb6013ac572 -46:00.005 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bca56907-0f50-d1a8-944a-abb6013ac572 -46:00.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:00.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.020 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.020 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -2022-11-07T22:46:00.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.027Z [INFO] agent.router.manager: shutting down -46:00.030 [INFO] TestAgent: consul server down -46:00.030 [INFO] TestAgent: shutdown complete -46:00.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11087 network=tcp -46:00.030 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:00.030 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11087 network=udp -46:00.031 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11088 network=tcp -46:00.531 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.531 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_DELETE_ConflictingFlags (0.81s) -=== CONT TestKVSEndpoint_GET_Raw -2022-11-07T22:45:59.752Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.771Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4762aefe-5f2e-0b56-e980-01ee889054bc Address:127.0.0.1:11127}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11127 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4762aefe-5f2e-0b56-e980-01ee889054bc.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4762aefe-5f2e-0b56-e980-01ee889054bc 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-4762aefe-5f2e-0b56-e980-01ee889054bc (Addr: tcp/127.0.0.1:11127) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4762aefe-5f2e-0b56-e980-01ee889054bc.dc1 area=wan -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11115 network=udp -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11115 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11116 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11128 network=tcp -45:59.896 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.896 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11127 [Candidate]" term=2 -45:59.931 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.931 [DEBUG] TestAgent.server.raft: vote granted: from=4762aefe-5f2e-0b56-e980-01ee889054bc term=2 tally=1 -45:59.931 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.931 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11127 [Leader]" -45:59.931 [INFO] TestAgent.server: cluster leadership acquired -45:59.931 [INFO] TestAgent.server: New leader elected: payload=Node-4762aefe-5f2e-0b56-e980-01ee889054bc -45:59.952 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11127 -45:59.985 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.006 [INFO] TestAgent: Requesting shutdown -46:00.006 [INFO] TestAgent.server: shutting down server -46:00.006 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.006 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:00.026 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.026 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.026 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.026 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.026 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.026 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4762aefe-5f2e-0b56-e980-01ee889054bc -46:00.026 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4762aefe-5f2e-0b56-e980-01ee889054bc -2022-11-07T22:46:00.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.027Z [INFO] agent.router.manager: shutting down -46:00.039 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-4762aefe-5f2e-0b56-e980-01ee889054bc 127.0.0.1 11118 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:4762aefe-5f2e-0b56-e980-01ee889054bc port:11127 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11119] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:00.039 [INFO] TestAgent: consul server down -46:00.039 [INFO] TestAgent: shutdown complete -46:00.039 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.039 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.039 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11115 network=tcp -46:00.039 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.039 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.039 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11115 network=udp -46:00.039 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.040 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11116 network=tcp -46:00.540 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.540 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_Keyring_InvalidRelayFactor (0.82s) -=== CONT TestKVSEndpoint_AcquireRelease -2022-11-07T22:45:59.746Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.769Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.814 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58091152-1628-1e5f-1e97-44dc3c41df5b Address:127.0.0.1:11078}]" -45:59.814 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11078 [Follower]" leader= -45:59.814 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-58091152-1628-1e5f-1e97-44dc3c41df5b.dc1 127.0.0.1 -45:59.815 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-58091152-1628-1e5f-1e97-44dc3c41df5b 127.0.0.1 -2022-11-07T22:45:59.815Z [INFO] agent.router: Initializing LAN area manager -45:59.815 [INFO] TestAgent.server: Adding LAN server: server="Node-58091152-1628-1e5f-1e97-44dc3c41df5b (Addr: tcp/127.0.0.1:11078) (DC: dc1)" -45:59.815 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-58091152-1628-1e5f-1e97-44dc3c41df5b.dc1 area=wan -45:59.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11073 network=tcp -45:59.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11073 network=udp -45:59.815 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11074 network=tcp -45:59.815 [INFO] TestAgent: started state syncer -45:59.815 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11079 network=tcp -45:59.858 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.858 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11078 [Candidate]" term=2 -45:59.868 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.868 [DEBUG] TestAgent.server.raft: vote granted: from=58091152-1628-1e5f-1e97-44dc3c41df5b term=2 tally=1 -45:59.868 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.868 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11078 [Leader]" -45:59.868 [INFO] TestAgent.server: cluster leadership acquired -45:59.868 [INFO] TestAgent.server: New leader elected: payload=Node-58091152-1628-1e5f-1e97-44dc3c41df5b -45:59.880 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11078 -45:59.912 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.975 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.975 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.975 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.975 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.975 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.975 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58091152-1628-1e5f-1e97-44dc3c41df5b -45:59.975 [INFO] TestAgent.server: member joined, marking health alive: member=Node-58091152-1628-1e5f-1e97-44dc3c41df5b -45:59.999 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.037 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:00.037 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-58091152-1628-1e5f-1e97-44dc3c41df5b -46:00.037 [INFO] TestAgent: Requesting shutdown -46:00.037 [INFO] TestAgent.server: shutting down server -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.037 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.037 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.037 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.047Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.047Z [INFO] agent.router.manager: shutting down -46:00.047 [INFO] TestAgent: consul server down -46:00.047 [INFO] TestAgent: shutdown complete -46:00.047 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11073 network=tcp -46:00.047 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11073 network=udp -46:00.047 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11074 network=tcp -46:00.548 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.548 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:59.759Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.777Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a8f4b6fc-efd3-1e98-8b11-c671fabfc024 Address:127.0.0.1:11134}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11134 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 (Addr: tcp/127.0.0.1:11134) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 area=wan -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11129 network=udp -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11129 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11130 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11135 network=tcp -45:59.908 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.909 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11134 [Candidate]" term=2 -45:59.938 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.938 [DEBUG] TestAgent.server.raft: vote granted: from=a8f4b6fc-efd3-1e98-8b11-c671fabfc024 term=2 tally=1 -45:59.938 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.938 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11134 [Leader]" -45:59.939 [INFO] TestAgent.server: cluster leadership acquired -45:59.939 [INFO] TestAgent.server: New leader elected: payload=Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -45:59.954 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11134 -45:59.954 [INFO] TestAgent: Synced node info -45:59.954 [DEBUG] TestAgent: Node info in sync -45:59.997 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.023 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.023 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.023 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.023 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.023 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.023 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.023 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.034 [INFO] TestAgent.server.serf.wan: serf: Received install-key query -46:00.034 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 -46:00.035 [INFO] TestAgent.server.serf.lan: serf: Received install-key query -46:00.035 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.035 [INFO] TestAgent.server.serf.wan: serf: Received use-key query -46:00.035 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 -46:00.035 [INFO] TestAgent.server.serf.lan: serf: Received use-key query -46:00.036 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.036 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query -46:00.036 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 -46:00.036 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query -46:00.036 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.036 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:00.037 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024.dc1 -46:00.037 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:00.037 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-a8f4b6fc-efd3-1e98-8b11-c671fabfc024 -46:00.037 [INFO] TestAgent: Requesting shutdown -46:00.037 [INFO] TestAgent.server: shutting down server -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.037 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.037 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.037 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.042 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.047Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.047Z [INFO] agent.router.manager: shutting down -46:00.047 [INFO] TestAgent: consul server down -46:00.047 [INFO] TestAgent: shutdown complete -46:00.047 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11129 network=tcp -46:00.047 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11129 network=udp -46:00.047 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11130 network=tcp -46:00.549 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.549 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_Keyring_LocalOnly (0.83s) -=== CONT TestKVSEndpoint_ListKeys ---- PASS: TestOperator_KeyringUse (0.83s) -=== CONT TestKVSEndpoint_CAS -2022-11-07T22:45:59.738Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.755Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.809 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3ed0a17-1318-72ff-4239-ce4698ba2856 Address:127.0.0.1:11120}]" -45:59.809 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11120 [Follower]" leader= -45:59.810 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3ed0a17-1318-72ff-4239-ce4698ba2856.dc1 127.0.0.1 -45:59.810 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3ed0a17-1318-72ff-4239-ce4698ba2856 127.0.0.1 -2022-11-07T22:45:59.810Z [INFO] agent.router: Initializing LAN area manager -45:59.811 [INFO] TestAgent.server: Adding LAN server: server="Node-e3ed0a17-1318-72ff-4239-ce4698ba2856 (Addr: tcp/127.0.0.1:11120) (DC: dc1)" -45:59.811 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3ed0a17-1318-72ff-4239-ce4698ba2856.dc1 area=wan -45:59.811 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11108 network=udp -45:59.811 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11108 network=tcp -45:59.811 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11109 network=tcp -45:59.811 [INFO] TestAgent: started state syncer -45:59.811 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11121 network=tcp -45:59.855 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.855 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11120 [Candidate]" term=2 -45:59.872 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.872 [DEBUG] TestAgent.server.raft: vote granted: from=e3ed0a17-1318-72ff-4239-ce4698ba2856 term=2 tally=1 -45:59.872 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.872 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11120 [Leader]" -45:59.872 [INFO] TestAgent.server: cluster leadership acquired -45:59.872 [INFO] TestAgent.server: New leader elected: payload=Node-e3ed0a17-1318-72ff-4239-ce4698ba2856 -45:59.895 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11120 -45:59.927 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.990 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.990 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.990 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.990 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.990 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.990 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3ed0a17-1318-72ff-4239-ce4698ba2856 -45:59.990 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3ed0a17-1318-72ff-4239-ce4698ba2856 -46:00.019 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.062 [INFO] TestAgent: Requesting shutdown -46:00.062 [INFO] TestAgent.server: shutting down server -46:00.062 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.062 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.062 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.062 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.063 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.063 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.063 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.066 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.071Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.071Z [INFO] agent.router.manager: shutting down -46:00.071 [INFO] TestAgent: consul server down -46:00.071 [INFO] TestAgent: shutdown complete -46:00.071 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11108 network=tcp -46:00.071 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11108 network=udp -46:00.071 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11109 network=tcp -46:00.571 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.571 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsCheck_noSource (0.85s) -=== CONT TestKVSEndpoint_DELETE_CAS -2022-11-07T22:45:59.732Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.761Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.801 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:99c413a3-bca6-db36-73d2-27425d168963 Address:127.0.0.1:11043}]" -45:59.801 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11043 [Follower]" leader= -45:59.802 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-99c413a3-bca6-db36-73d2-27425d168963.dc1 127.0.0.1 -45:59.802 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-99c413a3-bca6-db36-73d2-27425d168963 127.0.0.1 -2022-11-07T22:45:59.802Z [INFO] agent.router: Initializing LAN area manager -45:59.802 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11038 network=udp -45:59.802 [INFO] TestAgent.server: Adding LAN server: server="Node-99c413a3-bca6-db36-73d2-27425d168963 (Addr: tcp/127.0.0.1:11043) (DC: dc1)" -45:59.802 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-99c413a3-bca6-db36-73d2-27425d168963.dc1 area=wan -45:59.803 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11038 network=tcp -45:59.803 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11039 network=tcp -45:59.803 [INFO] TestAgent: started state syncer -45:59.803 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11044 network=tcp -45:59.863 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.863 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11043 [Candidate]" term=2 -45:59.892 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.892 [DEBUG] TestAgent.server.raft: vote granted: from=99c413a3-bca6-db36-73d2-27425d168963 term=2 tally=1 -45:59.892 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.892 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11043 [Leader]" -45:59.893 [INFO] TestAgent.server: cluster leadership acquired -45:59.893 [INFO] TestAgent.server: New leader elected: payload=Node-99c413a3-bca6-db36-73d2-27425d168963 -45:59.912 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11043 -45:59.949 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.999 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.999 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.999 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.999 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.999 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.999 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-99c413a3-bca6-db36-73d2-27425d168963 -45:59.999 [INFO] TestAgent.server: member joined, marking health alive: member=Node-99c413a3-bca6-db36-73d2-27425d168963 -46:00.025 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.063 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=157.619µs client=127.0.0.1:55479 client_network=udp -46:00.063 [DEBUG] TestAgent.dns: request served from client: name=740a56ab-3848-a052-760d-1b345d53fbce.query.consul. type=ANY class=IN latency=137.523µs client=127.0.0.1:41594 client_network=udp -46:00.063 [INFO] TestAgent: Requesting shutdown -46:00.063 [INFO] TestAgent.server: shutting down server -46:00.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.063 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.063 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.063 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.063 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.063 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.063 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.068 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.073Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.073Z [INFO] agent.router.manager: shutting down -46:00.073 [INFO] TestAgent: consul server down -46:00.073 [INFO] TestAgent: shutdown complete -46:00.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11038 network=tcp -46:00.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11038 network=udp -46:00.073 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11039 network=tcp -46:00.574 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.574 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_FilterCritical (0.85s) -=== CONT TestParseConsistency_Invalid -2022-11-07T22:45:59.771Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.785Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.845 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 Address:127.0.0.1:11141}]" -45:59.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11141 [Follower]" leader= -45:59.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5.dc1 127.0.0.1 -45:59.846 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 127.0.0.1 -2022-11-07T22:45:59.846Z [INFO] agent.router: Initializing LAN area manager -45:59.847 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5.dc1 area=wan -45:59.847 [INFO] TestAgent.server: Adding LAN server: server="Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 (Addr: tcp/127.0.0.1:11141) (DC: dc1)" -45:59.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11136 network=udp -45:59.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11136 network=tcp -45:59.847 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11137 network=tcp -45:59.847 [INFO] TestAgent: started state syncer -45:59.847 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11142 network=tcp -45:59.914 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.914 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11141 [Candidate]" term=2 -45:59.943 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.943 [DEBUG] TestAgent.server.raft: vote granted: from=fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 term=2 tally=1 -45:59.943 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.943 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11141 [Leader]" -45:59.943 [INFO] TestAgent.server: cluster leadership acquired -45:59.943 [INFO] TestAgent.server: New leader elected: payload=Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 -45:59.957 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11141 -45:59.981 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.018 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.018 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.018 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.018 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.018 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.018 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 -46:00.018 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fbf6e18a-aa3d-b27c-b0c0-3d2584fa66b5 -46:00.030 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.072 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=103.404µs client=127.0.0.1:54063 client_network=udp -46:00.072 [DEBUG] TestAgent.dns: request served from client: name=08fb4a7a-4045-142c-aa7b-44dccafc54e5.query.consul. type=ANY class=IN latency=152.57µs client=127.0.0.1:55896 client_network=udp -46:00.072 [INFO] TestAgent: Requesting shutdown -46:00.073 [INFO] TestAgent.server: shutting down server -46:00.073 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.073 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.073 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.073 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.073 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.073 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.073 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.080 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.084Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.084Z [INFO] agent.router.manager: shutting down -46:00.084 [INFO] TestAgent: consul server down -46:00.084 [INFO] TestAgent: shutdown complete -46:00.084 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11136 network=tcp -46:00.084 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11136 network=udp -46:00.084 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11137 network=tcp -46:00.585 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.585 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_OnlyFailing (0.86s) -=== CONT TestIntentionsList_values -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:59.750Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.771Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.819 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:369050fc-7dce-2469-6927-307e1bee261c Address:127.0.0.1:11113}]" -45:59.819 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11113 [Follower]" leader= -45:59.820 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-369050fc-7dce-2469-6927-307e1bee261c.dc1 127.0.0.1 -45:59.820 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-369050fc-7dce-2469-6927-307e1bee261c 127.0.0.1 -2022-11-07T22:45:59.820Z [INFO] agent.router: Initializing LAN area manager -45:59.820 [INFO] TestAgent.server: Adding LAN server: server="Node-369050fc-7dce-2469-6927-307e1bee261c (Addr: tcp/127.0.0.1:11113) (DC: dc1)" -45:59.820 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11122 network=udp -45:59.820 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-369050fc-7dce-2469-6927-307e1bee261c.dc1 area=wan -45:59.820 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11122 network=tcp -45:59.821 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11123 network=tcp -45:59.838 [INFO] TestAgent: started state syncer -45:59.838 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11114 network=tcp -45:59.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11113 [Candidate]" term=2 -45:59.896 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.896 [DEBUG] TestAgent.server.raft: vote granted: from=369050fc-7dce-2469-6927-307e1bee261c term=2 tally=1 -45:59.896 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.896 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11113 [Leader]" -45:59.896 [INFO] TestAgent.server: cluster leadership acquired -45:59.896 [INFO] TestAgent.server: New leader elected: payload=Node-369050fc-7dce-2469-6927-307e1bee261c -45:59.916 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11113 -45:59.958 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.997 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.997 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.997 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.997 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.998 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.998 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-369050fc-7dce-2469-6927-307e1bee261c -45:59.998 [INFO] TestAgent.server: member joined, marking health alive: member=Node-369050fc-7dce-2469-6927-307e1bee261c -46:00.009 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.107 [INFO] TestAgent: Requesting shutdown -46:00.107 [INFO] TestAgent.server: shutting down server -46:00.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.107 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.107 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.107 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.107 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.107 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.107 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.107 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.110 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.114Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.114Z [INFO] agent.router.manager: shutting down -46:00.114 [INFO] TestAgent: consul server down -46:00.114 [INFO] TestAgent: shutdown complete -46:00.114 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11122 network=tcp -46:00.114 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11122 network=udp -46:00.114 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11123 network=tcp -46:00.614 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.614 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_Disabled_Response (0.89s) +2021-10-05T16:31:11.462Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.606 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ebca0108-437c-6e1f-b2b2-14a86f4d5f03 Address:127.0.0.1:29477}]" +31:11.606 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29477 [Follower]" leader= +31:11.606 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03.dc1 127.0.0.1 +31:11.607 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 127.0.0.1 +2021-10-05T16:31:11.607Z [INFO] agent.router: Initializing LAN area manager +31:11.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29486 network=udp +31:11.607 [INFO] TestAgent.server: Adding LAN server: server="Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 (Addr: tcp/127.0.0.1:29477) (DC: dc1)" +31:11.607 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03.dc1 area=wan +31:11.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29486 network=tcp +31:11.608 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29487 network=tcp +31:11.608 [INFO] TestAgent: started state syncer +31:11.608 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29478 network=tcp +31:11.672 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.672 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29477 [Candidate]" term=2 +31:11.691 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.691 [DEBUG] TestAgent.server.raft: vote granted: from=ebca0108-437c-6e1f-b2b2-14a86f4d5f03 term=2 tally=1 +31:11.691 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.691 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29477 [Leader]" +31:11.692 [INFO] TestAgent.server: cluster leadership acquired +31:11.692 [INFO] TestAgent.server: New leader elected: payload=Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 +31:11.703 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29477 +31:11.753 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.793 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:11.794 [INFO] TestAgent: Requesting shutdown +31:11.794 [INFO] TestAgent.server: shutting down server +31:11.794 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.794 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:11.799 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:11.805 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.805 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.805 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.805 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.805 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.805 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 +31:11.805 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 +2021-10-05T16:31:11.811Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.811Z [INFO] agent.router.manager: shutting down +31:11.811 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: raft is already shutdown" +31:11.816 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-ebca0108-437c-6e1f-b2b2-14a86f4d5f03 127.0.0.1 29489 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:ebca0108-437c-6e1f-b2b2-14a86f4d5f03 port:29477 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29490] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:11.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.816 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.816 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.816 [INFO] TestAgent: consul server down +31:11.816 [INFO] TestAgent: shutdown complete +31:11.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29486 network=tcp +31:11.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29486 network=udp +31:11.816 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29487 network=tcp +31:11.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.816 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.816 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.317 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.317 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#01 +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:11.423Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.467Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.612 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a3c44a1c-cfc8-56c8-59ff-ae2b194cee27 Address:127.0.0.1:29449}]" +31:11.612 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29449 [Follower]" leader= +31:11.612 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a3c44a1c-cfc8-56c8-59ff-ae2b194cee27.dc1 127.0.0.1 +31:11.613 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a3c44a1c-cfc8-56c8-59ff-ae2b194cee27 127.0.0.1 +2021-10-05T16:31:11.613Z [INFO] agent.router: Initializing LAN area manager +31:11.613 [INFO] TestAgent.server: Adding LAN server: server="Node-a3c44a1c-cfc8-56c8-59ff-ae2b194cee27 (Addr: tcp/127.0.0.1:29449) (DC: dc1)" +31:11.613 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29444 network=udp +31:11.613 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a3c44a1c-cfc8-56c8-59ff-ae2b194cee27.dc1 area=wan +31:11.613 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29444 network=tcp +31:11.614 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29445 network=tcp +31:11.614 [INFO] TestAgent: started state syncer +31:11.614 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29450 network=tcp +31:11.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29449 [Candidate]" term=2 +31:11.712 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.712 [DEBUG] TestAgent.server.raft: vote granted: from=a3c44a1c-cfc8-56c8-59ff-ae2b194cee27 term=2 tally=1 +31:11.712 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.712 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29449 [Leader]" +31:11.712 [INFO] TestAgent.server: cluster leadership acquired +31:11.712 [INFO] TestAgent.server: New leader elected: payload=Node-a3c44a1c-cfc8-56c8-59ff-ae2b194cee27 +31:11.738 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29449 +31:11.757 [INFO] TestAgent: Synced node info +31:11.757 [DEBUG] TestAgent: Node info in sync +31:11.799 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:11.799 [INFO] TestAgent: Requesting shutdown +31:11.799 [INFO] TestAgent.server: shutting down server +31:11.799 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.808 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:11.812 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:31:11.818Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.818Z [INFO] agent.router.manager: shutting down +31:11.820 [INFO] TestAgent: consul server down +31:11.820 [INFO] TestAgent: shutdown complete +31:11.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29444 network=tcp +31:11.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29444 network=udp +31:11.820 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29445 network=tcp +31:11.820 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:12.322 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.322 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_List/#01 +2021-10-05T16:31:11.436Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.469Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.601 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e85ccfb2-813a-976e-2100-e4fd69d56f1c Address:127.0.0.1:29484}]" +31:11.601 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c.dc1 127.0.0.1 +31:11.601 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29484 [Follower]" leader= +31:11.605 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c 127.0.0.1 +2021-10-05T16:31:11.605Z [INFO] agent.router: Initializing LAN area manager +31:11.605 [INFO] TestAgent.server: Adding LAN server: server="Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c (Addr: tcp/127.0.0.1:29484) (DC: dc1)" +31:11.605 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c.dc1 area=wan +31:11.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29472 network=udp +31:11.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29472 network=tcp +31:11.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29473 network=tcp +31:11.606 [INFO] TestAgent: started state syncer +31:11.606 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29485 network=tcp +31:11.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29484 [Candidate]" term=2 +31:11.712 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.712 [DEBUG] TestAgent.server.raft: vote granted: from=e85ccfb2-813a-976e-2100-e4fd69d56f1c term=2 tally=1 +31:11.712 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.712 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29484 [Leader]" +31:11.712 [INFO] TestAgent.server: cluster leadership acquired +31:11.712 [INFO] TestAgent.server: New leader elected: payload=Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c +31:11.736 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29484 +31:11.760 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.822 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:11.823 [INFO] TestAgent: Requesting shutdown +31:11.823 [INFO] TestAgent.server: shutting down server +31:11.823 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.823 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:11.831 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.831 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.831 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.831 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.831 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.831 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c +31:11.831 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c +31:11.833 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.844Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.844Z [INFO] agent.router.manager: shutting down +31:11.846 [INFO] TestAgent: consul server down +31:11.846 [INFO] TestAgent: shutdown complete +31:11.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29472 network=tcp +31:11.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29472 network=udp +31:11.846 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29473 network=tcp +31:11.846 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-e85ccfb2-813a-976e-2100-e4fd69d56f1c 127.0.0.1 29475 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:e85ccfb2-813a-976e-2100-e4fd69d56f1c port:29484 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29476] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:11.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.847 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.847 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.847 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.847 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.348 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.348 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_ExecuteCached (1.01s) +=== CONT TestPProfHandlers_DisableDebugNoACLs +2021-10-05T16:31:11.430Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.471Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.602 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a6b48650-e4ee-6e3b-400d-4619251b720b Address:127.0.0.1:29470}]" +31:11.603 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a6b48650-e4ee-6e3b-400d-4619251b720b.dc1 127.0.0.1 +31:11.603 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a6b48650-e4ee-6e3b-400d-4619251b720b 127.0.0.1 +2021-10-05T16:31:11.603Z [INFO] agent.router: Initializing LAN area manager +31:11.603 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29465 network=udp +31:11.603 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29470 [Follower]" leader= +31:11.604 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a6b48650-e4ee-6e3b-400d-4619251b720b.dc1 area=wan +31:11.604 [INFO] TestAgent.server: Adding LAN server: server="Node-a6b48650-e4ee-6e3b-400d-4619251b720b (Addr: tcp/127.0.0.1:29470) (DC: dc1)" +31:11.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29465 network=tcp +31:11.604 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29466 network=tcp +31:11.604 [INFO] TestAgent: started state syncer +31:11.604 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29471 network=tcp +31:11.671 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.671 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29470 [Candidate]" term=2 +31:11.699 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.699 [DEBUG] TestAgent.server.raft: vote granted: from=a6b48650-e4ee-6e3b-400d-4619251b720b term=2 tally=1 +31:11.699 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.699 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29470 [Leader]" +31:11.699 [INFO] TestAgent.server: cluster leadership acquired +31:11.700 [INFO] TestAgent.server: New leader elected: payload=Node-a6b48650-e4ee-6e3b-400d-4619251b720b +31:11.710 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29470 +31:11.751 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.811 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.811 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.811 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.811 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.811 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.811 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a6b48650-e4ee-6e3b-400d-4619251b720b +31:11.811 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a6b48650-e4ee-6e3b-400d-4619251b720b +31:11.824 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:11.824 [INFO] TestAgent: Requesting shutdown +31:11.824 [INFO] TestAgent.server: shutting down server +31:11.824 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.824 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.824 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.824 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.824 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.825 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:11.825 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.825 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.825 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.827 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.827 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.831 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.845Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.845Z [INFO] agent.router.manager: shutting down +31:11.852 [INFO] TestAgent: consul server down +31:11.852 [INFO] TestAgent: shutdown complete +31:11.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29465 network=tcp +31:11.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29465 network=udp +31:11.852 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29466 network=tcp +31:11.852 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-a6b48650-e4ee-6e3b-400d-4619251b720b 127.0.0.1 29468 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:a6b48650-e4ee-6e3b-400d-4619251b720b port:29470 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29469] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:12.356 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.356 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Explain/#01 +2021-10-05T16:31:11.365Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:11.365Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.464Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:11.464Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.602 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf6a905b-b195-c514-588a-a34824c7217a Address:127.0.0.1:29435}]" +31:11.603 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 127.0.0.1 +31:11.603 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29435 [Follower]" leader= +31:11.604 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bf6a905b-b195-c514-588a-a34824c7217a 127.0.0.1 +2021-10-05T16:31:11.604Z [INFO] agent.router: Initializing LAN area manager +31:11.604 [INFO] TestAgent.server: Adding LAN server: server="Node-bf6a905b-b195-c514-588a-a34824c7217a (Addr: tcp/127.0.0.1:29435) (DC: dc1)" +31:11.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29423 network=udp +31:11.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29423 network=tcp +31:11.604 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 area=wan +31:11.605 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29424 network=tcp +31:11.605 [INFO] TestAgent: started state syncer +31:11.605 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29436 network=tcp +31:11.655 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:11.655 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.655 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29435 [Candidate]" term=2 +31:11.696 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.696 [DEBUG] TestAgent.server.raft: vote granted: from=bf6a905b-b195-c514-588a-a34824c7217a term=2 tally=1 +31:11.696 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29435 [Leader]" +31:11.697 [INFO] TestAgent.server: cluster leadership acquired +31:11.697 [INFO] TestAgent.server: New leader elected: payload=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.708 [INFO] TestAgent.server: initializing acls +31:11.719 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:11.719 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:11.734 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:11.755 [INFO] TestAgent.server: initializing acls +31:11.755 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:11.760 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:11.760 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:11.760 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:11.760 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.760 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 +31:11.760 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 area=wan +31:11.760 [INFO] TestAgent.server: Updating LAN server: server="Node-bf6a905b-b195-c514-588a-a34824c7217a (Addr: tcp/127.0.0.1:29435) (DC: dc1)" +31:11.769 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:11.769 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:11.769 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.770 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 +31:11.770 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bf6a905b-b195-c514-588a-a34824c7217a.dc1 area=wan +31:11.770 [INFO] TestAgent.server: Updating LAN server: server="Node-bf6a905b-b195-c514-588a-a34824c7217a (Addr: tcp/127.0.0.1:29435) (DC: dc1)" +31:11.791 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.832 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.832 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.832 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.832 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.832 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.832 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.832 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.853 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.853 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.865 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.874 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bf6a905b-b195-c514-588a-a34824c7217a +31:11.878 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=foo accessorID= +31:11.878 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=bar accessorID= +31:11.878 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=zip accessorID= +31:11.878 [DEBUG] TestAgent.server: User event: event=baz +31:11.878 [INFO] TestAgent: Requesting shutdown +31:11.878 [DEBUG] TestAgent: new event: event_name=baz event_id=90dedf1c-2a0f-1cb7-98bc-be748175432e +31:11.878 [INFO] TestAgent.server: shutting down server +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.878 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.879 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.879 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.879 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.879 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.884 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.888Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.888Z [INFO] agent.router.manager: shutting down +31:11.888 [INFO] TestAgent: consul server down +31:11.888 [INFO] TestAgent: shutdown complete +31:11.888 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29423 network=tcp +31:11.888 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29423 network=udp +31:11.888 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29424 network=tcp +31:12.389 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.389 [INFO] TestAgent: Endpoints down +--- PASS: TestUserEventToken (1.07s) +=== CONT TestPProfHandlers_EnableDebug +2021-10-05T16:31:11.400Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.474Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.598 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cbc59636-509a-8e64-121b-fcd10cac29a0 Address:127.0.0.1:29456}]" +31:11.599 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cbc59636-509a-8e64-121b-fcd10cac29a0.dc1 127.0.0.1 +31:11.599 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cbc59636-509a-8e64-121b-fcd10cac29a0 127.0.0.1 +2021-10-05T16:31:11.600Z [INFO] agent.router: Initializing LAN area manager +31:11.600 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29456 [Follower]" leader= +31:11.600 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29451 network=udp +31:11.600 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cbc59636-509a-8e64-121b-fcd10cac29a0.dc1 area=wan +31:11.600 [INFO] TestAgent.server: Adding LAN server: server="Node-cbc59636-509a-8e64-121b-fcd10cac29a0 (Addr: tcp/127.0.0.1:29456) (DC: dc1)" +31:11.600 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29451 network=tcp +31:11.601 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29452 network=tcp +31:11.601 [INFO] TestAgent: started state syncer +31:11.601 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29457 network=tcp +31:11.668 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.668 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29456 [Candidate]" term=2 +31:11.700 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.700 [DEBUG] TestAgent.server.raft: vote granted: from=cbc59636-509a-8e64-121b-fcd10cac29a0 term=2 tally=1 +31:11.700 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.700 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29456 [Leader]" +31:11.701 [INFO] TestAgent.server: cluster leadership acquired +31:11.701 [INFO] TestAgent.server: New leader elected: payload=Node-cbc59636-509a-8e64-121b-fcd10cac29a0 +31:11.719 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29456 +31:11.743 [INFO] TestAgent: Synced node info +31:11.743 [DEBUG] TestAgent: Node info in sync +31:11.779 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.821 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.821 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.821 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.821 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.821 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cbc59636-509a-8e64-121b-fcd10cac29a0 +31:11.821 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cbc59636-509a-8e64-121b-fcd10cac29a0 +31:11.853 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.896 [INFO] TestAgent: Requesting shutdown +31:11.896 [INFO] TestAgent.server: shutting down server +31:11.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.896 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.896 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.896 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.896 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:11.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.896 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.896 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.904 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.912Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.912Z [INFO] agent.router.manager: shutting down +31:11.912 [INFO] TestAgent: consul server down +31:11.912 [INFO] TestAgent: shutdown complete +31:11.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29451 network=tcp +31:11.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29451 network=udp +31:11.912 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29452 network=tcp +31:12.431 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.431 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsCheck_noSource (1.09s) +=== CONT TestParseWait +--- PASS: TestParseWait (0.00s) +=== CONT TestParseSource +2021-10-05T16:31:11.371Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.464Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.573 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 Address:127.0.0.1:29442}]" +31:11.574 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87.dc1 127.0.0.1 +31:11.575 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 127.0.0.1 +2021-10-05T16:31:11.575Z [INFO] agent.router: Initializing LAN area manager +31:11.576 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29442 [Follower]" leader= +31:11.577 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87.dc1 area=wan +31:11.577 [INFO] TestAgent.server: Adding LAN server: server="Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 (Addr: tcp/127.0.0.1:29442) (DC: dc1)" +31:11.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29437 network=udp +31:11.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29437 network=tcp +31:11.578 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29438 network=tcp +31:11.578 [INFO] TestAgent: started state syncer +31:11.579 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29443 network=tcp +31:11.614 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.615 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29442 [Candidate]" term=2 +31:11.624 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.624 [DEBUG] TestAgent.server.raft: vote granted: from=e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 term=2 tally=1 +31:11.624 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.624 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29442 [Leader]" +31:11.625 [INFO] TestAgent.server: cluster leadership acquired +31:11.625 [INFO] TestAgent.server: New leader elected: payload=Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 +31:11.631 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29442 +31:11.659 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.702 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.703 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.703 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.703 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.703 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.703 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 +31:11.703 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e1608fcf-2ac0-8932-2a2f-ed3a9ee59e87 +31:11.741 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.912 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:11.932 [INFO] TestAgent: Synced node info +31:11.946 [INFO] TestAgent: Requesting shutdown +31:11.946 [INFO] TestAgent.server: shutting down server +31:11.946 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.946 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.946 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.946 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.946 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.946 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:11.946 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.946 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.946 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.946 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.952 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.962Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.962Z [INFO] agent.router.manager: shutting down +31:11.962 [INFO] TestAgent: consul server down +31:11.962 [INFO] TestAgent: shutdown complete +31:11.962 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29437 network=tcp +31:11.962 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29437 network=udp +31:11.962 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29438 network=tcp +31:12.464 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.464 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_PUT_ConflictingFlags (1.13s) +=== CONT TestPrettyPrintBare +2021-10-05T16:31:11.460Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.606 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:158cfd8d-0dae-7f51-b82d-8603833a76c5 Address:127.0.0.1:29421}]" +31:11.606 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29421 [Follower]" leader= +31:11.607 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-158cfd8d-0dae-7f51-b82d-8603833a76c5.dc1 127.0.0.1 +31:11.608 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-158cfd8d-0dae-7f51-b82d-8603833a76c5 127.0.0.1 +2021-10-05T16:31:11.608Z [INFO] agent.router: Initializing LAN area manager +31:11.608 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29430 network=udp +31:11.608 [INFO] TestAgent.server: Adding LAN server: server="Node-158cfd8d-0dae-7f51-b82d-8603833a76c5 (Addr: tcp/127.0.0.1:29421) (DC: dc1)" +31:11.608 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-158cfd8d-0dae-7f51-b82d-8603833a76c5.dc1 area=wan +31:11.608 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29430 network=tcp +31:11.609 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29431 network=tcp +31:11.609 [INFO] TestAgent: started state syncer +31:11.609 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29422 network=tcp +31:11.646 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.646 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29421 [Candidate]" term=2 +31:11.672 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.672 [DEBUG] TestAgent.server.raft: vote granted: from=158cfd8d-0dae-7f51-b82d-8603833a76c5 term=2 tally=1 +31:11.672 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.672 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29421 [Leader]" +31:11.672 [INFO] TestAgent.server: cluster leadership acquired +31:11.672 [INFO] TestAgent.server: New leader elected: payload=Node-158cfd8d-0dae-7f51-b82d-8603833a76c5 +31:11.691 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29421 +31:11.732 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.776 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.776 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.776 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.776 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.776 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.776 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-158cfd8d-0dae-7f51-b82d-8603833a76c5 +31:11.776 [INFO] TestAgent.server: member joined, marking health alive: member=Node-158cfd8d-0dae-7f51-b82d-8603833a76c5 +31:11.813 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.854 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:11.863 [INFO] TestAgent: Synced node info +31:11.863 [DEBUG] TestAgent: Node info in sync +31:11.963 [INFO] TestAgent: Requesting shutdown +31:11.963 [INFO] TestAgent.server: shutting down server +31:11.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.963 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.963 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.963 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.963 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:11.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.963 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.963 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.969 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.975Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.975Z [INFO] agent.router.manager: shutting down +31:11.975 [INFO] TestAgent: consul server down +31:11.975 [INFO] TestAgent: shutdown complete +31:11.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29430 network=tcp +31:11.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29430 network=udp +31:11.975 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29431 network=tcp +31:12.475 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.475 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionDeleteDestroy (1.15s) +=== CONT TestPrettyPrint +2021-10-05T16:31:11.378Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.449Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.593 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba2615cb-1f03-2c7f-25de-cccc5e932578 Address:127.0.0.1:29428}]" +31:11.594 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba2615cb-1f03-2c7f-25de-cccc5e932578.dc1 127.0.0.1 +31:11.594 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29428 [Follower]" leader= +31:11.595 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba2615cb-1f03-2c7f-25de-cccc5e932578 127.0.0.1 +2021-10-05T16:31:11.595Z [INFO] agent.router: Initializing LAN area manager +31:11.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29416 network=udp +31:11.595 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba2615cb-1f03-2c7f-25de-cccc5e932578.dc1 area=wan +31:11.595 [INFO] TestAgent.server: Adding LAN server: server="Node-ba2615cb-1f03-2c7f-25de-cccc5e932578 (Addr: tcp/127.0.0.1:29428) (DC: dc1)" +31:11.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29416 network=tcp +31:11.596 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29417 network=tcp +31:11.596 [INFO] TestAgent: started state syncer +31:11.596 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29429 network=tcp +31:11.633 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.633 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29428 [Candidate]" term=2 +31:11.659 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.659 [DEBUG] TestAgent.server.raft: vote granted: from=ba2615cb-1f03-2c7f-25de-cccc5e932578 term=2 tally=1 +31:11.659 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.659 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29428 [Leader]" +31:11.660 [INFO] TestAgent.server: cluster leadership acquired +31:11.660 [INFO] TestAgent.server: New leader elected: payload=Node-ba2615cb-1f03-2c7f-25de-cccc5e932578 +31:11.691 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29428 +31:11.733 [INFO] TestAgent: Synced node info +31:11.733 [DEBUG] TestAgent: Node info in sync +31:11.758 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.817 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.817 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.817 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.817 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.817 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.817 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ba2615cb-1f03-2c7f-25de-cccc5e932578 +31:11.817 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ba2615cb-1f03-2c7f-25de-cccc5e932578 +31:11.854 [INFO] TestAgent.server: federation state anti-entropy synced +31:12.000 [INFO] TestAgent: Requesting shutdown +31:12.000 [INFO] TestAgent.server: shutting down server +31:12.000 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:12.000 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:12.000 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:12.000 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:12.000 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.000 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:12.001 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:12.001 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:12.001 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.001 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:12.005 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.010Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.010Z [INFO] agent.router.manager: shutting down +31:12.010 [INFO] TestAgent: consul server down +31:12.010 [INFO] TestAgent: shutdown complete +31:12.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29416 network=tcp +31:12.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29416 network=udp +31:12.010 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29417 network=tcp +31:12.510 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.510 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_Disabled_Response (1.18s) --- PASS: TestACL_Legacy_Disabled_Response/0 (0.00s) --- PASS: TestACL_Legacy_Disabled_Response/1 (0.00s) --- PASS: TestACL_Legacy_Disabled_Response/2 (0.00s) --- PASS: TestACL_Legacy_Disabled_Response/3 (0.00s) --- PASS: TestACL_Legacy_Disabled_Response/4 (0.00s) --- PASS: TestACL_Legacy_Disabled_Response/5 (0.00s) -=== CONT TestParseConsistency -2022-11-07T22:45:59.750Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.774Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08f6ecf8-5230-b9d9-1b77-53dd5b146f43 Address:127.0.0.1:11099}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11099 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43.dc1 area=wan -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 (Addr: tcp/127.0.0.1:11099) (DC: dc1)" -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11094 network=udp -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11094 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11095 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11100 network=tcp -45:59.897 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.897 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11099 [Candidate]" term=2 -45:59.923 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.923 [DEBUG] TestAgent.server.raft: vote granted: from=08f6ecf8-5230-b9d9-1b77-53dd5b146f43 term=2 tally=1 -45:59.923 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.923 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11099 [Leader]" -45:59.923 [INFO] TestAgent.server: cluster leadership acquired -45:59.923 [INFO] TestAgent.server: New leader elected: payload=Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 -45:59.949 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11099 -45:59.975 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.025 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.025 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.025 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.025 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.025 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.025 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 -46:00.025 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 -46:00.035 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.110 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:00.111 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-08f6ecf8-5230-b9d9-1b77-53dd5b146f43 -46:00.111 [INFO] TestAgent: Requesting shutdown -46:00.111 [INFO] TestAgent.server: shutting down server -46:00.111 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.111 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.111 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.111 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.111 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.111 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.111 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.111 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.111 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.111 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.111 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.117 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.123Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.123Z [INFO] agent.router.manager: shutting down -46:00.123 [INFO] TestAgent: consul server down -46:00.123 [INFO] TestAgent: shutdown complete -46:00.123 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11094 network=tcp -46:00.123 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11094 network=udp -46:00.123 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11095 network=tcp -46:00.624 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.624 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_KeyringListLocalOnly (0.90s) -=== CONT TestIntentionsList_empty -2022-11-07T22:45:59.739Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.764Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.812 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5208baf4-d27d-1c97-e09d-59e189572993 Address:127.0.0.1:11071}]" -45:59.812 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11071 [Follower]" leader= -45:59.812 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5208baf4-d27d-1c97-e09d-59e189572993.dc1 127.0.0.1 -45:59.813 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5208baf4-d27d-1c97-e09d-59e189572993 127.0.0.1 -2022-11-07T22:45:59.813Z [INFO] agent.router: Initializing LAN area manager -45:59.813 [INFO] TestAgent.server: Adding LAN server: server="Node-5208baf4-d27d-1c97-e09d-59e189572993 (Addr: tcp/127.0.0.1:11071) (DC: dc1)" -45:59.813 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5208baf4-d27d-1c97-e09d-59e189572993.dc1 area=wan -45:59.813 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11059 network=udp -45:59.813 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11059 network=tcp -45:59.813 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11060 network=tcp -45:59.813 [INFO] TestAgent: started state syncer -45:59.813 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11072 network=tcp -45:59.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11071 [Candidate]" term=2 -45:59.909 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.909 [DEBUG] TestAgent.server.raft: vote granted: from=5208baf4-d27d-1c97-e09d-59e189572993 term=2 tally=1 -45:59.909 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.909 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11071 [Leader]" -45:59.909 [INFO] TestAgent.server: cluster leadership acquired -45:59.909 [INFO] TestAgent.server: New leader elected: payload=Node-5208baf4-d27d-1c97-e09d-59e189572993 -45:59.927 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11071 -45:59.965 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.009 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.009 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.009 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.009 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.009 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.009 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5208baf4-d27d-1c97-e09d-59e189572993 -46:00.009 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5208baf4-d27d-1c97-e09d-59e189572993 -46:00.019 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.113 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [DEBUG] TestAgent: new event: event_name=test event_id= -46:00.114 [INFO] TestAgent: Requesting shutdown -46:00.114 [INFO] TestAgent.server: shutting down server -46:00.114 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.114 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.114 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.114 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.114 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.114 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.114 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.114 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.114 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.114 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.114 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.120 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.126Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.126Z [INFO] agent.router.manager: shutting down -46:00.126 [INFO] TestAgent: consul server down -46:00.126 [INFO] TestAgent: shutdown complete -46:00.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11059 network=tcp -46:00.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11059 network=udp -46:00.126 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11060 network=tcp -46:00.627 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.627 [INFO] TestAgent: Endpoints down ---- PASS: TestIngestUserEvent (0.90s) -=== CONT TestParseWait_InvalidIndex ---- PASS: TestParseWait_InvalidIndex (0.00s) -=== CONT TestRPC_HTTPSMaxConnsPerClient -=== RUN TestRPC_HTTPSMaxConnsPerClient/HTTP -2022-11-07T22:45:59.746Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.770Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.821 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa Address:127.0.0.1:11106}]" -45:59.821 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11106 [Follower]" leader= -45:59.838 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa (Addr: tcp/127.0.0.1:11106) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa.dc1 area=wan -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11101 network=udp -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11101 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11102 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11107 network=tcp -45:59.866 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.866 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11106 [Candidate]" term=2 -45:59.892 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.892 [DEBUG] TestAgent.server.raft: vote granted: from=68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa term=2 tally=1 -45:59.892 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.892 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11106 [Leader]" -45:59.892 [INFO] TestAgent.server: cluster leadership acquired -45:59.892 [INFO] TestAgent.server: New leader elected: payload=Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa -45:59.909 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11106 -45:59.949 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.004 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.004 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.004 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.004 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.004 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.004 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa -46:00.004 [INFO] TestAgent.server: member joined, marking health alive: member=Node-68ceb0ae-aa4d-8a33-aa99-4e3887c1feaa -46:00.016 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.136 [INFO] TestAgent: Requesting shutdown -46:00.136 [INFO] TestAgent.server: shutting down server -46:00.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.136 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.136 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.136 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.136 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.136 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.136 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.136 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.136 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.136 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.140 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.143Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.143Z [INFO] agent.router.manager: shutting down -46:00.143 [INFO] TestAgent: consul server down -46:00.143 [INFO] TestAgent: shutdown complete -46:00.143 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11101 network=tcp -46:00.143 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11101 network=udp -46:00.143 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11102 network=tcp -46:00.643 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.643 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_Recurse (0.92s) -=== CONT TestParseWait_InvalidTime ---- PASS: TestParseWait_InvalidTime (0.00s) -=== CONT TestHTTPServer_HandshakeTimeout -2022-11-07T22:45:59.756Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.774Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0e617cb-a2ac-0131-036e-a132c7371d7a Address:127.0.0.1:11162}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11162 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a0e617cb-a2ac-0131-036e-a132c7371d7a.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a0e617cb-a2ac-0131-036e-a132c7371d7a 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-a0e617cb-a2ac-0131-036e-a132c7371d7a (Addr: tcp/127.0.0.1:11162) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a0e617cb-a2ac-0131-036e-a132c7371d7a.dc1 area=wan -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11157 network=udp -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11157 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11158 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11163 network=tcp -45:59.881 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.881 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11162 [Candidate]" term=2 -45:59.916 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.916 [DEBUG] TestAgent.server.raft: vote granted: from=a0e617cb-a2ac-0131-036e-a132c7371d7a term=2 tally=1 -45:59.916 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.916 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11162 [Leader]" -45:59.916 [INFO] TestAgent.server: cluster leadership acquired -45:59.916 [INFO] TestAgent.server: New leader elected: payload=Node-a0e617cb-a2ac-0131-036e-a132c7371d7a -45:59.931 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11162 -45:59.957 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.993 [INFO] TestAgent: Synced node info -45:59.993 [DEBUG] TestAgent: Node info in sync -46:00.015 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.015 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.015 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.015 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.015 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.015 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0e617cb-a2ac-0131-036e-a132c7371d7a -46:00.015 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a0e617cb-a2ac-0131-036e-a132c7371d7a -46:00.024 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.135 [INFO] TestAgent: Requesting shutdown -46:00.135 [INFO] TestAgent.server: shutting down server -46:00.135 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.135 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.135 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.135 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.135 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.135 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.135 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.135 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.135 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.135 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.140 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.146Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.146Z [INFO] agent.router.manager: shutting down -46:00.146 [INFO] TestAgent: consul server down -46:00.146 [INFO] TestAgent: shutdown complete -46:00.146 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11157 network=tcp -46:00.146 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11157 network=udp -46:00.146 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11158 network=tcp -46:00.646 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.646 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_PUT_ConflictingFlags (0.92s) -=== CONT TestPProfHandlers_ACLs -2022-11-07T22:45:59.762Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.773Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b554cdf4-3bc7-0501-4f84-4f3a211215b9 Address:127.0.0.1:11050}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11050 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9 (Addr: tcp/127.0.0.1:11050) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9.dc1 area=wan -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11045 network=udp -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11045 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11046 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11051 network=tcp -45:59.880 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.880 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11050 [Candidate]" term=2 -45:59.909 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.909 [DEBUG] TestAgent.server.raft: vote granted: from=b554cdf4-3bc7-0501-4f84-4f3a211215b9 term=2 tally=1 -45:59.909 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.909 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11050 [Leader]" -45:59.911 [INFO] TestAgent.server: cluster leadership acquired -45:59.911 [INFO] TestAgent.server: New leader elected: payload=Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9 -45:59.931 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11050 -45:59.965 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.015 [INFO] TestAgent: Synced node info -46:00.015 [DEBUG] TestAgent: Node info in sync -46:00.015 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.015 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.015 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.015 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.015 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.015 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9 -46:00.015 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b554cdf4-3bc7-0501-4f84-4f3a211215b9 -46:00.033 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.175 [INFO] TestAgent: Requesting shutdown -46:00.175 [INFO] TestAgent.server: shutting down server -46:00.175 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.175 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.175 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.175 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.175 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.175 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.175 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.175 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.175 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.175 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.179 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.183Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.183Z [INFO] agent.router.manager: shutting down -46:00.183 [INFO] TestAgent: consul server down -46:00.183 [INFO] TestAgent: shutdown complete -46:00.183 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11045 network=tcp -46:00.183 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11045 network=udp -46:00.183 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11046 network=tcp -46:00.683 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.683 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsSpecificUpdate_good (0.96s) -=== CONT TestEnableWebUI -2022-11-07T22:45:59.740Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.762Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.799 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9cf3730c-7f7e-530c-735c-eccffb962827 Address:127.0.0.1:11085}]" -45:59.799 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11085 [Follower]" leader= -45:59.799 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9cf3730c-7f7e-530c-735c-eccffb962827.dc1 127.0.0.1 -45:59.800 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9cf3730c-7f7e-530c-735c-eccffb962827 127.0.0.1 -2022-11-07T22:45:59.800Z [INFO] agent.router: Initializing LAN area manager -45:59.800 [INFO] TestAgent.server: Adding LAN server: server="Node-9cf3730c-7f7e-530c-735c-eccffb962827 (Addr: tcp/127.0.0.1:11085) (DC: dc1)" -45:59.800 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9cf3730c-7f7e-530c-735c-eccffb962827.dc1 area=wan -45:59.800 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11080 network=udp -45:59.800 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11080 network=tcp -45:59.801 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11081 network=tcp -45:59.801 [INFO] TestAgent: started state syncer -45:59.801 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11086 network=tcp -45:59.854 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.854 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11085 [Candidate]" term=2 -45:59.866 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.866 [DEBUG] TestAgent.server.raft: vote granted: from=9cf3730c-7f7e-530c-735c-eccffb962827 term=2 tally=1 -45:59.866 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.866 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11085 [Leader]" -45:59.866 [INFO] TestAgent.server: cluster leadership acquired -45:59.866 [INFO] TestAgent.server: New leader elected: payload=Node-9cf3730c-7f7e-530c-735c-eccffb962827 -45:59.887 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11085 -45:59.971 [INFO] TestAgent: Synced node info -45:59.971 [DEBUG] TestAgent: Node info in sync -45:59.971 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.997 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.997 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.997 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.997 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.998 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.998 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9cf3730c-7f7e-530c-735c-eccffb962827 -45:59.998 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9cf3730c-7f7e-530c-735c-eccffb962827 -46:00.018 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.199 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=160.996µs client=127.0.0.1:53458 client_network=udp -46:00.200 [DEBUG] TestAgent.dns: request served from client: name=4c9447ab-450a-af05-9be7-3ab7995ccc6e.query.consul. type=ANY class=IN latency=130.992µs client=127.0.0.1:46132 client_network=udp -46:00.200 [INFO] TestAgent: Requesting shutdown -46:00.200 [INFO] TestAgent.server: shutting down server -46:00.200 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.200 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.204 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.211Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.211Z [INFO] agent.router.manager: shutting down -46:00.211 [INFO] TestAgent: consul server down -46:00.211 [INFO] TestAgent: shutdown complete -46:00.211 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11080 network=tcp -46:00.211 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11080 network=udp -46:00.211 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11081 network=tcp -46:00.711 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.711 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_OnlyPassing (0.99s) -=== CONT TestACLResolution -2022-11-07T22:45:59.772Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:59.772Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.786Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:45:59.786Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.845 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f86b0012-a994-aabf-e3bf-c780466bdc8d Address:127.0.0.1:11148}]" -45:59.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11148 [Follower]" leader= -45:59.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 127.0.0.1 -45:59.846 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d 127.0.0.1 -2022-11-07T22:45:59.846Z [INFO] agent.router: Initializing LAN area manager -45:59.847 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 area=wan -45:59.847 [INFO] TestAgent.server: Adding LAN server: server="Node-f86b0012-a994-aabf-e3bf-c780466bdc8d (Addr: tcp/127.0.0.1:11148) (DC: dc1)" -45:59.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11143 network=udp -45:59.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11143 network=tcp -45:59.847 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11144 network=tcp -45:59.847 [INFO] TestAgent: started state syncer -45:59.847 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11149 network=tcp -45:59.897 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:59.913 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.913 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11148 [Candidate]" term=2 -45:59.939 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.939 [DEBUG] TestAgent.server.raft: vote granted: from=f86b0012-a994-aabf-e3bf-c780466bdc8d term=2 tally=1 -45:59.939 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.939 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11148 [Leader]" -45:59.939 [INFO] TestAgent.server: cluster leadership acquired -45:59.939 [INFO] TestAgent.server: New leader elected: payload=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -45:59.941 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -45:59.956 [INFO] TestAgent.server: initializing acls -45:59.961 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:59.961 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:59.975 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:59.997 [INFO] TestAgent.server: initializing acls -45:59.997 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:00.015 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:00.015 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:00.015 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:00.015 [INFO] TestAgent: Synced node info -46:00.015 [DEBUG] TestAgent: Node info in sync -46:00.015 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:00.015 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:00.015 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.015 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 -46:00.015 [INFO] TestAgent.server: Updating LAN server: server="Node-f86b0012-a994-aabf-e3bf-c780466bdc8d (Addr: tcp/127.0.0.1:11148) (DC: dc1)" -46:00.015 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 area=wan -46:00.015 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.016 [INFO] TestAgent.server: Updating LAN server: server="Node-f86b0012-a994-aabf-e3bf-c780466bdc8d (Addr: tcp/127.0.0.1:11148) (DC: dc1)" -46:00.016 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 -46:00.016 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d.dc1 area=wan -46:00.032 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.052 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.052 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.052 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.052 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.052 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.052 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.052 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.054 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.063 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.063 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.206 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f86b0012-a994-aabf-e3bf-c780466bdc8d -46:00.210 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=foo accessorID= -46:00.210 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=bar accessorID= -46:00.211 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=zip accessorID= -46:00.211 [INFO] TestAgent: Requesting shutdown -46:00.211 [INFO] TestAgent.server: shutting down server -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.211 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.211 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.211 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.211 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.214 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.216Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.216Z [INFO] agent.router.manager: shutting down -46:00.217 [INFO] TestAgent: consul server down -46:00.217 [INFO] TestAgent: shutdown complete -46:00.217 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11143 network=tcp -46:00.217 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11143 network=udp -46:00.217 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11144 network=tcp -46:00.717 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.717 [INFO] TestAgent: Endpoints down ---- PASS: TestUserEventToken (0.99s) -=== CONT TestPProfHandlers_DisableDebugNoACLs -2022-11-07T22:45:59.746Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.770Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 Address:127.0.0.1:11057}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11057 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 (Addr: tcp/127.0.0.1:11057) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 area=wan -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11066 network=udp -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11066 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11067 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11058 network=tcp -45:59.899 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.899 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11057 [Candidate]" term=2 -45:59.932 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.932 [DEBUG] TestAgent.server.raft: vote granted: from=b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 term=2 tally=1 -45:59.932 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.932 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11057 [Leader]" -45:59.932 [INFO] TestAgent.server: cluster leadership acquired -45:59.932 [INFO] TestAgent.server: New leader elected: payload=Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -45:59.947 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11057 -45:59.970 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.014 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.014 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.014 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.014 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.014 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.014 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.014 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.036 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.228 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:00.231 [INFO] TestAgent: Synced node info -46:00.235 [INFO] TestAgent.server.serf.wan: serf: Received install-key query -46:00.243 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 -46:00.243 [INFO] TestAgent.server.serf.lan: serf: Received install-key query -46:00.248 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.248 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:00.249 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 -46:00.249 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:00.249 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.249 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query -46:00.249 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 -46:00.249 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query -46:00.250 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.250 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:00.250 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7.dc1 -46:00.250 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:00.250 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-b0a81e16-3fca-e6d7-31ed-9355bd84ebd7 -46:00.250 [INFO] TestAgent: Requesting shutdown -46:00.250 [INFO] TestAgent.server: shutting down server -46:00.250 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.250 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.250 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.250 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.250 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.250 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.250 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.250 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.250 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.250 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.255 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.259Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.259Z [INFO] agent.router.manager: shutting down -46:00.259 [INFO] TestAgent: consul server down -46:00.259 [INFO] TestAgent: shutdown complete -46:00.259 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11066 network=tcp -46:00.259 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11066 network=udp -46:00.259 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11067 network=tcp -46:00.759 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.759 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_KeyringRemove (1.04s) -=== CONT TestPrettyPrint -2022-11-07T22:45:59.758Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.780Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:68890afd-0543-d92a-f86f-bbafe5e3864d Address:127.0.0.1:11155}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11155 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-68890afd-0543-d92a-f86f-bbafe5e3864d.dc1 127.0.0.1 -45:59.840 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-68890afd-0543-d92a-f86f-bbafe5e3864d 127.0.0.1 -2022-11-07T22:45:59.840Z [INFO] agent.router: Initializing LAN area manager -45:59.840 [INFO] TestAgent.server: Adding LAN server: server="Node-68890afd-0543-d92a-f86f-bbafe5e3864d (Addr: tcp/127.0.0.1:11155) (DC: dc1)" -45:59.840 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-68890afd-0543-d92a-f86f-bbafe5e3864d.dc1 area=wan -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11150 network=udp -45:59.840 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11150 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11151 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11156 network=tcp -45:59.897 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.897 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11155 [Candidate]" term=2 -45:59.920 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.920 [DEBUG] TestAgent.server.raft: vote granted: from=68890afd-0543-d92a-f86f-bbafe5e3864d term=2 tally=1 -45:59.920 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.920 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11155 [Leader]" -45:59.920 [INFO] TestAgent.server: cluster leadership acquired -45:59.920 [INFO] TestAgent.server: New leader elected: payload=Node-68890afd-0543-d92a-f86f-bbafe5e3864d -45:59.941 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11155 -45:59.975 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.019 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.019 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.019 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.019 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.019 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.019 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-68890afd-0543-d92a-f86f-bbafe5e3864d -46:00.019 [INFO] TestAgent.server: member joined, marking health alive: member=Node-68890afd-0543-d92a-f86f-bbafe5e3864d -46:00.039 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.073 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:00.079 [INFO] TestAgent: Synced node info -46:00.079 [DEBUG] TestAgent: Node info in sync -46:00.267 [DEBUG] TestAgent.server: User event: event=deploy -46:00.267 [DEBUG] TestAgent.server: User event: event=deploy -46:00.267 [DEBUG] TestAgent: new event: event_name=deploy event_id=75abed63-bd6f-0f67-0631-c0e2b706ce79 -46:00.293 [INFO] TestAgent: Requesting shutdown -46:00.293 [INFO] TestAgent.server: shutting down server -46:00.293 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.293 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.293 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.293 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.293 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.293 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.293 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.293 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.293 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.293 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.296 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.300Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.300Z [INFO] agent.router.manager: shutting down -46:00.300 [INFO] TestAgent: consul server down -46:00.300 [INFO] TestAgent: shutdown complete -46:00.300 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11150 network=tcp -46:00.300 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11150 network=udp -46:00.300 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11151 network=tcp -46:00.800 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.800 [INFO] TestAgent: Endpoints down ---- PASS: TestFireReceiveEvent (1.08s) -=== CONT TestPProfHandlers_EnableDebug -[WARN] freeport: 10 out of 80 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:45:59.747Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.769Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.838 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:21144b96-1c07-e27c-5aaa-0daf8bbd7a81 Address:127.0.0.1:11064}]" -45:59.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11064 [Follower]" leader= -45:59.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81.dc1 127.0.0.1 -45:59.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81 127.0.0.1 -2022-11-07T22:45:59.839Z [INFO] agent.router: Initializing LAN area manager -45:59.839 [INFO] TestAgent.server: Adding LAN server: server="Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81 (Addr: tcp/127.0.0.1:11064) (DC: dc1)" -45:59.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81.dc1 area=wan -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11052 network=udp -45:59.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11052 network=tcp -45:59.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11053 network=tcp -45:59.840 [INFO] TestAgent: started state syncer -45:59.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11065 network=tcp -45:59.894 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.894 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11064 [Candidate]" term=2 -45:59.922 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.922 [DEBUG] TestAgent.server.raft: vote granted: from=21144b96-1c07-e27c-5aaa-0daf8bbd7a81 term=2 tally=1 -45:59.922 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.922 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11064 [Leader]" -45:59.922 [INFO] TestAgent.server: cluster leadership acquired -45:59.922 [INFO] TestAgent.server: New leader elected: payload=Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81 -45:59.937 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11064 -45:59.970 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.997 [INFO] TestAgent: Synced node info -45:59.997 [DEBUG] TestAgent: Node info in sync -46:00.013 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.013 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.013 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.013 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.013 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.013 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81 -46:00.014 [INFO] TestAgent.server: member joined, marking health alive: member=Node-21144b96-1c07-e27c-5aaa-0daf8bbd7a81 -46:00.035 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.145 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:00.145 [DEBUG] TestAgent: Node info in sync -46:00.465 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=764.606µs client=127.0.0.1:54080 client_network=udp -46:00.465 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=656.625µs client=127.0.0.1:38777 client_network=udp -46:00.466 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=673.414µs client=127.0.0.1:49879 client_network=udp -46:00.467 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=670.847µs client=127.0.0.1:59268 client_network=udp -46:00.468 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=675.892µs client=127.0.0.1:53885 client_network=udp -46:00.469 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=674.403µs client=127.0.0.1:58399 client_network=udp -46:00.470 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=675.351µs client=127.0.0.1:51831 client_network=udp -46:00.470 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=680.327µs client=127.0.0.1:38995 client_network=udp -46:00.471 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=666.217µs client=127.0.0.1:44686 client_network=udp -46:00.472 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=667.208µs client=127.0.0.1:46748 client_network=udp -46:00.473 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=779.309µs client=127.0.0.1:60981 client_network=udp -46:00.474 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=754.781µs client=127.0.0.1:48060 client_network=udp -46:00.475 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=710.652µs client=127.0.0.1:41668 client_network=udp -46:00.476 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=723.356µs client=127.0.0.1:38442 client_network=udp -46:00.477 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=713.242µs client=127.0.0.1:57064 client_network=udp -46:00.477 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=721.071µs client=127.0.0.1:35929 client_network=udp -46:00.478 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=723.755µs client=127.0.0.1:49492 client_network=udp -46:00.479 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=698.838µs client=127.0.0.1:59405 client_network=udp -46:00.480 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=690.365µs client=127.0.0.1:38746 client_network=udp -46:00.481 [DEBUG] TestAgent.dns: request served from client: name=273bf573-8498-7cce-d497-37030d7837d9.query.consul. type=ANY class=IN latency=723.001µs client=127.0.0.1:59334 client_network=udp -46:00.481 [INFO] TestAgent: Requesting shutdown -46:00.481 [INFO] TestAgent.server: shutting down server -46:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.481 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.481 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.481 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.481 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.485 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.488Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.488Z [INFO] agent.router.manager: shutting down -46:00.488 [INFO] TestAgent: consul server down -46:00.488 [INFO] TestAgent: shutdown complete -46:00.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11052 network=tcp -46:00.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11052 network=udp -46:00.488 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11053 network=tcp -46:00.988 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.988 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_Randomize (1.27s) === CONT TestHTTP_wrap_obfuscateLog -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022/11/07 22:46:01 http: TLS handshake error from 127.0.0.1:58820: read tcp 127.0.0.1:11236->127.0.0.1:58820: i/o timeout -=== RUN TestPProfHandlers_ACLs/case_0_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) -=== RUN TestPProfHandlers_ACLs/case_1_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/heap",_nilResponse:true}) -=== RUN TestPProfHandlers_ACLs/case_2_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/",_nilResponse:true}) -=== RUN TestPProfHandlers_ACLs/case_3_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"",_endpoint:"/debug/pprof/",_nilResponse:true}) -=== RUN TestPProfHandlers_ACLs/case_4_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) -=== RUN TestPProfHandlers_ACLs/case_5_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"towel",_endpoint:"/debug/pprof/heap",_nilResponse:true}) +2021-10-05T16:31:11.456Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:11.494Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:11.581 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab8fdbbe-58a1-affa-32ec-ab709bf18b4f Address:127.0.0.1:29463}]" +31:11.581 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f.dc1 127.0.0.1 +31:11.582 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f 127.0.0.1 +2021-10-05T16:31:11.582Z [INFO] agent.router: Initializing LAN area manager +31:11.592 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29463 [Follower]" leader= +31:11.592 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f.dc1 area=wan +31:11.592 [INFO] TestAgent.server: Adding LAN server: server="Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f (Addr: tcp/127.0.0.1:29463) (DC: dc1)" +31:11.593 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29458 network=udp +31:11.593 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29458 network=tcp +31:11.594 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29459 network=tcp +31:11.594 [INFO] TestAgent: started state syncer +31:11.594 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29464 network=tcp +31:11.644 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:11.644 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29463 [Candidate]" term=2 +31:11.692 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:11.692 [DEBUG] TestAgent.server.raft: vote granted: from=ab8fdbbe-58a1-affa-32ec-ab709bf18b4f term=2 tally=1 +31:11.692 [INFO] TestAgent.server.raft: election won: tally=1 +31:11.692 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29463 [Leader]" +31:11.692 [INFO] TestAgent.server: cluster leadership acquired +31:11.692 [INFO] TestAgent.server: New leader elected: payload=Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f +31:11.717 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29463 +31:11.759 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.844 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.844 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.844 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.844 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.844 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.844 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f +31:11.844 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab8fdbbe-58a1-affa-32ec-ab709bf18b4f +31:11.868 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.912 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:11.928 [INFO] TestAgent: Synced node info +31:11.928 [DEBUG] TestAgent: Node info in sync +31:12.020 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:12.020 [INFO] TestAgent: Requesting shutdown +31:12.020 [INFO] TestAgent.server: shutting down server +31:12.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:12.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:12.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:12.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:12.020 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.020 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:12.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:12.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:12.020 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:12.020 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.028 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.036Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.036Z [INFO] agent.router.manager: shutting down +31:12.036 [INFO] TestAgent: consul server down +31:12.036 [INFO] TestAgent: shutdown complete +31:12.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29458 network=tcp +31:12.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29458 network=udp +31:12.036 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29459 network=tcp +31:12.537 [INFO] TestAgent: Waiting for endpoints to shut down +31:12.537 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Get/#01 +[WARN] freeport: 10 out of 73 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestHTTP_wrap_obfuscateLog//some/url?token=secret1&token=secret2 === RUN TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1 === RUN TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1?token=secret2 @@ -23648,614 +22724,394 @@ === RUN TestHTTP_wrap_obfuscateLog//v1/acl/destroy/secret1?token=secret2 === RUN TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1 === RUN TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1?token=secret2 -2022-11-07T22:46:00.597Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.607Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.639 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5af36c44-66a2-cddc-3a4d-9b21d4b40034 Address:127.0.0.1:11211}]" -46:00.640 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11211 [Follower]" leader= -46:00.640 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034.dc1 127.0.0.1 -46:00.641 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 127.0.0.1 -2022-11-07T22:46:00.641Z [INFO] agent.router: Initializing LAN area manager -46:00.641 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11206 network=udp -46:00.641 [INFO] TestAgent.server: Adding LAN server: server="Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 (Addr: tcp/127.0.0.1:11211) (DC: dc1)" -46:00.641 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034.dc1 area=wan -46:00.641 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11206 network=tcp -46:00.642 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11207 network=tcp -46:00.642 [INFO] TestAgent: started state syncer -46:00.642 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11212 network=tcp -46:00.697 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.697 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11211 [Candidate]" term=2 -46:00.715 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.715 [DEBUG] TestAgent.server.raft: vote granted: from=5af36c44-66a2-cddc-3a4d-9b21d4b40034 term=2 tally=1 -46:00.715 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.715 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11211 [Leader]" -46:00.715 [INFO] TestAgent.server: cluster leadership acquired -46:00.715 [INFO] TestAgent.server: New leader elected: payload=Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 -46:00.722 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11211 -46:00.761 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.790 [INFO] TestAgent: Requesting shutdown -46:00.790 [INFO] TestAgent.server: shutting down server -46:00.790 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.790 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.803 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:00.808 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.808 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.808 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.808 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.808 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.808 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 -46:00.808 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 -2022-11-07T22:46:00.814Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.814Z [INFO] agent.router.manager: shutting down -46:00.814 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: raft is already shutdown" -46:00.816 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-5af36c44-66a2-cddc-3a4d-9b21d4b40034 127.0.0.1 11209 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:5af36c44-66a2-cddc-3a4d-9b21d4b40034 port:11211 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11210] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:00.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.816 [INFO] TestAgent: consul server down -46:00.816 [INFO] TestAgent: shutdown complete -46:00.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11206 network=tcp -46:00.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.816 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.816 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11206 network=udp -46:00.816 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.816 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.816 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11207 network=tcp -46:01.317 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.317 [INFO] TestAgent: Endpoints down -=== CONT TestParseWait ---- PASS: TestIntentionsList_values (0.73s) ---- PASS: TestParseWait (0.00s) +2021-10-05T16:31:12.391Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.463Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.554 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a57429d-13af-c063-ed92-4d8e1c83b6b3 Address:127.0.0.1:29505}]" +31:12.555 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a57429d-13af-c063-ed92-4d8e1c83b6b3.dc1 127.0.0.1 +31:12.555 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29505 [Follower]" leader= +31:12.564 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a57429d-13af-c063-ed92-4d8e1c83b6b3 127.0.0.1 +2021-10-05T16:31:12.564Z [INFO] agent.router: Initializing LAN area manager +31:12.564 [INFO] TestAgent.server: Adding LAN server: server="Node-1a57429d-13af-c063-ed92-4d8e1c83b6b3 (Addr: tcp/127.0.0.1:29505) (DC: dc1)" +31:12.564 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a57429d-13af-c063-ed92-4d8e1c83b6b3.dc1 area=wan +31:12.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29514 network=tcp +31:12.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29514 network=udp +31:12.565 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29515 network=tcp +31:12.565 [INFO] TestAgent: started state syncer +31:12.565 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29506 network=tcp +31:12.604 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.604 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29505 [Candidate]" term=2 +31:12.618 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.618 [DEBUG] TestAgent.server.raft: vote granted: from=1a57429d-13af-c063-ed92-4d8e1c83b6b3 term=2 tally=1 +31:12.618 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.618 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29505 [Leader]" +31:12.619 [INFO] TestAgent.server: cluster leadership acquired +31:12.619 [INFO] TestAgent.server: New leader elected: payload=Node-1a57429d-13af-c063-ed92-4d8e1c83b6b3 +31:12.624 [INFO] TestAgent: Requesting shutdown +31:12.624 [INFO] TestAgent.server: shutting down server +31:12.624 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.624 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:12.629 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29505 +31:12.639 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.651Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.651Z [INFO] agent.router.manager: shutting down +31:12.660 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +31:12.660 [INFO] TestAgent: consul server down +31:12.660 [INFO] TestAgent: shutdown complete +31:12.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29514 network=tcp +31:12.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29514 network=udp +31:12.660 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29515 network=tcp +31:13.164 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.164 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Explain/#02 +2021-10-05T16:31:12.351Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.437Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.512 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0476889f-c667-bba3-05b3-5964842e4872 Address:127.0.0.1:29491}]" +31:12.512 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0476889f-c667-bba3-05b3-5964842e4872.dc1 127.0.0.1 +31:12.513 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0476889f-c667-bba3-05b3-5964842e4872 127.0.0.1 +2021-10-05T16:31:12.513Z [INFO] agent.router: Initializing LAN area manager +31:12.513 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29500 network=udp +31:12.513 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29491 [Follower]" leader= +31:12.514 [INFO] TestAgent.server: Adding LAN server: server="Node-0476889f-c667-bba3-05b3-5964842e4872 (Addr: tcp/127.0.0.1:29491) (DC: dc1)" +31:12.514 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0476889f-c667-bba3-05b3-5964842e4872.dc1 area=wan +31:12.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29500 network=tcp +31:12.515 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29501 network=tcp +31:12.515 [INFO] TestAgent: started state syncer +31:12.515 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29492 network=tcp +31:12.573 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.573 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29491 [Candidate]" term=2 +31:12.598 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.598 [DEBUG] TestAgent.server.raft: vote granted: from=0476889f-c667-bba3-05b3-5964842e4872 term=2 tally=1 +31:12.598 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.598 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29491 [Leader]" +31:12.604 [INFO] TestAgent.server: cluster leadership acquired +31:12.604 [INFO] TestAgent.server: New leader elected: payload=Node-0476889f-c667-bba3-05b3-5964842e4872 +31:12.619 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29491 +31:12.648 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.668 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:12.668 [INFO] TestAgent: Requesting shutdown +31:12.668 [INFO] TestAgent.server: shutting down server +31:12.668 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.668 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:12.679 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.692Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.692Z [INFO] agent.router.manager: shutting down +31:12.697 [INFO] TestAgent: consul server down +31:12.697 [INFO] TestAgent: shutdown complete +31:12.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29500 network=tcp +31:12.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29500 network=udp +31:12.697 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29501 network=tcp +31:12.697 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:12.697 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:12.697 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:12.697 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:12.697 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:13.200 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.200 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_List (1.86s) + --- PASS: TestPreparedQuery_List/#00 (0.99s) + --- PASS: TestPreparedQuery_List/#01 (0.88s) === CONT TestContentTypeIsJSON -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:00.563Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.573Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.605 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c4be5878-5057-9fdc-f0ca-b260aec63b83 Address:127.0.0.1:11169}]" -46:00.605 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11169 [Follower]" leader= -46:00.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c4be5878-5057-9fdc-f0ca-b260aec63b83.dc1 127.0.0.1 -46:00.606 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c4be5878-5057-9fdc-f0ca-b260aec63b83 127.0.0.1 -2022-11-07T22:46:00.606Z [INFO] agent.router: Initializing LAN area manager -46:00.606 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c4be5878-5057-9fdc-f0ca-b260aec63b83.dc1 area=wan -46:00.606 [INFO] TestAgent.server: Adding LAN server: server="Node-c4be5878-5057-9fdc-f0ca-b260aec63b83 (Addr: tcp/127.0.0.1:11169) (DC: dc1)" -46:00.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11178 network=udp -46:00.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11178 network=tcp -46:00.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11179 network=tcp -46:00.606 [INFO] TestAgent: started state syncer -46:00.606 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11170 network=tcp -46:00.671 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.671 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11169 [Candidate]" term=2 -46:00.693 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.693 [DEBUG] TestAgent.server.raft: vote granted: from=c4be5878-5057-9fdc-f0ca-b260aec63b83 term=2 tally=1 -46:00.693 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.693 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11169 [Leader]" -46:00.693 [INFO] TestAgent.server: cluster leadership acquired -46:00.693 [INFO] TestAgent.server: New leader elected: payload=Node-c4be5878-5057-9fdc-f0ca-b260aec63b83 -46:00.701 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11169 -46:00.721 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.757 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.757 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.757 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.757 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.757 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.757 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c4be5878-5057-9fdc-f0ca-b260aec63b83 -46:00.757 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c4be5878-5057-9fdc-f0ca-b260aec63b83 -46:00.786 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.825 [INFO] TestAgent: Requesting shutdown -46:00.825 [INFO] TestAgent.server: shutting down server -46:00.825 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.825 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.825 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.825 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.825 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.825 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.825 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.825 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.825 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.825 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.825 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.832 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.842Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.842Z [INFO] agent.router.manager: shutting down -46:00.843 [INFO] TestAgent: consul server down -46:00.843 [INFO] TestAgent: shutdown complete -46:00.843 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11178 network=tcp -46:00.843 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11178 network=udp -46:00.843 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11179 network=tcp -46:01.343 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.343 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_AcquireRelease (0.80s) -=== CONT TestParseSource -2022-11-07T22:46:00.542Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.562Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.586 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf71b4a8-82be-ad5b-cbba-7539b721f2be Address:127.0.0.1:11176}]" -46:00.587 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11176 [Follower]" leader= -46:00.587 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be.dc1 127.0.0.1 -46:00.588 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be 127.0.0.1 -2022-11-07T22:46:00.588Z [INFO] agent.router: Initializing LAN area manager -46:00.588 [INFO] TestAgent.server: Adding LAN server: server="Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be (Addr: tcp/127.0.0.1:11176) (DC: dc1)" -46:00.588 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be.dc1 area=wan -46:00.588 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11164 network=udp -46:00.588 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11164 network=tcp -46:00.589 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11165 network=tcp -46:00.589 [INFO] TestAgent: started state syncer -46:00.589 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11177 network=tcp -46:00.626 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.626 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11176 [Candidate]" term=2 -46:00.638 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.638 [DEBUG] TestAgent.server.raft: vote granted: from=cf71b4a8-82be-ad5b-cbba-7539b721f2be term=2 tally=1 -46:00.638 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.638 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11176 [Leader]" -46:00.640 [INFO] TestAgent.server: cluster leadership acquired -46:00.640 [INFO] TestAgent.server: New leader elected: payload=Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be -46:00.645 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11176 -46:00.657 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.695 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.695 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.695 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.695 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.695 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.695 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be -46:00.695 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cf71b4a8-82be-ad5b-cbba-7539b721f2be -46:00.713 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.828 [INFO] TestAgent: Requesting shutdown -46:00.828 [INFO] TestAgent.server: shutting down server -46:00.828 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.828 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.828 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.828 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.828 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.828 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.828 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.828 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.828 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.828 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.828 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.838 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.849Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.849Z [INFO] agent.router.manager: shutting down -46:00.849 [INFO] TestAgent: consul server down -46:00.849 [INFO] TestAgent: shutdown complete -46:00.849 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11164 network=tcp -46:00.849 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11164 network=udp -46:00.849 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11165 network=tcp -46:01.349 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.349 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_GET_Raw (0.82s) -=== CONT TestPrettyPrintBare -2022-11-07T22:46:00.580Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.591Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.628 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9746e30b-3e66-aa18-2a60-be8923737275 Address:127.0.0.1:11197}]" -46:00.628 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11197 [Follower]" leader= -46:00.629 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9746e30b-3e66-aa18-2a60-be8923737275.dc1 127.0.0.1 -46:00.629 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9746e30b-3e66-aa18-2a60-be8923737275 127.0.0.1 -2022-11-07T22:46:00.629Z [INFO] agent.router: Initializing LAN area manager -46:00.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11192 network=udp -46:00.629 [INFO] TestAgent.server: Adding LAN server: server="Node-9746e30b-3e66-aa18-2a60-be8923737275 (Addr: tcp/127.0.0.1:11197) (DC: dc1)" -46:00.629 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9746e30b-3e66-aa18-2a60-be8923737275.dc1 area=wan -46:00.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11192 network=tcp -46:00.630 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11193 network=tcp -46:00.630 [INFO] TestAgent: started state syncer -46:00.630 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11198 network=tcp -46:00.665 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.665 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11197 [Candidate]" term=2 -46:00.682 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.682 [DEBUG] TestAgent.server.raft: vote granted: from=9746e30b-3e66-aa18-2a60-be8923737275 term=2 tally=1 -46:00.682 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.682 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11197 [Leader]" -46:00.682 [INFO] TestAgent.server: cluster leadership acquired -46:00.682 [INFO] TestAgent.server: New leader elected: payload=Node-9746e30b-3e66-aa18-2a60-be8923737275 -46:00.695 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11197 -46:00.722 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.761 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.761 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.761 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.761 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.761 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.761 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9746e30b-3e66-aa18-2a60-be8923737275 -46:00.761 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9746e30b-3e66-aa18-2a60-be8923737275 -46:00.779 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.848 [INFO] TestAgent: Requesting shutdown -46:00.848 [INFO] TestAgent.server: shutting down server -46:00.848 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.848 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.848 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.848 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.848 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.848 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.848 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.859 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.876Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.876Z [INFO] agent.router.manager: shutting down -46:00.876 [INFO] TestAgent: consul server down -46:00.877 [INFO] TestAgent: shutdown complete -46:00.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11192 network=tcp -46:00.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11192 network=udp -46:00.877 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11193 network=tcp -46:01.377 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.377 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_DELETE_CAS (0.81s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:12.332Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.364Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.485 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06804c88-1e15-b32b-3582-f4eaecefa609 Address:127.0.0.1:29498}]" +31:12.485 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29498 [Follower]" leader= +31:12.505 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-06804c88-1e15-b32b-3582-f4eaecefa609.dc1 127.0.0.1 +31:12.506 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-06804c88-1e15-b32b-3582-f4eaecefa609 127.0.0.1 +2021-10-05T16:31:12.506Z [INFO] agent.router: Initializing LAN area manager +31:12.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29479 network=udp +31:12.506 [INFO] TestAgent.server: Adding LAN server: server="Node-06804c88-1e15-b32b-3582-f4eaecefa609 (Addr: tcp/127.0.0.1:29498) (DC: dc1)" +31:12.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29479 network=tcp +31:12.506 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-06804c88-1e15-b32b-3582-f4eaecefa609.dc1 area=wan +31:12.525 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.525 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29498 [Candidate]" term=2 +31:12.525 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29480 network=tcp +31:12.526 [INFO] TestAgent: started state syncer +31:12.526 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29499 network=tcp +31:12.592 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.592 [DEBUG] TestAgent.server.raft: vote granted: from=06804c88-1e15-b32b-3582-f4eaecefa609 term=2 tally=1 +31:12.592 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.592 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29498 [Leader]" +31:12.593 [INFO] TestAgent.server: cluster leadership acquired +31:12.593 [INFO] TestAgent.server: New leader elected: payload=Node-06804c88-1e15-b32b-3582-f4eaecefa609 +31:12.618 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29498 +31:12.719 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.720 [INFO] TestAgent: Synced node info +31:12.720 [DEBUG] TestAgent: Node info in sync +31:12.759 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.759 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.759 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.759 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.759 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.759 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06804c88-1e15-b32b-3582-f4eaecefa609 +31:12.759 [INFO] TestAgent.server: member joined, marking health alive: member=Node-06804c88-1e15-b32b-3582-f4eaecefa609 +31:12.786 [INFO] TestAgent.server: federation state anti-entropy synced +31:12.884 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:12.884 [INFO] TestAgent: Requesting shutdown +31:12.884 [INFO] TestAgent.server: shutting down server +31:12.884 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:12.884 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:12.884 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:12.884 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:12.884 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.884 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:12.884 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:12.884 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:12.884 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:12.884 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.890 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.900Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.900Z [INFO] agent.router.manager: shutting down +31:12.901 [INFO] TestAgent: consul server down +31:12.901 [INFO] TestAgent: shutdown complete +31:12.901 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29479 network=tcp +31:12.901 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29479 network=udp +31:12.901 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29480 network=tcp +31:13.404 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.404 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#02 +2021-10-05T16:31:12.529Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.545Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.605 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5cf3dde9-5e08-3ff0-adf7-70467e260f00 Address:127.0.0.1:29540}]" +31:12.606 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29540 [Follower]" leader= +31:12.610 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00.dc1 127.0.0.1 +31:12.610 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00 127.0.0.1 +2021-10-05T16:31:12.610Z [INFO] agent.router: Initializing LAN area manager +31:12.611 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29535 network=udp +31:12.611 [INFO] TestAgent.server: Adding LAN server: server="Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00 (Addr: tcp/127.0.0.1:29540) (DC: dc1)" +31:12.611 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00.dc1 area=wan +31:12.611 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29535 network=tcp +31:12.612 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29536 network=tcp +31:12.612 [INFO] TestAgent: started state syncer +31:12.612 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29541 network=tcp +31:12.662 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.662 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29540 [Candidate]" term=2 +31:12.704 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.704 [DEBUG] TestAgent.server.raft: vote granted: from=5cf3dde9-5e08-3ff0-adf7-70467e260f00 term=2 tally=1 +31:12.704 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.704 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29540 [Leader]" +31:12.706 [INFO] TestAgent.server: cluster leadership acquired +31:12.706 [INFO] TestAgent.server: New leader elected: payload=Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00 +31:12.722 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29540 +31:12.757 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.830 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.830 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.830 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.830 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.830 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.830 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00 +31:12.830 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5cf3dde9-5e08-3ff0-adf7-70467e260f00 +31:12.864 [INFO] TestAgent.server: federation state anti-entropy synced +31:12.927 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key?pretty=1 from= latency=138.194µs +31:12.927 [INFO] TestAgent: Requesting shutdown +31:12.927 [INFO] TestAgent.server: shutting down server +31:12.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:12.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:12.927 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:12.927 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:12.927 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.927 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:12.927 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:12.927 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:12.927 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:12.927 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:12.927 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.934 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.941Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.941Z [INFO] agent.router.manager: shutting down +31:12.941 [INFO] TestAgent: consul server down +31:12.941 [INFO] TestAgent: shutdown complete +31:12.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29535 network=tcp +31:12.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29535 network=udp +31:12.941 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29536 network=tcp +31:13.444 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.444 [INFO] TestAgent: Endpoints down +--- PASS: TestPrettyPrint (0.97s) === CONT TestUIResponseHeaders -2022-11-07T22:46:00.567Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.582Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.614 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d98c986e-acc4-6ea8-46bf-e9ea7c1d214a Address:127.0.0.1:11190}]" -46:00.615 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a.dc1 127.0.0.1 -46:00.615 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a 127.0.0.1 -2022-11-07T22:46:00.615Z [INFO] agent.router: Initializing LAN area manager -46:00.615 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11190 [Follower]" leader= -46:00.615 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a.dc1 area=wan -46:00.616 [INFO] TestAgent.server: Adding LAN server: server="Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a (Addr: tcp/127.0.0.1:11190) (DC: dc1)" -46:00.616 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11185 network=udp -46:00.616 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11185 network=tcp -46:00.616 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11186 network=tcp -46:00.616 [INFO] TestAgent: started state syncer -46:00.616 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11191 network=tcp -46:00.653 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.653 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11190 [Candidate]" term=2 -46:00.663 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.663 [DEBUG] TestAgent.server.raft: vote granted: from=d98c986e-acc4-6ea8-46bf-e9ea7c1d214a term=2 tally=1 -46:00.663 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.663 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11190 [Leader]" -46:00.664 [INFO] TestAgent.server: cluster leadership acquired -46:00.664 [INFO] TestAgent.server: New leader elected: payload=Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a -46:00.671 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11190 -46:00.694 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.732 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.732 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.732 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.732 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.732 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.732 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a -46:00.732 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d98c986e-acc4-6ea8-46bf-e9ea7c1d214a -46:00.757 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.882 [INFO] TestAgent: Requesting shutdown -46:00.882 [INFO] TestAgent.server: shutting down server -46:00.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.882 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.882 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.882 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.882 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.882 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.882 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.882 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.890 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.900Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.900Z [INFO] agent.router.manager: shutting down -46:00.900 [INFO] TestAgent: consul server down -46:00.900 [INFO] TestAgent: shutdown complete -46:00.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11185 network=tcp -46:00.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11185 network=udp -46:00.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11186 network=tcp -46:01.401 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.401 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_CAS (0.85s) +2021-10-05T16:31:12.391Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.467Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.532 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 Address:127.0.0.1:29512}]" +31:12.532 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29512 [Follower]" leader= +31:12.547 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6.dc1 127.0.0.1 +31:12.549 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 127.0.0.1 +2021-10-05T16:31:12.549Z [INFO] agent.router: Initializing LAN area manager +31:12.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29493 network=udp +31:12.550 [INFO] TestAgent.server: Adding LAN server: server="Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 (Addr: tcp/127.0.0.1:29512) (DC: dc1)" +31:12.550 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6.dc1 area=wan +31:12.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29493 network=tcp +31:12.551 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29494 network=tcp +31:12.551 [INFO] TestAgent: started state syncer +31:12.552 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29513 network=tcp +31:12.584 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.584 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29512 [Candidate]" term=2 +31:12.619 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.619 [DEBUG] TestAgent.server.raft: vote granted: from=7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 term=2 tally=1 +31:12.619 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.619 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29512 [Leader]" +31:12.619 [INFO] TestAgent.server: cluster leadership acquired +31:12.619 [INFO] TestAgent.server: New leader elected: payload=Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 +31:12.647 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29512 +31:12.704 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.752 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.752 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.752 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.752 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.752 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.752 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 +31:12.752 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7cdaeaaa-d8bb-4b2c-1303-e0c70620fdf6 +31:12.775 [INFO] TestAgent.server: federation state anti-entropy synced +31:12.815 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:12.839 [INFO] TestAgent: Synced node info +31:12.959 [INFO] TestAgent: Requesting shutdown +31:12.959 [INFO] TestAgent.server: shutting down server +31:12.959 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:12.959 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:12.959 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:12.959 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:12.959 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:12.959 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:12.959 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:12.959 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:12.959 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:12.959 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:12.963 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:12.968Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:12.968Z [INFO] agent.router.manager: shutting down +31:12.968 [INFO] TestAgent: consul server down +31:12.968 [INFO] TestAgent: shutdown complete +31:12.968 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29493 network=tcp +31:12.968 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29493 network=udp +31:12.968 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29494 network=tcp +31:13.469 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.469 [INFO] TestAgent: Endpoints down +--- PASS: TestPProfHandlers_DisableDebugNoACLs (1.12s) === CONT TestHTTPAPIResponseHeaders -2022-11-07T22:46:00.634Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.650Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.699 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e316659-11e5-bfb5-1534-00c944ef6eab Address:127.0.0.1:11225}]" -46:00.699 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11225 [Follower]" leader= -46:00.700 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9e316659-11e5-bfb5-1534-00c944ef6eab.dc1 127.0.0.1 -46:00.700 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9e316659-11e5-bfb5-1534-00c944ef6eab 127.0.0.1 -2022-11-07T22:46:00.700Z [INFO] agent.router: Initializing LAN area manager -46:00.701 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11220 network=udp -46:00.701 [INFO] TestAgent.server: Adding LAN server: server="Node-9e316659-11e5-bfb5-1534-00c944ef6eab (Addr: tcp/127.0.0.1:11225) (DC: dc1)" -46:00.701 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11220 network=tcp -46:00.701 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9e316659-11e5-bfb5-1534-00c944ef6eab.dc1 area=wan -46:00.701 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11221 network=tcp -46:00.701 [INFO] TestAgent: started state syncer -46:00.701 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11226 network=tcp -46:00.738 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.738 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11225 [Candidate]" term=2 -46:00.753 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.753 [DEBUG] TestAgent.server.raft: vote granted: from=9e316659-11e5-bfb5-1534-00c944ef6eab term=2 tally=1 -46:00.753 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.753 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11225 [Leader]" -46:00.753 [INFO] TestAgent.server: cluster leadership acquired -46:00.753 [INFO] TestAgent.server: New leader elected: payload=Node-9e316659-11e5-bfb5-1534-00c944ef6eab -46:00.763 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11225 -46:00.802 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.852 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.852 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.852 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.852 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.852 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.852 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9e316659-11e5-bfb5-1534-00c944ef6eab -46:00.852 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9e316659-11e5-bfb5-1534-00c944ef6eab -46:00.863 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.890 [INFO] TestAgent: Requesting shutdown -46:00.890 [INFO] TestAgent.server: shutting down server -46:00.890 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.890 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.890 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.890 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.890 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.890 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.890 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.890 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.890 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.890 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.890 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.900 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.908Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.908Z [INFO] agent.router.manager: shutting down -46:00.908 [INFO] TestAgent: consul server down -46:00.908 [INFO] TestAgent: shutdown complete -46:00.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11220 network=tcp -46:00.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11220 network=udp -46:00.908 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11221 network=tcp -46:01.408 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.408 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsList_empty (0.78s) +2021-10-05T16:31:12.526Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.549Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.617 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7a6ab3a-726e-16a9-c414-b861bb248a06 Address:127.0.0.1:29533}]" +31:12.617 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29533 [Follower]" leader= +31:12.618 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7a6ab3a-726e-16a9-c414-b861bb248a06.dc1 127.0.0.1 +31:12.618 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7a6ab3a-726e-16a9-c414-b861bb248a06 127.0.0.1 +2021-10-05T16:31:12.618Z [INFO] agent.router: Initializing LAN area manager +31:12.619 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29528 network=udp +31:12.619 [INFO] TestAgent.server: Adding LAN server: server="Node-c7a6ab3a-726e-16a9-c414-b861bb248a06 (Addr: tcp/127.0.0.1:29533) (DC: dc1)" +31:12.619 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29528 network=tcp +31:12.619 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7a6ab3a-726e-16a9-c414-b861bb248a06.dc1 area=wan +31:12.619 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29529 network=tcp +31:12.619 [INFO] TestAgent: started state syncer +31:12.619 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29534 network=tcp +31:12.660 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.660 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29533 [Candidate]" term=2 +31:12.693 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.693 [DEBUG] TestAgent.server.raft: vote granted: from=c7a6ab3a-726e-16a9-c414-b861bb248a06 term=2 tally=1 +31:12.693 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.693 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29533 [Leader]" +31:12.694 [INFO] TestAgent.server: cluster leadership acquired +31:12.694 [INFO] TestAgent.server: New leader elected: payload=Node-c7a6ab3a-726e-16a9-c414-b861bb248a06 +31:12.716 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29533 +31:12.736 [INFO] TestAgent: Synced node info +31:12.736 [DEBUG] TestAgent: Node info in sync +31:12.767 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.815 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.816 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.816 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.816 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.816 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.816 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7a6ab3a-726e-16a9-c414-b861bb248a06 +31:12.816 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7a6ab3a-726e-16a9-c414-b861bb248a06 +31:12.852 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.021 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key?pretty from= latency=29.101µs +31:13.021 [INFO] TestAgent: Requesting shutdown +31:13.021 [INFO] TestAgent.server: shutting down server +31:13.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.021 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.021 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.021 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.021 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.021 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.021 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.032 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.037Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.037Z [INFO] agent.router.manager: shutting down +31:13.037 [INFO] TestAgent: consul server down +31:13.037 [INFO] TestAgent: shutdown complete +31:13.037 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29528 network=tcp +31:13.037 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29528 network=udp +31:13.037 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29529 network=tcp +31:13.537 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.537 [INFO] TestAgent: Endpoints down +--- PASS: TestPrettyPrintBare (1.07s) === CONT TestHTTPAPI_TranslateAddrHeader -2022-11-07T22:46:00.644Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.657Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.714 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5e3b4588-706a-b933-acff-90136befd10b Address:127.0.0.1:11232}]" -46:00.714 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11232 [Follower]" leader= -46:00.715 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5e3b4588-706a-b933-acff-90136befd10b.dc1 127.0.0.1 -46:00.715 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5e3b4588-706a-b933-acff-90136befd10b 127.0.0.1 -2022-11-07T22:46:00.715Z [INFO] agent.router: Initializing LAN area manager -46:00.715 [INFO] TestAgent.server: Adding LAN server: server="Node-5e3b4588-706a-b933-acff-90136befd10b (Addr: tcp/127.0.0.1:11232) (DC: dc1)" -46:00.715 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5e3b4588-706a-b933-acff-90136befd10b.dc1 area=wan -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11227 network=udp -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11227 network=tcp -46:00.716 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11228 network=tcp -46:00.716 [INFO] TestAgent: started state syncer -46:00.716 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11233 network=tcp -46:00.765 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.765 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11232 [Candidate]" term=2 -46:00.797 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.797 [DEBUG] TestAgent.server.raft: vote granted: from=5e3b4588-706a-b933-acff-90136befd10b term=2 tally=1 -46:00.797 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.797 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11232 [Leader]" -46:00.797 [INFO] TestAgent.server: cluster leadership acquired -46:00.797 [INFO] TestAgent.server: New leader elected: payload=Node-5e3b4588-706a-b933-acff-90136befd10b -46:00.810 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11232 -46:00.877 [INFO] TestAgent: Synced node info -46:00.877 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.900 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.900 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.900 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.900 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.900 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.900 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e3b4588-706a-b933-acff-90136befd10b -46:00.900 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5e3b4588-706a-b933-acff-90136befd10b -46:00.922 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.941 [INFO] TestAgent: Requesting shutdown -46:00.941 [INFO] TestAgent.server: shutting down server -46:00.941 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.941 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.947 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.955Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.955Z [INFO] agent.router.manager: shutting down -46:00.955 [INFO] TestAgent: consul server down -46:00.955 [INFO] TestAgent: shutdown complete -46:00.955 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11227 network=tcp -46:00.955 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11227 network=udp -46:00.955 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11228 network=tcp -46:01.455 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.455 [INFO] TestAgent: Endpoints down -=== RUN TestRPC_HTTPSMaxConnsPerClient/HTTPS -2022-11-07T22:46:00.583Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.593Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.628 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f9730011-76c8-735b-60af-b5441c99856b Address:127.0.0.1:11204}]" -46:00.628 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11204 [Follower]" leader= -46:00.629 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f9730011-76c8-735b-60af-b5441c99856b.dc1 127.0.0.1 -46:00.629 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f9730011-76c8-735b-60af-b5441c99856b 127.0.0.1 -2022-11-07T22:46:00.629Z [INFO] agent.router: Initializing LAN area manager -46:00.629 [INFO] TestAgent.server: Adding LAN server: server="Node-f9730011-76c8-735b-60af-b5441c99856b (Addr: tcp/127.0.0.1:11204) (DC: dc1)" -46:00.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11199 network=udp -46:00.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11199 network=tcp -46:00.629 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f9730011-76c8-735b-60af-b5441c99856b.dc1 area=wan -46:00.630 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11200 network=tcp -46:00.630 [INFO] TestAgent: started state syncer -46:00.630 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11205 network=tcp -46:00.678 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.678 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11204 [Candidate]" term=2 -46:00.701 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.701 [DEBUG] TestAgent.server.raft: vote granted: from=f9730011-76c8-735b-60af-b5441c99856b term=2 tally=1 -46:00.701 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.701 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11204 [Leader]" -46:00.701 [INFO] TestAgent.server: cluster leadership acquired -46:00.701 [INFO] TestAgent.server: New leader elected: payload=Node-f9730011-76c8-735b-60af-b5441c99856b -46:00.713 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11204 -46:00.730 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.797 [INFO] TestAgent: Synced node info -46:00.797 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.797 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.797 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.797 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.797 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.797 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9730011-76c8-735b-60af-b5441c99856b -46:00.797 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f9730011-76c8-735b-60af-b5441c99856b -46:00.816 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.948 [INFO] TestAgent: Requesting shutdown -46:00.948 [INFO] TestAgent.server: shutting down server -46:00.948 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.948 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.948 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.948 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.948 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.948 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.948 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.948 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.948 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.948 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:00.955 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.962Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.962Z [INFO] agent.router.manager: shutting down -46:00.962 [INFO] TestAgent: consul server down -46:00.962 [INFO] TestAgent: shutdown complete -46:00.962 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11199 network=tcp -46:00.962 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11199 network=udp -46:00.962 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11200 network=tcp -46:01.462 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.462 [INFO] TestAgent: Endpoints down ---- PASS: TestParseConsistency_Invalid (0.89s) -=== CONT TestSetIndex ---- PASS: TestSetIndex (0.00s) +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:12.526Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.557Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.652 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0d48f913-f773-7df0-e52a-2b9dd3eb1f46 Address:127.0.0.1:29526}]" +31:12.652 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46.dc1 127.0.0.1 +31:12.653 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46 127.0.0.1 +2021-10-05T16:31:12.653Z [INFO] agent.router: Initializing LAN area manager +31:12.656 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29526 [Follower]" leader= +31:12.660 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46.dc1 area=wan +31:12.660 [INFO] TestAgent.server: Adding LAN server: server="Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46 (Addr: tcp/127.0.0.1:29526) (DC: dc1)" +31:12.664 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29521 network=udp +31:12.664 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29521 network=tcp +31:12.674 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29522 network=tcp +31:12.674 [INFO] TestAgent: started state syncer +31:12.674 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29527 network=tcp +31:12.704 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.704 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29526 [Candidate]" term=2 +31:12.749 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.749 [DEBUG] TestAgent.server.raft: vote granted: from=0d48f913-f773-7df0-e52a-2b9dd3eb1f46 term=2 tally=1 +31:12.749 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.749 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29526 [Leader]" +31:12.751 [INFO] TestAgent.server: cluster leadership acquired +31:12.751 [INFO] TestAgent.server: New leader elected: payload=Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46 +31:12.788 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29526 +31:12.833 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.884 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.884 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.884 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.884 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.884 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.884 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46 +31:12.884 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0d48f913-f773-7df0-e52a-2b9dd3eb1f46 +31:12.913 [INFO] TestAgent.server: federation state anti-entropy synced +31:12.975 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:12.978 [INFO] TestAgent: Synced node info +31:13.092 [INFO] TestAgent: Requesting shutdown +31:13.092 [INFO] TestAgent.server: shutting down server +31:13.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.092 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.092 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.092 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.092 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.092 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.092 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.100 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.107Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.107Z [INFO] agent.router.manager: shutting down +31:13.107 [INFO] TestAgent: consul server down +31:13.107 [INFO] TestAgent: shutdown complete +31:13.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29521 network=tcp +31:13.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29521 network=udp +31:13.107 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29522 network=tcp +31:13.608 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.608 [INFO] TestAgent: Endpoints down +--- PASS: TestParseSource (1.18s) === CONT TestHTTPAPI_BlockEndpoints -2022-11-07T22:46:00.774Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.788Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.839 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:27ec629c-d1b8-ab9c-fe70-766992e94f9d Address:127.0.0.1:11267}]" -46:00.839 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11267 [Follower]" leader= -46:00.840 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-27ec629c-d1b8-ab9c-fe70-766992e94f9d.dc1 127.0.0.1 -46:00.841 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-27ec629c-d1b8-ab9c-fe70-766992e94f9d 127.0.0.1 -2022-11-07T22:46:00.841Z [INFO] agent.router: Initializing LAN area manager -46:00.841 [INFO] TestAgent.server: Adding LAN server: server="Node-27ec629c-d1b8-ab9c-fe70-766992e94f9d (Addr: tcp/127.0.0.1:11267) (DC: dc1)" -46:00.841 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-27ec629c-d1b8-ab9c-fe70-766992e94f9d.dc1 area=wan -46:00.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11276 network=tcp -46:00.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11276 network=udp -46:00.841 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11277 network=tcp -46:00.841 [INFO] TestAgent: started state syncer -46:00.841 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11268 network=tcp -46:00.894 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.894 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11267 [Candidate]" term=2 -46:00.908 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.908 [DEBUG] TestAgent.server.raft: vote granted: from=27ec629c-d1b8-ab9c-fe70-766992e94f9d term=2 tally=1 -46:00.908 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.908 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11267 [Leader]" -46:00.908 [INFO] TestAgent.server: cluster leadership acquired -46:00.908 [INFO] TestAgent.server: New leader elected: payload=Node-27ec629c-d1b8-ab9c-fe70-766992e94f9d -46:00.921 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11267 -46:00.949 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.953 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key?pretty=1 from= latency=78.604µs -46:00.953 [INFO] TestAgent: Requesting shutdown -46:00.953 [INFO] TestAgent.server: shutting down server -46:00.953 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.953 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.958 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:00.964Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:00.964Z [INFO] agent.router.manager: shutting down -46:00.964 [INFO] TestAgent: consul server down -46:00.964 [INFO] TestAgent: shutdown complete -46:00.964 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11276 network=tcp -46:00.964 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11276 network=udp -46:00.964 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11277 network=tcp -46:00.964 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" -46:01.465 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.465 [INFO] TestAgent: Endpoints down ---- PASS: TestPrettyPrint (0.71s) -=== CONT TestOperator_KeyringList -2022-11-07T22:46:00.560Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.569Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.600 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c89f26af-fab5-bd14-3d11-93d7f793bd95 Address:127.0.0.1:11183}]" -46:00.600 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11183 [Follower]" leader= -46:00.601 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c89f26af-fab5-bd14-3d11-93d7f793bd95.dc1 127.0.0.1 -46:00.602 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c89f26af-fab5-bd14-3d11-93d7f793bd95 127.0.0.1 -2022-11-07T22:46:00.602Z [INFO] agent.router: Initializing LAN area manager -46:00.602 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c89f26af-fab5-bd14-3d11-93d7f793bd95.dc1 area=wan -46:00.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11171 network=udp -46:00.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11171 network=tcp -46:00.602 [INFO] TestAgent.server: Adding LAN server: server="Node-c89f26af-fab5-bd14-3d11-93d7f793bd95 (Addr: tcp/127.0.0.1:11183) (DC: dc1)" -46:00.602 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11172 network=tcp -46:00.602 [INFO] TestAgent: started state syncer -46:00.602 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11184 network=tcp -46:00.667 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.667 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11183 [Candidate]" term=2 -46:00.685 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.685 [DEBUG] TestAgent.server.raft: vote granted: from=c89f26af-fab5-bd14-3d11-93d7f793bd95 term=2 tally=1 -46:00.685 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.685 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11183 [Leader]" -46:00.685 [INFO] TestAgent.server: cluster leadership acquired -46:00.685 [INFO] TestAgent.server: New leader elected: payload=Node-c89f26af-fab5-bd14-3d11-93d7f793bd95 -46:00.699 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11183 -46:00.722 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.760 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.760 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.760 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.760 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.760 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.760 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c89f26af-fab5-bd14-3d11-93d7f793bd95 -46:00.760 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c89f26af-fab5-bd14-3d11-93d7f793bd95 -46:00.795 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.016 [INFO] TestAgent: Requesting shutdown -46:01.016 [INFO] TestAgent.server: shutting down server -46:01.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.016 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.016 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.016 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.016 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.016 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.016 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.016 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.017 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.016 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.026 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.033Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.033Z [INFO] agent.router.manager: shutting down -46:01.033 [INFO] TestAgent: consul server down -46:01.033 [INFO] TestAgent: shutdown complete -46:01.033 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11171 network=tcp -46:01.033 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11171 network=udp -46:01.033 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11172 network=tcp -46:01.533 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.533 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_ListKeys (0.99s) -=== CONT TestHTTPServer_UnixSocket_FileExists -2022-11-07T22:46:00.628Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.645Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.678 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e2ac16e-fd11-23f7-2b29-a5feb40f218d Address:127.0.0.1:11218}]" -46:00.679 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11218 [Follower]" leader= -46:00.679 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d.dc1 127.0.0.1 -46:00.680 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d 127.0.0.1 -2022-11-07T22:46:00.680Z [INFO] agent.router: Initializing LAN area manager -46:00.680 [INFO] TestAgent.server: Adding LAN server: server="Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d (Addr: tcp/127.0.0.1:11218) (DC: dc1)" -46:00.680 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d.dc1 area=wan -46:00.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11213 network=tcp -46:00.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11213 network=udp -46:00.680 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11214 network=tcp -46:00.680 [INFO] TestAgent: started state syncer -46:00.680 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11219 network=tcp -46:00.746 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.746 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11218 [Candidate]" term=2 -46:00.765 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.765 [DEBUG] TestAgent.server.raft: vote granted: from=2e2ac16e-fd11-23f7-2b29-a5feb40f218d term=2 tally=1 -46:00.765 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.765 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11218 [Leader]" -46:00.765 [INFO] TestAgent.server: cluster leadership acquired -46:00.765 [INFO] TestAgent.server: New leader elected: payload=Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d -46:00.780 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11218 -46:00.808 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.852 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.852 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.852 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.852 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.852 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.852 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d -46:00.852 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2e2ac16e-fd11-23f7-2b29-a5feb40f218d -46:00.881 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.991 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:00.995 [INFO] TestAgent: Synced node info -46:01.069 [INFO] TestAgent: Requesting shutdown -46:01.070 [INFO] TestAgent.server: shutting down server -46:01.070 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.070 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.070 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.070 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.070 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.070 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.070 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.070 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.070 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.070 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.073 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.078Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.078Z [INFO] agent.router.manager: shutting down -46:01.078 [INFO] TestAgent: consul server down -46:01.078 [INFO] TestAgent: shutdown complete -46:01.078 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11213 network=tcp -46:01.078 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11213 network=udp -46:01.078 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11214 network=tcp -46:01.579 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.579 [INFO] TestAgent: Endpoints down ---- PASS: TestParseConsistency (0.97s) +--- PASS: TestHTTP_wrap_obfuscateLog (1.14s) + --- PASS: TestHTTP_wrap_obfuscateLog//some/url?token=secret1&token=secret2 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1?token=secret2 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/destroy/secret1 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/destroy/secret1?token=secret2 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1 (0.00s) + --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1?token=secret2 (0.00s) === CONT TestSetMeta --- PASS: TestSetMeta (0.00s) === CONT TestSetLastContact @@ -24268,4019 +23124,3823 @@ --- PASS: TestSetLastContact/zero (0.00s) --- PASS: TestSetLastContact/pos (0.00s) --- PASS: TestSetLastContact/pos_ms_only (0.00s) +=== CONT TestSetKnownLeader +--- PASS: TestSetKnownLeader (0.00s) +=== CONT TestSetIndex +--- PASS: TestSetIndex (0.00s) +=== CONT TestHTTPServer_UnixSocket_FileExists +2021-10-05T16:31:12.573Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.620Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.711 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c336e0b8-069c-5861-b782-85b2a7b5d35c Address:127.0.0.1:29554}]" +31:12.711 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29554 [Follower]" leader= +31:12.713 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c336e0b8-069c-5861-b782-85b2a7b5d35c.dc1 127.0.0.1 +31:12.714 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c336e0b8-069c-5861-b782-85b2a7b5d35c 127.0.0.1 +2021-10-05T16:31:12.714Z [INFO] agent.router: Initializing LAN area manager +31:12.714 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29549 network=udp +31:12.715 [INFO] TestAgent.server: Adding LAN server: server="Node-c336e0b8-069c-5861-b782-85b2a7b5d35c (Addr: tcp/127.0.0.1:29554) (DC: dc1)" +31:12.715 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c336e0b8-069c-5861-b782-85b2a7b5d35c.dc1 area=wan +31:12.715 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29549 network=tcp +31:12.715 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29550 network=tcp +31:12.716 [INFO] TestAgent: started state syncer +31:12.716 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29555 network=tcp +31:12.768 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.768 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29554 [Candidate]" term=2 +31:12.816 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.816 [DEBUG] TestAgent.server.raft: vote granted: from=c336e0b8-069c-5861-b782-85b2a7b5d35c term=2 tally=1 +31:12.816 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.816 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29554 [Leader]" +31:12.817 [INFO] TestAgent.server: cluster leadership acquired +31:12.828 [INFO] TestAgent.server: New leader elected: payload=Node-c336e0b8-069c-5861-b782-85b2a7b5d35c +31:12.844 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29554 +31:12.884 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.912 [INFO] TestAgent: Synced node info +31:12.932 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.932 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.932 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.932 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.932 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.932 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c336e0b8-069c-5861-b782-85b2a7b5d35c +31:12.932 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c336e0b8-069c-5861-b782-85b2a7b5d35c +31:12.944 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.148 [INFO] TestAgent: Requesting shutdown +31:13.148 [INFO] TestAgent.server: shutting down server +31:13.148 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.148 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.148 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.148 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.148 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.148 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.155 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.168Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.168Z [INFO] agent.router.manager: shutting down +31:13.168 [INFO] TestAgent: consul server down +31:13.168 [INFO] TestAgent: shutdown complete +31:13.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29549 network=tcp +31:13.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29549 network=udp +31:13.168 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29550 network=tcp +31:13.668 [INFO] TestAgent: Waiting for endpoints to shut down +31:13.668 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Get (2.33s) + --- PASS: TestPreparedQuery_Get/#00 (1.20s) + --- PASS: TestPreparedQuery_Get/#01 (1.12s) === CONT TestHTTPServer_UnixSocket -[WARN] freeport: 10 out of 80 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:00.727Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.742Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.810 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88a82ac4-fb59-5734-6a3e-42695ba5b952 Address:127.0.0.1:11274}]" -46:00.810 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11274 [Follower]" leader= -46:00.810 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88a82ac4-fb59-5734-6a3e-42695ba5b952.dc1 127.0.0.1 -46:00.811 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88a82ac4-fb59-5734-6a3e-42695ba5b952 127.0.0.1 -2022-11-07T22:46:00.811Z [INFO] agent.router: Initializing LAN area manager -46:00.811 [INFO] TestAgent.server: Adding LAN server: server="Node-88a82ac4-fb59-5734-6a3e-42695ba5b952 (Addr: tcp/127.0.0.1:11274) (DC: dc1)" -46:00.811 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11262 network=udp -46:00.811 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88a82ac4-fb59-5734-6a3e-42695ba5b952.dc1 area=wan -46:00.811 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11262 network=tcp -46:00.811 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11263 network=tcp -46:00.811 [INFO] TestAgent: started state syncer -46:00.811 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11275 network=tcp -46:00.852 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.852 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11274 [Candidate]" term=2 -46:00.879 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.879 [DEBUG] TestAgent.server.raft: vote granted: from=88a82ac4-fb59-5734-6a3e-42695ba5b952 term=2 tally=1 -46:00.879 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.879 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11274 [Leader]" -46:00.879 [INFO] TestAgent.server: cluster leadership acquired -46:00.879 [INFO] TestAgent.server: New leader elected: payload=Node-88a82ac4-fb59-5734-6a3e-42695ba5b952 -46:00.886 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11274 -46:00.906 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.955 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.955 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.955 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.955 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.955 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.955 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88a82ac4-fb59-5734-6a3e-42695ba5b952 -46:00.955 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88a82ac4-fb59-5734-6a3e-42695ba5b952 -46:00.968 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.120 [INFO] TestAgent: Requesting shutdown -46:01.120 [INFO] TestAgent.server: shutting down server -46:01.120 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.120 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.120 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.120 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.120 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.120 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.120 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.123 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.128Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.128Z [INFO] agent.router.manager: shutting down -46:01.128 [INFO] TestAgent: consul server down -46:01.128 [INFO] TestAgent: shutdown complete -46:01.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11262 network=tcp -46:01.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11262 network=udp -46:01.128 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11263 network=tcp -46:01.628 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.628 [INFO] TestAgent: Endpoints down ---- PASS: TestPProfHandlers_DisableDebugNoACLs (0.91s) -=== CONT TestOperator_KeyringInstall -2022-11-07T22:46:00.693Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.703Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.748 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f51e6c2-d767-618c-fc32-bd1a0682d752 Address:127.0.0.1:11253}]" -46:00.748 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11253 [Follower]" leader= -46:00.748 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0f51e6c2-d767-618c-fc32-bd1a0682d752.dc1 127.0.0.1 -46:00.749 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0f51e6c2-d767-618c-fc32-bd1a0682d752 127.0.0.1 -2022-11-07T22:46:00.749Z [INFO] agent.router: Initializing LAN area manager -46:00.749 [INFO] TestAgent.server: Adding LAN server: server="Node-0f51e6c2-d767-618c-fc32-bd1a0682d752 (Addr: tcp/127.0.0.1:11253) (DC: dc1)" -46:00.749 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0f51e6c2-d767-618c-fc32-bd1a0682d752.dc1 area=wan -46:00.749 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11248 network=udp -46:00.749 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11248 network=tcp -46:00.750 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11249 network=tcp -46:00.750 [INFO] TestAgent: started state syncer -46:00.750 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11254 network=tcp -46:00.801 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.801 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11253 [Candidate]" term=2 -46:00.819 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.819 [DEBUG] TestAgent.server.raft: vote granted: from=0f51e6c2-d767-618c-fc32-bd1a0682d752 term=2 tally=1 -46:00.819 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.819 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11253 [Leader]" -46:00.819 [INFO] TestAgent.server: cluster leadership acquired -46:00.819 [INFO] TestAgent.server: New leader elected: payload=Node-0f51e6c2-d767-618c-fc32-bd1a0682d752 -46:00.830 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11253 -46:00.851 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.900 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.900 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.900 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.900 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.900 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.900 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f51e6c2-d767-618c-fc32-bd1a0682d752 -46:00.900 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0f51e6c2-d767-618c-fc32-bd1a0682d752 -46:00.909 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.099 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.104 [INFO] TestAgent: Synced node info -46:01.104 [DEBUG] TestAgent: Node info in sync -46:01.154 [INFO] TestAgent: Requesting shutdown -46:01.154 [INFO] TestAgent.server: shutting down server -46:01.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.154 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.154 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.154 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.154 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.154 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.154 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.158 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.162Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.162Z [INFO] agent.router.manager: shutting down -46:01.162 [INFO] TestAgent: consul server down -46:01.162 [INFO] TestAgent: shutdown complete -46:01.162 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11248 network=tcp -46:01.162 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11248 network=udp -46:01.162 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11249 network=tcp -46:01.662 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.662 [INFO] TestAgent: Endpoints down ---- PASS: TestEnableWebUI (0.98s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:13.242Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.285Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.330 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:853672ee-9f85-fb30-bc4a-ff2b8217dae5 Address:127.0.0.1:29568}]" +31:13.330 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29568 [Follower]" leader= +31:13.331 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5.dc1 127.0.0.1 +31:13.331 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5 127.0.0.1 +2021-10-05T16:31:13.331Z [INFO] agent.router: Initializing LAN area manager +31:13.332 [INFO] TestAgent.server: Adding LAN server: server="Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5 (Addr: tcp/127.0.0.1:29568) (DC: dc1)" +31:13.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29563 network=udp +31:13.332 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5.dc1 area=wan +31:13.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29563 network=tcp +31:13.333 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29564 network=tcp +31:13.333 [INFO] TestAgent: started state syncer +31:13.333 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29569 network=tcp +31:13.370 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.370 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29568 [Candidate]" term=2 +31:13.384 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.384 [DEBUG] TestAgent.server.raft: vote granted: from=853672ee-9f85-fb30-bc4a-ff2b8217dae5 term=2 tally=1 +31:13.384 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.384 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29568 [Leader]" +31:13.384 [INFO] TestAgent.server: cluster leadership acquired +31:13.385 [INFO] TestAgent.server: New leader elected: payload=Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5 +31:13.420 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29568 +31:13.468 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.520 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.520 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.520 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.520 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.520 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.520 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5 +31:13.520 [INFO] TestAgent.server: member joined, marking health alive: member=Node-853672ee-9f85-fb30-bc4a-ff2b8217dae5 +31:13.552 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.564 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key from= latency=35.068µs +31:13.564 [INFO] TestAgent: Requesting shutdown +31:13.564 [INFO] TestAgent.server: shutting down server +31:13.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.564 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.564 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.564 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.564 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.564 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:13.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.564 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.564 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.573 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.589Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.589Z [INFO] agent.router.manager: shutting down +31:13.589 [INFO] TestAgent: consul server down +31:13.589 [INFO] TestAgent: shutdown complete +31:13.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29563 network=tcp +31:13.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29563 network=udp +31:13.589 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29564 network=tcp +31:14.092 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.092 [INFO] TestAgent: Endpoints down +--- PASS: TestContentTypeIsJSON (0.89s) === CONT TestFilterNonPassing --- PASS: TestFilterNonPassing (0.00s) -=== CONT TestSetKnownLeader ---- PASS: TestSetKnownLeader (0.00s) -=== CONT TestOperator_RaftPeer -=== RUN TestOperator_RaftPeer/#00 -2022-11-07T22:46:00.662Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:00.662Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.672Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:00.672Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.715 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58dab99f-45e4-b523-9fb0-c48201594101 Address:127.0.0.1:11246}]" -46:00.715 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11246 [Follower]" leader= -46:00.715 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-58dab99f-45e4-b523-9fb0-c48201594101.dc1 127.0.0.1 -46:00.716 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-58dab99f-45e4-b523-9fb0-c48201594101 127.0.0.1 -2022-11-07T22:46:00.716Z [INFO] agent.router: Initializing LAN area manager -46:00.716 [INFO] TestAgent.server: Adding LAN server: server="Node-58dab99f-45e4-b523-9fb0-c48201594101 (Addr: tcp/127.0.0.1:11246) (DC: dc1)" -46:00.716 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-58dab99f-45e4-b523-9fb0-c48201594101.dc1 area=wan -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11241 network=udp -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11241 network=tcp -46:00.717 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11242 network=tcp -46:00.717 [INFO] TestAgent: started state syncer -46:00.717 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11247 network=tcp -46:00.767 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:00.769 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.769 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11246 [Candidate]" term=2 -46:00.795 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.795 [DEBUG] TestAgent.server.raft: vote granted: from=58dab99f-45e4-b523-9fb0-c48201594101 term=2 tally=1 -46:00.795 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.795 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11246 [Leader]" -46:00.795 [INFO] TestAgent.server: cluster leadership acquired -46:00.795 [INFO] TestAgent.server: New leader elected: payload=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:00.810 [INFO] TestAgent.server: initializing acls -46:00.814 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:00.814 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:00.820 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:00.831 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:00.852 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:00.852 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:00.852 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:00.852 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:00.852 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-58dab99f-45e4-b523-9fb0-c48201594101.dc1 -46:00.852 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-58dab99f-45e4-b523-9fb0-c48201594101.dc1 area=wan -46:00.852 [INFO] TestAgent.server: Updating LAN server: server="Node-58dab99f-45e4-b523-9fb0-c48201594101 (Addr: tcp/127.0.0.1:11246) (DC: dc1)" -46:00.872 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.903 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.903 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.903 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.903 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.903 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.903 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:00.903 [INFO] TestAgent.server: member joined, marking health alive: member=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:00.906 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.916 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:01.134 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:01.134 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-58dab99f-45e4-b523-9fb0-c48201594101 -46:01.174 [INFO] TestAgent: Requesting shutdown -46:01.174 [INFO] TestAgent.server: shutting down server -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.174 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.174 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.174 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.178 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.184Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.184Z [INFO] agent.router.manager: shutting down -46:01.184 [INFO] TestAgent: consul server down -46:01.184 [INFO] TestAgent: shutdown complete -46:01.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11241 network=tcp -46:01.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11241 network=udp -46:01.184 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11242 network=tcp -46:01.684 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.684 [INFO] TestAgent: Endpoints down ---- PASS: TestPProfHandlers_ACLs (1.04s) - --- PASS: TestPProfHandlers_ACLs/case_0_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) (0.02s) - --- PASS: TestPProfHandlers_ACLs/case_1_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/heap",_nilResponse:true}) (0.00s) - --- PASS: TestPProfHandlers_ACLs/case_2_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/",_nilResponse:true}) (0.00s) - --- PASS: TestPProfHandlers_ACLs/case_3_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"",_endpoint:"/debug/pprof/",_nilResponse:true}) (0.00s) - --- PASS: TestPProfHandlers_ACLs/case_4_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) (0.02s) - --- PASS: TestPProfHandlers_ACLs/case_5_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"towel",_endpoint:"/debug/pprof/heap",_nilResponse:true}) (0.00s) -=== CONT TestOperator_RaftConfiguration ---- PASS: TestHTTP_wrap_obfuscateLog (0.70s) - --- PASS: TestHTTP_wrap_obfuscateLog//some/url?token=secret1&token=secret2 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/clone/secret1?token=secret2 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/destroy/secret1 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/destroy/secret1?token=secret2 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1 (0.00s) - --- PASS: TestHTTP_wrap_obfuscateLog//v1/acl/info/secret1?token=secret2 (0.00s) === CONT TestHealthConnectServiceNodes_PassingFilter -2022-11-07T22:46:00.723Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.738Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.807 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:830c428d-f635-9ed3-8cdd-5ec460aa6dd8 Address:127.0.0.1:11260}]" -46:00.807 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11260 [Follower]" leader= -46:00.808 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8.dc1 127.0.0.1 -46:00.808 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8 127.0.0.1 -2022-11-07T22:46:00.808Z [INFO] agent.router: Initializing LAN area manager -46:00.808 [INFO] TestAgent.server: Adding LAN server: server="Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8 (Addr: tcp/127.0.0.1:11260) (DC: dc1)" -46:00.808 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8.dc1 area=wan -46:00.809 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11255 network=udp -46:00.809 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11255 network=tcp -46:00.809 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11256 network=tcp -46:00.809 [INFO] TestAgent: started state syncer -46:00.809 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11261 network=tcp -46:00.863 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.863 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11260 [Candidate]" term=2 -46:00.880 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.880 [DEBUG] TestAgent.server.raft: vote granted: from=830c428d-f635-9ed3-8cdd-5ec460aa6dd8 term=2 tally=1 -46:00.880 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.880 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11260 [Leader]" -46:00.880 [INFO] TestAgent.server: cluster leadership acquired -46:00.880 [INFO] TestAgent.server: New leader elected: payload=Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8 -46:00.887 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11260 -46:00.906 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.946 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.946 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.946 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.946 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.946 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.946 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8 -46:00.946 [INFO] TestAgent.server: member joined, marking health alive: member=Node-830c428d-f635-9ed3-8cdd-5ec460aa6dd8 -46:00.957 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.178 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.182 [INFO] TestAgent: Synced node info -46:01.214 [INFO] TestAgent: Requesting shutdown -46:01.214 [INFO] TestAgent.server: shutting down server -46:01.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.214 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.214 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.214 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.214 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.214 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.214 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.214 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.214 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.217 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.223Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.223Z [INFO] agent.router.manager: shutting down -46:01.223 [INFO] TestAgent: consul server down -46:01.223 [INFO] TestAgent: shutdown complete -46:01.223 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11255 network=tcp -46:01.223 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11255 network=udp -46:01.223 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11256 network=tcp -46:01.723 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.723 [INFO] TestAgent: Endpoints down ---- PASS: TestACLResolution (1.01s) -=== CONT TestHealthServiceNodes_WanTranslation -2022-11-07T22:46:00.653Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.664Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.715 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3f34bc6f-6750-6f79-c9b2-28c377e3cf23 Address:127.0.0.1:11239}]" -46:00.715 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11239 [Follower]" leader= -46:00.715 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23.dc1 127.0.0.1 -46:00.716 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23 127.0.0.1 -2022-11-07T22:46:00.716Z [INFO] agent.router: Initializing LAN area manager -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11234 network=udp -46:00.716 [INFO] TestAgent.server: Adding LAN server: server="Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23 (Addr: tcp/127.0.0.1:11239) (DC: dc1)" -46:00.716 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23.dc1 area=wan -46:00.716 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11234 network=tcp -46:00.717 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:11236 network=tcp -46:00.717 [INFO] TestAgent: started state syncer -46:00.717 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11240 network=tcp -46:00.775 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.775 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11239 [Candidate]" term=2 -46:00.802 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.802 [DEBUG] TestAgent.server.raft: vote granted: from=3f34bc6f-6750-6f79-c9b2-28c377e3cf23 term=2 tally=1 -46:00.802 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.802 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11239 [Leader]" -46:00.802 [INFO] TestAgent.server: cluster leadership acquired -46:00.802 [INFO] TestAgent.server: New leader elected: payload=Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23 -46:00.813 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11239 -46:00.836 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.885 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.885 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.885 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.885 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.885 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.885 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23 -46:00.885 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3f34bc6f-6750-6f79-c9b2-28c377e3cf23 -46:00.893 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.984 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:00.987 [INFO] TestAgent: Synced node info -46:00.987 [DEBUG] TestAgent: Node info in sync -46:01.216 [INFO] TestAgent: Requesting shutdown -46:01.216 [INFO] TestAgent.server: shutting down server -46:01.216 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.216 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.216 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.216 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.216 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.216 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.216 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.216 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.216 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.216 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.220 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.225Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.226Z [INFO] agent.router.manager: shutting down -46:01.226 [INFO] TestAgent: consul server down -46:01.226 [INFO] TestAgent: shutdown complete -46:01.226 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11234 network=tcp -46:01.226 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11234 network=udp -46:01.226 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:11236 network=tcp -46:01.726 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.726 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPServer_HandshakeTimeout (1.08s) +2021-10-05T16:31:13.179Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.202Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.268 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f4f7b45e-43b1-c518-b6a9-c7a1054abf26 Address:127.0.0.1:29561}]" +31:13.269 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26.dc1 127.0.0.1 +31:13.270 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26 127.0.0.1 +2021-10-05T16:31:13.270Z [INFO] agent.router: Initializing LAN area manager +31:13.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29556 network=udp +31:13.270 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29561 [Follower]" leader= +31:13.270 [INFO] TestAgent.server: Adding LAN server: server="Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26 (Addr: tcp/127.0.0.1:29561) (DC: dc1)" +31:13.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26.dc1 area=wan +31:13.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29556 network=tcp +31:13.271 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29557 network=tcp +31:13.271 [INFO] TestAgent: started state syncer +31:13.271 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29562 network=tcp +31:13.310 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.310 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29561 [Candidate]" term=2 +31:13.328 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.328 [DEBUG] TestAgent.server.raft: vote granted: from=f4f7b45e-43b1-c518-b6a9-c7a1054abf26 term=2 tally=1 +31:13.328 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.328 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29561 [Leader]" +31:13.328 [INFO] TestAgent.server: cluster leadership acquired +31:13.329 [INFO] TestAgent.server: New leader elected: payload=Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26 +31:13.335 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29561 +31:13.348 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.373 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.373 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.373 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.373 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.373 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.373 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26 +31:13.373 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4f7b45e-43b1-c518-b6a9-c7a1054abf26 +31:13.396 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.484 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:13.494 [INFO] TestAgent: Synced node info +31:13.494 [DEBUG] TestAgent: Node info in sync +31:13.581 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:13.581 [INFO] TestAgent: Requesting shutdown +31:13.581 [INFO] TestAgent.server: shutting down server +31:13.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.581 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.581 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.581 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.581 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.581 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.581 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.587 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.595Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.595Z [INFO] agent.router.manager: shutting down +31:13.595 [INFO] TestAgent: consul server down +31:13.595 [INFO] TestAgent: shutdown complete +31:13.595 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29556 network=tcp +31:13.595 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29556 network=udp +31:13.595 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29557 network=tcp +31:14.096 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.096 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Explain (2.76s) + --- PASS: TestPreparedQuery_Explain/#00 (1.02s) + --- PASS: TestPreparedQuery_Explain/#01 (0.81s) + --- PASS: TestPreparedQuery_Explain/#02 (0.94s) === CONT TestHealthConnectServiceNodes_Filter -2022/11/07 22:46:01 http: TLS handshake error from 127.0.0.1:34914: read tcp 127.0.0.1:11334->127.0.0.1:34914: use of closed network connection -2022/11/07 22:46:01 http: TLS handshake error from 127.0.0.1:34916: EOF -2022/11/07 22:46:01 http: TLS handshake error from 127.0.0.1:34910: EOF -2022/11/07 22:46:01 http: TLS handshake error from 127.0.0.1:34912: EOF -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:13.491Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.507Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.584 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ff1a5866-8df1-c9a7-c473-c43ef2d54cc7 Address:127.0.0.1:29589}]" +31:13.584 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29589 [Follower]" leader= +31:13.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ff1a5866-8df1-c9a7-c473-c43ef2d54cc7.dc1 127.0.0.1 +31:13.586 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ff1a5866-8df1-c9a7-c473-c43ef2d54cc7 127.0.0.1 +2021-10-05T16:31:13.586Z [INFO] agent.router: Initializing LAN area manager +31:13.586 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29584 network=udp +31:13.586 [INFO] TestAgent.server: Adding LAN server: server="Node-ff1a5866-8df1-c9a7-c473-c43ef2d54cc7 (Addr: tcp/127.0.0.1:29589) (DC: dc1)" +31:13.586 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ff1a5866-8df1-c9a7-c473-c43ef2d54cc7.dc1 area=wan +31:13.586 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29584 network=tcp +31:13.587 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29585 network=tcp +31:13.587 [INFO] TestAgent: started state syncer +31:13.587 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29590 network=tcp +31:13.656 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.656 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29589 [Candidate]" term=2 +31:13.696 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.696 [DEBUG] TestAgent.server.raft: vote granted: from=ff1a5866-8df1-c9a7-c473-c43ef2d54cc7 term=2 tally=1 +31:13.696 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29589 [Leader]" +31:13.697 [INFO] TestAgent.server: cluster leadership acquired +31:13.697 [INFO] TestAgent.server: New leader elected: payload=Node-ff1a5866-8df1-c9a7-c473-c43ef2d54cc7 +31:13.740 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29589 +31:13.772 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=3.121µs +31:13.772 [INFO] TestAgent: Requesting shutdown +31:13.772 [INFO] TestAgent.server: shutting down server +31:13.772 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.772 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:13.784 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:13.790 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:31:13.796Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.796Z [INFO] agent.router.manager: shutting down +31:13.812 [INFO] TestAgent: consul server down +31:13.812 [INFO] TestAgent: shutdown complete +31:13.812 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29584 network=tcp +31:13.812 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29584 network=udp +31:13.812 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29585 network=tcp +31:13.812 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:14.316 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.316 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPIResponseHeaders (0.85s) +=== CONT TestConfig_Apply_IngressGateway +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestHealthConnectServiceNodes_PassingFilter/bc_no_query_value === RUN TestHealthConnectServiceNodes_PassingFilter/passing_true === RUN TestHealthConnectServiceNodes_PassingFilter/passing_false === RUN TestHealthConnectServiceNodes_PassingFilter/passing_bad -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:01.358Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.367Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.393 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5cb01637-5cfd-e39f-1457-dc91173f5aed Address:127.0.0.1:11309}]" -46:01.393 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11309 [Follower]" leader= -46:01.393 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5cb01637-5cfd-e39f-1457-dc91173f5aed.dc1 127.0.0.1 -46:01.394 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5cb01637-5cfd-e39f-1457-dc91173f5aed 127.0.0.1 -2022-11-07T22:46:01.394Z [INFO] agent.router: Initializing LAN area manager -46:01.394 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5cb01637-5cfd-e39f-1457-dc91173f5aed.dc1 area=wan -46:01.394 [INFO] TestAgent.server: Adding LAN server: server="Node-5cb01637-5cfd-e39f-1457-dc91173f5aed (Addr: tcp/127.0.0.1:11309) (DC: dc1)" -46:01.394 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11304 network=udp -46:01.394 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11304 network=tcp -46:01.395 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11305 network=tcp -46:01.395 [INFO] TestAgent: started state syncer -46:01.395 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11310 network=tcp -46:01.438 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.438 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11309 [Candidate]" term=2 -46:01.448 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.448 [DEBUG] TestAgent.server.raft: vote granted: from=5cb01637-5cfd-e39f-1457-dc91173f5aed term=2 tally=1 -46:01.448 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.448 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11309 [Leader]" -46:01.448 [INFO] TestAgent.server: cluster leadership acquired -46:01.449 [INFO] TestAgent.server: New leader elected: payload=Node-5cb01637-5cfd-e39f-1457-dc91173f5aed -46:01.460 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11309 -46:01.477 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.505 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.505 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.505 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.505 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.505 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.505 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5cb01637-5cfd-e39f-1457-dc91173f5aed -46:01.505 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5cb01637-5cfd-e39f-1457-dc91173f5aed -46:01.532 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.644 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key?pretty from= latency=18.317µs -46:01.644 [INFO] TestAgent: Requesting shutdown -46:01.644 [INFO] TestAgent.server: shutting down server -46:01.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.644 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.644 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.644 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.644 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.644 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.644 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.644 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.649 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.658Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.658Z [INFO] agent.router.manager: shutting down -46:01.658 [INFO] TestAgent: consul server down -46:01.658 [INFO] TestAgent: shutdown complete -46:01.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11304 network=tcp -46:01.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11304 network=udp -46:01.658 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11305 network=tcp -46:02.158 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.158 [INFO] TestAgent: Endpoints down ---- PASS: TestPrettyPrintBare (0.81s) -=== CONT TestHealthServiceNodes_Filter -2022-11-07T22:46:01.328Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.337Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.358 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba6d69b5-06e6-24bb-dc71-47e152f6c58f Address:127.0.0.1:11295}]" -46:01.358 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11295 [Follower]" leader= -46:01.358 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f.dc1 127.0.0.1 -46:01.359 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f 127.0.0.1 -2022-11-07T22:46:01.359Z [INFO] agent.router: Initializing LAN area manager -46:01.359 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f.dc1 area=wan -46:01.359 [INFO] TestAgent.server: Adding LAN server: server="Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f (Addr: tcp/127.0.0.1:11295) (DC: dc1)" -46:01.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11290 network=udp -46:01.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11290 network=tcp -46:01.360 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11291 network=tcp -46:01.360 [INFO] TestAgent: started state syncer -46:01.360 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11296 network=tcp -46:01.419 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.419 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11295 [Candidate]" term=2 -46:01.428 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.428 [DEBUG] TestAgent.server.raft: vote granted: from=ba6d69b5-06e6-24bb-dc71-47e152f6c58f term=2 tally=1 -46:01.428 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11295 [Leader]" -46:01.428 [INFO] TestAgent.server: cluster leadership acquired -46:01.428 [INFO] TestAgent.server: New leader elected: payload=Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f -46:01.435 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11295 -46:01.451 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.481 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.481 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.481 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.481 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.481 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.481 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f -46:01.481 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ba6d69b5-06e6-24bb-dc71-47e152f6c58f -46:01.488 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.703 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/key from= latency=20.821µs -46:01.703 [INFO] TestAgent: Requesting shutdown -46:01.703 [INFO] TestAgent.server: shutting down server -46:01.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.703 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.703 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.703 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.703 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.703 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.703 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.703 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.707 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.714Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.714Z [INFO] agent.router.manager: shutting down -46:01.714 [INFO] TestAgent: consul server down -46:01.714 [INFO] TestAgent: shutdown complete -46:01.714 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11290 network=tcp -46:01.715 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11290 network=udp -46:01.715 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11291 network=tcp -46:02.215 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.215 [INFO] TestAgent: Endpoints down ---- PASS: TestContentTypeIsJSON (0.90s) -=== CONT TestHealthServiceChecks_Filtering -2022-11-07T22:46:01.466Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.485Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.527 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55967ade-1b57-6d07-f070-f698714808a3 Address:127.0.0.1:11337}]" -46:01.527 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11337 [Follower]" leader= -46:01.528 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55967ade-1b57-6d07-f070-f698714808a3.dc1 127.0.0.1 -46:01.529 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55967ade-1b57-6d07-f070-f698714808a3 127.0.0.1 -2022-11-07T22:46:01.529Z [INFO] agent.router: Initializing LAN area manager -46:01.529 [INFO] TestAgent.server: Adding LAN server: server="Node-55967ade-1b57-6d07-f070-f698714808a3 (Addr: tcp/127.0.0.1:11337) (DC: dc1)" -46:01.529 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55967ade-1b57-6d07-f070-f698714808a3.dc1 area=wan -46:01.529 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11332 network=udp -46:01.529 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11332 network=tcp -46:01.529 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:11334 network=tcp -46:01.530 [INFO] TestAgent: started state syncer -46:01.530 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11338 network=tcp -46:01.571 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.571 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11337 [Candidate]" term=2 -46:01.584 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.584 [DEBUG] TestAgent.server.raft: vote granted: from=55967ade-1b57-6d07-f070-f698714808a3 term=2 tally=1 -46:01.584 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.584 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11337 [Leader]" -46:01.584 [INFO] TestAgent.server: cluster leadership acquired -46:01.584 [INFO] TestAgent.server: New leader elected: payload=Node-55967ade-1b57-6d07-f070-f698714808a3 -46:01.590 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11337 -46:01.601 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.629 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.629 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.629 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.629 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.629 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.629 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55967ade-1b57-6d07-f070-f698714808a3 -46:01.629 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55967ade-1b57-6d07-f070-f698714808a3 -46:01.649 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.739 [INFO] TestAgent: Requesting shutdown -46:01.739 [INFO] TestAgent.server: shutting down server -46:01.739 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.739 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.739 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.743 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.747Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.747Z [INFO] agent.router.manager: shutting down -46:01.748 [INFO] TestAgent: consul server down -46:01.748 [INFO] TestAgent: shutdown complete -46:01.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11332 network=tcp -46:01.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11332 network=udp -46:01.748 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:11334 network=tcp -46:02.248 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.248 [INFO] TestAgent: Endpoints down ---- PASS: TestRPC_HTTPSMaxConnsPerClient (1.62s) - --- PASS: TestRPC_HTTPSMaxConnsPerClient/HTTP (0.83s) - --- PASS: TestRPC_HTTPSMaxConnsPerClient/HTTPS (0.79s) -=== CONT TestHealthServiceNodes_CheckType -2022-11-07T22:46:01.591Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.600Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.633 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18e2a67e-60ec-45a1-bcc0-056ad45d3eff Address:127.0.0.1:11365}]" -46:01.633 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11365 [Follower]" leader= -46:01.633 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff.dc1 127.0.0.1 -46:01.634 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff 127.0.0.1 -2022-11-07T22:46:01.634Z [INFO] agent.router: Initializing LAN area manager -46:01.634 [INFO] TestAgent.server: Adding LAN server: server="Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff (Addr: tcp/127.0.0.1:11365) (DC: dc1)" -46:01.634 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11360 network=tcp -46:01.634 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff.dc1 area=wan -46:01.634 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11360 network=udp -46:01.635 [INFO] TestAgent: Started HTTP server: address=/tmp/consul-test/TestHTTPServer_UnixSocket-consul109569648/test.sock network=unix -46:01.635 [INFO] TestAgent: started state syncer -46:01.635 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11366 network=tcp -46:01.685 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.685 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11365 [Candidate]" term=2 -46:01.693 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.693 [DEBUG] TestAgent.server.raft: vote granted: from=18e2a67e-60ec-45a1-bcc0-056ad45d3eff term=2 tally=1 -46:01.693 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.693 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11365 [Leader]" -46:01.693 [INFO] TestAgent.server: cluster leadership acquired -46:01.693 [INFO] TestAgent.server: New leader elected: payload=Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff -46:01.700 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11365 -46:01.712 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.739 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.739 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.739 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.739 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.739 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.739 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff -46:01.739 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18e2a67e-60ec-45a1-bcc0-056ad45d3eff -46:01.742 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=@ latency=1.413459ms -46:01.742 [INFO] TestAgent: Requesting shutdown -46:01.742 [INFO] TestAgent.server: shutting down server -46:01.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.742 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.742 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.742 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.742 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.742 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.742 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.742 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.747 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.756Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.756Z [INFO] agent.router.manager: shutting down -46:01.756 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.756 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.757 [INFO] TestAgent: consul server down -46:01.757 [INFO] TestAgent: shutdown complete -46:01.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11360 network=tcp -46:01.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11360 network=udp -46:01.757 [INFO] TestAgent: Stopping server: protocol=HTTP address=/tmp/consul-test/TestHTTPServer_UnixSocket-consul109569648/test.sock network=unix -46:02.257 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.257 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPServer_UnixSocket (0.68s) -=== CONT TestHealthServiceNodes_NodeMetaFilter -2022-11-07T22:46:01.390Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.399Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.419 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b77eadbf-aa8c-0c77-c622-05994e639041 Address:127.0.0.1:11316}]" -46:01.419 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11316 [Follower]" leader= -46:01.420 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b77eadbf-aa8c-0c77-c622-05994e639041.dc1 127.0.0.1 -46:01.420 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b77eadbf-aa8c-0c77-c622-05994e639041 127.0.0.1 -2022-11-07T22:46:01.420Z [INFO] agent.router: Initializing LAN area manager -46:01.420 [INFO] TestAgent.server: Adding LAN server: server="Node-b77eadbf-aa8c-0c77-c622-05994e639041 (Addr: tcp/127.0.0.1:11316) (DC: dc1)" -46:01.421 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b77eadbf-aa8c-0c77-c622-05994e639041.dc1 area=wan -46:01.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11311 network=udp -46:01.421 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11311 network=tcp -46:01.421 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11312 network=tcp -46:01.421 [INFO] TestAgent: started state syncer -46:01.421 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11317 network=tcp -46:01.488 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.488 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11316 [Candidate]" term=2 -46:01.500 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.500 [DEBUG] TestAgent.server.raft: vote granted: from=b77eadbf-aa8c-0c77-c622-05994e639041 term=2 tally=1 -46:01.500 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.500 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11316 [Leader]" -46:01.500 [INFO] TestAgent.server: cluster leadership acquired -46:01.500 [INFO] TestAgent.server: New leader elected: payload=Node-b77eadbf-aa8c-0c77-c622-05994e639041 -46:01.511 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11316 -46:01.547 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.547 [INFO] TestAgent: Synced node info -46:01.565 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.565 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.565 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.565 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.565 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.565 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b77eadbf-aa8c-0c77-c622-05994e639041 -46:01.565 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b77eadbf-aa8c-0c77-c622-05994e639041 -46:01.581 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.759 [DEBUG] TestAgent.http: Request finished: method=GET url=/ui from= latency=1.552µs -46:01.759 [INFO] TestAgent: Requesting shutdown -46:01.759 [INFO] TestAgent.server: shutting down server -46:01.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.759 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.759 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.759 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.759 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.759 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.759 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.759 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.759 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.763 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.770Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.770Z [INFO] agent.router.manager: shutting down -46:01.770 [INFO] TestAgent: consul server down -46:01.770 [INFO] TestAgent: shutdown complete -46:01.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11311 network=tcp -46:01.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11311 network=udp -46:01.770 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11312 network=tcp -46:02.271 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.271 [INFO] TestAgent: Endpoints down ---- PASS: TestUIResponseHeaders (0.89s) +2021-10-05T16:31:13.440Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.455Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.519 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:728237e0-790c-b61d-7335-7b338ce484a2 Address:127.0.0.1:29575}]" +31:13.520 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-728237e0-790c-b61d-7335-7b338ce484a2.dc1 127.0.0.1 +31:13.520 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29575 [Follower]" leader= +31:13.527 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-728237e0-790c-b61d-7335-7b338ce484a2 127.0.0.1 +2021-10-05T16:31:13.527Z [INFO] agent.router: Initializing LAN area manager +31:13.527 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29570 network=udp +31:13.528 [INFO] TestAgent.server: Adding LAN server: server="Node-728237e0-790c-b61d-7335-7b338ce484a2 (Addr: tcp/127.0.0.1:29575) (DC: dc1)" +31:13.528 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-728237e0-790c-b61d-7335-7b338ce484a2.dc1 area=wan +31:13.528 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29570 network=tcp +31:13.529 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29571 network=tcp +31:13.529 [INFO] TestAgent: started state syncer +31:13.529 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29576 network=tcp +31:13.572 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.572 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29575 [Candidate]" term=2 +31:13.603 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.603 [DEBUG] TestAgent.server.raft: vote granted: from=728237e0-790c-b61d-7335-7b338ce484a2 term=2 tally=1 +31:13.603 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.603 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29575 [Leader]" +31:13.603 [INFO] TestAgent.server: cluster leadership acquired +31:13.603 [INFO] TestAgent.server: New leader elected: payload=Node-728237e0-790c-b61d-7335-7b338ce484a2 +31:13.645 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29575 +31:13.694 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.792 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.792 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.792 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.792 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.792 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.792 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-728237e0-790c-b61d-7335-7b338ce484a2 +31:13.792 [INFO] TestAgent.server: member joined, marking health alive: member=Node-728237e0-790c-b61d-7335-7b338ce484a2 +31:13.823 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.859 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:13.859 [INFO] TestAgent: Requesting shutdown +31:13.860 [INFO] TestAgent.server: shutting down server +31:13.860 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.860 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.860 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.860 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.860 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.860 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.860 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:13.860 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.860 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.860 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.860 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.876 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.883Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.883Z [INFO] agent.router.manager: shutting down +31:13.883 [INFO] TestAgent: consul server down +31:13.883 [INFO] TestAgent: shutdown complete +31:13.883 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29570 network=tcp +31:13.883 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29570 network=udp +31:13.883 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29571 network=tcp +31:14.384 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.384 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#03 +2021-10-05T16:31:13.466Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.484Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.532 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb0299a2-67cf-9372-114a-b16ed03f0e82 Address:127.0.0.1:29582}]" +31:13.533 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cb0299a2-67cf-9372-114a-b16ed03f0e82.dc1 127.0.0.1 +31:13.533 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cb0299a2-67cf-9372-114a-b16ed03f0e82 127.0.0.1 +2021-10-05T16:31:13.533Z [INFO] agent.router: Initializing LAN area manager +31:13.533 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29577 network=udp +31:13.533 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29582 [Follower]" leader= +31:13.534 [INFO] TestAgent.server: Adding LAN server: server="Node-cb0299a2-67cf-9372-114a-b16ed03f0e82 (Addr: tcp/127.0.0.1:29582) (DC: dc1)" +31:13.534 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cb0299a2-67cf-9372-114a-b16ed03f0e82.dc1 area=wan +31:13.534 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29577 network=tcp +31:13.534 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29578 network=tcp +31:13.534 [INFO] TestAgent: started state syncer +31:13.534 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29583 network=tcp +31:13.588 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.588 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29582 [Candidate]" term=2 +31:13.632 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.632 [DEBUG] TestAgent.server.raft: vote granted: from=cb0299a2-67cf-9372-114a-b16ed03f0e82 term=2 tally=1 +31:13.632 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.632 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29582 [Leader]" +31:13.632 [INFO] TestAgent.server: cluster leadership acquired +31:13.632 [INFO] TestAgent.server: New leader elected: payload=Node-cb0299a2-67cf-9372-114a-b16ed03f0e82 +31:13.664 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29582 +31:13.687 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.747 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.748 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.748 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.748 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.748 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.748 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cb0299a2-67cf-9372-114a-b16ed03f0e82 +31:13.748 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cb0299a2-67cf-9372-114a-b16ed03f0e82 +31:13.782 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.879 [DEBUG] TestAgent.http: Request finished: method=GET url=/ui from= latency=2.059µs +31:13.879 [INFO] TestAgent: Requesting shutdown +31:13.879 [INFO] TestAgent.server: shutting down server +31:13.879 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.879 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.879 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.879 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.879 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.879 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.879 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:13.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.879 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.879 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.888 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.905Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.905Z [INFO] agent.router.manager: shutting down +31:13.905 [INFO] TestAgent: consul server down +31:13.905 [INFO] TestAgent: shutdown complete +31:13.905 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29577 network=tcp +31:13.905 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29577 network=udp +31:13.905 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29578 network=tcp +31:14.405 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.405 [INFO] TestAgent: Endpoints down +--- PASS: TestUIResponseHeaders (0.96s) === CONT TestHealthIngressServiceNodes -2022-11-07T22:46:01.358Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.373Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.401 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae46fc27-8815-c670-6424-e52e9c79004d Address:127.0.0.1:11302}]" -46:01.401 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11302 [Follower]" leader= -46:01.402 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ae46fc27-8815-c670-6424-e52e9c79004d.dc1 127.0.0.1 -46:01.403 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ae46fc27-8815-c670-6424-e52e9c79004d 127.0.0.1 -2022-11-07T22:46:01.403Z [INFO] agent.router: Initializing LAN area manager -46:01.403 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ae46fc27-8815-c670-6424-e52e9c79004d.dc1 area=wan -46:01.403 [INFO] TestAgent.server: Adding LAN server: server="Node-ae46fc27-8815-c670-6424-e52e9c79004d (Addr: tcp/127.0.0.1:11302) (DC: dc1)" -46:01.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11297 network=udp -46:01.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11297 network=tcp -46:01.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11298 network=tcp -46:01.403 [INFO] TestAgent: started state syncer -46:01.403 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11303 network=tcp -46:01.447 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.447 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11302 [Candidate]" term=2 -46:01.463 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.463 [DEBUG] TestAgent.server.raft: vote granted: from=ae46fc27-8815-c670-6424-e52e9c79004d term=2 tally=1 -46:01.463 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.463 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11302 [Leader]" -46:01.463 [INFO] TestAgent.server: cluster leadership acquired -46:01.463 [INFO] TestAgent.server: New leader elected: payload=Node-ae46fc27-8815-c670-6424-e52e9c79004d -46:01.472 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11302 -46:01.486 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.516 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.516 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.516 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.516 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.516 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.516 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae46fc27-8815-c670-6424-e52e9c79004d -46:01.516 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ae46fc27-8815-c670-6424-e52e9c79004d -46:01.538 [INFO] TestAgent: Synced node info -46:01.542 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.784 [INFO] TestAgent: Requesting shutdown -46:01.784 [INFO] TestAgent.server: shutting down server -46:01.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.784 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.784 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.784 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.784 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.784 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.784 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.789 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.797Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.797Z [INFO] agent.router.manager: shutting down -46:01.797 [INFO] TestAgent: consul server down -46:01.797 [INFO] TestAgent: shutdown complete -46:01.797 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11297 network=tcp -46:01.797 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11297 network=udp -46:01.797 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11298 network=tcp -46:02.298 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.298 [INFO] TestAgent: Endpoints down ---- PASS: TestParseSource (0.95s) -=== CONT TestHealthServiceNodes_PassingFilter -2022-11-07T22:46:01.544Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.554Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.582 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3a27475-ec3b-d8c8-b735-a43cc0549651 Address:127.0.0.1:11358}]" -46:01.582 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11358 [Follower]" leader= -46:01.583 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3a27475-ec3b-d8c8-b735-a43cc0549651.dc1 127.0.0.1 -46:01.583 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3a27475-ec3b-d8c8-b735-a43cc0549651 127.0.0.1 -2022-11-07T22:46:01.583Z [INFO] agent.router: Initializing LAN area manager -46:01.584 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11353 network=udp -46:01.584 [INFO] TestAgent.server: Adding LAN server: server="Node-e3a27475-ec3b-d8c8-b735-a43cc0549651 (Addr: tcp/127.0.0.1:11358) (DC: dc1)" -46:01.584 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11353 network=tcp -46:01.584 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3a27475-ec3b-d8c8-b735-a43cc0549651.dc1 area=wan -46:01.584 [WARN] TestAgent: Replacing socket: path=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul254053390/test.sock -46:01.584 [INFO] TestAgent: Started HTTP server: address=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul254053390/test.sock network=unix -46:01.584 [INFO] TestAgent: started state syncer -46:01.584 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11359 network=tcp -46:01.620 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.620 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11358 [Candidate]" term=2 -46:01.634 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.634 [DEBUG] TestAgent.server.raft: vote granted: from=e3a27475-ec3b-d8c8-b735-a43cc0549651 term=2 tally=1 -46:01.634 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.634 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11358 [Leader]" -46:01.634 [INFO] TestAgent.server: cluster leadership acquired -46:01.634 [INFO] TestAgent.server: New leader elected: payload=Node-e3a27475-ec3b-d8c8-b735-a43cc0549651 -46:01.643 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11358 -46:01.659 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.690 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.690 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.690 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.690 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.690 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.690 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3a27475-ec3b-d8c8-b735-a43cc0549651 -46:01.690 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3a27475-ec3b-d8c8-b735-a43cc0549651 -46:01.704 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.789 [INFO] TestAgent: Requesting shutdown -46:01.789 [INFO] TestAgent.server: shutting down server -46:01.789 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.789 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.789 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.789 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.789 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.789 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.789 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.789 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.789 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.789 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.789 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.797 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.802Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.802Z [INFO] agent.router.manager: shutting down -46:01.802 [INFO] TestAgent: consul server down -46:01.802 [INFO] TestAgent: shutdown complete -46:01.802 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11353 network=tcp -46:01.802 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11353 network=udp -46:01.802 [INFO] TestAgent: Stopping server: protocol=HTTP address=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul254053390/test.sock network=unix -46:02.302 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.302 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPServer_UnixSocket_FileExists (0.77s) +2021-10-05T16:31:12.472Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:12.522Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:12.604 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a8d866f0-4982-3824-a284-11cc01bc48fa Address:127.0.0.1:29519}]" +31:12.604 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29519 [Follower]" leader= +31:12.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a8d866f0-4982-3824-a284-11cc01bc48fa.dc1 127.0.0.1 +31:12.605 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a8d866f0-4982-3824-a284-11cc01bc48fa 127.0.0.1 +2021-10-05T16:31:12.605Z [INFO] agent.router: Initializing LAN area manager +31:12.605 [INFO] TestAgent.server: Adding LAN server: server="Node-a8d866f0-4982-3824-a284-11cc01bc48fa (Addr: tcp/127.0.0.1:29519) (DC: dc1)" +31:12.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29507 network=udp +31:12.606 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a8d866f0-4982-3824-a284-11cc01bc48fa.dc1 area=wan +31:12.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29507 network=tcp +31:12.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29508 network=tcp +31:12.606 [INFO] TestAgent: started state syncer +31:12.606 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29520 network=tcp +31:12.660 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:12.660 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29519 [Candidate]" term=2 +31:12.695 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:12.695 [DEBUG] TestAgent.server.raft: vote granted: from=a8d866f0-4982-3824-a284-11cc01bc48fa term=2 tally=1 +31:12.695 [INFO] TestAgent.server.raft: election won: tally=1 +31:12.695 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29519 [Leader]" +31:12.695 [INFO] TestAgent.server: cluster leadership acquired +31:12.696 [INFO] TestAgent.server: New leader elected: payload=Node-a8d866f0-4982-3824-a284-11cc01bc48fa +31:12.720 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29519 +31:12.755 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:12.816 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:12.816 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:12.816 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:12.816 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:12.816 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:12.816 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a8d866f0-4982-3824-a284-11cc01bc48fa +31:12.816 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a8d866f0-4982-3824-a284-11cc01bc48fa +31:12.890 [INFO] TestAgent.server: federation state anti-entropy synced +31:13.020 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:13.032 [INFO] TestAgent: Synced node info +31:13.032 [DEBUG] TestAgent: Node info in sync +31:13.885 [INFO] TestAgent: Requesting shutdown +31:13.885 [INFO] TestAgent.server: shutting down server +31:13.885 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:13.885 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:13.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:13.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:13.885 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:13.885 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:13.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:13.885 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:13.885 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:13.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:13.904 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:13.913Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:13.913Z [INFO] agent.router.manager: shutting down +31:13.913 [INFO] TestAgent: consul server down +31:13.913 [INFO] TestAgent: shutdown complete +31:13.913 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29507 network=tcp +31:13.913 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29507 network=udp +31:13.913 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29508 network=tcp +31:14.420 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.420 [INFO] TestAgent: Endpoints down +--- PASS: TestPProfHandlers_EnableDebug (2.01s) === CONT TestHealthConnectServiceNodes -2022-11-07T22:46:01.482Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.497Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.541 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:da781942-f168-578b-317e-420ef42ccae6 Address:127.0.0.1:11344}]" -46:01.541 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11344 [Follower]" leader= -46:01.541 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-da781942-f168-578b-317e-420ef42ccae6.dc1 127.0.0.1 -46:01.542 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-da781942-f168-578b-317e-420ef42ccae6 127.0.0.1 -2022-11-07T22:46:01.542Z [INFO] agent.router: Initializing LAN area manager -46:01.542 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-da781942-f168-578b-317e-420ef42ccae6.dc1 area=wan -46:01.542 [INFO] TestAgent.server: Adding LAN server: server="Node-da781942-f168-578b-317e-420ef42ccae6 (Addr: tcp/127.0.0.1:11344) (DC: dc1)" -46:01.542 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11339 network=udp -46:01.542 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11339 network=tcp -46:01.543 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11340 network=tcp -46:01.543 [INFO] TestAgent: started state syncer -46:01.543 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11345 network=tcp -46:01.606 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.606 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11344 [Candidate]" term=2 -46:01.612 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.612 [DEBUG] TestAgent.server.raft: vote granted: from=da781942-f168-578b-317e-420ef42ccae6 term=2 tally=1 -46:01.612 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.612 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11344 [Leader]" -46:01.612 [INFO] TestAgent.server: cluster leadership acquired -46:01.612 [INFO] TestAgent.server: New leader elected: payload=Node-da781942-f168-578b-317e-420ef42ccae6 -46:01.620 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11344 -46:01.642 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.669 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.669 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.669 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.669 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.669 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.669 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-da781942-f168-578b-317e-420ef42ccae6 -46:01.669 [INFO] TestAgent.server: member joined, marking health alive: member=Node-da781942-f168-578b-317e-420ef42ccae6 -46:01.688 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.810 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/self from= error="Endpoint is blocked by agent configuration" -46:01.810 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from= latency=1.458µs -46:01.810 [INFO] TestAgent: Requesting shutdown -46:01.810 [INFO] TestAgent.server: shutting down server -46:01.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.810 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.810 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.810 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.810 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.810 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.810 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.810 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.817 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.830Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.830Z [INFO] agent.router.manager: shutting down -46:01.830 [INFO] TestAgent: consul server down -46:01.830 [INFO] TestAgent: shutdown complete -46:01.830 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11339 network=tcp -46:01.830 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11339 network=udp -46:01.830 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11340 network=tcp -46:02.330 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.330 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_BlockEndpoints (0.87s) -=== CONT TestHealthServiceNodes -[WARN] freeport: 10 out of 66 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:01.412Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.423Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.450 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f8af60d-a428-d912-43fe-c8e7aad6aa71 Address:127.0.0.1:11323}]" -46:01.450 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11323 [Follower]" leader= -46:01.450 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71.dc1 127.0.0.1 -46:01.451 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71 127.0.0.1 -2022-11-07T22:46:01.451Z [INFO] agent.router: Initializing LAN area manager -46:01.451 [INFO] TestAgent.server: Adding LAN server: server="Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71 (Addr: tcp/127.0.0.1:11323) (DC: dc1)" -46:01.451 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71.dc1 area=wan -46:01.451 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11318 network=udp -46:01.451 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11318 network=tcp -46:01.451 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11319 network=tcp -46:01.452 [INFO] TestAgent: started state syncer -46:01.452 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11324 network=tcp -46:01.495 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.495 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11323 [Candidate]" term=2 -46:01.511 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.511 [DEBUG] TestAgent.server.raft: vote granted: from=0f8af60d-a428-d912-43fe-c8e7aad6aa71 term=2 tally=1 -46:01.511 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.511 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11323 [Leader]" -46:01.511 [INFO] TestAgent.server: cluster leadership acquired -46:01.511 [INFO] TestAgent.server: New leader elected: payload=Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71 -46:01.520 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11323 -46:01.546 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.567 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.568 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.568 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.568 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.568 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.568 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71 -46:01.568 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0f8af60d-a428-d912-43fe-c8e7aad6aa71 -46:01.576 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.844 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=1.474µs -46:01.844 [INFO] TestAgent: Requesting shutdown -46:01.844 [INFO] TestAgent.server: shutting down server -46:01.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.844 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.844 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.844 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.844 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.844 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.844 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.844 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.850 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.859Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.859Z [INFO] agent.router.manager: shutting down -46:01.859 [INFO] TestAgent: consul server down -46:01.859 [INFO] TestAgent: shutdown complete -46:01.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11318 network=tcp -46:01.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11318 network=udp -46:01.859 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11319 network=tcp -46:02.359 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.359 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPIResponseHeaders (0.96s) -=== CONT TestHealthServiceNodes_DistanceSort -2022-11-07T22:46:01.485Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.499Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.545 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c598f60f-4cec-bd3e-0b2d-230741e45ab9 Address:127.0.0.1:11351}]" -46:01.545 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11351 [Follower]" leader= -46:01.545 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9.dc1 127.0.0.1 -46:01.546 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 127.0.0.1 -2022-11-07T22:46:01.546Z [INFO] agent.router: Initializing LAN area manager -46:01.546 [INFO] TestAgent.server: Adding LAN server: server="Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 (Addr: tcp/127.0.0.1:11351) (DC: dc1)" -46:01.546 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9.dc1 area=wan -46:01.546 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11346 network=tcp -46:01.546 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11346 network=udp -46:01.547 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11347 network=tcp -46:01.547 [INFO] TestAgent: started state syncer -46:01.547 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11352 network=tcp -46:01.605 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.605 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11351 [Candidate]" term=2 -46:01.611 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.611 [DEBUG] TestAgent.server.raft: vote granted: from=c598f60f-4cec-bd3e-0b2d-230741e45ab9 term=2 tally=1 -46:01.611 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.611 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11351 [Leader]" -46:01.611 [INFO] TestAgent.server: cluster leadership acquired -46:01.611 [INFO] TestAgent.server: New leader elected: payload=Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 -46:01.618 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11351 -46:01.637 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.667 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.667 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.667 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.667 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.667 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.667 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 -46:01.667 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 -46:01.676 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.858 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:01.858 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9.dc1 -46:01.858 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:01.859 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-c598f60f-4cec-bd3e-0b2d-230741e45ab9 -46:01.859 [INFO] TestAgent: Requesting shutdown -46:01.859 [INFO] TestAgent.server: shutting down server -46:01.859 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.859 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.859 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.859 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.859 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.859 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.859 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.859 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.859 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.864 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.873Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.873Z [INFO] agent.router.manager: shutting down -46:01.873 [INFO] TestAgent: consul server down -46:01.873 [INFO] TestAgent: shutdown complete -46:01.873 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11346 network=tcp -46:01.873 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11346 network=udp -46:01.873 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11347 network=tcp -46:02.374 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.374 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_KeyringList (0.91s) -=== CONT TestHealthServiceChecks_NodeMetaFilter -2022-11-07T22:46:01.676Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.692Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.722 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 Address:127.0.0.1:11386}]" -46:01.722 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11386 [Follower]" leader= -46:01.723 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4.dc1 127.0.0.1 -46:01.724 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 127.0.0.1 -2022-11-07T22:46:01.724Z [INFO] agent.router: Initializing LAN area manager -46:01.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11381 network=udp -46:01.724 [INFO] TestAgent.server: Adding LAN server: server="Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 (Addr: tcp/127.0.0.1:11386) (DC: dc1)" -46:01.724 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4.dc1 area=wan -46:01.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11381 network=tcp -46:01.725 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11382 network=tcp -46:01.725 [INFO] TestAgent: started state syncer -46:01.725 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11387 network=tcp -46:01.765 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.765 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11386 [Candidate]" term=2 -46:01.783 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.783 [DEBUG] TestAgent.server.raft: vote granted: from=80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 term=2 tally=1 -46:01.783 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.783 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11386 [Leader]" -46:01.783 [INFO] TestAgent.server: cluster leadership acquired -46:01.783 [INFO] TestAgent.server: New leader elected: payload=Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 -46:01.790 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11386 -46:01.803 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.839 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.839 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.839 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.839 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.839 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.840 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 -46:01.840 [INFO] TestAgent.server: member joined, marking health alive: member=Node-80bfcbdc-2dc5-072f-ec6b-8664b89efeb4 -46:01.858 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.904 [INFO] TestAgent: Requesting shutdown -46:01.904 [INFO] TestAgent.server: shutting down server -46:01.904 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.904 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.904 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.904 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.904 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.904 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.904 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.904 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.904 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.904 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.904 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.909 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.912Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.912Z [INFO] agent.router.manager: shutting down -46:01.912 [INFO] TestAgent: consul server down -46:01.912 [INFO] TestAgent: shutdown complete -46:01.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11381 network=tcp -46:01.912 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11381 network=udp -46:01.912 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11382 network=tcp -46:02.413 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.413 [INFO] TestAgent: Endpoints down -=== RUN TestOperator_RaftPeer/#01 -2022-11-07T22:46:01.697Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.706Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.738 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:598b7b6f-c895-7f80-49a5-f6cd7cb8a116 Address:127.0.0.1:11407}]" -46:01.738 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11407 [Follower]" leader= -46:01.739 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116.dc1 127.0.0.1 -46:01.739 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116 127.0.0.1 -2022-11-07T22:46:01.739Z [INFO] agent.router: Initializing LAN area manager -46:01.739 [INFO] TestAgent.server: Adding LAN server: server="Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116 (Addr: tcp/127.0.0.1:11407) (DC: dc1)" -46:01.739 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116.dc1 area=wan -46:01.739 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11388 network=tcp -46:01.739 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11388 network=udp -46:01.740 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11389 network=tcp -46:01.740 [INFO] TestAgent: started state syncer -46:01.740 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11408 network=tcp -46:01.802 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.802 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11407 [Candidate]" term=2 -46:01.813 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.813 [DEBUG] TestAgent.server.raft: vote granted: from=598b7b6f-c895-7f80-49a5-f6cd7cb8a116 term=2 tally=1 -46:01.813 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.813 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11407 [Leader]" -46:01.814 [INFO] TestAgent.server: cluster leadership acquired -46:01.814 [INFO] TestAgent.server: New leader elected: payload=Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116 -46:01.823 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11407 -46:01.840 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.876 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.876 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.876 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.876 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.876 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.876 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116 -46:01.876 [INFO] TestAgent.server: member joined, marking health alive: member=Node-598b7b6f-c895-7f80-49a5-f6cd7cb8a116 -46:01.890 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.934 [INFO] TestAgent: Requesting shutdown -46:01.934 [INFO] TestAgent.server: shutting down server -46:01.934 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.934 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.934 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.934 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.934 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.934 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.934 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.934 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.934 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.934 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.934 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.939 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.944Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.944Z [INFO] agent.router.manager: shutting down -46:01.944 [INFO] TestAgent: consul server down -46:01.944 [INFO] TestAgent: shutdown complete -46:01.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11388 network=tcp -46:01.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11388 network=udp -46:01.944 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11389 network=tcp -46:02.445 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.445 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_RaftConfiguration (0.76s) -=== CONT TestHealthServiceChecks_DistanceSort -2022-11-07T22:46:01.742Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.756Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.793 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1fafcb12-52f8-e697-46f4-8473f4eea7f2 Address:127.0.0.1:11421}]" -46:01.793 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11421 [Follower]" leader= -46:01.793 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2.dc1 127.0.0.1 -46:01.794 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2 127.0.0.1 -2022-11-07T22:46:01.794Z [INFO] agent.router: Initializing LAN area manager -46:01.794 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2.dc1 area=wan -46:01.794 [INFO] TestAgent.server: Adding LAN server: server="Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2 (Addr: tcp/127.0.0.1:11421) (DC: dc1)" -46:01.794 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11402 network=udp -46:01.794 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11402 network=tcp -46:01.794 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11403 network=tcp -46:01.794 [INFO] TestAgent: started state syncer -46:01.794 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11422 network=tcp -46:01.843 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.843 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11421 [Candidate]" term=2 -46:01.853 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.853 [DEBUG] TestAgent.server.raft: vote granted: from=1fafcb12-52f8-e697-46f4-8473f4eea7f2 term=2 tally=1 -46:01.853 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.853 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11421 [Leader]" -46:01.853 [INFO] TestAgent.server: cluster leadership acquired -46:01.853 [INFO] TestAgent.server: New leader elected: payload=Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2 -46:01.863 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11421 -46:01.879 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.903 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.903 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.903 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.903 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.903 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.903 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2 -46:01.903 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1fafcb12-52f8-e697-46f4-8473f4eea7f2 -46:01.913 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.938 [INFO] TestAgent: Requesting shutdown -46:01.938 [INFO] TestAgent.server: shutting down server -46:01.938 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:01.938 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:01.938 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:01.938 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:01.938 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.938 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.938 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:01.938 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.938 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:01.938 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:01.938 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.942 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.947Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.947Z [INFO] agent.router.manager: shutting down -46:01.947 [INFO] TestAgent: consul server down -46:01.947 [INFO] TestAgent: shutdown complete -46:01.947 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11402 network=tcp -46:01.947 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11402 network=udp -46:01.947 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11403 network=tcp -46:02.447 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.447 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthConnectServiceNodes_Filter (0.72s) -=== CONT TestHealthNodeChecks -2022-11-07T22:46:01.640Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.649Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.685 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c909f68e-cf73-cda3-fd14-bd214f19e0b3 Address:127.0.0.1:11372}]" -46:01.685 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11372 [Follower]" leader= -46:01.686 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3.dc1 127.0.0.1 -46:01.687 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 127.0.0.1 -2022-11-07T22:46:01.687Z [INFO] agent.router: Initializing LAN area manager -46:01.687 [INFO] TestAgent.server: Adding LAN server: server="Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 (Addr: tcp/127.0.0.1:11372) (DC: dc1)" -46:01.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11367 network=udp -46:01.687 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3.dc1 area=wan -46:01.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11367 network=tcp -46:01.687 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11368 network=tcp -46:01.687 [INFO] TestAgent: started state syncer -46:01.687 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11373 network=tcp -46:01.748 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.748 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11372 [Candidate]" term=2 -46:01.762 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.762 [DEBUG] TestAgent.server.raft: vote granted: from=c909f68e-cf73-cda3-fd14-bd214f19e0b3 term=2 tally=1 -46:01.762 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.762 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11372 [Leader]" -46:01.762 [INFO] TestAgent.server: cluster leadership acquired -46:01.762 [INFO] TestAgent.server: New leader elected: payload=Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 -46:01.769 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11372 -46:01.787 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.817 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.817 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.817 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.817 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.817 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.817 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 -46:01.817 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 -46:01.836 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.886 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.890 [INFO] TestAgent: Synced node info -46:01.890 [DEBUG] TestAgent: Node info in sync -46:02.018 [INFO] TestAgent.server.serf.wan: serf: Received install-key query -46:02.019 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3.dc1 -46:02.019 [INFO] TestAgent.server.serf.lan: serf: Received install-key query -46:02.019 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 -46:02.019 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:02.019 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3.dc1 -46:02.019 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:02.020 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-c909f68e-cf73-cda3-fd14-bd214f19e0b3 -46:02.020 [INFO] TestAgent: Requesting shutdown -46:02.020 [INFO] TestAgent.server: shutting down server -46:02.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.020 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.020 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.020 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.020 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.024 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.029Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.029Z [INFO] agent.router.manager: shutting down -46:02.030 [INFO] TestAgent: consul server down -46:02.030 [INFO] TestAgent: shutdown complete -46:02.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11367 network=tcp -46:02.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11367 network=udp -46:02.030 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11368 network=tcp -46:02.530 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.530 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_KeyringInstall (0.90s) -=== CONT TestHealthServiceChecks -2022-11-07T22:46:01.707Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.722Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.753 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9430b371-9b01-594c-50cf-e8d860993ba5 Address:127.0.0.1:11393}]" -46:01.753 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11393 [Follower]" leader= -46:01.754 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9430b371-9b01-594c-50cf-e8d860993ba5.dc1 127.0.0.1 -46:01.754 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9430b371-9b01-594c-50cf-e8d860993ba5 127.0.0.1 -2022-11-07T22:46:01.754Z [INFO] agent.router: Initializing LAN area manager -46:01.755 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9430b371-9b01-594c-50cf-e8d860993ba5.dc1 area=wan -46:01.755 [INFO] TestAgent.server: Adding LAN server: server="Node-9430b371-9b01-594c-50cf-e8d860993ba5 (Addr: tcp/127.0.0.1:11393) (DC: dc1)" -46:01.755 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11409 network=tcp -46:01.755 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11409 network=udp -46:01.755 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11410 network=tcp -46:01.755 [INFO] TestAgent: started state syncer -46:01.755 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11394 network=tcp -46:01.806 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.806 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11393 [Candidate]" term=2 -46:01.817 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.817 [DEBUG] TestAgent.server.raft: vote granted: from=9430b371-9b01-594c-50cf-e8d860993ba5 term=2 tally=1 -46:01.817 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.817 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11393 [Leader]" -46:01.817 [INFO] TestAgent.server: cluster leadership acquired -46:01.817 [INFO] TestAgent.server: New leader elected: payload=Node-9430b371-9b01-594c-50cf-e8d860993ba5 -46:01.826 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11393 -46:01.841 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.873 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.873 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.873 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.873 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.873 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.873 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9430b371-9b01-594c-50cf-e8d860993ba5 -46:01.873 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9430b371-9b01-594c-50cf-e8d860993ba5 -46:01.888 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.050 [INFO] TestAgent: Requesting shutdown -46:02.050 [INFO] TestAgent.server: shutting down server -46:02.050 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.050 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.050 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.050 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.050 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.050 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.050 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.050 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.050 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.050 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.050 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.053 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.057Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.057Z [INFO] agent.router.manager: shutting down -46:02.057 [INFO] TestAgent: consul server down -46:02.057 [INFO] TestAgent: shutdown complete -46:02.057 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11409 network=tcp -46:02.057 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11409 network=udp -46:02.057 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11410 network=tcp -46:02.557 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.557 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthConnectServiceNodes_PassingFilter (0.86s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:13.691Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.743Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.793 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b49badd8-ec0a-7c1b-f5e8-8a58a63d078e Address:127.0.0.1:29610}]" +31:13.794 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e.dc1 127.0.0.1 +31:13.796 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29610 [Follower]" leader= +31:13.796 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e 127.0.0.1 +2021-10-05T16:31:13.796Z [INFO] agent.router: Initializing LAN area manager +31:13.797 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29605 network=udp +31:13.797 [INFO] TestAgent.server: Adding LAN server: server="Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e (Addr: tcp/127.0.0.1:29610) (DC: dc1)" +31:13.797 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e.dc1 area=wan +31:13.797 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29605 network=tcp +31:13.797 [WARN] TestAgent: Replacing socket: path=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul774297133/test.sock +31:13.798 [INFO] TestAgent: Started HTTP server: address=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul774297133/test.sock network=unix +31:13.798 [INFO] TestAgent: started state syncer +31:13.799 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29611 network=tcp +31:13.838 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.838 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29610 [Candidate]" term=2 +31:13.880 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.880 [DEBUG] TestAgent.server.raft: vote granted: from=b49badd8-ec0a-7c1b-f5e8-8a58a63d078e term=2 tally=1 +31:13.880 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.880 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29610 [Leader]" +31:13.880 [INFO] TestAgent.server: cluster leadership acquired +31:13.880 [INFO] TestAgent.server: New leader elected: payload=Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e +31:13.901 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29610 +31:13.937 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.962 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.962 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.962 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.962 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.962 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.962 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e +31:13.963 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b49badd8-ec0a-7c1b-f5e8-8a58a63d078e +31:13.975 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.084 [INFO] TestAgent: Requesting shutdown +31:14.084 [INFO] TestAgent.server: shutting down server +31:14.084 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.084 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.084 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.084 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.084 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.084 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:14.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.084 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.084 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.084 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.092 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.100Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.100Z [INFO] agent.router.manager: shutting down +31:14.100 [INFO] TestAgent: consul server down +31:14.100 [INFO] TestAgent: shutdown complete +31:14.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29605 network=tcp +31:14.100 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29605 network=udp +31:14.100 [INFO] TestAgent: Stopping server: protocol=HTTP address=/tmp/consul-test/TestHTTPServer_UnixSocket_FileExists-consul774297133/test.sock network=unix +31:14.600 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.600 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPServer_UnixSocket_FileExists (0.95s) +=== CONT TestHealthServiceNodes_WanTranslation +2021-10-05T16:31:13.729Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.761Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.826 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18d47604-945b-685b-2920-6b6adbfc0030 Address:127.0.0.1:29617}]" +31:13.827 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18d47604-945b-685b-2920-6b6adbfc0030.dc1 127.0.0.1 +31:13.828 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29617 [Follower]" leader= +31:13.828 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18d47604-945b-685b-2920-6b6adbfc0030 127.0.0.1 +2021-10-05T16:31:13.828Z [INFO] agent.router: Initializing LAN area manager +31:13.828 [INFO] TestAgent.server: Adding LAN server: server="Node-18d47604-945b-685b-2920-6b6adbfc0030 (Addr: tcp/127.0.0.1:29617) (DC: dc1)" +31:13.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29612 network=udp +31:13.828 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18d47604-945b-685b-2920-6b6adbfc0030.dc1 area=wan +31:13.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29612 network=tcp +31:13.829 [INFO] TestAgent: Started HTTP server: address=/tmp/consul-test/TestHTTPServer_UnixSocket-consul263894247/test.sock network=unix +31:13.829 [INFO] TestAgent: started state syncer +31:13.829 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29618 network=tcp +31:13.891 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.891 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29617 [Candidate]" term=2 +31:13.926 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.926 [DEBUG] TestAgent.server.raft: vote granted: from=18d47604-945b-685b-2920-6b6adbfc0030 term=2 tally=1 +31:13.926 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.926 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29617 [Leader]" +31:13.926 [INFO] TestAgent.server: cluster leadership acquired +31:13.926 [INFO] TestAgent.server: New leader elected: payload=Node-18d47604-945b-685b-2920-6b6adbfc0030 +31:13.939 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29617 +31:13.969 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.999 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.999 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.999 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.999 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.999 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.000 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18d47604-945b-685b-2920-6b6adbfc0030 +31:14.000 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18d47604-945b-685b-2920-6b6adbfc0030 +31:14.010 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.169 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=@ latency=2.503851ms +31:14.170 [INFO] TestAgent: Requesting shutdown +31:14.170 [INFO] TestAgent.server: shutting down server +31:14.170 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.170 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.170 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.170 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.170 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.170 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.171 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:14.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.171 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.171 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.179 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.186Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.186Z [INFO] agent.router.manager: shutting down +31:14.186 [INFO] TestAgent: consul server down +31:14.186 [INFO] TestAgent: shutdown complete +31:14.186 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29612 network=tcp +31:14.186 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29612 network=udp +31:14.186 [INFO] TestAgent: Stopping server: protocol=HTTP address=/tmp/consul-test/TestHTTPServer_UnixSocket-consul263894247/test.sock network=unix +31:14.688 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.688 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPServer_UnixSocket (1.02s) +=== CONT TestHealthServiceNodes_CheckType +2021-10-05T16:31:13.626Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.673Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.767 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ac814954-052b-3984-c405-3e22bfde2188 Address:127.0.0.1:29603}]" +31:13.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ac814954-052b-3984-c405-3e22bfde2188.dc1 127.0.0.1 +31:13.768 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29603 [Follower]" leader= +31:13.768 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ac814954-052b-3984-c405-3e22bfde2188 127.0.0.1 +2021-10-05T16:31:13.768Z [INFO] agent.router: Initializing LAN area manager +31:13.768 [INFO] TestAgent.server: Adding LAN server: server="Node-ac814954-052b-3984-c405-3e22bfde2188 (Addr: tcp/127.0.0.1:29603) (DC: dc1)" +31:13.769 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ac814954-052b-3984-c405-3e22bfde2188.dc1 area=wan +31:13.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29598 network=udp +31:13.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29598 network=tcp +31:13.769 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29599 network=tcp +31:13.769 [INFO] TestAgent: started state syncer +31:13.769 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29604 network=tcp +31:13.808 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.808 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29603 [Candidate]" term=2 +31:13.849 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.849 [DEBUG] TestAgent.server.raft: vote granted: from=ac814954-052b-3984-c405-3e22bfde2188 term=2 tally=1 +31:13.849 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.849 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29603 [Leader]" +31:13.850 [INFO] TestAgent.server: cluster leadership acquired +31:13.851 [INFO] TestAgent.server: New leader elected: payload=Node-ac814954-052b-3984-c405-3e22bfde2188 +31:13.867 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29603 +31:13.889 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.946 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.946 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.946 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.946 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.946 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.946 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ac814954-052b-3984-c405-3e22bfde2188 +31:13.946 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ac814954-052b-3984-c405-3e22bfde2188 +31:13.959 [INFO] TestAgent: Synced node info +31:13.959 [DEBUG] TestAgent: Node info in sync +31:13.971 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.178 [ERROR] TestAgent.http: Request error: method=GET url=/v1/agent/self from= error="Endpoint is blocked by agent configuration" +31:14.178 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from= latency=1.726µs +31:14.178 [INFO] TestAgent: Requesting shutdown +31:14.178 [INFO] TestAgent.server: shutting down server +31:14.178 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.178 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.178 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.178 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.178 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.178 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.178 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.178 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.178 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.178 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.186 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.200Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.200Z [INFO] agent.router.manager: shutting down +31:14.200 [INFO] TestAgent: consul server down +31:14.200 [INFO] TestAgent: shutdown complete +31:14.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29598 network=tcp +31:14.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29598 network=udp +31:14.200 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29599 network=tcp +31:14.700 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.700 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_BlockEndpoints (1.09s) +=== CONT TestHealthServiceNodes_PassingFilter +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestHealthIngressServiceNodes/associated_service +=== RUN TestHealthIngressServiceNodes/non-associated_service +2021-10-05T16:31:14.119Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.152Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.228 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6b34529f-167f-be07-0c46-cf79d83f94cf Address:127.0.0.1:29624}]" +31:14.229 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6b34529f-167f-be07-0c46-cf79d83f94cf.dc1 127.0.0.1 +31:14.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6b34529f-167f-be07-0c46-cf79d83f94cf 127.0.0.1 +2021-10-05T16:31:14.229Z [INFO] agent.router: Initializing LAN area manager +31:14.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29619 network=udp +31:14.229 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29624 [Follower]" leader= +31:14.230 [INFO] TestAgent.server: Adding LAN server: server="Node-6b34529f-167f-be07-0c46-cf79d83f94cf (Addr: tcp/127.0.0.1:29624) (DC: dc1)" +31:14.230 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6b34529f-167f-be07-0c46-cf79d83f94cf.dc1 area=wan +31:14.230 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29619 network=tcp +31:14.230 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29620 network=tcp +31:14.230 [INFO] TestAgent: started state syncer +31:14.230 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29625 network=tcp +31:14.282 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.283 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29624 [Candidate]" term=2 +31:14.308 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.308 [DEBUG] TestAgent.server.raft: vote granted: from=6b34529f-167f-be07-0c46-cf79d83f94cf term=2 tally=1 +31:14.308 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.308 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29624 [Leader]" +31:14.308 [INFO] TestAgent.server: cluster leadership acquired +31:14.308 [INFO] TestAgent.server: New leader elected: payload=Node-6b34529f-167f-be07-0c46-cf79d83f94cf +31:14.331 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29624 +31:14.353 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.382 [INFO] TestAgent: Requesting shutdown +31:14.383 [INFO] TestAgent.server: shutting down server +31:14.383 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.383 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:14.392 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:14.393 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.393 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.393 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.393 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.393 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.393 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6b34529f-167f-be07-0c46-cf79d83f94cf +31:14.393 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6b34529f-167f-be07-0c46-cf79d83f94cf +2021-10-05T16:31:14.399Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.399Z [INFO] agent.router.manager: shutting down +31:14.417 [INFO] TestAgent: consul server down +31:14.417 [INFO] TestAgent: shutdown complete +31:14.417 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29619 network=tcp +31:14.417 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29619 network=udp +31:14.417 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29620 network=tcp +31:14.417 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-6b34529f-167f-be07-0c46-cf79d83f94cf 127.0.0.1 29622 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:6b34529f-167f-be07-0c46-cf79d83f94cf port:29624 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29623] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:14.417 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.417 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.417 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.417 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.418 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.418 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.418 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.918 [INFO] TestAgent: Waiting for endpoints to shut down +31:14.918 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthConnectServiceNodes_PassingFilter (0.83s) --- PASS: TestHealthConnectServiceNodes_PassingFilter/bc_no_query_value (0.00s) --- PASS: TestHealthConnectServiceNodes_PassingFilter/passing_true (0.00s) --- PASS: TestHealthConnectServiceNodes_PassingFilter/passing_false (0.00s) --- PASS: TestHealthConnectServiceNodes_PassingFilter/passing_bad (0.00s) -=== CONT TestHealthChecksInState_NodeMetaFilter -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:00.810Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.820Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.882 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9a97907-7afc-71d1-6b2d-674b5dfbc7bb Address:127.0.0.1:11281}]" -46:00.882 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11281 [Follower]" leader= -46:00.883 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb.dc1 127.0.0.1 -46:00.883 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb 127.0.0.1 -2022-11-07T22:46:00.883Z [INFO] agent.router: Initializing LAN area manager -46:00.883 [INFO] TestAgent.server: Adding LAN server: server="Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb (Addr: tcp/127.0.0.1:11281) (DC: dc1)" -46:00.883 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb.dc1 area=wan -46:00.884 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11269 network=tcp -46:00.884 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11269 network=udp -46:00.884 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11270 network=tcp -46:00.884 [INFO] TestAgent: started state syncer -46:00.884 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11282 network=tcp -46:00.924 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.924 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11281 [Candidate]" term=2 -46:00.945 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.945 [DEBUG] TestAgent.server.raft: vote granted: from=c9a97907-7afc-71d1-6b2d-674b5dfbc7bb term=2 tally=1 -46:00.945 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.945 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11281 [Leader]" -46:00.945 [INFO] TestAgent.server: cluster leadership acquired -46:00.945 [INFO] TestAgent.server: New leader elected: payload=Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb -46:00.953 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11281 -46:00.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.986 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.986 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.986 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.986 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.986 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.986 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb -46:00.986 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9a97907-7afc-71d1-6b2d-674b5dfbc7bb -46:00.996 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.097 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.100 [INFO] TestAgent: Synced node info -46:01.100 [DEBUG] TestAgent: Node info in sync -46:02.131 [INFO] TestAgent: Requesting shutdown -46:02.131 [INFO] TestAgent.server: shutting down server -46:02.131 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.131 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.131 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.131 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.131 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.131 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.131 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.131 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.131 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.131 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.134 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.137Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.137Z [INFO] agent.router.manager: shutting down -46:02.137 [INFO] TestAgent: consul server down -46:02.137 [INFO] TestAgent: shutdown complete -46:02.137 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11269 network=tcp -46:02.137 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11269 network=udp -46:02.137 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11270 network=tcp -46:02.637 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.637 [INFO] TestAgent: Endpoints down ---- PASS: TestPProfHandlers_EnableDebug (1.84s) -=== CONT TestHealthChecksInState_DistanceSort +=== CONT TestHealthServiceNodes_DistanceSort +2021-10-05T16:31:14.139Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.179Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.227 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:66ae3dff-0bee-ea52-1c8d-b87ed964cfef Address:127.0.0.1:29638}]" +31:14.228 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29638 [Follower]" leader= +31:14.228 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef.dc1 127.0.0.1 +31:14.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef 127.0.0.1 +2021-10-05T16:31:14.229Z [INFO] agent.router: Initializing LAN area manager +31:14.229 [INFO] TestAgent.server: Adding LAN server: server="Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef (Addr: tcp/127.0.0.1:29638) (DC: dc1)" +31:14.229 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef.dc1 area=wan +31:14.230 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29626 network=tcp +31:14.230 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29626 network=udp +31:14.230 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29627 network=tcp +31:14.231 [INFO] TestAgent: started state syncer +31:14.231 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29639 network=tcp +31:14.283 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.283 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29638 [Candidate]" term=2 +31:14.304 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.304 [DEBUG] TestAgent.server.raft: vote granted: from=66ae3dff-0bee-ea52-1c8d-b87ed964cfef term=2 tally=1 +31:14.304 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.304 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29638 [Leader]" +31:14.304 [INFO] TestAgent.server: cluster leadership acquired +31:14.304 [INFO] TestAgent.server: New leader elected: payload=Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef +31:14.329 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29638 +31:14.351 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.395 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.395 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.395 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.395 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.395 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.395 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef +31:14.395 [INFO] TestAgent.server: member joined, marking health alive: member=Node-66ae3dff-0bee-ea52-1c8d-b87ed964cfef +31:14.436 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:14.447 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.454 [INFO] TestAgent: Synced node info +31:14.454 [DEBUG] TestAgent: Node info in sync +31:14.489 [INFO] TestAgent: Requesting shutdown +31:14.489 [INFO] TestAgent.server: shutting down server +31:14.489 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.489 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.489 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.490 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.490 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.490 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.504 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.512Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.512Z [INFO] agent.router.manager: shutting down +31:14.512 [INFO] TestAgent: consul server down +31:14.512 [INFO] TestAgent: shutdown complete +31:14.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29626 network=tcp +31:14.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29626 network=udp +31:14.512 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29627 network=tcp +31:15.012 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.012 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthConnectServiceNodes_Filter (0.91s) +=== CONT TestHealthServiceNodes_Filter === RUN TestHealthServiceNodes_PassingFilter/bc_no_query_value === RUN TestHealthServiceNodes_PassingFilter/passing_true === RUN TestHealthServiceNodes_PassingFilter/passing_false === RUN TestHealthServiceNodes_PassingFilter/passing_bad -=== RUN TestHealthIngressServiceNodes/associated_service -=== RUN TestHealthIngressServiceNodes/non-associated_service +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:14.347Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.360Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.438 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:94bbf6ae-8d27-f0e6-d140-28448f4b5ebf Address:127.0.0.1:29645}]" +31:14.439 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf.dc1 127.0.0.1 +31:14.439 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf 127.0.0.1 +2021-10-05T16:31:14.439Z [INFO] agent.router: Initializing LAN area manager +31:14.440 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29633 network=udp +31:14.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29645 [Follower]" leader= +31:14.444 [INFO] TestAgent.server: Adding LAN server: server="Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf (Addr: tcp/127.0.0.1:29645) (DC: dc1)" +31:14.445 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf.dc1 area=wan +31:14.445 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29633 network=tcp +31:14.446 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29634 network=tcp +31:14.446 [INFO] TestAgent: started state syncer +31:14.446 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29646 network=tcp +31:14.493 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.493 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29645 [Candidate]" term=2 +31:14.512 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.512 [DEBUG] TestAgent.server.raft: vote granted: from=94bbf6ae-8d27-f0e6-d140-28448f4b5ebf term=2 tally=1 +31:14.512 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.512 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29645 [Leader]" +31:14.512 [INFO] TestAgent.server: cluster leadership acquired +31:14.512 [INFO] TestAgent.server: New leader elected: payload=Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf +31:14.532 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29645 +31:14.573 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.628 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.628 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.628 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.628 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.628 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.628 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf +31:14.628 [INFO] TestAgent.server: member joined, marking health alive: member=Node-94bbf6ae-8d27-f0e6-d140-28448f4b5ebf +31:14.640 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.675 [INFO] TestAgent: Requesting shutdown +31:14.675 [INFO] TestAgent.server: shutting down server +31:14.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.675 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.675 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.675 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.675 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.675 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:14.676 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.676 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.676 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.683 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.695Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.695Z [INFO] agent.router.manager: shutting down +31:14.695 [INFO] TestAgent: consul server down +31:14.695 [INFO] TestAgent: shutdown complete +31:14.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29633 network=tcp +31:14.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29633 network=udp +31:14.695 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29634 network=tcp +31:15.196 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.196 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_IngressGateway (0.88s) +=== CONT TestHealthServiceNodes_NodeMetaFilter [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:01.648Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.692 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:266c81c4-483e-83a7-b789-ad8c58ee1549 Address:127.0.0.1:11379}]" -46:01.692 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11379 [Follower]" leader= -46:01.693 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-266c81c4-483e-83a7-b789-ad8c58ee1549.dc1 127.0.0.1 -46:01.693 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-266c81c4-483e-83a7-b789-ad8c58ee1549 127.0.0.1 -2022-11-07T22:46:01.693Z [INFO] agent.router: Initializing LAN area manager -46:01.693 [INFO] TestAgent.server: Adding LAN server: server="Node-266c81c4-483e-83a7-b789-ad8c58ee1549 (Addr: tcp/127.0.0.1:11379) (DC: dc1)" -46:01.693 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-266c81c4-483e-83a7-b789-ad8c58ee1549.dc1 area=wan -46:01.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11374 network=udp -46:01.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11374 network=tcp -46:01.694 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11375 network=tcp -46:01.694 [INFO] TestAgent: started state syncer -46:01.694 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11380 network=tcp -46:01.747 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.747 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11379 [Candidate]" term=2 -46:01.759 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.759 [DEBUG] TestAgent.server.raft: vote granted: from=266c81c4-483e-83a7-b789-ad8c58ee1549 term=2 tally=1 -46:01.759 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.759 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11379 [Leader]" -46:01.759 [INFO] TestAgent.server: cluster leadership acquired -46:01.759 [INFO] TestAgent.server: New leader elected: payload=Node-266c81c4-483e-83a7-b789-ad8c58ee1549 -46:01.769 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11379 -46:01.787 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.821 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.821 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.821 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.821 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.821 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.822 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-266c81c4-483e-83a7-b789-ad8c58ee1549 -46:01.822 [INFO] TestAgent.server: member joined, marking health alive: member=Node-266c81c4-483e-83a7-b789-ad8c58ee1549 -46:01.832 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.850 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.858 [INFO] TestAgent: Synced node info -46:02.018 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=13.625µs -46:02.018 [INFO] TestAgent: Requesting shutdown -46:02.018 [INFO] TestAgent.server: shutting down server -46:02.018 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.018 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.018 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.018 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.018 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.018 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.018 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.018 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.018 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.018 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.022 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.027Z [INFO] agent.router.manager: shutting down -46:02.027 [INFO] TestAgent: consul server down -46:02.027 [INFO] TestAgent: shutdown complete -46:02.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11374 network=tcp -46:02.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11374 network=udp -46:02.027 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11375 network=tcp -46:02.527 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.527 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:01.418Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.428Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.460 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 Address:127.0.0.1:11330}]" -46:01.460 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11330 [Follower]" leader= -46:01.460 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2.dc1 127.0.0.1 -46:01.461 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 127.0.0.1 -2022-11-07T22:46:01.461Z [INFO] agent.router: Initializing LAN area manager -46:01.461 [INFO] TestAgent.server: Adding LAN server: server="Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 (Addr: tcp/127.0.0.1:11330) (DC: dc1)" -46:01.461 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2.dc1 area=wan -46:01.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11325 network=udp -46:01.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11325 network=tcp -46:01.462 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11326 network=tcp -46:01.462 [INFO] TestAgent: started state syncer -46:01.462 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11331 network=tcp -46:01.519 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.519 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11330 [Candidate]" term=2 -46:01.533 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.533 [DEBUG] TestAgent.server.raft: vote granted: from=b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 term=2 tally=1 -46:01.533 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.533 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11330 [Leader]" -46:01.533 [INFO] TestAgent.server: cluster leadership acquired -46:01.534 [INFO] TestAgent.server: New leader elected: payload=Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 -46:01.543 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11330 -46:01.559 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.584 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.584 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.584 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.584 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.584 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.584 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 -46:01.584 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b3e49408-ebb6-1ad7-04b4-77cde8b27ac2 -46:01.589 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.638 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=1.522µs -46:01.895 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.898 [INFO] TestAgent: Synced node info -46:02.527 [INFO] TestAgent: Requesting shutdown -46:02.527 [INFO] TestAgent.server: shutting down server -46:02.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.527 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.527 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.531 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.535Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.535Z [INFO] agent.router.manager: shutting down -46:02.535 [INFO] TestAgent: consul server down -46:02.535 [INFO] TestAgent: shutdown complete -46:02.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11325 network=tcp -46:02.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11325 network=udp -46:02.535 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11326 network=tcp -46:03.035 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.035 [INFO] TestAgent: Endpoints down ---- PASS: TestHTTPAPI_TranslateAddrHeader (1.63s) -=== CONT TestHealthNodeChecks_Filtering -2022-11-07T22:46:02.312Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.323Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.360 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 Address:127.0.0.1:11470}]" -46:02.360 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11470 [Follower]" leader= -46:02.360 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6.dc1 127.0.0.1 -46:02.360 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 127.0.0.1 -2022-11-07T22:46:02.360Z [INFO] agent.router: Initializing LAN area manager -46:02.360 [INFO] TestAgent.server: Adding LAN server: server="Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 (Addr: tcp/127.0.0.1:11470) (DC: dc1)" -46:02.361 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6.dc1 area=wan -46:02.361 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11465 network=udp -46:02.361 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11465 network=tcp -46:02.361 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11466 network=tcp -46:02.361 [INFO] TestAgent: started state syncer -46:02.361 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11471 network=tcp -46:02.427 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.427 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11470 [Candidate]" term=2 -46:02.441 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.441 [DEBUG] TestAgent.server.raft: vote granted: from=2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 term=2 tally=1 -46:02.441 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.441 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11470 [Leader]" -46:02.441 [INFO] TestAgent.server: cluster leadership acquired -46:02.441 [INFO] TestAgent.server: New leader elected: payload=Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 -46:02.448 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11470 -46:02.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.512 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.512 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.512 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.512 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.512 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.512 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 -46:02.512 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b975bc6-95de-8c68-d74e-3e24bd4fd9a6 -46:02.517 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.534 [INFO] TestAgent: Synced node info -46:02.534 [DEBUG] TestAgent: Node info in sync -46:02.534 [INFO] TestAgent: Requesting shutdown -46:02.534 [INFO] TestAgent.server: shutting down server -46:02.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.534 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.534 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.534 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.534 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.534 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.534 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.538 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.545Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.545Z [INFO] agent.router.manager: shutting down -46:02.545 [INFO] TestAgent: consul server down -46:02.545 [INFO] TestAgent: shutdown complete -46:02.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11465 network=tcp -46:02.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11465 network=udp -46:02.545 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11466 network=tcp -46:03.045 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.045 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthConnectServiceNodes (0.74s) -=== CONT TestUUIDToUint64 ---- PASS: TestUUIDToUint64 (0.00s) -=== CONT TestHealthChecksInState_Filter -2022-11-07T22:46:02.177Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.188Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.209 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:53310d9e-368d-74eb-27e8-1315d61abc69 Address:127.0.0.1:11456}]" -46:02.209 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11456 [Follower]" leader= -46:02.209 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-53310d9e-368d-74eb-27e8-1315d61abc69.dc1 127.0.0.1 -46:02.210 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-53310d9e-368d-74eb-27e8-1315d61abc69 127.0.0.1 -2022-11-07T22:46:02.210Z [INFO] agent.router: Initializing LAN area manager -46:02.210 [INFO] TestAgent.server: Adding LAN server: server="Node-53310d9e-368d-74eb-27e8-1315d61abc69 (Addr: tcp/127.0.0.1:11456) (DC: dc1)" -46:02.210 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-53310d9e-368d-74eb-27e8-1315d61abc69.dc1 area=wan -46:02.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11423 network=udp -46:02.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11423 network=tcp -46:02.210 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11424 network=tcp -46:02.210 [INFO] TestAgent: started state syncer -46:02.210 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11457 network=tcp -46:02.260 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.260 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11456 [Candidate]" term=2 -46:02.268 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.268 [DEBUG] TestAgent.server.raft: vote granted: from=53310d9e-368d-74eb-27e8-1315d61abc69 term=2 tally=1 -46:02.268 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.269 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11456 [Leader]" -46:02.269 [INFO] TestAgent.server: cluster leadership acquired -46:02.269 [INFO] TestAgent.server: New leader elected: payload=Node-53310d9e-368d-74eb-27e8-1315d61abc69 -46:02.275 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11456 -46:02.290 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.323 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.323 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.323 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.323 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.323 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.323 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-53310d9e-368d-74eb-27e8-1315d61abc69 -46:02.323 [INFO] TestAgent.server: member joined, marking health alive: member=Node-53310d9e-368d-74eb-27e8-1315d61abc69 -46:02.330 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.508 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.513 [INFO] TestAgent: Synced node info -46:02.551 [INFO] TestAgent: Requesting shutdown -46:02.551 [INFO] TestAgent.server: shutting down server -46:02.551 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.551 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.551 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.551 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.551 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.551 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.551 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.551 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.551 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.551 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.560 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.571Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.571Z [INFO] agent.router.manager: shutting down -46:02.571 [INFO] TestAgent: consul server down -46:02.571 [INFO] TestAgent: shutdown complete -46:02.571 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11423 network=tcp -46:02.571 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11423 network=udp -46:02.571 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11424 network=tcp -46:03.071 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.071 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_Filter (0.91s) -=== CONT TestEventList_Blocking -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:02.226Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.235Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.255 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:41c8eef6-0a18-578b-2437-3ab7b84f6116 Address:127.0.0.1:11449}]" -46:02.255 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11449 [Follower]" leader= -46:02.257 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-41c8eef6-0a18-578b-2437-3ab7b84f6116.dc1 127.0.0.1 -46:02.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-41c8eef6-0a18-578b-2437-3ab7b84f6116 127.0.0.1 -2022-11-07T22:46:02.258Z [INFO] agent.router: Initializing LAN area manager -46:02.258 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-41c8eef6-0a18-578b-2437-3ab7b84f6116.dc1 area=wan -46:02.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11416 network=udp -46:02.258 [INFO] TestAgent.server: Adding LAN server: server="Node-41c8eef6-0a18-578b-2437-3ab7b84f6116 (Addr: tcp/127.0.0.1:11449) (DC: dc1)" -46:02.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11416 network=tcp -46:02.258 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11417 network=tcp -46:02.258 [INFO] TestAgent: started state syncer -46:02.258 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11450 network=tcp -46:02.291 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.291 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11449 [Candidate]" term=2 -46:02.306 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.306 [DEBUG] TestAgent.server.raft: vote granted: from=41c8eef6-0a18-578b-2437-3ab7b84f6116 term=2 tally=1 -46:02.306 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.306 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11449 [Leader]" -46:02.306 [INFO] TestAgent.server: cluster leadership acquired -46:02.307 [INFO] TestAgent.server: New leader elected: payload=Node-41c8eef6-0a18-578b-2437-3ab7b84f6116 -46:02.314 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11449 -46:02.331 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.363 [INFO] TestAgent: Synced node info -46:02.363 [DEBUG] TestAgent: Node info in sync -46:02.363 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.363 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.363 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.363 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.363 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.363 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-41c8eef6-0a18-578b-2437-3ab7b84f6116 -46:02.363 [INFO] TestAgent.server: member joined, marking health alive: member=Node-41c8eef6-0a18-578b-2437-3ab7b84f6116 -46:02.379 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.604 [INFO] TestAgent: Requesting shutdown -46:02.604 [INFO] TestAgent.server: shutting down server -46:02.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.604 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.604 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.604 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.604 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.604 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.604 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.612 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.619Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.619Z [INFO] agent.router.manager: shutting down -46:02.619 [INFO] TestAgent: consul server down -46:02.619 [INFO] TestAgent: shutdown complete -46:02.619 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11416 network=tcp -46:02.619 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11416 network=udp -46:02.619 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11417 network=tcp -46:03.119 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.119 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceChecks_Filtering (0.90s) -=== CONT TestEventList_EventBufOrder -2022-11-07T22:46:02.383Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.392Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.428 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ec123322-4d71-63cb-697e-f3984a47d290 Address:127.0.0.1:11484}]" -46:02.428 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11484 [Follower]" leader= -46:02.429 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ec123322-4d71-63cb-697e-f3984a47d290.dc1 127.0.0.1 -46:02.429 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec123322-4d71-63cb-697e-f3984a47d290 127.0.0.1 -2022-11-07T22:46:02.430Z [INFO] agent.router: Initializing LAN area manager -46:02.430 [INFO] TestAgent.server: Adding LAN server: server="Node-ec123322-4d71-63cb-697e-f3984a47d290 (Addr: tcp/127.0.0.1:11484) (DC: dc1)" -46:02.430 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11493 network=udp -46:02.430 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ec123322-4d71-63cb-697e-f3984a47d290.dc1 area=wan -46:02.430 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11493 network=tcp -46:02.430 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11494 network=tcp -46:02.430 [INFO] TestAgent: started state syncer -46:02.430 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11485 network=tcp -46:02.481 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.481 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11484 [Candidate]" term=2 -46:02.494 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.494 [DEBUG] TestAgent.server.raft: vote granted: from=ec123322-4d71-63cb-697e-f3984a47d290 term=2 tally=1 -46:02.494 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.494 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11484 [Leader]" -46:02.494 [INFO] TestAgent.server: cluster leadership acquired -46:02.494 [INFO] TestAgent.server: New leader elected: payload=Node-ec123322-4d71-63cb-697e-f3984a47d290 -46:02.505 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11484 -46:02.521 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.547 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.547 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.547 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.547 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.547 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.547 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec123322-4d71-63cb-697e-f3984a47d290 -46:02.547 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec123322-4d71-63cb-697e-f3984a47d290 -46:02.562 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.609 [INFO] TestAgent: Requesting shutdown -46:02.609 [INFO] TestAgent.server: shutting down server -46:02.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.609 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.609 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.609 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.609 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.609 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.609 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.609 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.616 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.622Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.622Z [INFO] agent.router.manager: shutting down -46:02.622 [INFO] TestAgent: consul server down -46:02.622 [INFO] TestAgent: shutdown complete -46:02.622 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11493 network=tcp -46:02.622 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11493 network=udp -46:02.622 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11494 network=tcp -46:03.123 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.123 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceChecks_NodeMetaFilter (0.75s) +2021-10-05T16:31:14.447Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.504Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.592 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dfb4eaa1-017d-143c-d566-1dbe6f97a68e Address:127.0.0.1:29659}]" +31:14.593 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e.dc1 127.0.0.1 +31:14.593 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e 127.0.0.1 +2021-10-05T16:31:14.593Z [INFO] agent.router: Initializing LAN area manager +31:14.593 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29654 network=udp +31:14.593 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29659 [Follower]" leader= +31:14.594 [INFO] TestAgent.server: Adding LAN server: server="Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e (Addr: tcp/127.0.0.1:29659) (DC: dc1)" +31:14.594 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e.dc1 area=wan +31:14.594 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29654 network=tcp +31:14.594 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29655 network=tcp +31:14.594 [INFO] TestAgent: started state syncer +31:14.594 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29660 network=tcp +31:14.644 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.644 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29659 [Candidate]" term=2 +31:14.696 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.696 [DEBUG] TestAgent.server.raft: vote granted: from=dfb4eaa1-017d-143c-d566-1dbe6f97a68e term=2 tally=1 +31:14.696 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29659 [Leader]" +31:14.696 [INFO] TestAgent.server: cluster leadership acquired +31:14.696 [INFO] TestAgent.server: New leader elected: payload=Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e +31:14.728 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29659 +31:14.766 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.798 [INFO] TestAgent: Synced node info +31:14.798 [DEBUG] TestAgent: Node info in sync +31:14.824 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.824 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.824 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.824 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.824 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.824 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e +31:14.824 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dfb4eaa1-017d-143c-d566-1dbe6f97a68e +31:14.847 [INFO] TestAgent: Requesting shutdown +31:14.847 [INFO] TestAgent.server: shutting down server +31:14.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.847 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.847 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.847 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.847 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.847 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.847 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.847 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.851 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.860Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.860Z [INFO] agent.router.manager: shutting down +31:14.860 [INFO] TestAgent: consul server down +31:14.860 [INFO] TestAgent: shutdown complete +31:14.860 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29654 network=tcp +31:14.860 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29654 network=udp +31:14.860 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29655 network=tcp +31:15.360 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.360 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthIngressServiceNodes (0.96s) + --- PASS: TestHealthIngressServiceNodes/associated_service (0.00s) + --- PASS: TestHealthIngressServiceNodes/non-associated_service (0.00s) +=== CONT TestHealthServiceNodes +2021-10-05T16:31:14.401Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.431Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.555 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3105e377-e6ca-5f2a-b882-8e182bf459c7 Address:127.0.0.1:29652}]" +31:14.556 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3105e377-e6ca-5f2a-b882-8e182bf459c7.dc1 127.0.0.1 +31:14.557 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3105e377-e6ca-5f2a-b882-8e182bf459c7 127.0.0.1 +2021-10-05T16:31:14.557Z [INFO] agent.router: Initializing LAN area manager +31:14.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29647 network=udp +31:14.557 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29652 [Follower]" leader= +31:14.557 [INFO] TestAgent.server: Adding LAN server: server="Node-3105e377-e6ca-5f2a-b882-8e182bf459c7 (Addr: tcp/127.0.0.1:29652) (DC: dc1)" +31:14.557 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3105e377-e6ca-5f2a-b882-8e182bf459c7.dc1 area=wan +31:14.558 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29647 network=tcp +31:14.558 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29648 network=tcp +31:14.558 [INFO] TestAgent: started state syncer +31:14.558 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29653 network=tcp +31:14.596 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.596 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29652 [Candidate]" term=2 +31:14.634 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.634 [DEBUG] TestAgent.server.raft: vote granted: from=3105e377-e6ca-5f2a-b882-8e182bf459c7 term=2 tally=1 +31:14.634 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.634 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29652 [Leader]" +31:14.636 [INFO] TestAgent.server: cluster leadership acquired +31:14.636 [INFO] TestAgent.server: New leader elected: payload=Node-3105e377-e6ca-5f2a-b882-8e182bf459c7 +31:14.663 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29652 +31:14.692 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.755 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.755 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.755 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.755 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.755 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.755 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3105e377-e6ca-5f2a-b882-8e182bf459c7 +31:14.755 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3105e377-e6ca-5f2a-b882-8e182bf459c7 +31:14.772 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.802 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:14.812 [INFO] TestAgent: Synced node info +31:14.880 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:14.880 [INFO] TestAgent: Requesting shutdown +31:14.880 [INFO] TestAgent.server: shutting down server +31:14.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.880 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.880 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.880 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.880 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.880 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.880 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.880 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.880 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.895 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.908Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.908Z [INFO] agent.router.manager: shutting down +31:14.908 [INFO] TestAgent: consul server down +31:14.908 [INFO] TestAgent: shutdown complete +31:14.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29647 network=tcp +31:14.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29647 network=udp +31:14.908 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29648 network=tcp +31:15.409 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.409 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#04 +2021-10-05T16:31:14.715Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.750Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.820 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b33c8d6-f754-5048-c859-15bece5e8a2d Address:127.0.0.1:29680}]" +31:14.820 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29680 [Follower]" leader= +31:14.821 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b33c8d6-f754-5048-c859-15bece5e8a2d.dc1 127.0.0.1 +31:14.822 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b33c8d6-f754-5048-c859-15bece5e8a2d 127.0.0.1 +2021-10-05T16:31:14.822Z [INFO] agent.router: Initializing LAN area manager +31:14.822 [INFO] TestAgent.server: Adding LAN server: server="Node-2b33c8d6-f754-5048-c859-15bece5e8a2d (Addr: tcp/127.0.0.1:29680) (DC: dc1)" +31:14.822 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b33c8d6-f754-5048-c859-15bece5e8a2d.dc1 area=wan +31:14.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29675 network=udp +31:14.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29675 network=tcp +31:14.823 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29676 network=tcp +31:14.823 [INFO] TestAgent: started state syncer +31:14.823 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29681 network=tcp +31:14.886 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.886 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29680 [Candidate]" term=2 +31:14.907 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.907 [DEBUG] TestAgent.server.raft: vote granted: from=2b33c8d6-f754-5048-c859-15bece5e8a2d term=2 tally=1 +31:14.907 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.907 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29680 [Leader]" +31:14.907 [INFO] TestAgent.server: cluster leadership acquired +31:14.907 [INFO] TestAgent.server: New leader elected: payload=Node-2b33c8d6-f754-5048-c859-15bece5e8a2d +31:14.917 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29680 +31:14.936 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.959 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.959 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.959 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.959 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.959 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.959 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b33c8d6-f754-5048-c859-15bece5e8a2d +31:14.959 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b33c8d6-f754-5048-c859-15bece5e8a2d +31:14.976 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.025 [INFO] TestAgent: Requesting shutdown +31:15.025 [INFO] TestAgent.server: shutting down server +31:15.025 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.025 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.025 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.025 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.025 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:15.025 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.025 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.035 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.044Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.044Z [INFO] agent.router.manager: shutting down +31:15.044 [INFO] TestAgent: consul server down +31:15.044 [INFO] TestAgent: shutdown complete +31:15.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29675 network=tcp +31:15.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29675 network=udp +31:15.044 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29676 network=tcp +31:15.550 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.550 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_CheckType (0.86s) +=== CONT TestHealthServiceChecks_DistanceSort +2021-10-05T16:31:14.487Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.509Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.592 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:718bd739-baa6-2969-59aa-cba7bb38e4c8 Address:127.0.0.1:29666}]" +31:14.593 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-718bd739-baa6-2969-59aa-cba7bb38e4c8.dc1 127.0.0.1 +31:14.594 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-718bd739-baa6-2969-59aa-cba7bb38e4c8 127.0.0.1 +2021-10-05T16:31:14.594Z [INFO] agent.router: Initializing LAN area manager +31:14.594 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29661 network=udp +31:14.594 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29666 [Follower]" leader= +31:14.594 [INFO] TestAgent.server: Adding LAN server: server="Node-718bd739-baa6-2969-59aa-cba7bb38e4c8 (Addr: tcp/127.0.0.1:29666) (DC: dc1)" +31:14.595 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-718bd739-baa6-2969-59aa-cba7bb38e4c8.dc1 area=wan +31:14.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29661 network=tcp +31:14.595 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29662 network=tcp +31:14.595 [INFO] TestAgent: started state syncer +31:14.595 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29667 network=tcp +31:14.655 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.655 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29666 [Candidate]" term=2 +31:14.672 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.672 [DEBUG] TestAgent.server.raft: vote granted: from=718bd739-baa6-2969-59aa-cba7bb38e4c8 term=2 tally=1 +31:14.672 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.672 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29666 [Leader]" +31:14.672 [INFO] TestAgent.server: cluster leadership acquired +31:14.672 [INFO] TestAgent.server: New leader elected: payload=Node-718bd739-baa6-2969-59aa-cba7bb38e4c8 +31:14.682 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29666 +31:14.720 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.793 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.793 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.793 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.793 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.793 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.793 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-718bd739-baa6-2969-59aa-cba7bb38e4c8 +31:14.793 [INFO] TestAgent.server: member joined, marking health alive: member=Node-718bd739-baa6-2969-59aa-cba7bb38e4c8 +31:14.809 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.830 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:14.835 [INFO] TestAgent: Synced node info +31:15.030 [INFO] TestAgent: Requesting shutdown +31:15.030 [INFO] TestAgent.server: shutting down server +31:15.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.030 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.030 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.030 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.031 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.031 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.031 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.031 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.031 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.040 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.048Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.048Z [INFO] agent.router.manager: shutting down +31:15.048 [INFO] TestAgent: consul server down +31:15.048 [INFO] TestAgent: shutdown complete +31:15.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29661 network=tcp +31:15.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29661 network=udp +31:15.048 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29662 network=tcp +31:15.552 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.552 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthConnectServiceNodes (1.13s) === CONT TestEventList_ACLFilter -2022-11-07T22:46:02.349Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.364Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.399 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4842cae-2e90-e644-fab3-b7ba77da949d Address:127.0.0.1:11477}]" -46:02.400 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11477 [Follower]" leader= -46:02.400 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e4842cae-2e90-e644-fab3-b7ba77da949d.dc1 127.0.0.1 -46:02.400 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e4842cae-2e90-e644-fab3-b7ba77da949d 127.0.0.1 -2022-11-07T22:46:02.400Z [INFO] agent.router: Initializing LAN area manager -46:02.400 [INFO] TestAgent.server: Adding LAN server: server="Node-e4842cae-2e90-e644-fab3-b7ba77da949d (Addr: tcp/127.0.0.1:11477) (DC: dc1)" -46:02.400 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e4842cae-2e90-e644-fab3-b7ba77da949d.dc1 area=wan -46:02.401 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11472 network=udp -46:02.401 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11472 network=tcp -46:02.401 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11473 network=tcp -46:02.401 [INFO] TestAgent: started state syncer -46:02.401 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11478 network=tcp -46:02.438 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.438 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11477 [Candidate]" term=2 -46:02.447 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.447 [DEBUG] TestAgent.server.raft: vote granted: from=e4842cae-2e90-e644-fab3-b7ba77da949d term=2 tally=1 -46:02.447 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.447 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11477 [Leader]" -46:02.448 [INFO] TestAgent.server: cluster leadership acquired -46:02.448 [INFO] TestAgent.server: New leader elected: payload=Node-e4842cae-2e90-e644-fab3-b7ba77da949d -46:02.456 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11477 -46:02.480 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.505 [INFO] TestAgent: Synced node info -46:02.505 [DEBUG] TestAgent: Node info in sync -46:02.515 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.515 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.515 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.515 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.515 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.515 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e4842cae-2e90-e644-fab3-b7ba77da949d -46:02.515 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e4842cae-2e90-e644-fab3-b7ba77da949d -46:02.527 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.612 [INFO] TestAgent: Requesting shutdown -46:02.612 [INFO] TestAgent.server: shutting down server -46:02.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.612 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.612 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.612 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.612 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.612 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.612 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.619 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.625Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.625Z [INFO] agent.router.manager: shutting down -46:02.625 [INFO] TestAgent: consul server down -46:02.625 [INFO] TestAgent: shutdown complete -46:02.625 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11472 network=tcp -46:02.625 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11472 network=udp -46:02.625 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11473 network=tcp -46:03.126 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.126 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes (0.79s) -=== CONT TestEventList -2022-11-07T22:46:02.457Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.466Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.514 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9cb06738-4249-52d0-339f-f287292e0af0 Address:127.0.0.1:11514}]" -46:02.514 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11514 [Follower]" leader= -46:02.515 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9cb06738-4249-52d0-339f-f287292e0af0.dc1 127.0.0.1 -46:02.515 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9cb06738-4249-52d0-339f-f287292e0af0 127.0.0.1 -2022-11-07T22:46:02.515Z [INFO] agent.router: Initializing LAN area manager -46:02.515 [INFO] TestAgent.server: Adding LAN server: server="Node-9cb06738-4249-52d0-339f-f287292e0af0 (Addr: tcp/127.0.0.1:11514) (DC: dc1)" -46:02.515 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9cb06738-4249-52d0-339f-f287292e0af0.dc1 area=wan -46:02.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11507 network=udp -46:02.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11507 network=tcp -46:02.516 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11508 network=tcp -46:02.516 [INFO] TestAgent: started state syncer -46:02.516 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11526 network=tcp -46:02.551 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.551 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11514 [Candidate]" term=2 -46:02.566 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.566 [DEBUG] TestAgent.server.raft: vote granted: from=9cb06738-4249-52d0-339f-f287292e0af0 term=2 tally=1 -46:02.566 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.566 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11514 [Leader]" -46:02.566 [INFO] TestAgent.server: cluster leadership acquired -46:02.566 [INFO] TestAgent.server: New leader elected: payload=Node-9cb06738-4249-52d0-339f-f287292e0af0 -46:02.574 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11514 -46:02.594 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.621 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.621 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.621 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.621 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.621 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.621 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9cb06738-4249-52d0-339f-f287292e0af0 -46:02.621 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9cb06738-4249-52d0-339f-f287292e0af0 -46:02.632 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.650 [INFO] TestAgent: Requesting shutdown -46:02.650 [INFO] TestAgent.server: shutting down server -46:02.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.651 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.651 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.651 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.651 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.651 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.651 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.651 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.654 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.657Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.657Z [INFO] agent.router.manager: shutting down -46:02.658 [INFO] TestAgent: consul server down -46:02.658 [INFO] TestAgent: shutdown complete -46:02.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11507 network=tcp -46:02.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11507 network=udp -46:02.658 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11508 network=tcp -46:03.158 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.158 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthNodeChecks (0.71s) -=== CONT TestDNS_ReloadConfig_DuringQuery -2022-11-07T22:46:02.308Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.317Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.353 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4995d5c6-2161-51b8-4cdd-6598abb69542 Address:127.0.0.1:11463}]" -46:02.353 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11463 [Follower]" leader= -46:02.354 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4995d5c6-2161-51b8-4cdd-6598abb69542.dc1 127.0.0.1 -46:02.354 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4995d5c6-2161-51b8-4cdd-6598abb69542 127.0.0.1 -2022-11-07T22:46:02.354Z [INFO] agent.router: Initializing LAN area manager -46:02.354 [INFO] TestAgent.server: Adding LAN server: server="Node-4995d5c6-2161-51b8-4cdd-6598abb69542 (Addr: tcp/127.0.0.1:11463) (DC: dc1)" -46:02.355 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4995d5c6-2161-51b8-4cdd-6598abb69542.dc1 area=wan -46:02.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11444 network=udp -46:02.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11444 network=tcp -46:02.355 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11445 network=tcp -46:02.355 [INFO] TestAgent: started state syncer -46:02.355 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11464 network=tcp -46:02.402 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.402 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11463 [Candidate]" term=2 -46:02.414 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.414 [DEBUG] TestAgent.server.raft: vote granted: from=4995d5c6-2161-51b8-4cdd-6598abb69542 term=2 tally=1 -46:02.414 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.414 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11463 [Leader]" -46:02.414 [INFO] TestAgent.server: cluster leadership acquired -46:02.414 [INFO] TestAgent.server: New leader elected: payload=Node-4995d5c6-2161-51b8-4cdd-6598abb69542 -46:02.420 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11463 -46:02.434 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.471 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.471 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.471 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.471 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.471 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.471 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4995d5c6-2161-51b8-4cdd-6598abb69542 -46:02.471 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4995d5c6-2161-51b8-4cdd-6598abb69542 -46:02.478 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.661 [INFO] TestAgent: Requesting shutdown -46:02.661 [INFO] TestAgent.server: shutting down server -46:02.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.661 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.661 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.661 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.661 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.661 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.661 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.661 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.666 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.671Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.671Z [INFO] agent.router.manager: shutting down -46:02.672 [INFO] TestAgent: consul server down -46:02.672 [INFO] TestAgent: shutdown complete -46:02.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11444 network=tcp -46:02.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11444 network=udp -46:02.672 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11445 network=tcp -46:03.172 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.172 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_PassingFilter (0.87s) +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:14.728Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.743Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.812 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 Address:127.0.0.1:29687}]" +31:14.813 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0.dc1 127.0.0.1 +31:14.813 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 127.0.0.1 +2021-10-05T16:31:14.813Z [INFO] agent.router: Initializing LAN area manager +31:14.813 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29682 network=udp +31:14.813 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29687 [Follower]" leader= +31:14.814 [INFO] TestAgent.server: Adding LAN server: server="Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 (Addr: tcp/127.0.0.1:29687) (DC: dc1)" +31:14.814 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29682 network=tcp +31:14.814 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0.dc1 area=wan +31:14.814 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29683 network=tcp +31:14.814 [INFO] TestAgent: started state syncer +31:14.814 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29688 network=tcp +31:14.869 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.869 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29687 [Candidate]" term=2 +31:14.883 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.883 [DEBUG] TestAgent.server.raft: vote granted: from=5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 term=2 tally=1 +31:14.883 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.883 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29687 [Leader]" +31:14.883 [INFO] TestAgent.server: cluster leadership acquired +31:14.883 [INFO] TestAgent.server: New leader elected: payload=Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 +31:14.897 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29687 +31:14.913 [WARN] TestAgent.server.fsm: EnsureRegistration failed: error="failed inserting check: Missing service registration on node "Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0"" +31:14.933 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.956 [WARN] TestAgent.server.fsm: EnsureRegistration failed: error="failed inserting check: Missing service registration on node "Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0"" +31:14.973 [INFO] TestAgent: Synced node info +31:14.992 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.992 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.992 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.992 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.992 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.992 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 +31:14.992 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0 +31:15.020 [WARN] TestAgent.server.fsm: EnsureRegistration failed: error="failed inserting check: Missing service registration on node "Node-5ccc2f2b-f2ff-26b4-69be-3704e415cdb0"" +31:15.020 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.069 [INFO] TestAgent: Requesting shutdown +31:15.069 [INFO] TestAgent.server: shutting down server +31:15.069 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.069 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.069 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.069 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.069 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.069 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.069 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.069 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.069 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.069 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.079 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.092Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.092Z [INFO] agent.router.manager: shutting down +31:15.092 [INFO] TestAgent: consul server down +31:15.092 [INFO] TestAgent: shutdown complete +31:15.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29682 network=tcp +31:15.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29682 network=udp +31:15.092 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29683 network=tcp +31:15.592 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.592 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_PassingFilter (0.89s) --- PASS: TestHealthServiceNodes_PassingFilter/bc_no_query_value (0.00s) --- PASS: TestHealthServiceNodes_PassingFilter/passing_true (0.00s) --- PASS: TestHealthServiceNodes_PassingFilter/passing_false (0.00s) --- PASS: TestHealthServiceNodes_PassingFilter/passing_bad (0.00s) +=== CONT TestHealthServiceChecks_Filtering +2021-10-05T16:31:14.146Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.165Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8166930-c19f-86e3-970d-ce8945e154f8 Address:127.0.0.1:29631}]" +31:14.253 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8166930-c19f-86e3-970d-ce8945e154f8.dc1 127.0.0.1 +31:14.253 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8166930-c19f-86e3-970d-ce8945e154f8 127.0.0.1 +2021-10-05T16:31:14.253Z [INFO] agent.router: Initializing LAN area manager +31:14.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29640 network=udp +31:14.253 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29631 [Follower]" leader= +31:14.254 [INFO] TestAgent.server: Adding LAN server: server="Node-b8166930-c19f-86e3-970d-ce8945e154f8 (Addr: tcp/127.0.0.1:29631) (DC: dc1)" +31:14.254 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8166930-c19f-86e3-970d-ce8945e154f8.dc1 area=wan +31:14.254 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29640 network=tcp +31:14.254 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29641 network=tcp +31:14.254 [INFO] TestAgent: started state syncer +31:14.254 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29632 network=tcp +31:14.300 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.300 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29631 [Candidate]" term=2 +31:14.348 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.348 [DEBUG] TestAgent.server.raft: vote granted: from=b8166930-c19f-86e3-970d-ce8945e154f8 term=2 tally=1 +31:14.348 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.348 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29631 [Leader]" +31:14.348 [INFO] TestAgent.server: cluster leadership acquired +31:14.348 [INFO] TestAgent.server: New leader elected: payload=Node-b8166930-c19f-86e3-970d-ce8945e154f8 +31:14.378 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29631 +31:14.397 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.484 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.484 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.484 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.484 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.484 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.484 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8166930-c19f-86e3-970d-ce8945e154f8 +31:14.484 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8166930-c19f-86e3-970d-ce8945e154f8 +31:14.532 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.588 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:14.600 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=3.596µs +31:14.600 [INFO] TestAgent: Requesting shutdown +31:14.600 [INFO] TestAgent.server: shutting down server +31:14.600 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:14.600 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:14.600 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:14.600 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:14.600 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:14.600 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:14.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:14.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:14.600 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:14.600 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:14.604 [INFO] TestAgent: Synced node info +31:14.608 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:14.616Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:14.616Z [INFO] agent.router.manager: shutting down +31:14.616 [INFO] TestAgent: consul server down +31:14.616 [INFO] TestAgent: shutdown complete +31:14.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29640 network=tcp +31:14.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29640 network=udp +31:14.616 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29641 network=tcp +31:15.120 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.120 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:13.568Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:13.582Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:13.680 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1bc6abe1-563e-0dcf-6193-6ed50c728cd0 Address:127.0.0.1:29596}]" +31:13.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0.dc1 127.0.0.1 +31:13.682 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0 127.0.0.1 +2021-10-05T16:31:13.682Z [INFO] agent.router: Initializing LAN area manager +31:13.684 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29596 [Follower]" leader= +31:13.685 [INFO] TestAgent.server: Adding LAN server: server="Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0 (Addr: tcp/127.0.0.1:29596) (DC: dc1)" +31:13.685 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0.dc1 area=wan +31:13.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29591 network=tcp +31:13.686 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29591 network=udp +31:13.687 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29592 network=tcp +31:13.687 [INFO] TestAgent: started state syncer +31:13.687 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29597 network=tcp +31:13.728 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:13.728 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29596 [Candidate]" term=2 +31:13.776 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:13.776 [DEBUG] TestAgent.server.raft: vote granted: from=1bc6abe1-563e-0dcf-6193-6ed50c728cd0 term=2 tally=1 +31:13.776 [INFO] TestAgent.server.raft: election won: tally=1 +31:13.776 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29596 [Leader]" +31:13.778 [INFO] TestAgent.server: cluster leadership acquired +31:13.778 [INFO] TestAgent.server: New leader elected: payload=Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0 +31:13.804 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29596 +31:13.844 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:13.912 [INFO] TestAgent: Synced node info +31:13.912 [DEBUG] TestAgent: Node info in sync +31:13.912 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:13.912 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:13.912 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:13.912 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:13.912 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:13.912 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0 +31:13.912 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1bc6abe1-563e-0dcf-6193-6ed50c728cd0 +31:13.940 [INFO] TestAgent.server: federation state anti-entropy synced +31:14.112 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from= latency=1.845µs +31:14.666 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:14.666 [DEBUG] TestAgent: Node info in sync +31:15.120 [INFO] TestAgent: Requesting shutdown +31:15.120 [INFO] TestAgent.server: shutting down server +31:15.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.120 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.120 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.120 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.120 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.120 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.120 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.125 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.133Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.133Z [INFO] agent.router.manager: shutting down +31:15.133 [INFO] TestAgent: consul server down +31:15.133 [INFO] TestAgent: shutdown complete +31:15.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29591 network=tcp +31:15.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29591 network=udp +31:15.133 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29592 network=tcp +31:15.633 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.633 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPAPI_TranslateAddrHeader (2.10s) +=== CONT TestHealthServiceChecks_NodeMetaFilter +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestEventList_ACLFilter/no_token +=== RUN TestEventList_ACLFilter/root_token +2021-10-05T16:31:14.935Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.952Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.010 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:26e77cfe-3e47-fed6-e39e-129a61fd44c4 Address:127.0.0.1:29694}]" +31:15.011 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4.dc1 127.0.0.1 +31:15.012 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4 127.0.0.1 +2021-10-05T16:31:15.012Z [INFO] agent.router: Initializing LAN area manager +31:15.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29689 network=udp +31:15.012 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29694 [Follower]" leader= +31:15.012 [INFO] TestAgent.server: Adding LAN server: server="Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4 (Addr: tcp/127.0.0.1:29694) (DC: dc1)" +31:15.012 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4.dc1 area=wan +31:15.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29689 network=tcp +31:15.013 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29690 network=tcp +31:15.013 [INFO] TestAgent: started state syncer +31:15.013 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29695 network=tcp +31:15.069 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.069 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29694 [Candidate]" term=2 +31:15.092 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.092 [DEBUG] TestAgent.server.raft: vote granted: from=26e77cfe-3e47-fed6-e39e-129a61fd44c4 term=2 tally=1 +31:15.092 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.092 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29694 [Leader]" +31:15.095 [INFO] TestAgent.server: cluster leadership acquired +31:15.095 [INFO] TestAgent.server: New leader elected: payload=Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4 +31:15.104 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29694 +31:15.122 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.187 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.188 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.188 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.188 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.188 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.188 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4 +31:15.188 [INFO] TestAgent.server: member joined, marking health alive: member=Node-26e77cfe-3e47-fed6-e39e-129a61fd44c4 +31:15.215 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.388 [INFO] TestAgent: Requesting shutdown +31:15.388 [INFO] TestAgent.server: shutting down server +31:15.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.388 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.388 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.388 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.388 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.388 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:15.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.388 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.388 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.400 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.412Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.412Z [INFO] agent.router.manager: shutting down +31:15.412 [INFO] TestAgent: consul server down +31:15.412 [INFO] TestAgent: shutdown complete +31:15.412 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29689 network=tcp +31:15.412 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29689 network=udp +31:15.412 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29690 network=tcp +31:15.913 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.913 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_DistanceSort (0.99s) +=== CONT TestHealthServiceChecks +2021-10-05T16:31:15.031Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.051Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.099 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2f67b19e-d30f-9eeb-1695-83490da357d9 Address:127.0.0.1:29701}]" +31:15.100 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2f67b19e-d30f-9eeb-1695-83490da357d9.dc1 127.0.0.1 +31:15.100 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2f67b19e-d30f-9eeb-1695-83490da357d9 127.0.0.1 +2021-10-05T16:31:15.100Z [INFO] agent.router: Initializing LAN area manager +31:15.100 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29696 network=udp +31:15.100 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29701 [Follower]" leader= +31:15.101 [INFO] TestAgent.server: Adding LAN server: server="Node-2f67b19e-d30f-9eeb-1695-83490da357d9 (Addr: tcp/127.0.0.1:29701) (DC: dc1)" +31:15.101 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2f67b19e-d30f-9eeb-1695-83490da357d9.dc1 area=wan +31:15.101 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29696 network=tcp +31:15.101 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29697 network=tcp +31:15.101 [INFO] TestAgent: started state syncer +31:15.101 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29702 network=tcp +31:15.140 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.140 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29701 [Candidate]" term=2 +31:15.180 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.180 [DEBUG] TestAgent.server.raft: vote granted: from=2f67b19e-d30f-9eeb-1695-83490da357d9 term=2 tally=1 +31:15.180 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.180 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29701 [Leader]" +31:15.184 [INFO] TestAgent.server: cluster leadership acquired +31:15.184 [INFO] TestAgent.server: New leader elected: payload=Node-2f67b19e-d30f-9eeb-1695-83490da357d9 +31:15.203 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29701 +31:15.235 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.292 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.292 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.292 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.292 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.292 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.292 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2f67b19e-d30f-9eeb-1695-83490da357d9 +31:15.292 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2f67b19e-d30f-9eeb-1695-83490da357d9 +31:15.310 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.377 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:15.382 [INFO] TestAgent: Synced node info +31:15.463 [INFO] TestAgent: Requesting shutdown +31:15.463 [INFO] TestAgent.server: shutting down server +31:15.463 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.463 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.463 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.463 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.463 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.463 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.463 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.463 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.463 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.463 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.469 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.475Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.475Z [INFO] agent.router.manager: shutting down +31:15.476 [INFO] TestAgent: consul server down +31:15.476 [INFO] TestAgent: shutdown complete +31:15.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29696 network=tcp +31:15.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29696 network=udp +31:15.476 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29697 network=tcp +31:15.976 [INFO] TestAgent: Waiting for endpoints to shut down +31:15.976 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_Filter (0.97s) +=== CONT TestHealthNodeChecks_Filtering +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:15.427Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.454Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.486 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6b6be8c3-242c-cf7a-bce3-43331b03569f Address:127.0.0.1:29736}]" +31:15.487 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6b6be8c3-242c-cf7a-bce3-43331b03569f.dc1 127.0.0.1 +31:15.487 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6b6be8c3-242c-cf7a-bce3-43331b03569f 127.0.0.1 +2021-10-05T16:31:15.487Z [INFO] agent.router: Initializing LAN area manager +31:15.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29724 network=udp +31:15.488 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29736 [Follower]" leader= +31:15.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29724 network=tcp +31:15.488 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6b6be8c3-242c-cf7a-bce3-43331b03569f.dc1 area=wan +31:15.488 [INFO] TestAgent.server: Adding LAN server: server="Node-6b6be8c3-242c-cf7a-bce3-43331b03569f (Addr: tcp/127.0.0.1:29736) (DC: dc1)" +31:15.488 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29725 network=tcp +31:15.488 [INFO] TestAgent: started state syncer +31:15.488 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29737 network=tcp +31:15.548 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.548 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29736 [Candidate]" term=2 +31:15.604 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.604 [DEBUG] TestAgent.server.raft: vote granted: from=6b6be8c3-242c-cf7a-bce3-43331b03569f term=2 tally=1 +31:15.604 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.604 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29736 [Leader]" +31:15.604 [INFO] TestAgent.server: cluster leadership acquired +31:15.604 [INFO] TestAgent.server: New leader elected: payload=Node-6b6be8c3-242c-cf7a-bce3-43331b03569f +31:15.632 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:15.632 [INFO] TestAgent: Requesting shutdown +31:15.632 [INFO] TestAgent.server: shutting down server +31:15.632 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.632 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:15.648 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.652Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.652Z [INFO] agent.router.manager: shutting down +31:15.652 [ERROR] TestAgent.server: failed to wait for barrier: error="leadership lost while committing log" +31:15.652 [INFO] TestAgent: consul server down +31:15.652 [INFO] TestAgent: shutdown complete +31:15.652 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29724 network=tcp +31:15.652 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29724 network=udp +31:15.652 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29725 network=tcp +31:16.155 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.155 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#05 +2021-10-05T16:31:15.231Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.248Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.286 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a802ba0c-d8e7-0248-49b1-f2b8cd67b506 Address:127.0.0.1:29715}]" +31:15.286 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506.dc1 127.0.0.1 +31:15.287 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506 127.0.0.1 +2021-10-05T16:31:15.287Z [INFO] agent.router: Initializing LAN area manager +31:15.287 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29710 network=udp +31:15.287 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29715 [Follower]" leader= +31:15.287 [INFO] TestAgent.server: Adding LAN server: server="Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506 (Addr: tcp/127.0.0.1:29715) (DC: dc1)" +31:15.287 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506.dc1 area=wan +31:15.288 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29710 network=tcp +31:15.288 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29711 network=tcp +31:15.288 [INFO] TestAgent: started state syncer +31:15.288 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29716 network=tcp +31:15.324 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.324 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29715 [Candidate]" term=2 +31:15.350 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.350 [DEBUG] TestAgent.server.raft: vote granted: from=a802ba0c-d8e7-0248-49b1-f2b8cd67b506 term=2 tally=1 +31:15.350 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.350 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29715 [Leader]" +31:15.351 [INFO] TestAgent.server: cluster leadership acquired +31:15.351 [INFO] TestAgent.server: New leader elected: payload=Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506 +31:15.356 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29715 +31:15.375 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.408 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.408 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.408 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.408 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.408 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.408 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506 +31:15.408 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a802ba0c-d8e7-0248-49b1-f2b8cd67b506 +31:15.436 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.667 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:15.672 [INFO] TestAgent: Synced node info +31:15.716 [INFO] TestAgent: Requesting shutdown +31:15.716 [INFO] TestAgent.server: shutting down server +31:15.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.716 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.717 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.717 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.717 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.721 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.721Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.721Z [INFO] agent.router.manager: shutting down +31:15.721 [INFO] TestAgent: consul server down +31:15.721 [INFO] TestAgent: shutdown complete +31:15.721 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29710 network=tcp +31:15.721 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29710 network=udp +31:15.721 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29711 network=tcp +31:16.221 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.221 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_NodeMetaFilter (1.02s) +=== CONT TestHealthNodeChecks +2021-10-05T16:31:15.375Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.390Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.447 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:41e49557-ba27-307b-81f6-b4776c5bf700 Address:127.0.0.1:29722}]" +31:15.447 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29722 [Follower]" leader= +31:15.448 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-41e49557-ba27-307b-81f6-b4776c5bf700.dc1 127.0.0.1 +31:15.449 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-41e49557-ba27-307b-81f6-b4776c5bf700 127.0.0.1 +2021-10-05T16:31:15.449Z [INFO] agent.router: Initializing LAN area manager +31:15.449 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29717 network=udp +31:15.449 [INFO] TestAgent.server: Adding LAN server: server="Node-41e49557-ba27-307b-81f6-b4776c5bf700 (Addr: tcp/127.0.0.1:29722) (DC: dc1)" +31:15.449 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-41e49557-ba27-307b-81f6-b4776c5bf700.dc1 area=wan +31:15.449 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29717 network=tcp +31:15.450 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29718 network=tcp +31:15.450 [INFO] TestAgent: started state syncer +31:15.450 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29723 network=tcp +31:15.508 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.508 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29722 [Candidate]" term=2 +31:15.550 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.550 [DEBUG] TestAgent.server.raft: vote granted: from=41e49557-ba27-307b-81f6-b4776c5bf700 term=2 tally=1 +31:15.550 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.550 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29722 [Leader]" +31:15.556 [INFO] TestAgent.server: cluster leadership acquired +31:15.556 [INFO] TestAgent.server: New leader elected: payload=Node-41e49557-ba27-307b-81f6-b4776c5bf700 +31:15.582 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29722 +31:15.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.704 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.704 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.704 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.704 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.704 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.704 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-41e49557-ba27-307b-81f6-b4776c5bf700 +31:15.704 [INFO] TestAgent.server: member joined, marking health alive: member=Node-41e49557-ba27-307b-81f6-b4776c5bf700 +31:15.724 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.754 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:15.770 [INFO] TestAgent: Synced node info +31:15.770 [DEBUG] TestAgent: Node info in sync +31:15.780 [INFO] TestAgent: Requesting shutdown +31:15.780 [INFO] TestAgent.server: shutting down server +31:15.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.780 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.780 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.780 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.780 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:15.780 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.780 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.780 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.780 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.786 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.788Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.788Z [INFO] agent.router.manager: shutting down +31:15.788 [INFO] TestAgent: consul server down +31:15.788 [INFO] TestAgent: shutdown complete +31:15.788 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29717 network=tcp +31:15.788 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29717 network=udp +31:15.788 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29718 network=tcp +31:16.289 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.289 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes (0.93s) +=== CONT TestHealthChecksInState_DistanceSort +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:15.572Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:15.572Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.586Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:15.586Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.660 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:11dd8592-aac3-1bec-f749-fd3352d02085 Address:127.0.0.1:29743}]" +31:15.661 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-11dd8592-aac3-1bec-f749-fd3352d02085.dc1 127.0.0.1 +31:15.661 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-11dd8592-aac3-1bec-f749-fd3352d02085 127.0.0.1 +2021-10-05T16:31:15.661Z [INFO] agent.router: Initializing LAN area manager +31:15.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29731 network=udp +31:15.662 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29743 [Follower]" leader= +31:15.662 [INFO] TestAgent.server: Adding LAN server: server="Node-11dd8592-aac3-1bec-f749-fd3352d02085 (Addr: tcp/127.0.0.1:29743) (DC: dc1)" +31:15.662 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-11dd8592-aac3-1bec-f749-fd3352d02085.dc1 area=wan +31:15.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29731 network=tcp +31:15.663 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29732 network=tcp +31:15.663 [INFO] TestAgent: started state syncer +31:15.663 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29744 network=tcp +31:15.709 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.709 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29743 [Candidate]" term=2 +31:15.712 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:15.736 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.736 [DEBUG] TestAgent.server.raft: vote granted: from=11dd8592-aac3-1bec-f749-fd3352d02085 term=2 tally=1 +31:15.736 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.736 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29743 [Leader]" +31:15.736 [INFO] TestAgent.server: cluster leadership acquired +31:15.736 [INFO] TestAgent.server: New leader elected: payload=Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.744 [INFO] TestAgent.server: initializing acls +31:15.748 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:15.748 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:15.758 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:15.767 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:15.767 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:15.767 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:15.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-11dd8592-aac3-1bec-f749-fd3352d02085.dc1 +31:15.767 [INFO] TestAgent.server: Updating LAN server: server="Node-11dd8592-aac3-1bec-f749-fd3352d02085 (Addr: tcp/127.0.0.1:29743) (DC: dc1)" +31:15.767 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-11dd8592-aac3-1bec-f749-fd3352d02085.dc1 area=wan +31:15.778 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.811 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.811 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.811 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.811 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.811 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.811 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.811 [INFO] TestAgent.server: member joined, marking health alive: member=Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.840 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.854 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-11dd8592-aac3-1bec-f749-fd3352d02085 +31:15.855 [DEBUG] TestAgent.server: User event: event=foo +31:15.855 [DEBUG] TestAgent: new event: event_name=foo event_id=95f52d37-6651-410d-3bd7-054f4e9bc29f +31:15.855 [INFO] TestAgent: Requesting shutdown +31:15.855 [INFO] TestAgent.server: shutting down server +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:15.855 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:15.855 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.855 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:15.856 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:15.856 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:15.856 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:15.856 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:15.856 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:15.860 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.860 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:15.864 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.865Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.865Z [INFO] agent.router.manager: shutting down +31:15.865 [INFO] TestAgent: consul server down +31:15.865 [INFO] TestAgent: shutdown complete +31:15.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29731 network=tcp +31:15.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29731 network=udp +31:15.865 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29732 network=tcp +31:16.366 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.366 [INFO] TestAgent: Endpoints down +--- PASS: TestEventList_ACLFilter (0.81s) + --- PASS: TestEventList_ACLFilter/no_token (0.00s) + --- PASS: TestEventList_ACLFilter/root_token (0.00s) +=== CONT TestHealthChecksInState_Filter +2021-10-05T16:31:15.679Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.708Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.801 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d73f0ebc-50da-f118-b6ee-7fc09def1220 Address:127.0.0.1:29757}]" +31:15.802 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29757 [Follower]" leader= +31:15.804 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d73f0ebc-50da-f118-b6ee-7fc09def1220.dc1 127.0.0.1 +31:15.804 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d73f0ebc-50da-f118-b6ee-7fc09def1220 127.0.0.1 +2021-10-05T16:31:15.804Z [INFO] agent.router: Initializing LAN area manager +31:15.804 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29752 network=udp +31:15.805 [INFO] TestAgent.server: Adding LAN server: server="Node-d73f0ebc-50da-f118-b6ee-7fc09def1220 (Addr: tcp/127.0.0.1:29757) (DC: dc1)" +31:15.805 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d73f0ebc-50da-f118-b6ee-7fc09def1220.dc1 area=wan +31:15.805 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29752 network=tcp +31:15.805 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29753 network=tcp +31:15.805 [INFO] TestAgent: started state syncer +31:15.805 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29758 network=tcp +31:15.848 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.848 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29757 [Candidate]" term=2 +31:15.864 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.864 [DEBUG] TestAgent.server.raft: vote granted: from=d73f0ebc-50da-f118-b6ee-7fc09def1220 term=2 tally=1 +31:15.864 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.864 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29757 [Leader]" +31:15.864 [INFO] TestAgent.server: cluster leadership acquired +31:15.864 [INFO] TestAgent.server: New leader elected: payload=Node-d73f0ebc-50da-f118-b6ee-7fc09def1220 +31:15.882 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29757 +31:15.900 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.965 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.965 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.965 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.965 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.965 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.965 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d73f0ebc-50da-f118-b6ee-7fc09def1220 +31:15.965 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d73f0ebc-50da-f118-b6ee-7fc09def1220 +31:15.982 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.002 [INFO] TestAgent: Requesting shutdown +31:16.002 [INFO] TestAgent.server: shutting down server +31:16.002 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.002 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.002 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.002 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.002 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.002 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.002 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.002 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.003 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.003 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.003 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.010 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.011Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.011Z [INFO] agent.router.manager: shutting down +31:16.012 [INFO] TestAgent: consul server down +31:16.012 [INFO] TestAgent: shutdown complete +31:16.012 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29752 network=tcp +31:16.012 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29752 network=udp +31:16.012 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29753 network=tcp +31:16.513 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.513 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceChecks_NodeMetaFilter (0.88s) +=== CONT TestHealthChecksInState_NodeMetaFilter +2021-10-05T16:31:15.207Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:15.207Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.239Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:15.239Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.332 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 Address:127.0.0.1:29708}]" +31:15.333 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 127.0.0.1 +31:15.333 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 127.0.0.1 +2021-10-05T16:31:15.333Z [INFO] agent.router: Initializing LAN area manager +31:15.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29703 network=udp +31:15.333 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29708 [Follower]" leader= +31:15.334 [INFO] TestAgent.server: Adding LAN server: server="Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 (Addr: tcp/127.0.0.1:29708) (DC: dc2)" +31:15.334 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 area=wan +31:15.334 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29703 network=tcp +31:15.335 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29704 network=tcp +31:15.335 [INFO] TestAgent: started state syncer +31:15.335 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29709 network=tcp +31:15.383 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.383 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29708 [Candidate]" term=2 +31:15.388 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:15.412 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.412 [DEBUG] TestAgent.server.raft: vote granted: from=7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 term=2 tally=1 +31:15.412 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.412 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29708 [Leader]" +31:15.412 [INFO] TestAgent.server: cluster leadership acquired +31:15.412 [INFO] TestAgent.server: New leader elected: payload=Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 +31:15.444 [INFO] TestAgent.server: initializing acls +31:15.458 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:15.466 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:15.466 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:15.466 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:15.466 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 +31:15.466 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 +31:15.467 [INFO] TestAgent.server: Updating LAN server: server="Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7 (Addr: tcp/127.0.0.1:29708) (DC: dc2)" +31:15.467 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 area=wan +31:15.468 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29672] +31:15.471 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29672 +31:15.472 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 127.0.0.1 +31:15.472 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:15.472 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 area=wan +31:15.500 [INFO] TestAgent: Requesting shutdown +31:15.500 [INFO] TestAgent.server: shutting down server +31:15.500 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:15.500 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:15.500 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:15.500 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:15.500 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:15.500 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:15.508 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.519 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:15.529Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.529Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:15.529Z [INFO] agent.router.manager: shutting down +31:15.531 [INFO] TestAgent: consul server down +31:15.531 [INFO] TestAgent: shutdown complete +31:15.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29703 network=tcp +31:15.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29703 network=udp +31:15.531 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29704 network=tcp +31:15.531 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +31:15.531 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:15.531 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:16.036 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.036 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:14.617Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:14.617Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:14.632Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:14.632Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:14.724 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a54e0129-5381-35c7-da81-f1f85d99a931 Address:127.0.0.1:29673}]" +31:14.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 127.0.0.1 +31:14.726 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a54e0129-5381-35c7-da81-f1f85d99a931 127.0.0.1 +2021-10-05T16:31:14.726Z [INFO] agent.router: Initializing LAN area manager +31:14.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29668 network=udp +31:14.726 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29673 [Follower]" leader= +31:14.726 [INFO] TestAgent.server: Adding LAN server: server="Node-a54e0129-5381-35c7-da81-f1f85d99a931 (Addr: tcp/127.0.0.1:29673) (DC: dc1)" +31:14.726 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 area=wan +31:14.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29668 network=tcp +31:14.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29669 network=tcp +31:14.727 [INFO] TestAgent: started state syncer +31:14.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29674 network=tcp +31:14.772 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:14.772 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29673 [Candidate]" term=2 +31:14.780 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:14.820 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:14.820 [DEBUG] TestAgent.server.raft: vote granted: from=a54e0129-5381-35c7-da81-f1f85d99a931 term=2 tally=1 +31:14.820 [INFO] TestAgent.server.raft: election won: tally=1 +31:14.820 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29673 [Leader]" +31:14.820 [INFO] TestAgent.server: cluster leadership acquired +31:14.820 [INFO] TestAgent.server: New leader elected: payload=Node-a54e0129-5381-35c7-da81-f1f85d99a931 +31:14.835 [INFO] TestAgent.server: initializing acls +31:14.839 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:14.850 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:14.850 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:14.850 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:14.850 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a54e0129-5381-35c7-da81-f1f85d99a931 +31:14.850 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 +31:14.851 [INFO] TestAgent.server: Updating LAN server: server="Node-a54e0129-5381-35c7-da81-f1f85d99a931 (Addr: tcp/127.0.0.1:29673) (DC: dc1)" +31:14.851 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a54e0129-5381-35c7-da81-f1f85d99a931.dc1 area=wan +31:14.873 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:14.900 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:14.900 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:14.900 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:14.900 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:14.900 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:14.900 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a54e0129-5381-35c7-da81-f1f85d99a931 +31:14.900 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a54e0129-5381-35c7-da81-f1f85d99a931 +31:14.914 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a54e0129-5381-35c7-da81-f1f85d99a931 +31:14.925 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.086 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:15.095 [INFO] TestAgent: Synced node info +31:15.471 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:44032 +31:15.472 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 127.0.0.1 +31:15.472 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7f91a93c-ab77-b65b-e6ce-bcc4804f16e7.dc2 area=wan +31:16.036 [INFO] TestAgent: Requesting shutdown +31:16.037 [INFO] TestAgent.server: shutting down server +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.037 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.037 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.037 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.037 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.050 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.052Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.052Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.052Z [INFO] agent.router.manager: shutting down +31:16.052 [INFO] TestAgent: consul server down +31:16.052 [INFO] TestAgent: shutdown complete +31:16.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29668 network=tcp +31:16.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29668 network=udp +31:16.052 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29669 network=tcp +31:16.552 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.552 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceNodes_WanTranslation (1.95s) +=== CONT TestUUIDToUint64 +--- PASS: TestUUIDToUint64 (0.00s) +=== CONT TestEventList_EventBufOrder +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:15.565Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.580Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.674 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3d77c4fe-1936-7065-f683-e1b28eb8a72f Address:127.0.0.1:29729}]" +31:15.674 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f.dc1 127.0.0.1 +31:15.675 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f 127.0.0.1 +2021-10-05T16:31:15.675Z [INFO] agent.router: Initializing LAN area manager +31:15.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29738 network=udp +31:15.675 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29729 [Follower]" leader= +31:15.675 [INFO] TestAgent.server: Adding LAN server: server="Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f (Addr: tcp/127.0.0.1:29729) (DC: dc1)" +31:15.675 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f.dc1 area=wan +31:15.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29738 network=tcp +31:15.676 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29739 network=tcp +31:15.676 [INFO] TestAgent: started state syncer +31:15.676 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29730 network=tcp +31:15.733 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.733 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29729 [Candidate]" term=2 +31:15.768 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.768 [DEBUG] TestAgent.server.raft: vote granted: from=3d77c4fe-1936-7065-f683-e1b28eb8a72f term=2 tally=1 +31:15.768 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.768 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29729 [Leader]" +31:15.768 [INFO] TestAgent.server: cluster leadership acquired +31:15.768 [INFO] TestAgent.server: New leader elected: payload=Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f +31:15.790 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29729 +31:15.826 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.907 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.907 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.907 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.907 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.907 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.907 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f +31:15.907 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3d77c4fe-1936-7065-f683-e1b28eb8a72f +31:15.960 [INFO] TestAgent.server: federation state anti-entropy synced +31:15.998 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:16.007 [INFO] TestAgent: Synced node info +31:16.209 [INFO] TestAgent: Requesting shutdown +31:16.209 [INFO] TestAgent.server: shutting down server +31:16.209 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.209 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.209 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.209 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.209 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.209 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.209 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.209 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.209 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.209 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.216 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.216Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.216Z [INFO] agent.router.manager: shutting down +31:16.216 [INFO] TestAgent: consul server down +31:16.216 [INFO] TestAgent: shutdown complete +31:16.216 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29738 network=tcp +31:16.216 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29738 network=udp +31:16.216 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29739 network=tcp +31:16.719 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.719 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceChecks_DistanceSort (1.17s) +=== CONT TestEventList_Blocking +2021-10-05T16:31:15.635Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.678Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.758 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2c188c00-c26d-b8dd-12a7-e1284c1324f0 Address:127.0.0.1:29750}]" +31:15.759 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29750 [Follower]" leader= +31:15.759 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0.dc1 127.0.0.1 +31:15.760 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0 127.0.0.1 +2021-10-05T16:31:15.760Z [INFO] agent.router: Initializing LAN area manager +31:15.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29745 network=udp +31:15.761 [INFO] TestAgent.server: Adding LAN server: server="Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0 (Addr: tcp/127.0.0.1:29750) (DC: dc1)" +31:15.761 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0.dc1 area=wan +31:15.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29745 network=tcp +31:15.761 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29746 network=tcp +31:15.761 [INFO] TestAgent: started state syncer +31:15.761 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29751 network=tcp +31:15.828 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:15.828 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29750 [Candidate]" term=2 +31:15.847 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:15.847 [DEBUG] TestAgent.server.raft: vote granted: from=2c188c00-c26d-b8dd-12a7-e1284c1324f0 term=2 tally=1 +31:15.847 [INFO] TestAgent.server.raft: election won: tally=1 +31:15.847 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29750 [Leader]" +31:15.847 [INFO] TestAgent.server: cluster leadership acquired +31:15.848 [INFO] TestAgent.server: New leader elected: payload=Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0 +31:15.855 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29750 +31:15.894 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:15.959 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:15.959 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:15.959 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:15.959 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:15.959 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:15.959 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0 +31:15.959 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2c188c00-c26d-b8dd-12a7-e1284c1324f0 +31:15.996 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.075 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:16.083 [INFO] TestAgent: Synced node info +31:16.083 [DEBUG] TestAgent: Node info in sync +31:16.211 [INFO] TestAgent: Requesting shutdown +31:16.211 [INFO] TestAgent.server: shutting down server +31:16.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.211 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.211 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.211 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.211 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.211 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.211 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.221 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.224Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.224Z [INFO] agent.router.manager: shutting down +31:16.224 [INFO] TestAgent: consul server down +31:16.224 [INFO] TestAgent: shutdown complete +31:16.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29745 network=tcp +31:16.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29745 network=udp +31:16.224 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29746 network=tcp +31:16.727 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.727 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceChecks_Filtering (1.14s) === CONT TestEventList_Filter -2022-11-07T22:46:02.455Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.463Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.511 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fb39b837-1f8b-0991-8d1e-039452e2325d Address:127.0.0.1:11505}]" -46:02.511 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11505 [Follower]" leader= -46:02.511 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fb39b837-1f8b-0991-8d1e-039452e2325d.dc1 127.0.0.1 -46:02.512 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fb39b837-1f8b-0991-8d1e-039452e2325d 127.0.0.1 -2022-11-07T22:46:02.512Z [INFO] agent.router: Initializing LAN area manager -46:02.512 [INFO] TestAgent.server: Adding LAN server: server="Node-fb39b837-1f8b-0991-8d1e-039452e2325d (Addr: tcp/127.0.0.1:11505) (DC: dc1)" -46:02.512 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fb39b837-1f8b-0991-8d1e-039452e2325d.dc1 area=wan -46:02.513 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11500 network=udp -46:02.513 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11500 network=tcp -46:02.513 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11501 network=tcp -46:02.513 [INFO] TestAgent: started state syncer -46:02.513 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11506 network=tcp -46:02.546 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.546 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11505 [Candidate]" term=2 -46:02.560 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.560 [DEBUG] TestAgent.server.raft: vote granted: from=fb39b837-1f8b-0991-8d1e-039452e2325d term=2 tally=1 -46:02.560 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.560 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11505 [Leader]" -46:02.560 [INFO] TestAgent.server: cluster leadership acquired -46:02.560 [INFO] TestAgent.server: New leader elected: payload=Node-fb39b837-1f8b-0991-8d1e-039452e2325d -46:02.571 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11505 -46:02.592 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.625 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.625 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.625 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.625 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.625 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.625 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fb39b837-1f8b-0991-8d1e-039452e2325d -46:02.625 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fb39b837-1f8b-0991-8d1e-039452e2325d -46:02.630 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.674 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.675 [INFO] TestAgent: Requesting shutdown -46:02.675 [INFO] TestAgent.server: shutting down server -46:02.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.675 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.675 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.675 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.675 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.675 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.675 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.677 [INFO] TestAgent: Synced node info -46:02.679 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.686Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.686Z [INFO] agent.router.manager: shutting down -46:02.686 [INFO] TestAgent: consul server down -46:02.686 [INFO] TestAgent: shutdown complete -46:02.686 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11500 network=tcp -46:02.686 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11500 network=udp -46:02.686 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11501 network=tcp -46:03.186 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.186 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceChecks_DistanceSort (0.74s) +2021-10-05T16:31:15.928Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:15.942Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:15.989 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:012636c4-6ceb-9afa-b6f0-f364c5de1150 Address:127.0.0.1:29764}]" +31:15.990 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-012636c4-6ceb-9afa-b6f0-f364c5de1150.dc1 127.0.0.1 +31:15.991 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-012636c4-6ceb-9afa-b6f0-f364c5de1150 127.0.0.1 +2021-10-05T16:31:15.991Z [INFO] agent.router: Initializing LAN area manager +31:15.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29764 [Follower]" leader= +31:15.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29759 network=udp +31:15.991 [INFO] TestAgent.server: Adding LAN server: server="Node-012636c4-6ceb-9afa-b6f0-f364c5de1150 (Addr: tcp/127.0.0.1:29764) (DC: dc1)" +31:15.991 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-012636c4-6ceb-9afa-b6f0-f364c5de1150.dc1 area=wan +31:15.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29759 network=tcp +31:15.992 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29760 network=tcp +31:15.992 [INFO] TestAgent: started state syncer +31:15.992 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29765 network=tcp +31:16.049 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.049 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29764 [Candidate]" term=2 +31:16.061 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.061 [DEBUG] TestAgent.server.raft: vote granted: from=012636c4-6ceb-9afa-b6f0-f364c5de1150 term=2 tally=1 +31:16.061 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.061 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29764 [Leader]" +31:16.062 [INFO] TestAgent.server: cluster leadership acquired +31:16.062 [INFO] TestAgent.server: New leader elected: payload=Node-012636c4-6ceb-9afa-b6f0-f364c5de1150 +31:16.068 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29764 +31:16.086 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.119 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.119 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.119 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.119 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.119 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.119 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-012636c4-6ceb-9afa-b6f0-f364c5de1150 +31:16.119 [INFO] TestAgent.server: member joined, marking health alive: member=Node-012636c4-6ceb-9afa-b6f0-f364c5de1150 +31:16.148 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.245 [INFO] TestAgent: Requesting shutdown +31:16.245 [INFO] TestAgent.server: shutting down server +31:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.245 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.245 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.245 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.251 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.255Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.255Z [INFO] agent.router.manager: shutting down +31:16.255 [INFO] TestAgent: consul server down +31:16.255 [INFO] TestAgent: shutdown complete +31:16.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29759 network=tcp +31:16.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29759 network=udp +31:16.256 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29760 network=tcp +31:16.756 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.756 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthServiceChecks (0.84s) +=== CONT TestEventList +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:16.169Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.224 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:155ff33b-d8d9-6582-9c89-92ecda08ebe2 Address:127.0.0.1:29785}]" +31:16.225 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29785 [Follower]" leader= +31:16.225 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-155ff33b-d8d9-6582-9c89-92ecda08ebe2.dc1 127.0.0.1 +31:16.226 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-155ff33b-d8d9-6582-9c89-92ecda08ebe2 127.0.0.1 +2021-10-05T16:31:16.226Z [INFO] agent.router: Initializing LAN area manager +31:16.226 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29780 network=udp +31:16.226 [INFO] TestAgent.server: Adding LAN server: server="Node-155ff33b-d8d9-6582-9c89-92ecda08ebe2 (Addr: tcp/127.0.0.1:29785) (DC: dc1)" +31:16.226 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-155ff33b-d8d9-6582-9c89-92ecda08ebe2.dc1 area=wan +31:16.226 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29780 network=tcp +31:16.226 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29781 network=tcp +31:16.227 [INFO] TestAgent: started state syncer +31:16.227 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29786 network=tcp +31:16.284 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.284 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29785 [Candidate]" term=2 +31:16.299 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.299 [DEBUG] TestAgent.server.raft: vote granted: from=155ff33b-d8d9-6582-9c89-92ecda08ebe2 term=2 tally=1 +31:16.299 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.299 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29785 [Leader]" +31:16.300 [INFO] TestAgent.server: cluster leadership acquired +31:16.300 [INFO] TestAgent.server: New leader elected: payload=Node-155ff33b-d8d9-6582-9c89-92ecda08ebe2 +31:16.306 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29785 +31:16.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.372 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:16.372 [INFO] TestAgent: Requesting shutdown +31:16.372 [INFO] TestAgent.server: shutting down server +31:16.372 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.372 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.384 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.388Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.388Z [INFO] agent.router.manager: shutting down +31:16.396 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +31:16.396 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:16.396 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:16.396 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:16.396 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:16.396 [INFO] TestAgent: consul server down +31:16.396 [INFO] TestAgent: shutdown complete +31:16.396 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29780 network=tcp +31:16.396 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29780 network=udp +31:16.396 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29781 network=tcp +31:16.900 [INFO] TestAgent: Waiting for endpoints to shut down +31:16.900 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#06 +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:16.238Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.257Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.332 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:307c4b64-1028-bf98-b5ae-1eeac81b322c Address:127.0.0.1:29834}]" +31:16.333 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-307c4b64-1028-bf98-b5ae-1eeac81b322c.dc1 127.0.0.1 +31:16.333 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-307c4b64-1028-bf98-b5ae-1eeac81b322c 127.0.0.1 +2021-10-05T16:31:16.333Z [INFO] agent.router: Initializing LAN area manager +31:16.333 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29834 [Follower]" leader= +31:16.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29787 network=udp +31:16.334 [INFO] TestAgent.server: Adding LAN server: server="Node-307c4b64-1028-bf98-b5ae-1eeac81b322c (Addr: tcp/127.0.0.1:29834) (DC: dc1)" +31:16.334 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-307c4b64-1028-bf98-b5ae-1eeac81b322c.dc1 area=wan +31:16.335 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29787 network=tcp +31:16.335 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29788 network=tcp +31:16.335 [INFO] TestAgent: started state syncer +31:16.335 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29835 network=tcp +31:16.372 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.372 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29834 [Candidate]" term=2 +31:16.415 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.415 [DEBUG] TestAgent.server.raft: vote granted: from=307c4b64-1028-bf98-b5ae-1eeac81b322c term=2 tally=1 +31:16.415 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.415 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29834 [Leader]" +31:16.415 [INFO] TestAgent.server: cluster leadership acquired +31:16.415 [INFO] TestAgent.server: New leader elected: payload=Node-307c4b64-1028-bf98-b5ae-1eeac81b322c +31:16.425 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29834 +31:16.456 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.526 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.526 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.526 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.526 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.526 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.526 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-307c4b64-1028-bf98-b5ae-1eeac81b322c +31:16.526 [INFO] TestAgent.server: member joined, marking health alive: member=Node-307c4b64-1028-bf98-b5ae-1eeac81b322c +31:16.549 [INFO] TestAgent: Requesting shutdown +31:16.549 [INFO] TestAgent.server: shutting down server +31:16.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.549 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.549 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.549 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.549 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.549 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.549 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.557 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.560Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.560Z [INFO] agent.router.manager: shutting down +31:16.583 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:16.583 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.583 [INFO] TestAgent: consul server down +31:16.583 [INFO] TestAgent: shutdown complete +31:16.583 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29787 network=tcp +31:16.583 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29787 network=udp +31:16.583 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29788 network=tcp +31:17.084 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.084 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthNodeChecks (0.86s) === CONT TestEventFire_token -2022-11-07T22:46:02.281Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.290Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.328 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:462c0b05-37d6-18fd-5f71-b53bccb295d9 Address:127.0.0.1:11442}]" -46:02.328 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11442 [Follower]" leader= -46:02.331 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-462c0b05-37d6-18fd-5f71-b53bccb295d9.dc1 127.0.0.1 -46:02.332 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-462c0b05-37d6-18fd-5f71-b53bccb295d9 127.0.0.1 -2022-11-07T22:46:02.332Z [INFO] agent.router: Initializing LAN area manager -46:02.333 [INFO] TestAgent.server: Adding LAN server: server="Node-462c0b05-37d6-18fd-5f71-b53bccb295d9 (Addr: tcp/127.0.0.1:11442) (DC: dc1)" -46:02.333 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-462c0b05-37d6-18fd-5f71-b53bccb295d9.dc1 area=wan -46:02.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11437 network=udp -46:02.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11437 network=tcp -46:02.334 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11438 network=tcp -46:02.334 [INFO] TestAgent: started state syncer -46:02.334 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11443 network=tcp -46:02.369 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.369 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11442 [Candidate]" term=2 -46:02.379 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.379 [DEBUG] TestAgent.server.raft: vote granted: from=462c0b05-37d6-18fd-5f71-b53bccb295d9 term=2 tally=1 -46:02.379 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.379 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11442 [Leader]" -46:02.380 [INFO] TestAgent.server: cluster leadership acquired -46:02.380 [INFO] TestAgent.server: New leader elected: payload=Node-462c0b05-37d6-18fd-5f71-b53bccb295d9 -46:02.390 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11442 -46:02.406 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.437 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.437 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.437 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.437 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.437 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.437 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-462c0b05-37d6-18fd-5f71-b53bccb295d9 -46:02.437 [INFO] TestAgent.server: member joined, marking health alive: member=Node-462c0b05-37d6-18fd-5f71-b53bccb295d9 -46:02.447 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.684 [INFO] TestAgent: Requesting shutdown -46:02.684 [INFO] TestAgent.server: shutting down server -46:02.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.684 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.684 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.684 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.684 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.684 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.684 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.684 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.691 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.697Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.697Z [INFO] agent.router.manager: shutting down -46:02.697 [INFO] TestAgent: consul server down -46:02.697 [INFO] TestAgent: shutdown complete -46:02.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11437 network=tcp -46:02.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11437 network=udp -46:02.697 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11438 network=tcp -46:03.197 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.197 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthIngressServiceNodes (0.93s) - --- PASS: TestHealthIngressServiceNodes/associated_service (0.00s) - --- PASS: TestHealthIngressServiceNodes/non-associated_service (0.00s) -=== CONT TestDNS_ConfigReload -2022-11-07T22:46:02.267Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.276Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b Address:127.0.0.1:11435}]" -46:02.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11435 [Follower]" leader= -46:02.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b.dc1 127.0.0.1 -46:02.322 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b 127.0.0.1 -2022-11-07T22:46:02.322Z [INFO] agent.router: Initializing LAN area manager -46:02.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b.dc1 area=wan -46:02.322 [INFO] TestAgent.server: Adding LAN server: server="Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b (Addr: tcp/127.0.0.1:11435) (DC: dc1)" -46:02.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11430 network=tcp -46:02.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11430 network=udp -46:02.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11431 network=tcp -46:02.323 [INFO] TestAgent: started state syncer -46:02.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11436 network=tcp -46:02.364 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.364 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11435 [Candidate]" term=2 -46:02.377 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.377 [DEBUG] TestAgent.server.raft: vote granted: from=57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b term=2 tally=1 -46:02.377 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.377 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11435 [Leader]" -46:02.377 [INFO] TestAgent.server: cluster leadership acquired -46:02.377 [INFO] TestAgent.server: New leader elected: payload=Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b -46:02.387 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11435 -46:02.406 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.439 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.439 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.439 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.439 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.439 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.439 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b -46:02.439 [INFO] TestAgent.server: member joined, marking health alive: member=Node-57a675fd-ca2d-d71c-e5c6-eea6f2b29b5b -46:02.456 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.544 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.549 [INFO] TestAgent: Synced node info -46:02.549 [DEBUG] TestAgent: Node info in sync -46:02.697 [INFO] TestAgent: Requesting shutdown -46:02.697 [INFO] TestAgent.server: shutting down server -46:02.697 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.697 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.697 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.697 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.697 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.697 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.697 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.697 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.697 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.697 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.702 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.706Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.706Z [INFO] agent.router.manager: shutting down -46:02.706 [INFO] TestAgent: consul server down -46:02.706 [INFO] TestAgent: shutdown complete -46:02.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11430 network=tcp -46:02.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11430 network=udp -46:02.706 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11431 network=tcp -46:03.206 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.206 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_NodeMetaFilter (0.95s) +2021-10-05T16:31:16.001Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.016Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.104 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:545a610f-c303-9ab5-e5b4-92d2d7676e4c Address:127.0.0.1:29778}]" +31:16.105 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c.dc1 127.0.0.1 +31:16.105 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c 127.0.0.1 +2021-10-05T16:31:16.105Z [INFO] agent.router: Initializing LAN area manager +31:16.106 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29766 network=udp +31:16.106 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29778 [Follower]" leader= +31:16.106 [INFO] TestAgent.server: Adding LAN server: server="Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c (Addr: tcp/127.0.0.1:29778) (DC: dc1)" +31:16.106 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c.dc1 area=wan +31:16.106 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29766 network=tcp +31:16.107 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29767 network=tcp +31:16.107 [INFO] TestAgent: started state syncer +31:16.107 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29779 network=tcp +31:16.168 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.168 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29778 [Candidate]" term=2 +31:16.188 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.188 [DEBUG] TestAgent.server.raft: vote granted: from=545a610f-c303-9ab5-e5b4-92d2d7676e4c term=2 tally=1 +31:16.188 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.188 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29778 [Leader]" +31:16.189 [INFO] TestAgent.server: cluster leadership acquired +31:16.189 [INFO] TestAgent.server: New leader elected: payload=Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c +31:16.204 [INFO] TestAgent: Synced node info +31:16.204 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29778 +31:16.240 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.265 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.265 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.265 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.265 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.265 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.265 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c +31:16.265 [INFO] TestAgent.server: member joined, marking health alive: member=Node-545a610f-c303-9ab5-e5b4-92d2d7676e4c +31:16.277 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.566 [INFO] TestAgent: Requesting shutdown +31:16.566 [INFO] TestAgent.server: shutting down server +31:16.566 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.566 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.566 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.566 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.566 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.566 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.566 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.566 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.566 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.566 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.582 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.584Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.584Z [INFO] agent.router.manager: shutting down +31:16.584 [INFO] TestAgent: consul server down +31:16.584 [INFO] TestAgent: shutdown complete +31:16.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29766 network=tcp +31:16.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29766 network=udp +31:16.584 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29767 network=tcp +31:17.088 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.088 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthNodeChecks_Filtering (1.11s) === CONT TestEventFire -2022-11-07T22:46:02.259Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.267Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.293 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:777c7a9d-6111-b72a-fce7-08f8fa96bd29 Address:127.0.0.1:11428}]" -46:02.293 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11428 [Follower]" leader= -46:02.294 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29.dc1 127.0.0.1 -46:02.295 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29 127.0.0.1 -2022-11-07T22:46:02.295Z [INFO] agent.router: Initializing LAN area manager -46:02.295 [INFO] TestAgent.server: Adding LAN server: server="Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29 (Addr: tcp/127.0.0.1:11428) (DC: dc1)" -46:02.295 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29.dc1 area=wan -46:02.295 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11451 network=tcp -46:02.295 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11451 network=udp -46:02.295 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11452 network=tcp -46:02.295 [INFO] TestAgent: started state syncer -46:02.295 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11429 network=tcp -46:02.331 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.331 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11428 [Candidate]" term=2 -46:02.343 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.343 [DEBUG] TestAgent.server.raft: vote granted: from=777c7a9d-6111-b72a-fce7-08f8fa96bd29 term=2 tally=1 -46:02.343 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.343 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11428 [Leader]" -46:02.343 [INFO] TestAgent.server: cluster leadership acquired -46:02.343 [INFO] TestAgent.server: New leader elected: payload=Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29 -46:02.352 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11428 -46:02.367 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.392 [INFO] TestAgent: Synced node info -46:02.392 [DEBUG] TestAgent: Node info in sync -46:02.403 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.403 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.403 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.403 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.403 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.403 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29 -46:02.403 [INFO] TestAgent.server: member joined, marking health alive: member=Node-777c7a9d-6111-b72a-fce7-08f8fa96bd29 -46:02.410 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.540 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.540 [DEBUG] TestAgent: Node info in sync -46:02.722 [INFO] TestAgent: Requesting shutdown -46:02.722 [INFO] TestAgent.server: shutting down server -46:02.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.722 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.722 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.722 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.722 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.722 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.722 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.728 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.734Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.734Z [INFO] agent.router.manager: shutting down -46:02.734 [INFO] TestAgent: consul server down -46:02.734 [INFO] TestAgent: shutdown complete -46:02.734 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11451 network=tcp -46:02.734 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11451 network=udp -46:02.734 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11452 network=tcp -46:03.234 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.234 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_CheckType (0.99s) -=== CONT TestDNS_Compression_ReverseLookup -2022-11-07T22:46:02.423Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.432Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.473 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4abbd056-6361-a48c-88f9-52b933c0bf9a Address:127.0.0.1:11498}]" -46:02.473 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11498 [Follower]" leader= -46:02.473 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4abbd056-6361-a48c-88f9-52b933c0bf9a.dc1 127.0.0.1 -46:02.474 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4abbd056-6361-a48c-88f9-52b933c0bf9a 127.0.0.1 -2022-11-07T22:46:02.474Z [INFO] agent.router: Initializing LAN area manager -46:02.474 [INFO] TestAgent.server: Adding LAN server: server="Node-4abbd056-6361-a48c-88f9-52b933c0bf9a (Addr: tcp/127.0.0.1:11498) (DC: dc1)" -46:02.474 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4abbd056-6361-a48c-88f9-52b933c0bf9a.dc1 area=wan -46:02.474 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11486 network=udp -46:02.474 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11486 network=tcp -46:02.474 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11487 network=tcp -46:02.474 [INFO] TestAgent: started state syncer -46:02.474 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11499 network=tcp -46:02.539 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.539 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11498 [Candidate]" term=2 -46:02.550 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.550 [DEBUG] TestAgent.server.raft: vote granted: from=4abbd056-6361-a48c-88f9-52b933c0bf9a term=2 tally=1 -46:02.550 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.550 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11498 [Leader]" -46:02.550 [INFO] TestAgent.server: cluster leadership acquired -46:02.550 [INFO] TestAgent.server: New leader elected: payload=Node-4abbd056-6361-a48c-88f9-52b933c0bf9a -46:02.559 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11498 -46:02.577 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.604 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.604 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.604 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.604 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.604 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.604 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4abbd056-6361-a48c-88f9-52b933c0bf9a -46:02.604 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4abbd056-6361-a48c-88f9-52b933c0bf9a -46:02.615 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.731 [INFO] TestAgent: Requesting shutdown -46:02.731 [INFO] TestAgent.server: shutting down server -46:02.731 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.731 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.731 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.731 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.731 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.731 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.731 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.731 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.731 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.731 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.731 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.736 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.740Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.740Z [INFO] agent.router.manager: shutting down -46:02.740 [INFO] TestAgent: consul server down -46:02.740 [INFO] TestAgent: shutdown complete -46:02.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11486 network=tcp -46:02.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11486 network=udp -46:02.740 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11487 network=tcp -46:03.241 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.241 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_RaftPeer (1.58s) - --- PASS: TestOperator_RaftPeer/#00 (0.75s) - --- PASS: TestOperator_RaftPeer/#01 (0.83s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:16.307Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.322Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.416 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c3598f4a-892a-550b-e7f7-7852e3f0d3cd Address:127.0.0.1:29820}]" +31:16.417 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd.dc1 127.0.0.1 +31:16.417 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd 127.0.0.1 +2021-10-05T16:31:16.417Z [INFO] agent.router: Initializing LAN area manager +31:16.417 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29836 network=udp +31:16.417 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29820 [Follower]" leader= +31:16.418 [INFO] TestAgent.server: Adding LAN server: server="Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd (Addr: tcp/127.0.0.1:29820) (DC: dc1)" +31:16.418 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd.dc1 area=wan +31:16.418 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29836 network=tcp +31:16.419 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29837 network=tcp +31:16.419 [INFO] TestAgent: started state syncer +31:16.419 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29821 network=tcp +31:16.456 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.456 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29820 [Candidate]" term=2 +31:16.505 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.505 [DEBUG] TestAgent.server.raft: vote granted: from=c3598f4a-892a-550b-e7f7-7852e3f0d3cd term=2 tally=1 +31:16.505 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.505 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29820 [Leader]" +31:16.506 [INFO] TestAgent.server: cluster leadership acquired +31:16.506 [INFO] TestAgent.server: New leader elected: payload=Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd +31:16.532 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29820 +31:16.624 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.725 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.725 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.725 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.725 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.725 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.725 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd +31:16.725 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c3598f4a-892a-550b-e7f7-7852e3f0d3cd +31:16.767 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.779 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:16.788 [INFO] TestAgent: Synced node info +31:16.851 [INFO] TestAgent: Requesting shutdown +31:16.851 [INFO] TestAgent.server: shutting down server +31:16.851 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.851 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.851 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.851 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.851 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.851 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.851 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.851 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.851 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.851 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.857 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.857Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.857Z [INFO] agent.router.manager: shutting down +31:16.858 [INFO] TestAgent: consul server down +31:16.858 [INFO] TestAgent: shutdown complete +31:16.858 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29836 network=tcp +31:16.858 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29836 network=udp +31:16.858 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29837 network=tcp +31:17.358 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.358 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthChecksInState_DistanceSort (1.07s) +=== CONT TestDNS_ReloadConfig_DuringQuery +2021-10-05T16:31:16.426Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.469Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.584 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2d194671-216b-6ab5-e573-3543690ea9ff Address:127.0.0.1:29813}]" +31:16.584 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29813 [Follower]" leader= +31:16.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2d194671-216b-6ab5-e573-3543690ea9ff.dc1 127.0.0.1 +31:16.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2d194671-216b-6ab5-e573-3543690ea9ff 127.0.0.1 +2021-10-05T16:31:16.585Z [INFO] agent.router: Initializing LAN area manager +31:16.585 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29822 network=udp +31:16.586 [INFO] TestAgent.server: Adding LAN server: server="Node-2d194671-216b-6ab5-e573-3543690ea9ff (Addr: tcp/127.0.0.1:29813) (DC: dc1)" +31:16.586 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2d194671-216b-6ab5-e573-3543690ea9ff.dc1 area=wan +31:16.586 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29822 network=tcp +31:16.586 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29823 network=tcp +31:16.586 [INFO] TestAgent: started state syncer +31:16.586 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29814 network=tcp +31:16.643 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.643 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29813 [Candidate]" term=2 +31:16.682 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.682 [DEBUG] TestAgent.server.raft: vote granted: from=2d194671-216b-6ab5-e573-3543690ea9ff term=2 tally=1 +31:16.682 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.682 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29813 [Leader]" +31:16.682 [INFO] TestAgent.server: cluster leadership acquired +31:16.682 [INFO] TestAgent.server: New leader elected: payload=Node-2d194671-216b-6ab5-e573-3543690ea9ff +31:16.692 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29813 +31:16.724 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.804 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.804 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.804 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.804 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.804 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.804 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2d194671-216b-6ab5-e573-3543690ea9ff +31:16.804 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2d194671-216b-6ab5-e573-3543690ea9ff +31:16.849 [INFO] TestAgent: Requesting shutdown +31:16.849 [INFO] TestAgent.server: shutting down server +31:16.849 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.849 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.849 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.849 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.849 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.849 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.849 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.849 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.849 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:16.853 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:16.855Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:16.855Z [INFO] agent.router.manager: shutting down +31:16.859 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:16.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.859 [INFO] TestAgent: consul server down +31:16.859 [INFO] TestAgent: shutdown complete +31:16.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29822 network=tcp +31:16.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29822 network=udp +31:16.859 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29823 network=tcp +31:17.360 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.360 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthChecksInState_Filter (0.99s) +=== CONT TestDNS_ConfigReload +2021-10-05T16:31:16.535Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.569Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.691 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:64962173-22c3-1051-7176-31498c502c93 Address:127.0.0.1:29827}]" +31:16.691 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29827 [Follower]" leader= +31:16.692 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-64962173-22c3-1051-7176-31498c502c93.dc1 127.0.0.1 +31:16.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-64962173-22c3-1051-7176-31498c502c93 127.0.0.1 +2021-10-05T16:31:16.693Z [INFO] agent.router: Initializing LAN area manager +31:16.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29815 network=udp +31:16.693 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-64962173-22c3-1051-7176-31498c502c93.dc1 area=wan +31:16.693 [INFO] TestAgent.server: Adding LAN server: server="Node-64962173-22c3-1051-7176-31498c502c93 (Addr: tcp/127.0.0.1:29827) (DC: dc1)" +31:16.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29815 network=tcp +31:16.693 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29816 network=tcp +31:16.693 [INFO] TestAgent: started state syncer +31:16.694 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29828 network=tcp +31:16.746 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.746 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29827 [Candidate]" term=2 +31:16.777 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.778 [DEBUG] TestAgent.server.raft: vote granted: from=64962173-22c3-1051-7176-31498c502c93 term=2 tally=1 +31:16.778 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.778 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29827 [Leader]" +31:16.778 [INFO] TestAgent.server: cluster leadership acquired +31:16.778 [INFO] TestAgent.server: New leader elected: payload=Node-64962173-22c3-1051-7176-31498c502c93 +31:16.791 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29827 +31:16.812 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.852 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.852 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.852 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.852 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.852 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.852 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-64962173-22c3-1051-7176-31498c502c93 +31:16.852 [INFO] TestAgent.server: member joined, marking health alive: member=Node-64962173-22c3-1051-7176-31498c502c93 +31:16.861 [INFO] TestAgent.server: federation state anti-entropy synced +31:16.994 [INFO] TestAgent: Requesting shutdown +31:16.994 [INFO] TestAgent.server: shutting down server +31:16.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:16.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:16.994 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:16.994 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:16.994 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:16.995 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:16.995 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:16.995 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:16.995 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:16.995 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:16.995 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.003 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.004Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.004Z [INFO] agent.router.manager: shutting down +31:17.004 [INFO] TestAgent: consul server down +31:17.004 [INFO] TestAgent: shutdown complete +31:17.004 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29815 network=tcp +31:17.004 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29815 network=udp +31:17.004 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29816 network=tcp +31:17.510 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.510 [INFO] TestAgent: Endpoints down +--- PASS: TestHealthChecksInState_NodeMetaFilter (1.00s) === CONT TestDNS_Compression_Recurse -2022-11-07T22:46:02.567Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.576Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.611 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:70865fe0-ca03-9e7a-8018-b57e6c5a9695 Address:127.0.0.1:11525}]" -46:02.611 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11525 [Follower]" leader= -46:02.611 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695.dc1 127.0.0.1 -46:02.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695 127.0.0.1 -2022-11-07T22:46:02.612Z [INFO] agent.router: Initializing LAN area manager -46:02.612 [INFO] TestAgent.server: Adding LAN server: server="Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695 (Addr: tcp/127.0.0.1:11525) (DC: dc1)" -46:02.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11520 network=udp -46:02.612 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695.dc1 area=wan -46:02.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11520 network=tcp -46:02.612 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11521 network=tcp -46:02.612 [INFO] TestAgent: started state syncer -46:02.612 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11533 network=tcp -46:02.669 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.669 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11525 [Candidate]" term=2 -46:02.680 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.680 [DEBUG] TestAgent.server.raft: vote granted: from=70865fe0-ca03-9e7a-8018-b57e6c5a9695 term=2 tally=1 -46:02.680 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.680 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11525 [Leader]" -46:02.680 [INFO] TestAgent.server: cluster leadership acquired -46:02.680 [INFO] TestAgent.server: New leader elected: payload=Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695 -46:02.686 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11525 -46:02.701 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.720 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.720 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.720 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.720 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.720 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.720 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695 -46:02.720 [INFO] TestAgent.server: member joined, marking health alive: member=Node-70865fe0-ca03-9e7a-8018-b57e6c5a9695 -46:02.733 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.830 [INFO] TestAgent: Requesting shutdown -46:02.830 [INFO] TestAgent.server: shutting down server -46:02.830 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.830 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.830 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.830 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.830 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.830 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.830 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.833 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.836Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.836Z [INFO] agent.router.manager: shutting down -46:02.836 [INFO] TestAgent: consul server down -46:02.836 [INFO] TestAgent: shutdown complete -46:02.836 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11520 network=tcp -46:02.837 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11520 network=udp -46:02.837 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11521 network=tcp -46:03.337 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.337 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthChecksInState_NodeMetaFilter (0.78s) +2021-10-05T16:31:16.743Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.757Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.802 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a Address:127.0.0.1:29799}]" +31:16.803 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a.dc1 127.0.0.1 +31:16.804 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29799 [Follower]" leader= +31:16.805 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a 127.0.0.1 +2021-10-05T16:31:16.805Z [INFO] agent.router: Initializing LAN area manager +31:16.805 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29794 network=udp +31:16.805 [INFO] TestAgent.server: Adding LAN server: server="Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a (Addr: tcp/127.0.0.1:29799) (DC: dc1)" +31:16.805 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a.dc1 area=wan +31:16.805 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29794 network=tcp +31:16.806 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29795 network=tcp +31:16.806 [INFO] TestAgent: started state syncer +31:16.806 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29800 network=tcp +31:16.856 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.856 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29799 [Candidate]" term=2 +31:16.888 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.888 [DEBUG] TestAgent.server.raft: vote granted: from=b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a term=2 tally=1 +31:16.888 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.888 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29799 [Leader]" +31:16.888 [INFO] TestAgent.server: cluster leadership acquired +31:16.888 [INFO] TestAgent.server: New leader elected: payload=Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a +31:16.903 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29799 +31:16.929 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.959 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.959 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.959 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.959 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.959 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.959 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a +31:16.959 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8390d63-fbf9-7bc4-1ca0-55b2fbf8899a +31:16.971 [DEBUG] TestAgent.server: User event: event=test +31:16.971 [DEBUG] TestAgent: new event: event_name=test event_id=9c96541c-d639-f905-465b-39f9b47bbd4b +31:16.980 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.022 [DEBUG] TestAgent.server: User event: event=second +31:17.023 [DEBUG] TestAgent: new event: event_name=second event_id=b1b5b921-3841-09e9-f7be-c7701600e98e +31:17.023 [INFO] TestAgent: Requesting shutdown +31:17.023 [INFO] TestAgent.server: shutting down server +31:17.023 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.023 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.023 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.023 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.023 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.023 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.023 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.023 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.023 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:17.023 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.023 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.033 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.039Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.039Z [INFO] agent.router.manager: shutting down +31:17.039 [INFO] TestAgent: consul server down +31:17.039 [INFO] TestAgent: shutdown complete +31:17.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29794 network=tcp +31:17.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29794 network=udp +31:17.039 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29795 network=tcp +31:17.540 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.540 [INFO] TestAgent: Endpoints down +--- PASS: TestEventList_Blocking (0.83s) +=== CONT TestDNS_Compression_ReverseLookup +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:16.594Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.648Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.764 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:28961e16-c6a6-8808-2b2d-7057096efe52 Address:127.0.0.1:29792}]" +31:16.765 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-28961e16-c6a6-8808-2b2d-7057096efe52.dc1 127.0.0.1 +31:16.765 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29792 [Follower]" leader= +31:16.765 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-28961e16-c6a6-8808-2b2d-7057096efe52 127.0.0.1 +2021-10-05T16:31:16.765Z [INFO] agent.router: Initializing LAN area manager +31:16.766 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29829 network=udp +31:16.766 [INFO] TestAgent.server: Adding LAN server: server="Node-28961e16-c6a6-8808-2b2d-7057096efe52 (Addr: tcp/127.0.0.1:29792) (DC: dc1)" +31:16.766 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29829 network=tcp +31:16.766 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-28961e16-c6a6-8808-2b2d-7057096efe52.dc1 area=wan +31:16.766 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29830 network=tcp +31:16.766 [INFO] TestAgent: started state syncer +31:16.766 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29793 network=tcp +31:16.832 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.832 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29792 [Candidate]" term=2 +31:16.856 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.856 [DEBUG] TestAgent.server.raft: vote granted: from=28961e16-c6a6-8808-2b2d-7057096efe52 term=2 tally=1 +31:16.856 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.856 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29792 [Leader]" +31:16.856 [INFO] TestAgent.server: cluster leadership acquired +31:16.856 [INFO] TestAgent.server: New leader elected: payload=Node-28961e16-c6a6-8808-2b2d-7057096efe52 +31:16.866 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29792 +31:16.891 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:16.923 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:16.923 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:16.923 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:16.923 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:16.923 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:16.923 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28961e16-c6a6-8808-2b2d-7057096efe52 +31:16.923 [INFO] TestAgent.server: member joined, marking health alive: member=Node-28961e16-c6a6-8808-2b2d-7057096efe52 +31:16.941 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.193 [DEBUG] TestAgent.server: User event: event=foo +31:17.193 [DEBUG] TestAgent.server: User event: event=bar +31:17.193 [DEBUG] TestAgent.server: User event: event=foo +31:17.193 [DEBUG] TestAgent.server: User event: event=foo +31:17.193 [DEBUG] TestAgent.server: User event: event=bar +31:17.193 [DEBUG] TestAgent: new event: event_name=foo event_id=8a0a9a01-695d-bdea-a386-f8ae47a22699 +31:17.193 [DEBUG] TestAgent: new event: event_name=bar event_id=ce3666a6-4715-36b2-e66b-48e15f609295 +31:17.193 [DEBUG] TestAgent: new event: event_name=foo event_id=9d71b9a2-ef39-cbe8-d2d4-0d5dc5f063c9 +31:17.193 [DEBUG] TestAgent: new event: event_name=foo event_id=dd76e2f3-eaf7-5f91-23b2-38c4333c90cf +31:17.193 [DEBUG] TestAgent: new event: event_name=bar event_id=a5212e43-4d0e-f227-228c-a79750135f26 +31:17.197 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:17.201 [INFO] TestAgent: Synced node info +31:17.201 [DEBUG] TestAgent: Node info in sync +31:17.220 [INFO] TestAgent: Requesting shutdown +31:17.220 [INFO] TestAgent.server: shutting down server +31:17.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.220 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.220 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.220 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.220 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.220 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.220 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.231Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.231Z [INFO] agent.router.manager: shutting down +31:17.232 [INFO] TestAgent: consul server down +31:17.232 [INFO] TestAgent: shutdown complete +31:17.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29829 network=tcp +31:17.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29829 network=udp +31:17.232 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29830 network=tcp +31:17.736 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.736 [INFO] TestAgent: Endpoints down +--- PASS: TestEventList_EventBufOrder (1.18s) === CONT TestDNS_Compression_Query -[WARN] freeport: 10 out of 87 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:02.541Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.550Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.590 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:52f82ef7-f725-e2d6-75e0-1db242b1800a Address:127.0.0.1:11532}]" -46:02.590 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11532 [Follower]" leader= -46:02.590 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-52f82ef7-f725-e2d6-75e0-1db242b1800a.dc1 127.0.0.1 -46:02.591 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-52f82ef7-f725-e2d6-75e0-1db242b1800a 127.0.0.1 -2022-11-07T22:46:02.591Z [INFO] agent.router: Initializing LAN area manager -46:02.591 [INFO] TestAgent.server: Adding LAN server: server="Node-52f82ef7-f725-e2d6-75e0-1db242b1800a (Addr: tcp/127.0.0.1:11532) (DC: dc1)" -46:02.591 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-52f82ef7-f725-e2d6-75e0-1db242b1800a.dc1 area=wan -46:02.591 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11527 network=udp -46:02.591 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11527 network=tcp -46:02.591 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11528 network=tcp -46:02.591 [INFO] TestAgent: started state syncer -46:02.591 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11519 network=tcp -46:02.637 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.637 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11532 [Candidate]" term=2 -46:02.645 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.645 [DEBUG] TestAgent.server.raft: vote granted: from=52f82ef7-f725-e2d6-75e0-1db242b1800a term=2 tally=1 -46:02.645 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.645 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11532 [Leader]" -46:02.645 [INFO] TestAgent.server: cluster leadership acquired -46:02.645 [INFO] TestAgent.server: New leader elected: payload=Node-52f82ef7-f725-e2d6-75e0-1db242b1800a -46:02.651 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11532 -46:02.662 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.689 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.689 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.689 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.689 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.689 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.689 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-52f82ef7-f725-e2d6-75e0-1db242b1800a -46:02.689 [INFO] TestAgent.server: member joined, marking health alive: member=Node-52f82ef7-f725-e2d6-75e0-1db242b1800a -46:02.702 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.863 [INFO] TestAgent: Requesting shutdown -46:02.863 [INFO] TestAgent.server: shutting down server -46:02.863 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.863 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.863 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.864 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.864 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.864 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.864 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.867 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.870Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.870Z [INFO] agent.router.manager: shutting down -46:02.870 [INFO] TestAgent: consul server down -46:02.870 [INFO] TestAgent: shutdown complete -46:02.870 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11527 network=tcp -46:02.870 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11527 network=udp -46:02.870 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11528 network=tcp -46:03.370 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.370 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceChecks (0.84s) +2021-10-05T16:31:16.768Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.794Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.880 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:39416120-ef2c-580a-fe6d-ea1382926361 Address:127.0.0.1:29806}]" +31:16.880 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29806 [Follower]" leader= +31:16.881 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-39416120-ef2c-580a-fe6d-ea1382926361.dc1 127.0.0.1 +31:16.881 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-39416120-ef2c-580a-fe6d-ea1382926361 127.0.0.1 +2021-10-05T16:31:16.881Z [INFO] agent.router: Initializing LAN area manager +31:16.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29801 network=udp +31:16.882 [INFO] TestAgent.server: Adding LAN server: server="Node-39416120-ef2c-580a-fe6d-ea1382926361 (Addr: tcp/127.0.0.1:29806) (DC: dc1)" +31:16.882 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-39416120-ef2c-580a-fe6d-ea1382926361.dc1 area=wan +31:16.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29801 network=tcp +31:16.883 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29802 network=tcp +31:16.883 [INFO] TestAgent: started state syncer +31:16.883 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29807 network=tcp +31:16.934 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.934 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29806 [Candidate]" term=2 +31:16.947 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.947 [DEBUG] TestAgent.server.raft: vote granted: from=39416120-ef2c-580a-fe6d-ea1382926361 term=2 tally=1 +31:16.947 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.947 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29806 [Leader]" +31:16.948 [INFO] TestAgent.server: cluster leadership acquired +31:16.948 [INFO] TestAgent.server: New leader elected: payload=Node-39416120-ef2c-580a-fe6d-ea1382926361 +31:16.957 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29806 +31:16.988 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.023 [INFO] TestAgent: Synced node info +31:17.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-39416120-ef2c-580a-fe6d-ea1382926361 +31:17.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-39416120-ef2c-580a-fe6d-ea1382926361 +31:17.064 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.161 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:17.161 [DEBUG] TestAgent: Node info in sync +31:17.161 [DEBUG] TestAgent: Node info in sync +31:17.260 [DEBUG] TestAgent.server: User event: event=test +31:17.260 [DEBUG] TestAgent.server: User event: event=foo +31:17.260 [DEBUG] TestAgent: new event: event_name=test event_id=b0ff06ce-a517-f997-1fff-1d555d7425b4 +31:17.260 [DEBUG] TestAgent: new event: event_name=foo event_id=b07ed4a6-4bc4-61e7-72bc-e33dd9336e95 +31:17.286 [INFO] TestAgent: Requesting shutdown +31:17.286 [INFO] TestAgent.server: shutting down server +31:17.286 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.286 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.286 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.286 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.286 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.286 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.286 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.286 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.286 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.286 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.293 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.295Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.295Z [INFO] agent.router.manager: shutting down +31:17.295 [INFO] TestAgent: consul server down +31:17.295 [INFO] TestAgent: shutdown complete +31:17.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29801 network=tcp +31:17.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29801 network=udp +31:17.295 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29802 network=tcp +31:17.796 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.796 [INFO] TestAgent: Endpoints down +--- PASS: TestEventList_Filter (1.06s) === CONT TestDNS_Compression_trimUDPResponse ---- PASS: TestDNS_Compression_trimUDPResponse (0.01s) -=== CONT TestKVSEndpoint_PUT_GET_DELETE -2022-11-07T22:46:02.369Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.379Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.418 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97dd31e1-5ff1-af68-ec70-453199f7af35 Address:127.0.0.1:11491}]" -46:02.418 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11491 [Follower]" leader= -46:02.418 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-97dd31e1-5ff1-af68-ec70-453199f7af35.dc1 127.0.0.1 -46:02.419 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-97dd31e1-5ff1-af68-ec70-453199f7af35 127.0.0.1 -2022-11-07T22:46:02.419Z [INFO] agent.router: Initializing LAN area manager -46:02.419 [INFO] TestAgent.server: Adding LAN server: server="Node-97dd31e1-5ff1-af68-ec70-453199f7af35 (Addr: tcp/127.0.0.1:11491) (DC: dc1)" -46:02.419 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-97dd31e1-5ff1-af68-ec70-453199f7af35.dc1 area=wan -46:02.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11479 network=udp -46:02.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11479 network=tcp -46:02.420 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11480 network=tcp -46:02.420 [INFO] TestAgent: started state syncer -46:02.420 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11492 network=tcp -46:02.471 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.471 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11491 [Candidate]" term=2 -46:02.486 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.486 [DEBUG] TestAgent.server.raft: vote granted: from=97dd31e1-5ff1-af68-ec70-453199f7af35 term=2 tally=1 -46:02.486 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.486 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11491 [Leader]" -46:02.486 [INFO] TestAgent.server: cluster leadership acquired -46:02.487 [INFO] TestAgent.server: New leader elected: payload=Node-97dd31e1-5ff1-af68-ec70-453199f7af35 -46:02.498 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11491 -46:02.518 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.542 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.542 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.542 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.542 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.542 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.542 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-97dd31e1-5ff1-af68-ec70-453199f7af35 -46:02.542 [INFO] TestAgent.server: member joined, marking health alive: member=Node-97dd31e1-5ff1-af68-ec70-453199f7af35 -46:02.558 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.568 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.572 [INFO] TestAgent: Synced node info -46:02.572 [DEBUG] TestAgent: Node info in sync -46:02.893 [INFO] TestAgent: Requesting shutdown -46:02.893 [INFO] TestAgent.server: shutting down server -46:02.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.893 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.893 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.893 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.893 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.893 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.893 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.897 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.900Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.900Z [INFO] agent.router.manager: shutting down -46:02.900 [INFO] TestAgent: consul server down -46:02.900 [INFO] TestAgent: shutdown complete -46:02.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11479 network=tcp -46:02.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11479 network=udp -46:02.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11480 network=tcp -46:03.400 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.400 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_DistanceSort (1.04s) -=== CONT TestAgentKeyring_ACL -2022-11-07T22:46:02.194Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:02.194Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.203Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:02.203Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.223 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3a08bfde-00ea-c5f9-b674-6a242b42b942 Address:127.0.0.1:11414}]" -46:02.223 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11414 [Follower]" leader= -46:02.224 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 127.0.0.1 -46:02.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 127.0.0.1 -2022-11-07T22:46:02.224Z [INFO] agent.router: Initializing LAN area manager -46:02.224 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 area=wan -46:02.224 [INFO] TestAgent.server: Adding LAN server: server="Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 (Addr: tcp/127.0.0.1:11414) (DC: dc2)" -46:02.224 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11458 network=udp -46:02.224 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11458 network=tcp -46:02.225 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11459 network=tcp -46:02.225 [INFO] TestAgent: started state syncer -46:02.225 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11415 network=tcp -46:02.274 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:02.293 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.293 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11414 [Candidate]" term=2 -46:02.305 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.305 [DEBUG] TestAgent.server.raft: vote granted: from=3a08bfde-00ea-c5f9-b674-6a242b42b942 term=2 tally=1 -46:02.305 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.305 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11414 [Leader]" -46:02.306 [INFO] TestAgent.server: cluster leadership acquired -46:02.306 [INFO] TestAgent.server: New leader elected: payload=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 -46:02.313 [INFO] TestAgent.server: initializing acls -46:02.316 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:02.328 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:02.328 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:02.328 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:02.328 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 -46:02.328 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 -46:02.328 [INFO] TestAgent.server: Updating LAN server: server="Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 (Addr: tcp/127.0.0.1:11414) (DC: dc2)" -46:02.328 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 area=wan -46:02.351 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.376 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.376 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.376 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.376 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.376 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.376 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 -46:02.376 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 -46:02.395 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.395 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942 -46:02.446 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11399] -46:02.446 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11399 -46:02.447 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 127.0.0.1 -46:02.447 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:02.447 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 area=wan -46:02.453 [INFO] TestAgent: Requesting shutdown -46:02.453 [INFO] TestAgent.server: shutting down server -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:02.453 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.453 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.453 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.453 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:02.453 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:02.458 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.462Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.462Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.462Z [INFO] agent.router.manager: shutting down -46:02.462 [INFO] TestAgent: consul server down -46:02.462 [INFO] TestAgent: shutdown complete -46:02.462 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11458 network=tcp -46:02.462 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11458 network=udp -46:02.462 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11459 network=tcp -46:02.962 [INFO] TestAgent: Waiting for endpoints to shut down -46:02.962 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:01.738Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:01.738Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.753Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:01.753Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.788 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 Address:127.0.0.1:11400}]" -46:01.789 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11400 [Follower]" leader= -46:01.789 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 127.0.0.1 -46:01.789 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 127.0.0.1 -2022-11-07T22:46:01.789Z [INFO] agent.router: Initializing LAN area manager -46:01.789 [INFO] TestAgent.server: Adding LAN server: server="Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 (Addr: tcp/127.0.0.1:11400) (DC: dc1)" -46:01.789 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 area=wan -46:01.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11395 network=udp -46:01.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11395 network=tcp -46:01.790 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11396 network=tcp -46:01.790 [INFO] TestAgent: started state syncer -46:01.790 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11401 network=tcp -46:01.840 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:01.852 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.852 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11400 [Candidate]" term=2 -46:01.866 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.866 [DEBUG] TestAgent.server.raft: vote granted: from=c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 term=2 tally=1 -46:01.866 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.866 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11400 [Leader]" -46:01.866 [INFO] TestAgent.server: cluster leadership acquired -46:01.866 [INFO] TestAgent.server: New leader elected: payload=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 -46:01.872 [INFO] TestAgent.server: initializing acls -46:01.875 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:01.883 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:01.884 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:01.884 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:01.884 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 -46:01.884 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 -46:01.884 [INFO] TestAgent.server: Updating LAN server: server="Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 (Addr: tcp/127.0.0.1:11400) (DC: dc1)" -46:01.884 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9.dc1 area=wan -46:01.902 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.916 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.916 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:01.916 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:01.916 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:01.916 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:01.916 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 -46:01.916 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 -46:01.927 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.927 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7a80ad5-a1a7-29dc-dd0b-d73e402c6ca9 -46:02.033 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.038 [INFO] TestAgent: Synced node info -46:02.446 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:49504 -46:02.447 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 127.0.0.1 -46:02.447 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3a08bfde-00ea-c5f9-b674-6a242b42b942.dc2 area=wan -46:02.962 [INFO] TestAgent: Requesting shutdown -46:02.962 [INFO] TestAgent.server: shutting down server -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:02.962 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:02.962 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:02.962 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:02.962 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:02.966 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:02.969Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.969Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:02.969Z [INFO] agent.router.manager: shutting down -46:02.970 [INFO] TestAgent: consul server down -46:02.970 [INFO] TestAgent: shutdown complete -46:02.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11395 network=tcp -46:02.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11395 network=udp -46:02.970 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11396 network=tcp -46:03.470 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.470 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthServiceNodes_WanTranslation (1.75s) +2021-10-05T16:31:16.805Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.857Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:16.890 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1c718b18-3695-a68b-0256-87e875292ed6 Address:127.0.0.1:29771}]" +31:16.890 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29771 [Follower]" leader= +31:16.890 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1c718b18-3695-a68b-0256-87e875292ed6.dc1 127.0.0.1 +31:16.891 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1c718b18-3695-a68b-0256-87e875292ed6 127.0.0.1 +2021-10-05T16:31:16.891Z [INFO] agent.router: Initializing LAN area manager +31:16.891 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29808 network=udp +31:16.891 [INFO] TestAgent.server: Adding LAN server: server="Node-1c718b18-3695-a68b-0256-87e875292ed6 (Addr: tcp/127.0.0.1:29771) (DC: dc1)" +31:16.892 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29808 network=tcp +31:16.891 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1c718b18-3695-a68b-0256-87e875292ed6.dc1 area=wan +31:16.892 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29809 network=tcp +31:16.892 [INFO] TestAgent: started state syncer +31:16.892 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29772 network=tcp +31:16.954 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:16.954 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29771 [Candidate]" term=2 +31:16.967 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:16.967 [DEBUG] TestAgent.server.raft: vote granted: from=1c718b18-3695-a68b-0256-87e875292ed6 term=2 tally=1 +31:16.967 [INFO] TestAgent.server.raft: election won: tally=1 +31:16.967 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29771 [Leader]" +31:16.967 [INFO] TestAgent.server: cluster leadership acquired +31:16.967 [INFO] TestAgent.server: New leader elected: payload=Node-1c718b18-3695-a68b-0256-87e875292ed6 +31:16.988 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29771 +31:17.017 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.042 [INFO] TestAgent: Synced node info +31:17.056 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.056 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.056 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.056 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.056 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.056 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1c718b18-3695-a68b-0256-87e875292ed6 +31:17.056 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1c718b18-3695-a68b-0256-87e875292ed6 +31:17.088 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.252 [DEBUG] TestAgent.server: User event: event=test +31:17.252 [DEBUG] TestAgent: new event: event_name=test event_id=7cf19f0a-9827-03f4-d732-4d6d8284f482 +31:17.280 [INFO] TestAgent: Requesting shutdown +31:17.281 [INFO] TestAgent.server: shutting down server +31:17.281 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.281 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.281 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.281 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.281 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.281 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.281 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.281 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.281 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.281 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.292 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.296Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.296Z [INFO] agent.router.manager: shutting down +31:17.296 [INFO] TestAgent: consul server down +31:17.296 [INFO] TestAgent: shutdown complete +31:17.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29808 network=tcp +31:17.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29808 network=udp +31:17.296 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29809 network=tcp +31:17.800 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.800 [INFO] TestAgent: Endpoints down +--- PASS: TestEventList (1.04s) === CONT TestDNS_syncExtra --- PASS: TestDNS_syncExtra (0.00s) === CONT TestDNS_trimUDPResponse_TrimSizeEDNS --- PASS: TestDNS_trimUDPResponse_TrimSizeEDNS (0.01s) -=== CONT TestDNS_trimUDPResponse_TrimLimit ---- PASS: TestDNS_trimUDPResponse_TrimLimit (0.01s) -=== CONT TestAgent_InitKeyring ---- PASS: TestAgent_InitKeyring (0.00s) -=== CONT TestDNS_PreparedQuery_AgentSource -=== RUN TestEventList_ACLFilter/no_token -=== RUN TestEventList_ACLFilter/root_token -2022-11-07T22:46:02.647Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:02.656Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:02.695 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c04a3d67-d1a3-b094-515c-14ca3bd6bcdd Address:127.0.0.1:11539}]" -46:02.695 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11539 [Follower]" leader= -46:02.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd.dc1 127.0.0.1 -46:02.697 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd 127.0.0.1 -2022-11-07T22:46:02.697Z [INFO] agent.router: Initializing LAN area manager -46:02.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11534 network=udp -46:02.697 [INFO] TestAgent.server: Adding LAN server: server="Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd (Addr: tcp/127.0.0.1:11539) (DC: dc1)" -46:02.697 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd.dc1 area=wan -46:02.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11534 network=tcp -46:02.698 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11535 network=tcp -46:02.698 [INFO] TestAgent: started state syncer -46:02.698 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11540 network=tcp -46:02.738 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:02.738 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11539 [Candidate]" term=2 -46:02.747 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:02.747 [DEBUG] TestAgent.server.raft: vote granted: from=c04a3d67-d1a3-b094-515c-14ca3bd6bcdd term=2 tally=1 -46:02.747 [INFO] TestAgent.server.raft: election won: tally=1 -46:02.747 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11539 [Leader]" -46:02.747 [INFO] TestAgent.server: cluster leadership acquired -46:02.747 [INFO] TestAgent.server: New leader elected: payload=Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd -46:02.752 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11539 -46:02.763 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:02.780 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:02.780 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:02.780 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:02.780 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:02.780 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:02.780 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd -46:02.780 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c04a3d67-d1a3-b094-515c-14ca3bd6bcdd -46:02.785 [INFO] TestAgent.server: federation state anti-entropy synced -46:02.789 [INFO] TestAgent: Synced node info -46:02.789 [DEBUG] TestAgent: Node info in sync -46:03.072 [INFO] TestAgent: Requesting shutdown -46:03.072 [INFO] TestAgent.server: shutting down server -46:03.072 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.072 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.072 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.072 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.072 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.072 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.072 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.072 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.072 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.072 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.077 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.083Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.083Z [INFO] agent.router.manager: shutting down -46:03.083 [INFO] TestAgent: consul server down -46:03.083 [INFO] TestAgent: shutdown complete -46:03.083 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11534 network=tcp -46:03.083 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11534 network=udp -46:03.083 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11535 network=tcp -46:03.583 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.583 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthChecksInState_DistanceSort (0.95s) === CONT TestDNS_trimUDPResponse_TrimSize -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! ---- PASS: TestDNS_trimUDPResponse_TrimSize (0.01s) +--- PASS: TestDNS_Compression_trimUDPResponse (0.02s) +=== CONT TestDNS_trimUDPResponse_TrimLimit +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +--- PASS: TestDNS_trimUDPResponse_TrimSize (0.02s) === CONT TestDNS_trimUDPResponse_NoTrim ---- PASS: TestDNS_trimUDPResponse_NoTrim (0.01s) -=== CONT TestAgent_InmemKeyrings -=== RUN TestAgent_InmemKeyrings/no_keys -2022-11-07T22:46:03.060Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.070Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.101 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f9a2e498-8f0a-4778-1b55-ef652939ef7d Address:127.0.0.1:11553}]" -46:03.101 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11553 [Follower]" leader= -46:03.102 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d.dc1 127.0.0.1 -46:03.102 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d 127.0.0.1 -2022-11-07T22:46:03.102Z [INFO] agent.router: Initializing LAN area manager -46:03.102 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d.dc1 area=wan -46:03.102 [INFO] TestAgent.server: Adding LAN server: server="Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d (Addr: tcp/127.0.0.1:11553) (DC: dc1)" -46:03.102 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11548 network=udp -46:03.102 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11548 network=tcp -46:03.103 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11549 network=tcp -46:03.103 [INFO] TestAgent: started state syncer -46:03.103 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11554 network=tcp -46:03.144 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.144 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11553 [Candidate]" term=2 -46:03.154 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.154 [DEBUG] TestAgent.server.raft: vote granted: from=f9a2e498-8f0a-4778-1b55-ef652939ef7d term=2 tally=1 -46:03.154 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.154 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11553 [Leader]" -46:03.154 [INFO] TestAgent.server: cluster leadership acquired -46:03.154 [INFO] TestAgent.server: New leader elected: payload=Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d -46:03.161 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11553 -46:03.185 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.213 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.213 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.213 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.213 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.213 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.213 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d -46:03.213 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f9a2e498-8f0a-4778-1b55-ef652939ef7d -46:03.231 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.269 [INFO] TestAgent: Requesting shutdown -46:03.269 [INFO] TestAgent.server: shutting down server -46:03.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.269 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.269 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.269 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.269 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.269 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.269 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.269 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.279 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.288Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.288Z [INFO] agent.router.manager: shutting down -46:03.288 [INFO] TestAgent: consul server down -46:03.288 [INFO] TestAgent: shutdown complete -46:03.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11548 network=tcp -46:03.289 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11548 network=udp -46:03.289 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11549 network=tcp -46:03.789 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.789 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthChecksInState_Filter (0.74s) +--- PASS: TestDNS_trimUDPResponse_NoTrim (0.03s) +=== CONT TestDNS_PreparedQuery_AgentSource +--- PASS: TestDNS_trimUDPResponse_TrimLimit (0.06s) === CONT TestDNS_InvalidQueries -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:03.045Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.060Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.083 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 Address:127.0.0.1:11546}]" -46:03.083 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11546 [Follower]" leader= -46:03.084 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5.dc1 127.0.0.1 -46:03.084 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 127.0.0.1 -2022-11-07T22:46:03.084Z [INFO] agent.router: Initializing LAN area manager -46:03.084 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11541 network=udp -46:03.084 [INFO] TestAgent.server: Adding LAN server: server="Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 (Addr: tcp/127.0.0.1:11546) (DC: dc1)" -46:03.084 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5.dc1 area=wan -46:03.085 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11541 network=tcp -46:03.085 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11542 network=tcp -46:03.085 [INFO] TestAgent: started state syncer -46:03.085 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11547 network=tcp -46:03.120 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.120 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11546 [Candidate]" term=2 -46:03.127 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.127 [DEBUG] TestAgent.server.raft: vote granted: from=fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 term=2 tally=1 -46:03.127 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.127 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11546 [Leader]" -46:03.127 [INFO] TestAgent.server: cluster leadership acquired -46:03.127 [INFO] TestAgent.server: New leader elected: payload=Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 -46:03.132 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11546 -46:03.142 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.176 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.176 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.176 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.176 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.176 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 -46:03.176 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fb6b5e67-893d-ffa7-1e1a-4a7520645bc5 -46:03.190 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.262 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.268 [INFO] TestAgent: Synced node info -46:03.268 [DEBUG] TestAgent: Node info in sync -46:03.367 [INFO] TestAgent: Requesting shutdown -46:03.367 [INFO] TestAgent.server: shutting down server -46:03.367 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.367 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.367 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.367 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.367 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.367 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.367 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.367 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.367 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.367 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.376 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.387Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.387Z [INFO] agent.router.manager: shutting down -46:03.387 [INFO] TestAgent: consul server down -46:03.387 [INFO] TestAgent: shutdown complete -46:03.387 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11541 network=tcp -46:03.387 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11541 network=udp -46:03.387 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11542 network=tcp -46:03.887 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.887 [INFO] TestAgent: Endpoints down ---- PASS: TestHealthNodeChecks_Filtering (0.85s) +2021-10-05T16:31:17.163Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.189Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.255 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:016029be-a0f2-a9a7-e961-8c8d988e4af0 Address:127.0.0.1:29855}]" +31:17.256 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29855 [Follower]" leader= +31:17.256 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-016029be-a0f2-a9a7-e961-8c8d988e4af0.dc1 127.0.0.1 +31:17.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-016029be-a0f2-a9a7-e961-8c8d988e4af0 127.0.0.1 +2021-10-05T16:31:17.257Z [INFO] agent.router: Initializing LAN area manager +31:17.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29850 network=udp +31:17.257 [INFO] TestAgent.server: Adding LAN server: server="Node-016029be-a0f2-a9a7-e961-8c8d988e4af0 (Addr: tcp/127.0.0.1:29855) (DC: dc1)" +31:17.257 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-016029be-a0f2-a9a7-e961-8c8d988e4af0.dc1 area=wan +31:17.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29850 network=tcp +31:17.258 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29851 network=tcp +31:17.258 [INFO] TestAgent: started state syncer +31:17.258 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29856 network=tcp +31:17.312 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.312 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29855 [Candidate]" term=2 +31:17.340 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.340 [DEBUG] TestAgent.server.raft: vote granted: from=016029be-a0f2-a9a7-e961-8c8d988e4af0 term=2 tally=1 +31:17.340 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.340 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29855 [Leader]" +31:17.343 [INFO] TestAgent.server: cluster leadership acquired +31:17.344 [INFO] TestAgent.server: New leader elected: payload=Node-016029be-a0f2-a9a7-e961-8c8d988e4af0 +31:17.346 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29855 +31:17.388 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.432 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.432 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.432 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.432 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.432 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.432 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-016029be-a0f2-a9a7-e961-8c8d988e4af0 +31:17.432 [INFO] TestAgent.server: member joined, marking health alive: member=Node-016029be-a0f2-a9a7-e961-8c8d988e4af0 +31:17.447 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.451 [INFO] TestAgent: Requesting shutdown +31:17.451 [INFO] TestAgent.server: shutting down server +31:17.451 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.451 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.451 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.451 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.451 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.451 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.451 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.451 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:17.451 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.451 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.451 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.458 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.459Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.459Z [INFO] agent.router.manager: shutting down +31:17.459 [INFO] TestAgent: consul server down +31:17.459 [INFO] TestAgent: shutdown complete +31:17.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29850 network=tcp +31:17.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29850 network=udp +31:17.459 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29851 network=tcp +31:17.960 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.960 [INFO] TestAgent: Endpoints down +--- PASS: TestEventFire (0.87s) === CONT TestDNS_PreparedQuery_AllowStale -2022-11-07T22:46:03.197Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.197Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.207Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.207Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.233 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f5558c2-4d30-bc57-a2e5-213680af63d8 Address:127.0.0.1:11595}]" -46:03.233 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11595 [Follower]" leader= -46:03.234 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4f5558c2-4d30-bc57-a2e5-213680af63d8.dc1 127.0.0.1 -46:03.234 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 127.0.0.1 -2022-11-07T22:46:03.234Z [INFO] agent.router: Initializing LAN area manager -46:03.234 [INFO] TestAgent.server: Adding LAN server: server="Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 (Addr: tcp/127.0.0.1:11595) (DC: dc1)" -46:03.234 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8.dc1 area=wan -46:03.235 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11590 network=udp -46:03.235 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11590 network=tcp -46:03.235 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11591 network=tcp -46:03.235 [INFO] TestAgent: started state syncer -46:03.235 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11617 network=tcp -46:03.269 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.269 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11595 [Candidate]" term=2 -46:03.285 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:03.288 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.288 [DEBUG] TestAgent.server.raft: vote granted: from=4f5558c2-4d30-bc57-a2e5-213680af63d8 term=2 tally=1 -46:03.288 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.288 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11595 [Leader]" -46:03.288 [INFO] TestAgent.server: cluster leadership acquired -46:03.288 [INFO] TestAgent.server: New leader elected: payload=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.296 [INFO] TestAgent.server: initializing acls -46:03.299 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:03.299 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:03.314 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:03.337 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:03.337 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:03.337 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:03.337 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.337 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4f5558c2-4d30-bc57-a2e5-213680af63d8.dc1 -46:03.337 [INFO] TestAgent.server: Updating LAN server: server="Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 (Addr: tcp/127.0.0.1:11595) (DC: dc1)" -46:03.337 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8.dc1 area=wan -46:03.356 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.391 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.391 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.391 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.391 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.391 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.391 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.391 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.410 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.424 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.430 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4f5558c2-4d30-bc57-a2e5-213680af63d8 -46:03.440 [INFO] TestAgent: Synced node info -46:03.440 [DEBUG] TestAgent: Node info in sync -46:03.440 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=foo accessorID= -46:03.440 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=bar accessorID= -46:03.440 [INFO] TestAgent: Requesting shutdown -46:03.440 [INFO] TestAgent.server: shutting down server -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.440 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:03.440 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.440 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.440 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:03.444 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.450Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.450Z [INFO] agent.router.manager: shutting down -46:03.450 [INFO] TestAgent: consul server down -46:03.450 [INFO] TestAgent: shutdown complete -46:03.450 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11590 network=tcp -46:03.450 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11590 network=udp -46:03.450 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11591 network=tcp -46:03.950 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.950 [INFO] TestAgent: Endpoints down ---- PASS: TestEventFire_token (0.77s) -=== CONT TestDNS_AltDomains_SOA -2022-11-07T22:46:03.182Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.191Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.222 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c46f6ee0-bb23-309a-6841-841b8252e758 Address:127.0.0.1:11602}]" -46:03.223 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11602 [Follower]" leader= -46:03.225 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c46f6ee0-bb23-309a-6841-841b8252e758.dc1 127.0.0.1 -46:03.227 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c46f6ee0-bb23-309a-6841-841b8252e758 127.0.0.1 -2022-11-07T22:46:03.227Z [INFO] agent.router: Initializing LAN area manager -46:03.227 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c46f6ee0-bb23-309a-6841-841b8252e758.dc1 area=wan -46:03.227 [INFO] TestAgent.server: Adding LAN server: server="Node-c46f6ee0-bb23-309a-6841-841b8252e758 (Addr: tcp/127.0.0.1:11602) (DC: dc1)" -46:03.227 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11597 network=udp -46:03.227 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11597 network=tcp -46:03.227 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11598 network=tcp -46:03.227 [INFO] TestAgent: started state syncer -46:03.227 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11589 network=tcp -46:03.288 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.288 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11602 [Candidate]" term=2 -46:03.301 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.301 [DEBUG] TestAgent.server.raft: vote granted: from=c46f6ee0-bb23-309a-6841-841b8252e758 term=2 tally=1 -46:03.301 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.301 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11602 [Leader]" -46:03.301 [INFO] TestAgent.server: cluster leadership acquired -46:03.301 [INFO] TestAgent.server: New leader elected: payload=Node-c46f6ee0-bb23-309a-6841-841b8252e758 -46:03.310 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11602 -46:03.338 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.378 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.378 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.378 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.378 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.378 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.378 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c46f6ee0-bb23-309a-6841-841b8252e758 -46:03.378 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c46f6ee0-bb23-309a-6841-841b8252e758 -46:03.396 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.458 [DEBUG] TestAgent.server: User event: event=test -46:03.458 [DEBUG] TestAgent.server: User event: event=foo -46:03.458 [DEBUG] TestAgent: new event: event_name=test event_id=7a78d341-6851-2337-2e23-0455ab0b1ab4 -46:03.458 [DEBUG] TestAgent: new event: event_name=foo event_id=dad90c58-161a-066a-f191-197841fce992 -46:03.484 [INFO] TestAgent: Requesting shutdown -46:03.484 [INFO] TestAgent.server: shutting down server -46:03.484 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.484 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.484 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.484 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.484 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.484 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.484 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.484 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.484 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.484 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.484 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.488 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.494Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.494Z [INFO] agent.router.manager: shutting down -46:03.494 [INFO] TestAgent: consul server down -46:03.494 [INFO] TestAgent: shutdown complete -46:03.494 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11597 network=tcp -46:03.494 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11597 network=udp -46:03.494 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11598 network=tcp -46:03.994 [INFO] TestAgent: Waiting for endpoints to shut down -46:03.994 [INFO] TestAgent: Endpoints down ---- PASS: TestEventList_Filter (0.82s) -=== CONT TestAgent_LoadKeyrings -=== RUN TestAgent_LoadKeyrings/no_keys -2022-11-07T22:46:03.129Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.138Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.174 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4acb539b-6a42-7e3a-04b3-aff60976856e Address:127.0.0.1:11567}]" -46:03.174 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11567 [Follower]" leader= -46:03.174 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4acb539b-6a42-7e3a-04b3-aff60976856e.dc1 127.0.0.1 -46:03.175 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4acb539b-6a42-7e3a-04b3-aff60976856e 127.0.0.1 -2022-11-07T22:46:03.175Z [INFO] agent.router: Initializing LAN area manager -46:03.175 [INFO] TestAgent.server: Adding LAN server: server="Node-4acb539b-6a42-7e3a-04b3-aff60976856e (Addr: tcp/127.0.0.1:11567) (DC: dc1)" -46:03.175 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4acb539b-6a42-7e3a-04b3-aff60976856e.dc1 area=wan -46:03.175 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11562 network=udp -46:03.175 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11562 network=tcp -46:03.175 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11563 network=tcp -46:03.175 [INFO] TestAgent: started state syncer -46:03.175 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11568 network=tcp -46:03.243 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.243 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11567 [Candidate]" term=2 -46:03.259 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.259 [DEBUG] TestAgent.server.raft: vote granted: from=4acb539b-6a42-7e3a-04b3-aff60976856e term=2 tally=1 -46:03.259 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.259 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11567 [Leader]" -46:03.259 [INFO] TestAgent.server: cluster leadership acquired -46:03.259 [INFO] TestAgent.server: New leader elected: payload=Node-4acb539b-6a42-7e3a-04b3-aff60976856e -46:03.273 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11567 -46:03.298 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.341 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.341 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.341 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.341 [INFO] TestAgent: Synced node info -46:03.341 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.341 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.341 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4acb539b-6a42-7e3a-04b3-aff60976856e -46:03.341 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4acb539b-6a42-7e3a-04b3-aff60976856e -46:03.361 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.480 [DEBUG] TestAgent.server: User event: event=foo -46:03.480 [DEBUG] TestAgent.server: User event: event=bar -46:03.480 [DEBUG] TestAgent.server: User event: event=foo -46:03.480 [DEBUG] TestAgent.server: User event: event=foo -46:03.480 [DEBUG] TestAgent.server: User event: event=bar -46:03.480 [DEBUG] TestAgent: new event: event_name=foo event_id=c3fa92d2-0b0f-27c1-f5e7-c6759413e97d -46:03.480 [DEBUG] TestAgent: new event: event_name=bar event_id=8caba363-2f2c-2219-b68c-d7963a03f079 -46:03.480 [DEBUG] TestAgent: new event: event_name=foo event_id=7e3969eb-86c7-c189-72af-cd8db4f3bcb1 -46:03.480 [DEBUG] TestAgent: new event: event_name=foo event_id=88ff9d38-7ac7-7807-ff4a-7785b66ecc68 -46:03.480 [DEBUG] TestAgent: new event: event_name=bar event_id=794c0b4f-3c42-7a42-7cd0-556986ead0d1 -46:03.505 [INFO] TestAgent: Requesting shutdown -46:03.505 [INFO] TestAgent.server: shutting down server -46:03.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.505 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.505 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.505 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.505 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.505 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.505 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.509 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.515Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.515Z [INFO] agent.router.manager: shutting down -46:03.515 [INFO] TestAgent: consul server down -46:03.515 [INFO] TestAgent: shutdown complete -46:03.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11562 network=tcp -46:03.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11562 network=udp -46:03.515 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11563 network=tcp -46:04.015 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.015 [INFO] TestAgent: Endpoints down ---- PASS: TestEventList_EventBufOrder (0.90s) +2021-10-05T16:31:16.939Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:16.979Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.027 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7de0b201-dbba-b8b8-9134-7ea3d697afa4 Address:127.0.0.1:29841}]" +31:17.027 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29841 [Follower]" leader= +31:17.028 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4.dc1 127.0.0.1 +31:17.033 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4 127.0.0.1 +2021-10-05T16:31:17.033Z [INFO] agent.router: Initializing LAN area manager +31:17.034 [INFO] TestAgent.server: Adding LAN server: server="Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4 (Addr: tcp/127.0.0.1:29841) (DC: dc1)" +31:17.034 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4.dc1 area=wan +31:17.034 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29773 network=udp +31:17.034 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29773 network=tcp +31:17.034 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29774 network=tcp +31:17.034 [INFO] TestAgent: started state syncer +31:17.035 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29842 network=tcp +31:17.128 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.128 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29841 [Candidate]" term=2 +31:17.151 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.151 [DEBUG] TestAgent.server.raft: vote granted: from=7de0b201-dbba-b8b8-9134-7ea3d697afa4 term=2 tally=1 +31:17.151 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.151 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29841 [Leader]" +31:17.152 [INFO] TestAgent.server: cluster leadership acquired +31:17.152 [INFO] TestAgent.server: New leader elected: payload=Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4 +31:17.166 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29841 +31:17.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.245 [INFO] TestAgent: Synced node info +31:17.245 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.245 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.245 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.245 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.245 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.245 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4 +31:17.245 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7de0b201-dbba-b8b8-9134-7ea3d697afa4 +31:17.262 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.464 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:17.464 [INFO] TestAgent: Requesting shutdown +31:17.464 [INFO] TestAgent.server: shutting down server +31:17.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.464 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.464 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.464 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.464 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.464 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.464 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.472 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.476Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.476Z [INFO] agent.router.manager: shutting down +31:17.476 [INFO] TestAgent: consul server down +31:17.476 [INFO] TestAgent: shutdown complete +31:17.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29773 network=tcp +31:17.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29773 network=udp +31:17.476 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29774 network=tcp +31:17.980 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.980 [INFO] TestAgent: Endpoints down +=== RUN TestPreparedQuery_Execute/#07 +2021-10-05T16:31:17.156Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:17.156Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.190Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:17.190Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.226 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7303199b-c3c4-0db7-a874-64abde82a5f3 Address:127.0.0.1:29848}]" +31:17.227 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 127.0.0.1 +31:17.228 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29848 [Follower]" leader= +31:17.228 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7303199b-c3c4-0db7-a874-64abde82a5f3 127.0.0.1 +2021-10-05T16:31:17.228Z [INFO] agent.router: Initializing LAN area manager +31:17.228 [INFO] TestAgent.server: Adding LAN server: server="Node-7303199b-c3c4-0db7-a874-64abde82a5f3 (Addr: tcp/127.0.0.1:29848) (DC: dc1)" +31:17.228 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29843 network=udp +31:17.228 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 area=wan +31:17.228 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29843 network=tcp +31:17.229 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29844 network=tcp +31:17.229 [INFO] TestAgent: started state syncer +31:17.229 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29849 network=tcp +31:17.271 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.271 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29848 [Candidate]" term=2 +31:17.281 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:17.295 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.295 [DEBUG] TestAgent.server.raft: vote granted: from=7303199b-c3c4-0db7-a874-64abde82a5f3 term=2 tally=1 +31:17.295 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29848 [Leader]" +31:17.296 [INFO] TestAgent.server: cluster leadership acquired +31:17.296 [INFO] TestAgent.server: New leader elected: payload=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.328 [INFO] TestAgent.server: initializing acls +31:17.344 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:17.344 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:17.368 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:17.384 [INFO] TestAgent.server: initializing acls +31:17.384 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:17.415 [INFO] TestAgent: Synced node info +31:17.415 [DEBUG] TestAgent: Node info in sync +31:17.415 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:17.415 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:17.415 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:17.415 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 +31:17.416 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:17.416 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:17.416 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 +31:17.416 [INFO] TestAgent.server: Updating LAN server: server="Node-7303199b-c3c4-0db7-a874-64abde82a5f3 (Addr: tcp/127.0.0.1:29848) (DC: dc1)" +31:17.416 [INFO] TestAgent.server: Updating LAN server: server="Node-7303199b-c3c4-0db7-a874-64abde82a5f3 (Addr: tcp/127.0.0.1:29848) (DC: dc1)" +31:17.417 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 area=wan +31:17.417 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7303199b-c3c4-0db7-a874-64abde82a5f3.dc1 area=wan +31:17.429 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.454 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.454 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.471 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=foo accessorID= +31:17.471 [WARN] TestAgent.server.internal: user event blocked by ACLs: event=bar accessorID= +31:17.471 [INFO] TestAgent: Requesting shutdown +31:17.471 [INFO] TestAgent.server: shutting down server +31:17.471 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:17.471 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:17.471 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.472 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:17.472 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:17.478 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:17.481 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.481 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.481 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.481 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.481 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.481 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.481 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +2021-10-05T16:31:17.484Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.484Z [INFO] agent.router.manager: shutting down +31:17.488 [INFO] TestAgent: consul server down +31:17.488 [INFO] TestAgent: shutdown complete +31:17.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29843 network=tcp +31:17.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29843 network=udp +31:17.488 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29844 network=tcp +31:17.488 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-7303199b-c3c4-0db7-a874-64abde82a5f3 127.0.0.1 29846 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:7303199b-c3c4-0db7-a874-64abde82a5f3 port:29848 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29847] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:17.488 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.488 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7303199b-c3c4-0db7-a874-64abde82a5f3 +31:17.489 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-7303199b-c3c4-0db7-a874-64abde82a5f3 127.0.0.1 29846 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:7303199b-c3c4-0db7-a874-64abde82a5f3 port:29848 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29847] alive 1 5 2 2 5 4}" error="raft is already shutdown" +31:17.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.489 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.489 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.489 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.489 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.489 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.489 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.991 [INFO] TestAgent: Waiting for endpoints to shut down +31:17.991 [INFO] TestAgent: Endpoints down +--- PASS: TestEventFire_token (0.91s) === CONT TestDNS_AltDomains_Overlap -2022-11-07T22:46:03.257Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.268Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.322 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d632528f-b506-2193-12b5-5b3771b6562e Address:127.0.0.1:11609}]" -46:03.322 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11609 [Follower]" leader= -46:03.323 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d632528f-b506-2193-12b5-5b3771b6562e.dc1 127.0.0.1 -46:03.323 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d632528f-b506-2193-12b5-5b3771b6562e 127.0.0.1 -2022-11-07T22:46:03.323Z [INFO] agent.router: Initializing LAN area manager -46:03.323 [INFO] TestAgent.server: Adding LAN server: server="Node-d632528f-b506-2193-12b5-5b3771b6562e (Addr: tcp/127.0.0.1:11609) (DC: dc1)" -46:03.323 [DEBUG] TestAgent.dns: recursor enabled -46:03.323 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d632528f-b506-2193-12b5-5b3771b6562e.dc1 area=wan -46:03.323 [DEBUG] TestAgent.dns: recursor enabled -46:03.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11604 network=udp -46:03.324 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11604 network=tcp -46:03.324 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11605 network=tcp -46:03.324 [INFO] TestAgent: started state syncer -46:03.324 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11638 network=tcp -46:03.392 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.392 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11609 [Candidate]" term=2 -46:03.407 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.407 [DEBUG] TestAgent.server.raft: vote granted: from=d632528f-b506-2193-12b5-5b3771b6562e term=2 tally=1 -46:03.407 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.407 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11609 [Leader]" -46:03.407 [INFO] TestAgent.server: cluster leadership acquired -46:03.407 [INFO] TestAgent.server: New leader elected: payload=Node-d632528f-b506-2193-12b5-5b3771b6562e -46:03.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11609 -46:03.438 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.465 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.465 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.465 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.465 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.465 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.465 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d632528f-b506-2193-12b5-5b3771b6562e -46:03.465 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d632528f-b506-2193-12b5-5b3771b6562e -46:03.477 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.510 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=121.886µs recursor=127.0.0.1:59931 -46:03.510 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=240.699µs client=127.0.0.1:33901 client_network=udp -46:03.511 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=97.047µs recursor=127.0.0.1:59931 -46:03.511 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=188.474µs client=127.0.0.1:33901 client_network=udp -46:03.511 [INFO] TestAgent: Requesting shutdown -46:03.511 [INFO] TestAgent.server: shutting down server -46:03.511 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.511 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.511 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.511 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.511 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.511 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.511 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.511 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.511 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.511 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.511 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.515 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.521Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.521Z [INFO] agent.router.manager: shutting down -46:03.521 [INFO] TestAgent: consul server down -46:03.521 [INFO] TestAgent: shutdown complete -46:03.521 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11604 network=tcp -46:03.521 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11604 network=udp -46:03.521 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11605 network=tcp -46:04.022 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.022 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Compression_Recurse (0.78s) -=== CONT TestDNS_NonExistingLookupEmptyAorAAAA -2022-11-07T22:46:03.081Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.090Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.115 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f0ef5e4a-8e48-eb92-c453-4142bab393d0 Address:127.0.0.1:11560}]" -46:03.115 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11560 [Follower]" leader= -46:03.115 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0.dc1 127.0.0.1 -46:03.116 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0 127.0.0.1 -2022-11-07T22:46:03.116Z [INFO] agent.router: Initializing LAN area manager -46:03.116 [INFO] TestAgent.server: Adding LAN server: server="Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0 (Addr: tcp/127.0.0.1:11560) (DC: dc1)" -46:03.116 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11555 network=udp -46:03.116 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0.dc1 area=wan -46:03.116 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11555 network=tcp -46:03.116 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11556 network=tcp -46:03.116 [INFO] TestAgent: started state syncer -46:03.116 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11561 network=tcp -46:03.157 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.157 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11560 [Candidate]" term=2 -46:03.170 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.170 [DEBUG] TestAgent.server.raft: vote granted: from=f0ef5e4a-8e48-eb92-c453-4142bab393d0 term=2 tally=1 -46:03.170 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.170 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11560 [Leader]" -46:03.170 [INFO] TestAgent.server: cluster leadership acquired -46:03.170 [INFO] TestAgent.server: New leader elected: payload=Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0 -46:03.177 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11560 -46:03.194 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.220 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.220 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.221 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.221 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.221 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.221 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0 -46:03.221 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f0ef5e4a-8e48-eb92-c453-4142bab393d0 -46:03.234 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.266 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.273 [INFO] TestAgent: Synced node info -46:03.474 [DEBUG] TestAgent.server: User event: event=test -46:03.474 [DEBUG] TestAgent: new event: event_name=test event_id=9bfc9bbb-a723-9418-a217-eda0c77a5d90 -46:03.524 [DEBUG] TestAgent.server: User event: event=second -46:03.524 [DEBUG] TestAgent: new event: event_name=second event_id=ff28ea0f-84b3-4084-4cdb-4ef27ec03dac -46:03.525 [INFO] TestAgent: Requesting shutdown -46:03.525 [INFO] TestAgent.server: shutting down server -46:03.525 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.525 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.525 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.525 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.525 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.525 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.525 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.525 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.525 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.525 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.530 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.537Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.537Z [INFO] agent.router.manager: shutting down -46:03.537 [INFO] TestAgent: consul server down -46:03.537 [INFO] TestAgent: shutdown complete -46:03.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11555 network=tcp -46:03.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11555 network=udp -46:03.537 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11556 network=tcp -46:04.037 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.037 [INFO] TestAgent: Endpoints down ---- PASS: TestEventList_Blocking (0.97s) -=== CONT TestDNS_NonExistentDC_RPC -2022-11-07T22:46:03.349Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.360Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.406 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b3e93860-7f55-7e81-babf-3e8e234d10e9 Address:127.0.0.1:11644}]" -46:03.406 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11644 [Follower]" leader= -46:03.406 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b3e93860-7f55-7e81-babf-3e8e234d10e9.dc1 127.0.0.1 -46:03.407 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b3e93860-7f55-7e81-babf-3e8e234d10e9 127.0.0.1 -2022-11-07T22:46:03.407Z [INFO] agent.router: Initializing LAN area manager -46:03.407 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b3e93860-7f55-7e81-babf-3e8e234d10e9.dc1 area=wan -46:03.407 [INFO] TestAgent.server: Adding LAN server: server="Node-b3e93860-7f55-7e81-babf-3e8e234d10e9 (Addr: tcp/127.0.0.1:11644) (DC: dc1)" -46:03.407 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11639 network=udp -46:03.407 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11639 network=tcp -46:03.407 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11640 network=tcp -46:03.407 [INFO] TestAgent: started state syncer -46:03.408 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11631 network=tcp -46:03.451 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.451 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11644 [Candidate]" term=2 -46:03.463 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.463 [DEBUG] TestAgent.server.raft: vote granted: from=b3e93860-7f55-7e81-babf-3e8e234d10e9 term=2 tally=1 -46:03.463 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.463 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11644 [Leader]" -46:03.463 [INFO] TestAgent.server: cluster leadership acquired -46:03.463 [INFO] TestAgent.server: New leader elected: payload=Node-b3e93860-7f55-7e81-babf-3e8e234d10e9 -46:03.470 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11644 -46:03.481 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.503 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.503 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.503 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3e93860-7f55-7e81-babf-3e8e234d10e9 -46:03.503 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b3e93860-7f55-7e81-babf-3e8e234d10e9 -46:03.517 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.529 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=111.989µs client=127.0.0.1:58051 client_network=udp -46:03.529 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=74.963µs client=127.0.0.1:58051 client_network=udp -46:03.529 [DEBUG] TestAgent.dns: request served from client: name=0cc4279d-bb08-2ebf-755d-5901e47a2e33.query.consul. type=SRV class=IN latency=96.454µs client=127.0.0.1:34894 client_network=udp -46:03.530 [DEBUG] TestAgent.dns: request served from client: name=0cc4279d-bb08-2ebf-755d-5901e47a2e33.query.consul. type=SRV class=IN latency=88.084µs client=127.0.0.1:34894 client_network=udp -46:03.530 [INFO] TestAgent: Requesting shutdown -46:03.530 [INFO] TestAgent.server: shutting down server -46:03.530 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.530 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.530 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.530 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.530 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.530 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.530 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.530 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.530 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.530 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.530 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.537 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.540Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.540Z [INFO] agent.router.manager: shutting down -46:03.540 [INFO] TestAgent: consul server down -46:03.540 [INFO] TestAgent: shutdown complete -46:03.540 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11639 network=tcp -46:03.540 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11639 network=udp -46:03.540 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11640 network=tcp -46:04.040 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.040 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Compression_Query (0.70s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:17.402Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.436Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.516 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0dcb57e3-abe6-b8b4-a2a6-36971988d42c Address:127.0.0.1:29862}]" +31:17.516 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29862 [Follower]" leader= +31:17.517 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c.dc1 127.0.0.1 +31:17.517 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c 127.0.0.1 +2021-10-05T16:31:17.517Z [INFO] agent.router: Initializing LAN area manager +31:17.517 [DEBUG] TestAgent.dns: recursor enabled +31:17.517 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29871 network=udp +31:17.518 [INFO] TestAgent.server: Adding LAN server: server="Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c (Addr: tcp/127.0.0.1:29862) (DC: dc1)" +31:17.518 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c.dc1 area=wan +31:17.518 [DEBUG] TestAgent.dns: recursor enabled +31:17.518 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29871 network=tcp +31:17.518 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29872 network=tcp +31:17.518 [INFO] TestAgent: started state syncer +31:17.518 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29863 network=tcp +31:17.570 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.570 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29862 [Candidate]" term=2 +31:17.583 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.583 [DEBUG] TestAgent.server.raft: vote granted: from=0dcb57e3-abe6-b8b4-a2a6-36971988d42c term=2 tally=1 +31:17.583 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.583 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29862 [Leader]" +31:17.584 [INFO] TestAgent.server: cluster leadership acquired +31:17.584 [INFO] TestAgent.server: New leader elected: payload=Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c +31:17.593 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29862 +31:17.612 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.641 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.641 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.641 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.641 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.641 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.641 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c +31:17.641 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0dcb57e3-abe6-b8b4-a2a6-36971988d42c +31:17.666 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.747 [INFO] TestAgent: Requesting shutdown +31:17.747 [INFO] TestAgent.server: shutting down server +31:17.747 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.747 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.747 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:17.753 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.757Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.757Z [INFO] agent.router.manager: shutting down +31:17.757 [INFO] TestAgent: consul server down +31:17.757 [INFO] TestAgent: shutdown complete +31:17.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29871 network=tcp +31:17.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29871 network=udp +31:17.757 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29872 network=tcp +31:18.260 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.260 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ConfigReload (0.91s) +=== CONT TestDNS_AltDomains_SOA +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:17.390Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.406Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.484 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4945891a-1343-1571-9055-a769bc0019d8 Address:127.0.0.1:29869}]" +31:17.484 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29869 [Follower]" leader= +31:17.484 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4945891a-1343-1571-9055-a769bc0019d8.dc1 127.0.0.1 +31:17.485 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4945891a-1343-1571-9055-a769bc0019d8 127.0.0.1 +2021-10-05T16:31:17.485Z [INFO] agent.router: Initializing LAN area manager +31:17.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29857 network=udp +31:17.486 [INFO] TestAgent.server: Adding LAN server: server="Node-4945891a-1343-1571-9055-a769bc0019d8 (Addr: tcp/127.0.0.1:29869) (DC: dc1)" +31:17.486 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4945891a-1343-1571-9055-a769bc0019d8.dc1 area=wan +31:17.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29857 network=tcp +31:17.486 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29858 network=tcp +31:17.486 [INFO] TestAgent: started state syncer +31:17.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29870 network=tcp +31:17.540 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.540 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29869 [Candidate]" term=2 +31:17.556 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.556 [DEBUG] TestAgent.server.raft: vote granted: from=4945891a-1343-1571-9055-a769bc0019d8 term=2 tally=1 +31:17.556 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.556 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29869 [Leader]" +31:17.560 [INFO] TestAgent.server: cluster leadership acquired +31:17.560 [INFO] TestAgent.server: New leader elected: payload=Node-4945891a-1343-1571-9055-a769bc0019d8 +31:17.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29869 +31:17.585 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.616 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.616 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.616 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.616 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.616 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.616 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4945891a-1343-1571-9055-a769bc0019d8 +31:17.616 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4945891a-1343-1571-9055-a769bc0019d8 +31:17.624 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.635 [INFO] TestAgent: Synced node info +31:17.857 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:17.959 [INFO] TestAgent: Requesting shutdown +31:17.959 [INFO] TestAgent.server: shutting down server +31:17.959 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.959 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.968 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:17.980Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:17.980Z [INFO] agent.router.manager: shutting down +31:17.980 [INFO] TestAgent: consul server down +31:17.980 [INFO] TestAgent: shutdown complete +31:17.980 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29857 network=tcp +31:17.980 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29857 network=udp +31:17.980 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29858 network=tcp +31:18.480 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.480 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ReloadConfig_DuringQuery (1.12s) === CONT TestDNS_AltDomains_Service -2022-11-07T22:46:03.137Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.137Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.151Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.151Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.192 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d Address:127.0.0.1:11574}]" -46:03.192 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11574 [Follower]" leader= -46:03.192 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d.dc1 127.0.0.1 -46:03.193 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d 127.0.0.1 -2022-11-07T22:46:03.193Z [INFO] agent.router: Initializing LAN area manager -46:03.193 [INFO] TestAgent.server: Adding LAN server: server="Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d (Addr: tcp/127.0.0.1:11574) (DC: dc1)" -46:03.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11569 network=udp -46:03.193 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d.dc1 area=wan -46:03.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11569 network=tcp -46:03.194 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11570 network=tcp -46:03.194 [INFO] TestAgent: started state syncer -46:03.194 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11575 network=tcp -46:03.238 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.238 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11574 [Candidate]" term=2 -46:03.243 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:03.251 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.251 [DEBUG] TestAgent.server.raft: vote granted: from=dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d term=2 tally=1 -46:03.251 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.251 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11574 [Leader]" -46:03.252 [INFO] TestAgent.server: cluster leadership acquired -46:03.252 [INFO] TestAgent.server: New leader elected: payload=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.262 [INFO] TestAgent.server: initializing acls -46:03.268 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:03.268 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:03.284 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:03.303 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:03.303 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:03.303 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:03.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d.dc1 -46:03.303 [INFO] TestAgent.server: Updating LAN server: server="Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d (Addr: tcp/127.0.0.1:11574) (DC: dc1)" -46:03.303 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d.dc1 area=wan -46:03.330 [INFO] TestAgent: Synced node info -46:03.335 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.369 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.369 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.369 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.369 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.369 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.369 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.369 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.391 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.391 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.537 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.537 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-dcf3c2bd-2d36-6ff9-7d27-b0b02bcc203d -46:03.537 [DEBUG] TestAgent.server: User event: event=foo -46:03.537 [DEBUG] TestAgent: new event: event_name=foo event_id=c75542fc-d2c6-c0f9-99b0-17265e4814f9 -46:03.537 [INFO] TestAgent: Requesting shutdown -46:03.537 [INFO] TestAgent.server: shutting down server -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:03.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.537 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.537 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:03.542 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.547Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.547Z [INFO] agent.router.manager: shutting down -46:03.548 [INFO] TestAgent: consul server down -46:03.548 [INFO] TestAgent: shutdown complete -46:03.548 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11569 network=tcp -46:03.548 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11569 network=udp -46:03.548 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11570 network=tcp -46:04.048 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.048 [INFO] TestAgent: Endpoints down ---- PASS: TestEventList_ACLFilter (0.93s) - --- PASS: TestEventList_ACLFilter/no_token (0.00s) - --- PASS: TestEventList_ACLFilter/root_token (0.00s) -=== CONT TestIntentionsSpecificDelete_good -2022-11-07T22:46:03.220Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.233Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.274 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b7070f0-6a2c-0747-43c2-71166acff99f Address:127.0.0.1:11616}]" -46:03.274 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11616 [Follower]" leader= -46:03.274 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b7070f0-6a2c-0747-43c2-71166acff99f.dc1 127.0.0.1 -46:03.275 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b7070f0-6a2c-0747-43c2-71166acff99f 127.0.0.1 -2022-11-07T22:46:03.275Z [INFO] agent.router: Initializing LAN area manager -46:03.275 [INFO] TestAgent.server: Adding LAN server: server="Node-4b7070f0-6a2c-0747-43c2-71166acff99f (Addr: tcp/127.0.0.1:11616) (DC: dc1)" -46:03.275 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b7070f0-6a2c-0747-43c2-71166acff99f.dc1 area=wan -46:03.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11611 network=udp -46:03.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11611 network=tcp -46:03.275 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11612 network=tcp -46:03.275 [INFO] TestAgent: started state syncer -46:03.275 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11624 network=tcp -46:03.334 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.334 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11616 [Candidate]" term=2 -46:03.350 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.350 [DEBUG] TestAgent.server.raft: vote granted: from=4b7070f0-6a2c-0747-43c2-71166acff99f term=2 tally=1 -46:03.350 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.350 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11616 [Leader]" -46:03.351 [INFO] TestAgent.server: cluster leadership acquired -46:03.351 [INFO] TestAgent.server: New leader elected: payload=Node-4b7070f0-6a2c-0747-43c2-71166acff99f -46:03.366 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11616 -46:03.390 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.434 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.434 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.434 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.434 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.434 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.434 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b7070f0-6a2c-0747-43c2-71166acff99f -46:03.434 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b7070f0-6a2c-0747-43c2-71166acff99f -46:03.451 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.556 [INFO] TestAgent: Requesting shutdown -46:03.556 [INFO] TestAgent.server: shutting down server -46:03.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.556 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.556 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.556 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.556 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.556 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.556 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.556 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.560 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.566Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.566Z [INFO] agent.router.manager: shutting down -46:03.566 [INFO] TestAgent: consul server down -46:03.566 [INFO] TestAgent: shutdown complete -46:03.566 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11611 network=tcp -46:03.566 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11611 network=udp -46:03.566 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11612 network=tcp -46:04.066 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.066 [INFO] TestAgent: Endpoints down ---- PASS: TestEventFire (0.86s) +2021-10-05T16:31:17.523Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.535Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.565 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b69b1b7c-4ed9-2806-6375-540e512a9e06 Address:127.0.0.1:29883}]" +31:17.566 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b69b1b7c-4ed9-2806-6375-540e512a9e06.dc1 127.0.0.1 +31:17.566 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b69b1b7c-4ed9-2806-6375-540e512a9e06 127.0.0.1 +2021-10-05T16:31:17.566Z [INFO] agent.router: Initializing LAN area manager +31:17.566 [DEBUG] TestAgent.dns: recursor enabled +31:17.567 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29864 network=udp +31:17.567 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29883 [Follower]" leader= +31:17.567 [INFO] TestAgent.server: Adding LAN server: server="Node-b69b1b7c-4ed9-2806-6375-540e512a9e06 (Addr: tcp/127.0.0.1:29883) (DC: dc1)" +31:17.567 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b69b1b7c-4ed9-2806-6375-540e512a9e06.dc1 area=wan +31:17.567 [DEBUG] TestAgent.dns: recursor enabled +31:17.567 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29864 network=tcp +31:17.568 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29865 network=tcp +31:17.568 [INFO] TestAgent: started state syncer +31:17.568 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29884 network=tcp +31:17.624 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.624 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29883 [Candidate]" term=2 +31:17.639 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.639 [DEBUG] TestAgent.server.raft: vote granted: from=b69b1b7c-4ed9-2806-6375-540e512a9e06 term=2 tally=1 +31:17.639 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.639 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29883 [Leader]" +31:17.639 [INFO] TestAgent.server: cluster leadership acquired +31:17.639 [INFO] TestAgent.server: New leader elected: payload=Node-b69b1b7c-4ed9-2806-6375-540e512a9e06 +31:17.648 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29883 +31:17.666 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b69b1b7c-4ed9-2806-6375-540e512a9e06 +31:17.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b69b1b7c-4ed9-2806-6375-540e512a9e06 +31:17.724 [INFO] TestAgent.server: federation state anti-entropy synced +31:17.932 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:17.944 [INFO] TestAgent: Synced node info +31:17.968 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=299.881µs recursor=127.0.0.1:45602 +31:17.968 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=454.826µs client=127.0.0.1:36935 client_network=udp +31:17.968 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=61.348µs recursor=127.0.0.1:45602 +31:17.968 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=132.685µs client=127.0.0.1:36935 client_network=udp +31:17.968 [INFO] TestAgent: Requesting shutdown +31:17.968 [INFO] TestAgent.server: shutting down server +31:17.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:17.968 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:17.968 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:17.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:17.968 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:17.969 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:17.969 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:17.969 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:17.969 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:17.969 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:17.991 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.000Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.000Z [INFO] agent.router.manager: shutting down +31:18.000 [INFO] TestAgent: consul server down +31:18.000 [INFO] TestAgent: shutdown complete +31:18.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29864 network=tcp +31:18.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29864 network=udp +31:18.000 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29865 network=tcp +31:18.502 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.502 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Compression_Recurse (0.99s) +=== CONT TestDNS_NonExistingLookupEmptyAorAAAA +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:17.591Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.621Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.716 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7f13133d-42aa-0b47-7c0b-d9d3f039c30e Address:127.0.0.1:29876}]" +31:17.717 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e.dc1 127.0.0.1 +31:17.717 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e 127.0.0.1 +2021-10-05T16:31:17.717Z [INFO] agent.router: Initializing LAN area manager +31:17.717 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29885 network=udp +31:17.717 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29876 [Follower]" leader= +31:17.718 [INFO] TestAgent.server: Adding LAN server: server="Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e (Addr: tcp/127.0.0.1:29876) (DC: dc1)" +31:17.718 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e.dc1 area=wan +31:17.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29885 network=tcp +31:17.718 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29886 network=tcp +31:17.718 [INFO] TestAgent: started state syncer +31:17.718 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29877 network=tcp +31:17.779 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.779 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29876 [Candidate]" term=2 +31:17.828 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.828 [DEBUG] TestAgent.server.raft: vote granted: from=7f13133d-42aa-0b47-7c0b-d9d3f039c30e term=2 tally=1 +31:17.828 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.828 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29876 [Leader]" +31:17.830 [INFO] TestAgent.server: cluster leadership acquired +31:17.856 [INFO] TestAgent.server: New leader elected: payload=Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e +31:17.876 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29876 +31:17.877 [INFO] TestAgent: Synced node info +31:17.877 [DEBUG] TestAgent: Node info in sync +31:17.936 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:17.996 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:17.996 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:17.996 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:17.996 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:17.996 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:17.996 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e +31:17.996 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7f13133d-42aa-0b47-7c0b-d9d3f039c30e +31:18.012 [DEBUG] TestAgent: Node info in sync +31:18.028 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.107 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=94.891µs client=127.0.0.1:47730 client_network=udp +31:18.107 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=63.382µs client=127.0.0.1:47730 client_network=udp +31:18.108 [INFO] TestAgent: Requesting shutdown +31:18.108 [INFO] TestAgent.server: shutting down server +31:18.108 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.108 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.108 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.108 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.108 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.108 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.120 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.128Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.128Z [INFO] agent.router.manager: shutting down +31:18.128 [INFO] TestAgent: consul server down +31:18.128 [INFO] TestAgent: shutdown complete +31:18.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29885 network=tcp +31:18.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29885 network=udp +31:18.128 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29886 network=tcp +31:18.628 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.628 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Compression_ReverseLookup (1.08s) === CONT TestDNS_NonExistingLookup -2022-11-07T22:46:03.208Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.222Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.260 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3186d686-49e2-1a1a-569b-3b18f9211f1c Address:127.0.0.1:11623}]" -46:03.260 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11623 [Follower]" leader= -46:03.261 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3186d686-49e2-1a1a-569b-3b18f9211f1c.dc1 127.0.0.1 -46:03.261 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3186d686-49e2-1a1a-569b-3b18f9211f1c 127.0.0.1 -2022-11-07T22:46:03.261Z [INFO] agent.router: Initializing LAN area manager -46:03.261 [DEBUG] TestAgent.dns: recursor enabled -46:03.261 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3186d686-49e2-1a1a-569b-3b18f9211f1c.dc1 area=wan -46:03.261 [DEBUG] TestAgent.dns: recursor enabled -46:03.261 [INFO] TestAgent.server: Adding LAN server: server="Node-3186d686-49e2-1a1a-569b-3b18f9211f1c (Addr: tcp/127.0.0.1:11623) (DC: dc1)" -46:03.261 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11618 network=udp -46:03.261 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11618 network=tcp -46:03.262 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11619 network=tcp -46:03.262 [INFO] TestAgent: started state syncer -46:03.262 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11610 network=tcp -46:03.322 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.322 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11623 [Candidate]" term=2 -46:03.340 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.340 [DEBUG] TestAgent.server.raft: vote granted: from=3186d686-49e2-1a1a-569b-3b18f9211f1c term=2 tally=1 -46:03.340 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.340 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11623 [Leader]" -46:03.340 [INFO] TestAgent.server: cluster leadership acquired -46:03.340 [INFO] TestAgent.server: New leader elected: payload=Node-3186d686-49e2-1a1a-569b-3b18f9211f1c -46:03.347 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11623 -46:03.375 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.416 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.416 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.416 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.416 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.416 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.416 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3186d686-49e2-1a1a-569b-3b18f9211f1c -46:03.416 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3186d686-49e2-1a1a-569b-3b18f9211f1c -46:03.435 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.516 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.521 [INFO] TestAgent: Synced node info -46:03.556 [INFO] TestAgent: Requesting shutdown -46:03.556 [INFO] TestAgent.server: shutting down server -46:03.556 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.556 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.563 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.569Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.569Z [INFO] agent.router.manager: shutting down -46:03.569 [INFO] TestAgent: consul server down -46:03.569 [INFO] TestAgent: shutdown complete -46:03.569 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11618 network=tcp -46:03.569 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11618 network=udp -46:03.569 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11619 network=tcp -46:04.069 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.069 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ConfigReload (0.87s) -=== CONT TestDNS_AddressLookupIPV6 -[WARN] freeport: 10 out of 80 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:03.410Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.410Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.429Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:03.429Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.464 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:29a27738-4a81-133f-a385-a20ee0d8ca55 Address:127.0.0.1:11658}]" -46:03.464 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11658 [Follower]" leader= -46:03.465 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 127.0.0.1 -46:03.465 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 127.0.0.1 -2022-11-07T22:46:03.465Z [INFO] agent.router: Initializing LAN area manager -46:03.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11653 network=udp -46:03.466 [INFO] TestAgent.server: Adding LAN server: server="Node-29a27738-4a81-133f-a385-a20ee0d8ca55 (Addr: tcp/127.0.0.1:11658) (DC: dc1)" -46:03.466 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 area=wan -46:03.466 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11653 network=tcp -46:03.466 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11654 network=tcp -46:03.466 [INFO] TestAgent: started state syncer -46:03.466 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11645 network=tcp -46:03.512 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.512 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11658 [Candidate]" term=2 -46:03.516 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:03.525 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.525 [DEBUG] TestAgent.server.raft: vote granted: from=29a27738-4a81-133f-a385-a20ee0d8ca55 term=2 tally=1 -46:03.525 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.525 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11658 [Leader]" -46:03.525 [INFO] TestAgent.server: cluster leadership acquired -46:03.525 [INFO] TestAgent.server: New leader elected: payload=Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.533 [INFO] TestAgent.server: initializing acls -46:03.537 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:03.537 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:03.543 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:03.552 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:03.552 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:03.552 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:03.552 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.552 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 -46:03.552 [INFO] TestAgent.server: Updating LAN server: server="Node-29a27738-4a81-133f-a385-a20ee0d8ca55 (Addr: tcp/127.0.0.1:11658) (DC: dc1)" -46:03.552 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 area=wan -46:03.561 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.577 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.577 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.577 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.577 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.577 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.577 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.577 [INFO] TestAgent.server: member joined, marking health alive: member=Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.580 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:03.580 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.580 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 -46:03.580 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:03.580 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.580 [INFO] TestAgent.server.serf.wan: serf: Received install-key query -46:03.581 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 -46:03.581 [INFO] TestAgent.server.serf.lan: serf: Received install-key query -46:03.582 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.582 [INFO] TestAgent.server.serf.wan: serf: Received use-key query -46:03.582 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 -46:03.582 [INFO] TestAgent.server.serf.lan: serf: Received use-key query -46:03.583 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.583 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query -46:03.583 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55.dc1 -46:03.583 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query -46:03.584 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.584 [INFO] TestAgent: Requesting shutdown -46:03.584 [INFO] TestAgent.server: shutting down server -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.584 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.584 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:03.584 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.584 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.585 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-29a27738-4a81-133f-a385-a20ee0d8ca55 -46:03.587 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.590Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.590Z [INFO] agent.router.manager: shutting down -46:03.590 [INFO] TestAgent: consul server down -46:03.590 [INFO] TestAgent: shutdown complete -46:03.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11653 network=tcp -46:03.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11653 network=udp -46:03.590 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11654 network=tcp -46:04.090 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.090 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentKeyring_ACL (0.69s) +2021-10-05T16:31:17.924Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.958Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.043 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 Address:127.0.0.1:29910}]" +31:18.044 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29910 [Follower]" leader= +31:18.044 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077.dc1 127.0.0.1 +31:18.046 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 127.0.0.1 +2021-10-05T16:31:18.046Z [INFO] agent.router: Initializing LAN area manager +31:18.047 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29905 network=udp +31:18.047 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077.dc1 area=wan +31:18.047 [INFO] TestAgent.server: Adding LAN server: server="Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 (Addr: tcp/127.0.0.1:29910) (DC: dc1)" +31:18.047 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29905 network=tcp +31:18.047 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29906 network=tcp +31:18.047 [INFO] TestAgent: started state syncer +31:18.047 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29897 network=tcp +31:18.109 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.109 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29910 [Candidate]" term=2 +31:18.119 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.119 [DEBUG] TestAgent.server.raft: vote granted: from=73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 term=2 tally=1 +31:18.119 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.119 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29910 [Leader]" +31:18.120 [INFO] TestAgent.server: cluster leadership acquired +31:18.120 [INFO] TestAgent.server: New leader elected: payload=Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 +31:18.132 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29910 +31:18.154 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 +31:18.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-73eb598b-ce0d-9d9f-b0c1-cee5a6f58077 +31:18.208 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.220 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:18.221 [DEBUG] TestAgent.dns: request served from client: name=foo.query.consul. type=SRV class=IN latency=61.277µs client=127.0.0.1:38063 client_network=udp +31:18.221 [INFO] TestAgent: Requesting shutdown +31:18.221 [INFO] TestAgent.server: shutting down server +31:18.221 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.221 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.221 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.221 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.221 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.221 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.221 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.221 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:18.221 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.221 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.221 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.236Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.236Z [INFO] agent.router.manager: shutting down +31:18.236 [INFO] TestAgent: consul server down +31:18.236 [INFO] TestAgent: shutdown complete +31:18.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29905 network=tcp +31:18.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29905 network=udp +31:18.236 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29906 network=tcp +31:18.736 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.736 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQuery_AgentSource (0.87s) +=== CONT TestDNS_NonExistentDC_RPC +2021-10-05T16:31:18.021Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.053Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.132 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a178cba2-5266-786d-32fb-5c1924c82ba8 Address:127.0.0.1:29924}]" +31:18.133 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a178cba2-5266-786d-32fb-5c1924c82ba8.dc1 127.0.0.1 +31:18.133 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a178cba2-5266-786d-32fb-5c1924c82ba8 127.0.0.1 +2021-10-05T16:31:18.133Z [INFO] agent.router: Initializing LAN area manager +31:18.134 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29919 network=udp +31:18.134 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29924 [Follower]" leader= +31:18.134 [INFO] TestAgent.server: Adding LAN server: server="Node-a178cba2-5266-786d-32fb-5c1924c82ba8 (Addr: tcp/127.0.0.1:29924) (DC: dc1)" +31:18.134 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a178cba2-5266-786d-32fb-5c1924c82ba8.dc1 area=wan +31:18.134 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29919 network=tcp +31:18.135 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29920 network=tcp +31:18.135 [INFO] TestAgent: started state syncer +31:18.135 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29925 network=tcp +31:18.176 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.176 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29924 [Candidate]" term=2 +31:18.212 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.212 [DEBUG] TestAgent.server.raft: vote granted: from=a178cba2-5266-786d-32fb-5c1924c82ba8 term=2 tally=1 +31:18.212 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.212 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29924 [Leader]" +31:18.216 [INFO] TestAgent.server: cluster leadership acquired +31:18.216 [INFO] TestAgent.server: New leader elected: payload=Node-a178cba2-5266-786d-32fb-5c1924c82ba8 +31:18.220 [INFO] TestAgent: Requesting shutdown +31:18.220 [INFO] TestAgent.server: shutting down server +31:18.220 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.220 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:18.223 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29924 +31:18.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.236Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.236Z [INFO] agent.router.manager: shutting down +31:18.241 [INFO] TestAgent: consul server down +31:18.241 [INFO] TestAgent: shutdown complete +31:18.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29919 network=tcp +31:18.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29919 network=udp +31:18.241 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29920 network=tcp +31:18.241 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:18.241 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:18.744 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.744 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Execute (7.41s) + --- PASS: TestPreparedQuery_Execute/#00 (0.98s) + --- PASS: TestPreparedQuery_Execute/#01 (1.09s) + --- PASS: TestPreparedQuery_Execute/#02 (0.98s) + --- PASS: TestPreparedQuery_Execute/#03 (1.03s) + --- PASS: TestPreparedQuery_Execute/#04 (0.75s) + --- PASS: TestPreparedQuery_Execute/#05 (0.75s) + --- PASS: TestPreparedQuery_Execute/#06 (1.08s) + --- PASS: TestPreparedQuery_Execute/#07 (0.76s) === CONT TestDNS_NonExistentDC_Server -2022-11-07T22:46:03.141Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.153Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.197 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4babe156-e4cc-8157-6bf2-861b2c0671e2 Address:127.0.0.1:11581}]" -46:03.197 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11581 [Follower]" leader= -46:03.197 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4babe156-e4cc-8157-6bf2-861b2c0671e2.dc1 127.0.0.1 -46:03.198 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4babe156-e4cc-8157-6bf2-861b2c0671e2 127.0.0.1 -2022-11-07T22:46:03.198Z [INFO] agent.router: Initializing LAN area manager -46:03.198 [INFO] TestAgent.server: Adding LAN server: server="Node-4babe156-e4cc-8157-6bf2-861b2c0671e2 (Addr: tcp/127.0.0.1:11581) (DC: dc1)" -46:03.198 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4babe156-e4cc-8157-6bf2-861b2c0671e2.dc1 area=wan -46:03.198 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11576 network=udp -46:03.198 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11576 network=tcp -46:03.199 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11577 network=tcp -46:03.199 [INFO] TestAgent: started state syncer -46:03.199 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11582 network=tcp -46:03.252 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.252 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11581 [Candidate]" term=2 -46:03.272 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.272 [DEBUG] TestAgent.server.raft: vote granted: from=4babe156-e4cc-8157-6bf2-861b2c0671e2 term=2 tally=1 -46:03.272 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.272 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11581 [Leader]" -46:03.272 [INFO] TestAgent.server: cluster leadership acquired -46:03.272 [INFO] TestAgent.server: New leader elected: payload=Node-4babe156-e4cc-8157-6bf2-861b2c0671e2 -46:03.285 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11581 -46:03.309 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.350 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.350 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.350 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.350 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.350 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.350 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4babe156-e4cc-8157-6bf2-861b2c0671e2 -46:03.350 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4babe156-e4cc-8157-6bf2-861b2c0671e2 -46:03.363 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.426 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.431 [INFO] TestAgent: Synced node info -46:03.610 [DEBUG] TestAgent.server: User event: event=test -46:03.610 [DEBUG] TestAgent: new event: event_name=test event_id=6a7a76af-fc80-270a-30ce-d68d109182f0 -46:03.635 [INFO] TestAgent: Requesting shutdown -46:03.636 [INFO] TestAgent.server: shutting down server -46:03.636 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.636 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.636 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.636 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.636 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.636 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.636 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.636 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.636 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.636 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.641 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.644Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.644Z [INFO] agent.router.manager: shutting down -46:03.644 [INFO] TestAgent: consul server down -46:03.644 [INFO] TestAgent: shutdown complete -46:03.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11576 network=tcp -46:03.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11576 network=udp -46:03.644 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11577 network=tcp -46:04.145 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.145 [INFO] TestAgent: Endpoints down ---- PASS: TestEventList (1.02s) +2021-10-05T16:31:17.760Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.807Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:17.900 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40704556-a9bb-282e-1c9a-591bdb3125d7 Address:127.0.0.1:29892}]" +31:17.901 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29892 [Follower]" leader= +31:17.901 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-40704556-a9bb-282e-1c9a-591bdb3125d7.dc1 127.0.0.1 +31:17.902 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-40704556-a9bb-282e-1c9a-591bdb3125d7 127.0.0.1 +2021-10-05T16:31:17.902Z [INFO] agent.router: Initializing LAN area manager +31:17.902 [INFO] TestAgent.server: Adding LAN server: server="Node-40704556-a9bb-282e-1c9a-591bdb3125d7 (Addr: tcp/127.0.0.1:29892) (DC: dc1)" +31:17.903 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-40704556-a9bb-282e-1c9a-591bdb3125d7.dc1 area=wan +31:17.903 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29878 network=tcp +31:17.903 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29878 network=udp +31:17.920 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29879 network=tcp +31:17.920 [INFO] TestAgent: started state syncer +31:17.920 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29904 network=tcp +31:17.952 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:17.952 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29892 [Candidate]" term=2 +31:17.993 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:17.993 [DEBUG] TestAgent.server.raft: vote granted: from=40704556-a9bb-282e-1c9a-591bdb3125d7 term=2 tally=1 +31:17.993 [INFO] TestAgent.server.raft: election won: tally=1 +31:17.993 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29892 [Leader]" +31:17.994 [INFO] TestAgent.server: cluster leadership acquired +31:17.994 [INFO] TestAgent.server: New leader elected: payload=Node-40704556-a9bb-282e-1c9a-591bdb3125d7 +31:18.024 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29892 +31:18.037 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.065 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.066 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.066 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.066 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.066 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.066 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-40704556-a9bb-282e-1c9a-591bdb3125d7 +31:18.066 [INFO] TestAgent.server: member joined, marking health alive: member=Node-40704556-a9bb-282e-1c9a-591bdb3125d7 +31:18.098 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.280 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=228.527µs client=127.0.0.1:50501 client_network=udp +31:18.280 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=86.799µs client=127.0.0.1:50501 client_network=udp +31:18.284 [DEBUG] TestAgent.dns: request served from client: name=a94a02d4-9754-b4ba-0e79-e22ac128c369.query.consul. type=SRV class=IN latency=211.573µs client=127.0.0.1:36076 client_network=udp +31:18.285 [DEBUG] TestAgent.dns: request served from client: name=a94a02d4-9754-b4ba-0e79-e22ac128c369.query.consul. type=SRV class=IN latency=170.944µs client=127.0.0.1:36076 client_network=udp +31:18.285 [INFO] TestAgent: Requesting shutdown +31:18.285 [INFO] TestAgent.server: shutting down server +31:18.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.285 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.285 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.285 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.285 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.285 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:18.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.285 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.285 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.292 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.297Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.297Z [INFO] agent.router.manager: shutting down +31:18.297 [INFO] TestAgent: consul server down +31:18.297 [INFO] TestAgent: shutdown complete +31:18.297 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29878 network=tcp +31:18.297 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29878 network=udp +31:18.297 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29879 network=tcp +31:18.800 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.800 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Compression_Query (1.06s) +=== CONT TestDNS_AddressLookupIPV6 +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:18.024Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.049Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.104 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ccb62742-af56-3e69-7b3d-b6b98d667e1e Address:127.0.0.1:29931}]" +31:18.104 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29931 [Follower]" leader= +31:18.104 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +31:18.105 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:31:18.105Z [INFO] agent.router: Initializing LAN area manager +31:18.105 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29926 network=udp +31:18.105 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29931) (DC: dc1)" +31:18.105 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29926 network=tcp +31:18.105 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +31:18.106 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29927 network=tcp +31:18.106 [INFO] TestAgent: started state syncer +31:18.106 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29932 network=tcp +31:18.148 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.148 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29931 [Candidate]" term=2 +31:18.175 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.175 [DEBUG] TestAgent.server.raft: vote granted: from=ccb62742-af56-3e69-7b3d-b6b98d667e1e term=2 tally=1 +31:18.175 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.175 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29931 [Leader]" +31:18.176 [INFO] TestAgent.server: cluster leadership acquired +31:18.176 [INFO] TestAgent.server: New leader elected: payload=test-node +31:18.185 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29931 +31:18.214 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.260 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.260 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.260 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.260 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.260 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.260 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:18.260 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +31:18.304 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.309 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.consul. type=A class=IN latency=106.063µs client=127.0.0.1:52787 client_network=udp +31:18.309 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.test.consul. type=A class=IN latency=51.18µs client=127.0.0.1:49538 client_network=udp +31:18.309 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.dc1.consul. type=A class=IN latency=42.52µs client=127.0.0.1:51183 client_network=udp +31:18.309 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.dc1.test.consul. type=A class=IN latency=44.146µs client=127.0.0.1:48903 client_network=udp +31:18.309 [INFO] TestAgent: Requesting shutdown +31:18.309 [INFO] TestAgent.server: shutting down server +31:18.309 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.310 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.310 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.310 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.310 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.310 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:18.310 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.310 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.310 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.315 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.323Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.323Z [INFO] agent.router.manager: shutting down +31:18.323 [INFO] TestAgent: consul server down +31:18.323 [INFO] TestAgent: shutdown complete +31:18.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29926 network=tcp +31:18.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29926 network=udp +31:18.323 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29927 network=tcp +31:18.824 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.824 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_AltDomains_Overlap (0.83s) === CONT TestDNS_AddressLookup -2022-11-07T22:46:03.168Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.177Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.209 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0513571-f986-3d7f-2f17-1f2051634217 Address:127.0.0.1:11588}]" -46:03.209 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11588 [Follower]" leader= -46:03.209 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0513571-f986-3d7f-2f17-1f2051634217.dc1 127.0.0.1 -46:03.210 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0513571-f986-3d7f-2f17-1f2051634217 127.0.0.1 -2022-11-07T22:46:03.210Z [INFO] agent.router: Initializing LAN area manager -46:03.210 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0513571-f986-3d7f-2f17-1f2051634217.dc1 area=wan -46:03.210 [INFO] TestAgent.server: Adding LAN server: server="Node-b0513571-f986-3d7f-2f17-1f2051634217 (Addr: tcp/127.0.0.1:11588) (DC: dc1)" -46:03.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11583 network=udp -46:03.210 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11583 network=tcp -46:03.211 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11584 network=tcp -46:03.211 [INFO] TestAgent: started state syncer -46:03.211 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11596 network=tcp -46:03.244 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.244 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11588 [Candidate]" term=2 -46:03.261 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.261 [DEBUG] TestAgent.server.raft: vote granted: from=b0513571-f986-3d7f-2f17-1f2051634217 term=2 tally=1 -46:03.261 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.261 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11588 [Leader]" -46:03.261 [INFO] TestAgent.server: cluster leadership acquired -46:03.261 [INFO] TestAgent.server: New leader elected: payload=Node-b0513571-f986-3d7f-2f17-1f2051634217 -46:03.274 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11588 -46:03.301 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.351 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.351 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.351 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.351 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.351 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.351 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0513571-f986-3d7f-2f17-1f2051634217 -46:03.351 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0513571-f986-3d7f-2f17-1f2051634217 -46:03.362 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.522 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.528 [INFO] TestAgent: Synced node info -46:03.528 [DEBUG] TestAgent: Node info in sync -46:03.562 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:03.663 [INFO] TestAgent: Requesting shutdown -46:03.663 [INFO] TestAgent.server: shutting down server -46:03.663 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.663 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.666 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.669Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.669Z [INFO] agent.router.manager: shutting down -46:03.669 [INFO] TestAgent: consul server down -46:03.669 [INFO] TestAgent: shutdown complete -46:03.669 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11583 network=tcp -46:03.669 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11583 network=udp -46:03.669 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11584 network=tcp -46:04.170 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.170 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ReloadConfig_DuringQuery (1.01s) +2021-10-05T16:31:18.000Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.041Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b15d4001-20c1-177b-bafd-2151ff42bd80 Address:127.0.0.1:29917}]" +31:18.081 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b15d4001-20c1-177b-bafd-2151ff42bd80.dc1 127.0.0.1 +31:18.081 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29917 [Follower]" leader= +31:18.082 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b15d4001-20c1-177b-bafd-2151ff42bd80 127.0.0.1 +2021-10-05T16:31:18.082Z [INFO] agent.router: Initializing LAN area manager +31:18.082 [INFO] TestAgent.server: Adding LAN server: server="Node-b15d4001-20c1-177b-bafd-2151ff42bd80 (Addr: tcp/127.0.0.1:29917) (DC: dc1)" +31:18.082 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29912 network=udp +31:18.082 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b15d4001-20c1-177b-bafd-2151ff42bd80.dc1 area=wan +31:18.082 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29912 network=tcp +31:18.083 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29913 network=tcp +31:18.083 [INFO] TestAgent: started state syncer +31:18.083 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29918 network=tcp +31:18.126 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.126 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29917 [Candidate]" term=2 +31:18.144 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.144 [DEBUG] TestAgent.server.raft: vote granted: from=b15d4001-20c1-177b-bafd-2151ff42bd80 term=2 tally=1 +31:18.144 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.144 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29917 [Leader]" +31:18.146 [INFO] TestAgent.server: cluster leadership acquired +31:18.146 [INFO] TestAgent.server: New leader elected: payload=Node-b15d4001-20c1-177b-bafd-2151ff42bd80 +31:18.157 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29917 +31:18.178 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.223 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.223 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.223 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.223 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.223 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.223 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b15d4001-20c1-177b-bafd-2151ff42bd80 +31:18.223 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b15d4001-20c1-177b-bafd-2151ff42bd80 +31:18.241 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:18.263 [INFO] TestAgent: Synced node info +31:18.263 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.392 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:18.393 [WARN] TestAgent.dns: Query results too stale, re-requesting +31:18.393 [DEBUG] TestAgent.dns: request served from client: name=nope.query.consul. type=SRV class=IN latency=126.13µs client=127.0.0.1:42105 client_network=udp +31:18.393 [INFO] TestAgent: Requesting shutdown +31:18.393 [INFO] TestAgent.server: shutting down server +31:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.393 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.393 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.401 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.407Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.407Z [INFO] agent.router.manager: shutting down +31:18.407 [INFO] TestAgent: consul server down +31:18.407 [INFO] TestAgent: shutdown complete +31:18.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29912 network=tcp +31:18.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29912 network=udp +31:18.407 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29913 network=tcp +31:18.908 [INFO] TestAgent: Waiting for endpoints to shut down +31:18.908 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQuery_AllowStale (0.95s) === CONT TestDNS_ServiceLookup_FilterACL === RUN TestDNS_ServiceLookup_FilterACL/ACLToken_==_root -2022-11-07T22:46:03.494Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.504Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.537 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b86d7e6-0e6d-87b9-432b-e9323271887d Address:127.0.0.1:11651}]" -46:03.537 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11651 [Follower]" leader= -46:03.537 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5b86d7e6-0e6d-87b9-432b-e9323271887d.dc1 127.0.0.1 -46:03.538 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5b86d7e6-0e6d-87b9-432b-e9323271887d 127.0.0.1 -2022-11-07T22:46:03.538Z [INFO] agent.router: Initializing LAN area manager -46:03.538 [INFO] TestAgent.server: Adding LAN server: server="Node-5b86d7e6-0e6d-87b9-432b-e9323271887d (Addr: tcp/127.0.0.1:11651) (DC: dc1)" -46:03.538 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5b86d7e6-0e6d-87b9-432b-e9323271887d.dc1 area=wan -46:03.538 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11646 network=udp -46:03.538 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11646 network=tcp -46:03.539 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11647 network=tcp -46:03.539 [INFO] TestAgent: started state syncer -46:03.539 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11673 network=tcp -46:03.599 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.599 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11651 [Candidate]" term=2 -46:03.607 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.607 [DEBUG] TestAgent.server.raft: vote granted: from=5b86d7e6-0e6d-87b9-432b-e9323271887d term=2 tally=1 -46:03.607 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.607 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11651 [Leader]" -46:03.607 [INFO] TestAgent.server: cluster leadership acquired -46:03.607 [INFO] TestAgent.server: New leader elected: payload=Node-5b86d7e6-0e6d-87b9-432b-e9323271887d -46:03.612 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11651 -46:03.624 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.652 [INFO] TestAgent: Synced node info -46:03.652 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.652 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.652 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.652 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.652 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.652 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b86d7e6-0e6d-87b9-432b-e9323271887d -46:03.652 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5b86d7e6-0e6d-87b9-432b-e9323271887d -46:03.658 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.675 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:03.675 [DEBUG] TestAgent.dns: request served from client: name=foo.query.consul. type=SRV class=IN latency=75.42µs client=127.0.0.1:41702 client_network=udp -46:03.675 [INFO] TestAgent: Requesting shutdown -46:03.675 [INFO] TestAgent.server: shutting down server -46:03.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.675 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.675 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.675 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.675 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.675 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.675 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.675 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.675 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.678 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.682Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.682Z [INFO] agent.router.manager: shutting down -46:03.682 [INFO] TestAgent: consul server down -46:03.682 [INFO] TestAgent: shutdown complete -46:03.682 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11646 network=tcp -46:03.682 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11646 network=udp -46:03.682 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11647 network=tcp -46:04.182 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.182 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQuery_AgentSource (0.70s) +2021-10-05T16:31:17.928Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:17.958Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.009 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 Address:127.0.0.1:29903}]" +31:18.010 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6.dc1 127.0.0.1 +31:18.011 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 127.0.0.1 +2021-10-05T16:31:18.011Z [INFO] agent.router: Initializing LAN area manager +31:18.011 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29898 network=udp +31:18.011 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29903 [Follower]" leader= +31:18.011 [INFO] TestAgent.server: Adding LAN server: server="Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 (Addr: tcp/127.0.0.1:29903) (DC: dc1)" +31:18.011 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6.dc1 area=wan +31:18.011 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29898 network=tcp +31:18.012 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29899 network=tcp +31:18.012 [INFO] TestAgent: started state syncer +31:18.012 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29911 network=tcp +31:18.069 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.069 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29903 [Candidate]" term=2 +31:18.109 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.109 [DEBUG] TestAgent.server.raft: vote granted: from=ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 term=2 tally=1 +31:18.109 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29903 [Leader]" +31:18.109 [INFO] TestAgent.server: cluster leadership acquired +31:18.109 [INFO] TestAgent.server: New leader elected: payload=Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 +31:18.148 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29903 +31:18.212 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.273 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.273 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.273 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.273 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.273 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.273 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 +31:18.273 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec6cfb99-35ac-3caf-4c77-2f8cadbd7dc6 +31:18.292 [INFO] TestAgent: Synced node info +31:18.298 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.506 [WARN] TestAgent.dns: QName invalid: qname= +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=consul. type=SRV class=IN latency=102.01µs client=127.0.0.1:47509 client_network=udp +31:18.506 [WARN] TestAgent.dns: QName invalid: qname=node. +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=node.consul. type=SRV class=IN latency=59.985µs client=127.0.0.1:52687 client_network=udp +31:18.506 [WARN] TestAgent.dns: QName invalid: qname=service. +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=service.consul. type=SRV class=IN latency=25.329µs client=127.0.0.1:52448 client_network=udp +31:18.506 [WARN] TestAgent.dns: QName invalid: qname=query. +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=query.consul. type=SRV class=IN latency=22.126µs client=127.0.0.1:36531 client_network=udp +31:18.506 [WARN] TestAgent.dns: QName invalid: qname=foo.node.dc1.extra.more. +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.extra.more.consul. type=SRV class=IN latency=27.302µs client=127.0.0.1:45951 client_network=udp +31:18.506 [WARN] TestAgent.dns: QName invalid: qname=foo.service.dc1.extra.more. +31:18.506 [DEBUG] TestAgent.dns: request served from client: name=foo.service.dc1.extra.more.consul. type=SRV class=IN latency=27.229µs client=127.0.0.1:40428 client_network=udp +31:18.507 [WARN] TestAgent.dns: QName invalid: qname=foo.query.dc1.extra.more. +31:18.507 [DEBUG] TestAgent.dns: request served from client: name=foo.query.dc1.extra.more.consul. type=SRV class=IN latency=30.748µs client=127.0.0.1:37788 client_network=udp +31:18.507 [INFO] TestAgent: Requesting shutdown +31:18.507 [INFO] TestAgent.server: shutting down server +31:18.507 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.507 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.507 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.507 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.507 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.507 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.507 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.507 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.507 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.507 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.517 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.544Z [INFO] agent.router.manager: shutting down +31:18.544 [INFO] TestAgent: consul server down +31:18.544 [INFO] TestAgent: shutdown complete +31:18.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29898 network=tcp +2021-10-05T16:31:18.544Z [INFO] agent.router.manager: shutting down +31:18.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29898 network=udp +31:18.544 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29899 network=tcp +31:19.045 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.045 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_InvalidQueries (1.17s) +=== CONT TestDNS_ServiceLookup_SRV_RFC_TCP_Default +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:18.289Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.303Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.335 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c97032bf-dcc4-6bd9-224c-1dbf64b2ae19 Address:127.0.0.1:29938}]" +31:18.336 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29938 [Follower]" leader= +31:18.336 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +31:18.337 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:31:18.337Z [INFO] agent.router: Initializing LAN area manager +31:18.337 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29933 network=udp +31:18.337 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29938) (DC: dc1)" +31:18.338 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +31:18.338 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29933 network=tcp +31:18.338 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29934 network=tcp +31:18.338 [INFO] TestAgent: started state syncer +31:18.338 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29939 network=tcp +31:18.393 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.393 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29938 [Candidate]" term=2 +31:18.412 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.412 [DEBUG] TestAgent.server.raft: vote granted: from=c97032bf-dcc4-6bd9-224c-1dbf64b2ae19 term=2 tally=1 +31:18.412 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.413 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29938 [Leader]" +31:18.413 [INFO] TestAgent.server: cluster leadership acquired +31:18.413 [INFO] TestAgent.server: New leader elected: payload=test-node +31:18.426 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29938 +31:18.449 [INFO] TestAgent: Synced node info +31:18.449 [DEBUG] TestAgent: Node info in sync +31:18.461 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:18.500 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +31:18.556 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.591 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.consul. type=SOA class=IN latency=164.207µs client=127.0.0.1:46784 client_network=udp +31:18.591 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.test-domain. type=SOA class=IN latency=118.972µs client=127.0.0.1:37131 client_network=udp +31:18.591 [INFO] TestAgent: Requesting shutdown +31:18.591 [INFO] TestAgent.server: shutting down server +31:18.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.591 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.591 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.592 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.592 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.600 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.600 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.608 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.616Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.616Z [INFO] agent.router.manager: shutting down +31:18.616 [INFO] TestAgent: consul server down +31:18.616 [INFO] TestAgent: shutdown complete +31:18.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29933 network=tcp +31:18.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29933 network=udp +31:18.616 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29934 network=tcp +31:19.117 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.117 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_AltDomains_SOA (0.86s) === CONT TestDNS_ServiceLookup_SRV_RFC -2022-11-07T22:46:03.245Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.259Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.311 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9aa1dcfe-4987-68da-a9d3-4610a26644c3 Address:127.0.0.1:11630}]" -46:03.311 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11630 [Follower]" leader= -46:03.312 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3.dc1 127.0.0.1 -46:03.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3 127.0.0.1 -2022-11-07T22:46:03.313Z [INFO] agent.router: Initializing LAN area manager -46:03.313 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3.dc1 area=wan -46:03.313 [INFO] TestAgent.server: Adding LAN server: server="Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3 (Addr: tcp/127.0.0.1:11630) (DC: dc1)" -46:03.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11625 network=udp -46:03.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11625 network=tcp -46:03.313 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11626 network=tcp -46:03.313 [INFO] TestAgent: started state syncer -46:03.313 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11603 network=tcp -46:03.358 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.358 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11630 [Candidate]" term=2 -46:03.382 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.382 [DEBUG] TestAgent.server.raft: vote granted: from=9aa1dcfe-4987-68da-a9d3-4610a26644c3 term=2 tally=1 -46:03.382 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.382 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11630 [Leader]" -46:03.382 [INFO] TestAgent.server: cluster leadership acquired -46:03.383 [INFO] TestAgent.server: New leader elected: payload=Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3 -46:03.392 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11630 -46:03.410 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.441 [INFO] TestAgent: Synced node info -46:03.441 [DEBUG] TestAgent: Node info in sync -46:03.449 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.449 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.449 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.449 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.449 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.449 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3 -46:03.449 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9aa1dcfe-4987-68da-a9d3-4610a26644c3 -46:03.456 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.747 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=79.793µs client=127.0.0.1:38942 client_network=udp -46:03.747 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=58.771µs client=127.0.0.1:38942 client_network=udp -46:03.747 [INFO] TestAgent: Requesting shutdown -46:03.747 [INFO] TestAgent.server: shutting down server -46:03.747 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.747 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.747 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.747 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.747 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.747 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.747 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.747 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.747 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.747 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.750 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.756Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.756Z [INFO] agent.router.manager: shutting down -46:03.757 [INFO] TestAgent: consul server down -46:03.757 [INFO] TestAgent: shutdown complete -46:03.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11625 network=tcp -46:03.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11625 network=udp -46:03.757 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11626 network=tcp -46:04.257 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.257 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Compression_ReverseLookup (1.02s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:18.508Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.549Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.621 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c82480eb-508f-f6b3-00b9-7374c61bb4ce Address:127.0.0.1:29945}]" +31:18.622 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29945 [Follower]" leader= +31:18.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce.dc1 127.0.0.1 +31:18.622 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce 127.0.0.1 +2021-10-05T16:31:18.622Z [INFO] agent.router: Initializing LAN area manager +31:18.622 [INFO] TestAgent.server: Adding LAN server: server="Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce (Addr: tcp/127.0.0.1:29945) (DC: dc1)" +31:18.623 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce.dc1 area=wan +31:18.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29940 network=udp +31:18.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29940 network=tcp +31:18.623 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29941 network=tcp +31:18.623 [INFO] TestAgent: started state syncer +31:18.623 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29946 network=tcp +31:18.661 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.661 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29945 [Candidate]" term=2 +31:18.680 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.680 [DEBUG] TestAgent.server.raft: vote granted: from=c82480eb-508f-f6b3-00b9-7374c61bb4ce term=2 tally=1 +31:18.680 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.680 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29945 [Leader]" +31:18.681 [INFO] TestAgent.server: cluster leadership acquired +31:18.681 [INFO] TestAgent.server: New leader elected: payload=Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce +31:18.687 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29945 +31:18.700 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.754 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.754 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.754 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.754 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.754 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.754 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce +31:18.754 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c82480eb-508f-f6b3-00b9-7374c61bb4ce +31:18.780 [INFO] TestAgent.server: federation state anti-entropy synced +31:18.860 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=313.18µs client=127.0.0.1:49748 client_network=udp +31:18.860 [DEBUG] TestAgent.dns: request served from client: name=db.service.test-domain. type=SRV class=IN latency=81.257µs client=127.0.0.1:49596 client_network=udp +31:18.861 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc1.consul. type=SRV class=IN latency=82.025µs client=127.0.0.1:49159 client_network=udp +31:18.861 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc1.test-domain. type=SRV class=IN latency=88.433µs client=127.0.0.1:35258 client_network=udp +31:18.861 [INFO] TestAgent: Requesting shutdown +31:18.861 [INFO] TestAgent.server: shutting down server +31:18.861 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:18.861 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:18.861 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:18.861 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:18.861 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:18.861 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:18.861 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:18.861 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:18.862 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:18.862 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:18.862 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:18.872 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:18.876Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:18.876Z [INFO] agent.router.manager: shutting down +31:18.877 [INFO] TestAgent: consul server down +31:18.877 [INFO] TestAgent: shutdown complete +31:18.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29940 network=tcp +31:18.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29940 network=udp +31:18.877 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29941 network=tcp +31:19.380 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.380 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_AltDomains_Service (0.90s) === CONT TestDNS_PreparedQuery_Failover -2022-11-07T22:46:03.387Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.396Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.433 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba16d3dd-d271-6f64-8a20-dabf9290750a Address:127.0.0.1:11637}]" -46:03.433 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11637 [Follower]" leader= -46:03.434 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba16d3dd-d271-6f64-8a20-dabf9290750a.dc1 127.0.0.1 -46:03.434 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba16d3dd-d271-6f64-8a20-dabf9290750a 127.0.0.1 -2022-11-07T22:46:03.434Z [INFO] agent.router: Initializing LAN area manager -46:03.434 [INFO] TestAgent.server: Adding LAN server: server="Node-ba16d3dd-d271-6f64-8a20-dabf9290750a (Addr: tcp/127.0.0.1:11637) (DC: dc1)" -46:03.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11632 network=udp -46:03.434 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba16d3dd-d271-6f64-8a20-dabf9290750a.dc1 area=wan -46:03.435 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11632 network=tcp -46:03.435 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11633 network=tcp -46:03.435 [INFO] TestAgent: started state syncer -46:03.435 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11652 network=tcp -46:03.481 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.481 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11637 [Candidate]" term=2 -46:03.490 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.490 [DEBUG] TestAgent.server.raft: vote granted: from=ba16d3dd-d271-6f64-8a20-dabf9290750a term=2 tally=1 -46:03.490 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.490 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11637 [Leader]" -46:03.490 [INFO] TestAgent.server: cluster leadership acquired -46:03.490 [INFO] TestAgent.server: New leader elected: payload=Node-ba16d3dd-d271-6f64-8a20-dabf9290750a -46:03.497 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11637 -46:03.507 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.536 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.536 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.536 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.536 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.536 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ba16d3dd-d271-6f64-8a20-dabf9290750a -46:03.536 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ba16d3dd-d271-6f64-8a20-dabf9290750a -46:03.547 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.562 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:03.568 [INFO] TestAgent: Synced node info -46:03.568 [DEBUG] TestAgent: Node info in sync -46:03.785 [INFO] TestAgent: Requesting shutdown -46:03.785 [INFO] TestAgent.server: shutting down server -46:03.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.785 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.785 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.785 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.785 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.785 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.785 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.786 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.785 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.789 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.792Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.792Z [INFO] agent.router.manager: shutting down -46:03.792 [INFO] TestAgent: consul server down -46:03.792 [INFO] TestAgent: shutdown complete -46:03.792 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11632 network=tcp -46:03.792 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11632 network=udp -46:03.792 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11633 network=tcp -46:04.293 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.293 [INFO] TestAgent: Endpoints down ---- PASS: TestKVSEndpoint_PUT_GET_DELETE (0.92s) -=== CONT TestDNS_ServiceLookup_SRV_RFC_TCP_Default -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:03.799Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.807Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.826 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:be9a702e-b47b-bfbe-7233-ee6077ca78b0 Address:127.0.0.1:11665}]" -46:03.827 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11665 [Follower]" leader= -46:03.827 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0.dc1 127.0.0.1 -46:03.827 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0 127.0.0.1 -2022-11-07T22:46:03.827Z [INFO] agent.router: Initializing LAN area manager -46:03.827 [INFO] TestAgent.server: Adding LAN server: server="Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0 (Addr: tcp/127.0.0.1:11665) (DC: dc1)" -46:03.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11660 network=udp -46:03.828 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0.dc1 area=wan -46:03.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11660 network=tcp -46:03.828 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11661 network=tcp -46:03.828 [INFO] TestAgent: started state syncer -46:03.828 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11680 network=tcp -46:03.892 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.892 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11665 [Candidate]" term=2 -46:03.900 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.900 [DEBUG] TestAgent.server.raft: vote granted: from=be9a702e-b47b-bfbe-7233-ee6077ca78b0 term=2 tally=1 -46:03.900 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.900 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11665 [Leader]" -46:03.900 [INFO] TestAgent.server: cluster leadership acquired -46:03.900 [INFO] TestAgent.server: New leader elected: payload=Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0 -46:03.905 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11665 -46:03.916 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.935 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.935 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.935 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.935 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.935 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.935 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0 -46:03.935 [INFO] TestAgent.server: member joined, marking health alive: member=Node-be9a702e-b47b-bfbe-7233-ee6077ca78b0 -46:03.940 [INFO] TestAgent.server: federation state anti-entropy synced -46:03.967 [WARN] TestAgent.dns: QName invalid: qname= -46:03.967 [DEBUG] TestAgent.dns: request served from client: name=consul. type=SRV class=IN latency=47.339µs client=127.0.0.1:41022 client_network=udp -46:03.967 [WARN] TestAgent.dns: QName invalid: qname=node. -46:03.967 [DEBUG] TestAgent.dns: request served from client: name=node.consul. type=SRV class=IN latency=35.114µs client=127.0.0.1:56072 client_network=udp -46:03.967 [WARN] TestAgent.dns: QName invalid: qname=service. -46:03.967 [DEBUG] TestAgent.dns: request served from client: name=service.consul. type=SRV class=IN latency=28.516µs client=127.0.0.1:49913 client_network=udp -46:03.967 [WARN] TestAgent.dns: QName invalid: qname=query. -46:03.967 [DEBUG] TestAgent.dns: request served from client: name=query.consul. type=SRV class=IN latency=40.474µs client=127.0.0.1:54535 client_network=udp -46:03.968 [WARN] TestAgent.dns: QName invalid: qname=foo.node.dc1.extra.more. -46:03.968 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.extra.more.consul. type=SRV class=IN latency=40.056µs client=127.0.0.1:40316 client_network=udp -46:03.968 [WARN] TestAgent.dns: QName invalid: qname=foo.service.dc1.extra.more. -46:03.968 [DEBUG] TestAgent.dns: request served from client: name=foo.service.dc1.extra.more.consul. type=SRV class=IN latency=42.854µs client=127.0.0.1:43294 client_network=udp -46:03.968 [WARN] TestAgent.dns: QName invalid: qname=foo.query.dc1.extra.more. -46:03.968 [DEBUG] TestAgent.dns: request served from client: name=foo.query.dc1.extra.more.consul. type=SRV class=IN latency=54.992µs client=127.0.0.1:51662 client_network=udp -46:03.968 [INFO] TestAgent: Requesting shutdown -46:03.968 [INFO] TestAgent.server: shutting down server -46:03.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:03.968 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:03.968 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:03.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:03.968 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:03.968 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:03.968 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:03.968 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:03.968 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:03.968 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:03.968 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:03.972 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:03.977Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:03.977Z [INFO] agent.router.manager: shutting down -46:03.977 [INFO] TestAgent: consul server down -46:03.977 [INFO] TestAgent: shutdown complete -46:03.977 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11660 network=tcp -46:03.977 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11660 network=udp -46:03.977 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11661 network=tcp -46:04.477 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.477 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_InvalidQueries (0.69s) +2021-10-05T16:31:18.577Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.611Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.678 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d6011653-8631-7ea1-042f-7a3f41ce7db3 Address:127.0.0.1:29952}]" +31:18.678 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29952 [Follower]" leader= +31:18.679 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d6011653-8631-7ea1-042f-7a3f41ce7db3.dc1 127.0.0.1 +31:18.680 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d6011653-8631-7ea1-042f-7a3f41ce7db3 127.0.0.1 +2021-10-05T16:31:18.680Z [INFO] agent.router: Initializing LAN area manager +31:18.680 [INFO] TestAgent.server: Adding LAN server: server="Node-d6011653-8631-7ea1-042f-7a3f41ce7db3 (Addr: tcp/127.0.0.1:29952) (DC: dc1)" +31:18.680 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d6011653-8631-7ea1-042f-7a3f41ce7db3.dc1 area=wan +31:18.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29947 network=udp +31:18.680 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29947 network=tcp +31:18.681 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29948 network=tcp +31:18.681 [INFO] TestAgent: started state syncer +31:18.681 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29960 network=tcp +31:18.748 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.748 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29952 [Candidate]" term=2 +31:18.795 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.795 [DEBUG] TestAgent.server.raft: vote granted: from=d6011653-8631-7ea1-042f-7a3f41ce7db3 term=2 tally=1 +31:18.795 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.795 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29952 [Leader]" +31:18.796 [INFO] TestAgent.server: cluster leadership acquired +31:18.796 [INFO] TestAgent.server: New leader elected: payload=Node-d6011653-8631-7ea1-042f-7a3f41ce7db3 +31:18.816 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29952 +31:18.850 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.910 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.910 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.910 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.910 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.910 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.911 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6011653-8631-7ea1-042f-7a3f41ce7db3 +31:18.911 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d6011653-8631-7ea1-042f-7a3f41ce7db3 +31:18.952 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.029 [DEBUG] TestAgent.dns: request served from client: name=webv4.service.consul. type=AAAA class=IN latency=165.87µs client=127.0.0.1:40369 client_network=udp +31:19.029 [DEBUG] TestAgent.dns: request served from client: name=webv4.query.consul. type=AAAA class=IN latency=98.11µs client=127.0.0.1:37562 client_network=udp +31:19.029 [DEBUG] TestAgent.dns: request served from client: name=webv6.service.consul. type=A class=IN latency=71.509µs client=127.0.0.1:42704 client_network=udp +31:19.030 [DEBUG] TestAgent.dns: request served from client: name=webv6.query.consul. type=A class=IN latency=247.541µs client=127.0.0.1:47504 client_network=udp +31:19.030 [INFO] TestAgent: Requesting shutdown +31:19.030 [INFO] TestAgent.server: shutting down server +31:19.030 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.030 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.030 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.030 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.030 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.030 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:19.030 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.030 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.030 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.039 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.045Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.045Z [INFO] agent.router.manager: shutting down +31:19.046 [INFO] TestAgent: consul server down +31:19.046 [INFO] TestAgent: shutdown complete +31:19.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29947 network=tcp +31:19.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29947 network=udp +31:19.046 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29948 network=tcp +31:19.548 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.548 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NonExistingLookupEmptyAorAAAA (1.05s) === CONT TestDNS_PreparedQuery_TTL -2022-11-07T22:46:03.606Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.616Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.643 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9d4aece-6a2e-e807-d4cd-7a5ed7056816 Address:127.0.0.1:11679}]" -46:03.643 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11679 [Follower]" leader= -46:03.644 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816.dc1 127.0.0.1 -46:03.644 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816 127.0.0.1 -2022-11-07T22:46:03.644Z [INFO] agent.router: Initializing LAN area manager -46:03.644 [INFO] TestAgent.server: Adding LAN server: server="Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816 (Addr: tcp/127.0.0.1:11679) (DC: dc1)" -46:03.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11674 network=udp -46:03.644 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816.dc1 area=wan -46:03.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11674 network=tcp -46:03.645 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11675 network=tcp -46:03.645 [INFO] TestAgent: started state syncer -46:03.645 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11659 network=tcp -46:03.706 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:03.706 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11679 [Candidate]" term=2 -46:03.714 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:03.714 [DEBUG] TestAgent.server.raft: vote granted: from=c9d4aece-6a2e-e807-d4cd-7a5ed7056816 term=2 tally=1 -46:03.714 [INFO] TestAgent.server.raft: election won: tally=1 -46:03.714 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11679 [Leader]" -46:03.714 [INFO] TestAgent.server: cluster leadership acquired -46:03.714 [INFO] TestAgent.server: New leader elected: payload=Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816 -46:03.719 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11679 -46:03.731 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:03.749 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:03.749 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:03.749 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:03.749 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:03.749 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:03.749 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816 -46:03.749 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9d4aece-6a2e-e807-d4cd-7a5ed7056816 -46:03.756 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.057 [INFO] TestAgent: Requesting shutdown -46:04.057 [INFO] TestAgent.server: shutting down server -46:04.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.057 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.057 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.057 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.057 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.057 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.068 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.080Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.080Z [INFO] agent.router.manager: shutting down -46:04.083 [INFO] TestAgent: consul server down -46:04.083 [INFO] TestAgent: shutdown complete -46:04.083 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11674 network=tcp -46:04.083 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11674 network=udp -46:04.083 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11675 network=tcp -46:04.583 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.583 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_InmemKeyrings/server_with_keys -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:03.904Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.920Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.941 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c61604f-3488-e1a0-d723-ee8a0ecc4402 Address:127.0.0.1:11686}]" -46:03.941 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11686 [Follower]" leader= -46:03.942 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402.dc1 127.0.0.1 -46:03.942 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402 127.0.0.1 -2022-11-07T22:46:03.942Z [INFO] agent.router: Initializing LAN area manager -46:03.942 [INFO] TestAgent.server: Adding LAN server: server="Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402 (Addr: tcp/127.0.0.1:11686) (DC: dc1)" -46:03.942 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402.dc1 area=wan -46:03.943 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11681 network=tcp -46:03.943 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11681 network=udp -46:03.943 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11682 network=tcp -46:03.943 [INFO] TestAgent: started state syncer -46:03.943 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11666 network=tcp -46:04.006 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.006 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11686 [Candidate]" term=2 -46:04.014 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.014 [DEBUG] TestAgent.server.raft: vote granted: from=0c61604f-3488-e1a0-d723-ee8a0ecc4402 term=2 tally=1 -46:04.014 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.014 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11686 [Leader]" -46:04.014 [INFO] TestAgent.server: cluster leadership acquired -46:04.014 [INFO] TestAgent.server: New leader elected: payload=Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402 -46:04.020 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11686 -46:04.032 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.056 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.056 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.056 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.056 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.056 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.056 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402 -46:04.056 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c61604f-3488-e1a0-d723-ee8a0ecc4402 -46:04.078 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.114 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:04.115 [WARN] TestAgent.dns: Query results too stale, re-requesting -46:04.115 [DEBUG] TestAgent.dns: request served from client: name=nope.query.consul. type=SRV class=IN latency=65.689µs client=127.0.0.1:60956 client_network=udp -46:04.115 [INFO] TestAgent: Requesting shutdown -46:04.115 [INFO] TestAgent.server: shutting down server -46:04.115 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.115 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.115 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.115 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.115 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.115 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.115 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.115 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.115 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.115 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.115 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.122 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.135Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.135Z [INFO] agent.router.manager: shutting down -46:04.135 [INFO] TestAgent: consul server down -46:04.135 [INFO] TestAgent: shutdown complete -46:04.135 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11681 network=tcp -46:04.135 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11681 network=udp -46:04.135 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11682 network=tcp -46:04.635 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.635 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQuery_AllowStale (0.75s) +2021-10-05T16:31:18.646Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.662Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.710 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:99c01123-d9a3-8075-f47f-f24c0064db1a Address:127.0.0.1:29966}]" +31:18.711 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-99c01123-d9a3-8075-f47f-f24c0064db1a.dc1 127.0.0.1 +31:18.711 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-99c01123-d9a3-8075-f47f-f24c0064db1a 127.0.0.1 +2021-10-05T16:31:18.711Z [INFO] agent.router: Initializing LAN area manager +31:18.711 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29961 network=udp +31:18.711 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29966 [Follower]" leader= +31:18.712 [INFO] TestAgent.server: Adding LAN server: server="Node-99c01123-d9a3-8075-f47f-f24c0064db1a (Addr: tcp/127.0.0.1:29966) (DC: dc1)" +31:18.712 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-99c01123-d9a3-8075-f47f-f24c0064db1a.dc1 area=wan +31:18.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29961 network=tcp +31:18.712 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29962 network=tcp +31:18.712 [INFO] TestAgent: started state syncer +31:18.713 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29953 network=tcp +31:18.776 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.776 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29966 [Candidate]" term=2 +31:18.816 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.816 [DEBUG] TestAgent.server.raft: vote granted: from=99c01123-d9a3-8075-f47f-f24c0064db1a term=2 tally=1 +31:18.816 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.816 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29966 [Leader]" +31:18.819 [INFO] TestAgent.server: cluster leadership acquired +31:18.819 [INFO] TestAgent.server: New leader elected: payload=Node-99c01123-d9a3-8075-f47f-f24c0064db1a +31:18.852 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29966 +31:18.856 [INFO] TestAgent: Synced node info +31:18.889 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:18.940 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:18.940 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:18.940 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:18.940 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:18.940 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:18.940 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-99c01123-d9a3-8075-f47f-f24c0064db1a +31:18.940 [INFO] TestAgent.server: member joined, marking health alive: member=Node-99c01123-d9a3-8075-f47f-f24c0064db1a +31:18.968 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.055 [WARN] TestAgent.dns: QName invalid: qname=nonexisting. +31:19.055 [DEBUG] TestAgent.dns: request served from client: name=nonexisting.consul. type=ANY class=IN latency=59.314µs client=127.0.0.1:54813 client_network=udp +31:19.055 [INFO] TestAgent: Requesting shutdown +31:19.055 [INFO] TestAgent.server: shutting down server +31:19.055 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.055 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.055 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.055 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.055 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.055 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.056 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.056 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.056 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.056 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.062 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.067Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.067Z [INFO] agent.router.manager: shutting down +31:19.067 [INFO] TestAgent: consul server down +31:19.067 [INFO] TestAgent: shutdown complete +31:19.067 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29961 network=tcp +31:19.068 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29961 network=udp +31:19.068 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29962 network=tcp +31:19.568 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.568 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NonExistingLookup (0.94s) === CONT TestDNS_ServiceLookup_TTL -2022-11-07T22:46:03.970Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.980Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.001 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4c85aea8-3295-cee5-48a9-c29ea384e8c8 Address:127.0.0.1:11672}]" -46:04.001 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11672 [Follower]" leader= -46:04.001 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:04.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:04.001Z [INFO] agent.router: Initializing LAN area manager -46:04.001 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:04.001 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11672) (DC: dc1)" -46:04.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11667 network=udp -46:04.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11667 network=tcp -46:04.002 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11668 network=tcp -46:04.002 [INFO] TestAgent: started state syncer -46:04.002 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11694 network=tcp -46:04.047 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.047 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11672 [Candidate]" term=2 -46:04.059 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.059 [DEBUG] TestAgent.server.raft: vote granted: from=4c85aea8-3295-cee5-48a9-c29ea384e8c8 term=2 tally=1 -46:04.059 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.059 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11672 [Leader]" -46:04.059 [INFO] TestAgent.server: cluster leadership acquired -46:04.059 [INFO] TestAgent.server: New leader elected: payload=test-node -46:04.070 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11672 -46:04.088 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.118 [INFO] TestAgent: Synced node info -46:04.118 [DEBUG] TestAgent: Node info in sync -46:04.128 [WARN] TestAgent.dns: no servers found -46:04.128 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.consul. type=SOA class=IN latency=164.452µs client=127.0.0.1:46656 client_network=udp -46:04.128 [WARN] TestAgent.dns: no servers found -46:04.129 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.test-domain. type=SOA class=IN latency=93.359µs client=127.0.0.1:34843 client_network=udp -46:04.129 [INFO] TestAgent: Requesting shutdown -46:04.129 [INFO] TestAgent.server: shutting down server -46:04.129 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.133 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.133 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.133 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.133 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.133 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.133 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:04.133 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:04.138 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:04.141 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.141 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.141 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.141 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.141 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.141 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.141 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -2022-11-07T22:46:04.146Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.146Z [INFO] agent.router.manager: shutting down -46:04.150 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:04.150 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.150 [INFO] TestAgent: consul server down -46:04.150 [INFO] TestAgent: shutdown complete -46:04.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11667 network=tcp -46:04.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11667 network=udp -46:04.150 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11668 network=tcp -46:04.650 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.650 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_AltDomains_SOA (0.70s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:18.777Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.868 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1be3134e-66f4-426c-06ff-c3be7e40cb29 Address:127.0.0.1:29973}]" +31:18.869 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1be3134e-66f4-426c-06ff-c3be7e40cb29.dc1 127.0.0.1 +31:18.869 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1be3134e-66f4-426c-06ff-c3be7e40cb29 127.0.0.1 +2021-10-05T16:31:18.869Z [INFO] agent.router: Initializing LAN area manager +31:18.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29968 network=udp +31:18.870 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29973 [Follower]" leader= +31:18.870 [INFO] TestAgent.server: Adding LAN server: server="Node-1be3134e-66f4-426c-06ff-c3be7e40cb29 (Addr: tcp/127.0.0.1:29973) (DC: dc1)" +31:18.870 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1be3134e-66f4-426c-06ff-c3be7e40cb29.dc1 area=wan +31:18.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29968 network=tcp +31:18.871 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29969 network=tcp +31:18.871 [INFO] TestAgent: started state syncer +31:18.871 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29974 network=tcp +31:18.927 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.927 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29973 [Candidate]" term=2 +31:18.968 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:18.968 [DEBUG] TestAgent.server.raft: vote granted: from=1be3134e-66f4-426c-06ff-c3be7e40cb29 term=2 tally=1 +31:18.968 [INFO] TestAgent.server.raft: election won: tally=1 +31:18.968 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29973 [Leader]" +31:18.968 [INFO] TestAgent.server: cluster leadership acquired +31:18.968 [INFO] TestAgent.server: New leader elected: payload=Node-1be3134e-66f4-426c-06ff-c3be7e40cb29 +31:18.996 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29973 +31:19.021 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.071 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.071 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.071 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.071 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.071 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.072 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1be3134e-66f4-426c-06ff-c3be7e40cb29 +31:19.072 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1be3134e-66f4-426c-06ff-c3be7e40cb29 +31:19.094 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.160 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:19.180 [INFO] TestAgent: Synced node info +31:19.180 [DEBUG] TestAgent: Node info in sync +31:19.285 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Health.ServiceNodes +31:19.285 [ERROR] TestAgent.dns: rpc error: error="No path to datacenter" +31:19.285 [DEBUG] TestAgent.dns: request served from client: name=consul.service.dc2.consul. type=ANY class=IN latency=137.895µs client=127.0.0.1:56569 client_network=udp +31:19.285 [INFO] TestAgent: Requesting shutdown +31:19.285 [INFO] TestAgent.server: shutting down server +31:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.285 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.285 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.293 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.295Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.295Z [INFO] agent.router.manager: shutting down +31:19.295 [INFO] TestAgent: consul server down +31:19.295 [INFO] TestAgent: shutdown complete +31:19.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29968 network=tcp +31:19.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29968 network=udp +31:19.295 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29969 network=tcp +31:19.796 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.796 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NonExistentDC_Server (1.05s) === CONT TestDNS_NodeLookup_TTL -2022-11-07T22:46:04.004Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.014Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.037 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46fa63c2-2996-de62-28ef-fe1cf0bc1dd7 Address:127.0.0.1:11700}]" -46:04.037 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11700 [Follower]" leader= -46:04.037 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46fa63c2-2996-de62-28ef-fe1cf0bc1dd7.dc1 127.0.0.1 -46:04.038 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46fa63c2-2996-de62-28ef-fe1cf0bc1dd7 127.0.0.1 -2022-11-07T22:46:04.038Z [INFO] agent.router: Initializing LAN area manager -46:04.038 [INFO] TestAgent.server: Adding LAN server: server="Node-46fa63c2-2996-de62-28ef-fe1cf0bc1dd7 (Addr: tcp/127.0.0.1:11700) (DC: dc1)" -46:04.038 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46fa63c2-2996-de62-28ef-fe1cf0bc1dd7.dc1 area=wan -46:04.038 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11695 network=udp -46:04.038 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11695 network=tcp -46:04.038 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11696 network=tcp -46:04.038 [INFO] TestAgent: started state syncer -46:04.038 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11687 network=tcp -46:04.100 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.101 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11700 [Candidate]" term=2 -46:04.113 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.113 [DEBUG] TestAgent.server.raft: vote granted: from=46fa63c2-2996-de62-28ef-fe1cf0bc1dd7 term=2 tally=1 -46:04.113 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.113 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11700 [Leader]" -46:04.113 [INFO] TestAgent.server: cluster leadership acquired -46:04.113 [INFO] TestAgent.server: New leader elected: payload=Node-46fa63c2-2996-de62-28ef-fe1cf0bc1dd7 -46:04.120 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11700 -46:04.143 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.152 [INFO] TestAgent: Requesting shutdown -46:04.152 [INFO] TestAgent.server: shutting down server -46:04.152 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.152 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.157 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.165Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.165Z [INFO] agent.router.manager: shutting down -46:04.170 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" -46:04.170 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" -46:04.170 [INFO] TestAgent: consul server down -46:04.170 [INFO] TestAgent: shutdown complete -46:04.170 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11695 network=tcp -46:04.170 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:04.170 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11695 network=udp -46:04.170 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11696 network=tcp -46:04.670 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.670 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_LoadKeyrings/server_with_keys -=== RUN TestDNS_PreparedQuery_TTL/db.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/db-ttl.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/dblb.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/dblb-ttl.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/dk.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/dk-ttl.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/api.query.consul. -=== RUN TestDNS_PreparedQuery_TTL/api-ttl.query.consul. -2022-11-07T22:46:04.050Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.060Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.104 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2faf6f04-939a-b5b4-344f-3139f2c87693 Address:127.0.0.1:11728}]" -46:04.104 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11728 [Follower]" leader= -46:04.105 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2faf6f04-939a-b5b4-344f-3139f2c87693.dc1 127.0.0.1 -46:04.106 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2faf6f04-939a-b5b4-344f-3139f2c87693 127.0.0.1 -2022-11-07T22:46:04.106Z [INFO] agent.router: Initializing LAN area manager -46:04.106 [INFO] TestAgent.server: Adding LAN server: server="Node-2faf6f04-939a-b5b4-344f-3139f2c87693 (Addr: tcp/127.0.0.1:11728) (DC: dc1)" -46:04.106 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2faf6f04-939a-b5b4-344f-3139f2c87693.dc1 area=wan -46:04.106 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11723 network=udp -46:04.106 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11723 network=tcp -46:04.106 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11724 network=tcp -46:04.106 [INFO] TestAgent: started state syncer -46:04.106 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11715 network=tcp -46:04.169 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.169 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11728 [Candidate]" term=2 -46:04.184 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.184 [DEBUG] TestAgent.server.raft: vote granted: from=2faf6f04-939a-b5b4-344f-3139f2c87693 term=2 tally=1 -46:04.184 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.184 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11728 [Leader]" -46:04.184 [INFO] TestAgent.server: cluster leadership acquired -46:04.184 [INFO] TestAgent.server: New leader elected: payload=Node-2faf6f04-939a-b5b4-344f-3139f2c87693 -46:04.195 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11728 -46:04.219 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.252 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.252 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.252 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.252 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.252 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.252 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2faf6f04-939a-b5b4-344f-3139f2c87693 -46:04.252 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2faf6f04-939a-b5b4-344f-3139f2c87693 -46:04.269 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.275 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=154.518µs client=127.0.0.1:58186 client_network=udp -46:04.275 [DEBUG] TestAgent.dns: request served from client: name=db.service.test-domain. type=SRV class=IN latency=105.285µs client=127.0.0.1:37102 client_network=udp -46:04.276 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc1.consul. type=SRV class=IN latency=158.166µs client=127.0.0.1:59675 client_network=udp -46:04.276 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc1.test-domain. type=SRV class=IN latency=114.202µs client=127.0.0.1:43738 client_network=udp -46:04.276 [INFO] TestAgent: Requesting shutdown -46:04.276 [INFO] TestAgent.server: shutting down server -46:04.276 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.276 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.276 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.276 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.276 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.276 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.276 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.276 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.276 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.276 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.276 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.285 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.292Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.292Z [INFO] agent.router.manager: shutting down -46:04.292 [INFO] TestAgent: consul server down -46:04.292 [INFO] TestAgent: shutdown complete -46:04.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11723 network=tcp -46:04.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11723 network=udp -46:04.292 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11724 network=tcp -46:04.792 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.792 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_AltDomains_Service (0.75s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:18.863Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.921Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.999 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:830ad23f-b2f5-e70a-b222-caee2c23abfd Address:127.0.0.1:30001}]" +31:19.000 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30001 [Follower]" leader= +31:19.000 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-830ad23f-b2f5-e70a-b222-caee2c23abfd.dc1 127.0.0.1 +31:19.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-830ad23f-b2f5-e70a-b222-caee2c23abfd 127.0.0.1 +2021-10-05T16:31:19.001Z [INFO] agent.router: Initializing LAN area manager +31:19.001 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-830ad23f-b2f5-e70a-b222-caee2c23abfd.dc1 area=wan +31:19.001 [INFO] TestAgent.server: Adding LAN server: server="Node-830ad23f-b2f5-e70a-b222-caee2c23abfd (Addr: tcp/127.0.0.1:30001) (DC: dc1)" +31:19.001 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29996 network=udp +31:19.001 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29996 network=tcp +31:19.002 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29997 network=tcp +31:19.002 [INFO] TestAgent: started state syncer +31:19.002 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29988 network=tcp +31:19.065 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.065 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30001 [Candidate]" term=2 +31:19.111 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.111 [DEBUG] TestAgent.server.raft: vote granted: from=830ad23f-b2f5-e70a-b222-caee2c23abfd term=2 tally=1 +31:19.111 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.111 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30001 [Leader]" +31:19.115 [INFO] TestAgent.server: cluster leadership acquired +31:19.117 [INFO] TestAgent.server: New leader elected: payload=Node-830ad23f-b2f5-e70a-b222-caee2c23abfd +31:19.140 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30001 +31:19.196 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.290 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.290 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.290 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.290 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.290 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.290 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-830ad23f-b2f5-e70a-b222-caee2c23abfd +31:19.290 [INFO] TestAgent.server: member joined, marking health alive: member=Node-830ad23f-b2f5-e70a-b222-caee2c23abfd +31:19.300 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.333 [DEBUG] TestAgent.dns: request served from client: name=7f000001.addr.dc1.consul. type=SRV class=IN latency=122.715µs client=127.0.0.1:60632 client_network=udp +31:19.333 [INFO] TestAgent: Requesting shutdown +31:19.334 [INFO] TestAgent.server: shutting down server +31:19.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.334 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.334 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.334 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.334 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.334 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:19.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.334 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.334 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.339 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.341Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.341Z [INFO] agent.router.manager: shutting down +31:19.341 [INFO] TestAgent: consul server down +31:19.341 [INFO] TestAgent: shutdown complete +31:19.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29996 network=tcp +31:19.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29996 network=udp +31:19.341 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29997 network=tcp +31:19.842 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.842 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_AddressLookup (1.02s) === CONT TestDNS_ServiceLookup_ServiceAddress_CNAME -2022-11-07T22:46:04.025Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.035Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.079 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ead668b7-20e9-667c-6c73-39148263d177 Address:127.0.0.1:11693}]" -46:04.079 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11693 [Follower]" leader= -46:04.079 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:04.080 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:04.080Z [INFO] agent.router: Initializing LAN area manager -46:04.080 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11693) (DC: dc1)" -46:04.080 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:04.080 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11688 network=udp -46:04.080 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11688 network=tcp -46:04.084 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11689 network=tcp -46:04.084 [INFO] TestAgent: started state syncer -46:04.084 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11701 network=tcp -46:04.126 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.126 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11693 [Candidate]" term=2 -46:04.139 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.139 [DEBUG] TestAgent.server.raft: vote granted: from=ead668b7-20e9-667c-6c73-39148263d177 term=2 tally=1 -46:04.139 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.139 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11693 [Leader]" -46:04.139 [INFO] TestAgent.server: cluster leadership acquired -46:04.139 [INFO] TestAgent.server: New leader elected: payload=test-node -46:04.149 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11693 -46:04.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.205 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.205 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.205 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.205 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.205 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.205 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:04.205 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:04.225 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.303 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.consul. type=A class=IN latency=72.306µs client=127.0.0.1:51833 client_network=udp -46:04.303 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.test.consul. type=A class=IN latency=54.878µs client=127.0.0.1:43970 client_network=udp -46:04.303 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.dc1.consul. type=A class=IN latency=60.145µs client=127.0.0.1:47063 client_network=udp -46:04.304 [DEBUG] TestAgent.dns: request served from client: name=test-node.node.dc1.test.consul. type=A class=IN latency=54.93µs client=127.0.0.1:53036 client_network=udp -46:04.304 [INFO] TestAgent: Requesting shutdown -46:04.304 [INFO] TestAgent.server: shutting down server -46:04.304 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.304 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.304 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.304 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.304 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.304 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.304 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.304 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.304 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.304 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.304 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.314 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.319Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.319Z [INFO] agent.router.manager: shutting down -46:04.319 [INFO] TestAgent: consul server down -46:04.319 [INFO] TestAgent: shutdown complete -46:04.319 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11688 network=tcp -46:04.319 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11688 network=udp -46:04.319 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11689 network=tcp -46:04.819 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.819 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_AltDomains_Overlap (0.80s) +2021-10-05T16:31:18.939Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:18.939Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.980Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:18.980Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:82730898-43ad-a510-7761-43d2da8b35e1 Address:127.0.0.1:29994}]" +31:19.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 127.0.0.1 +31:19.019 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-82730898-43ad-a510-7761-43d2da8b35e1 127.0.0.1 +2021-10-05T16:31:19.019Z [INFO] agent.router: Initializing LAN area manager +31:19.020 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29989 network=udp +31:19.020 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29994 [Follower]" leader= +31:19.020 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 area=wan +31:19.020 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29989 network=tcp +31:19.020 [INFO] TestAgent.server: Adding LAN server: server="Node-82730898-43ad-a510-7761-43d2da8b35e1 (Addr: tcp/127.0.0.1:29994) (DC: dc1)" +31:19.021 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29990 network=tcp +31:19.021 [INFO] TestAgent: started state syncer +31:19.021 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30002 network=tcp +31:19.071 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:19.091 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.091 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29994 [Candidate]" term=2 +31:19.137 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.137 [DEBUG] TestAgent.server.raft: vote granted: from=82730898-43ad-a510-7761-43d2da8b35e1 term=2 tally=1 +31:19.137 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29994 [Leader]" +31:19.139 [INFO] TestAgent.server: cluster leadership acquired +31:19.139 [INFO] TestAgent.server: New leader elected: payload=Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.147 [INFO] TestAgent.server: initializing acls +31:19.156 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:19.156 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:19.172 [INFO] TestAgent.server: initializing acls +31:19.172 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:19.208 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:19.212 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:19.280 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:19.280 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:19.280 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:19.280 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:19.280 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.280 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 +31:19.280 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 area=wan +31:19.281 [INFO] TestAgent: Synced node info +31:19.281 [DEBUG] TestAgent: Node info in sync +31:19.281 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:19.281 [INFO] TestAgent.server: Updating LAN server: server="Node-82730898-43ad-a510-7761-43d2da8b35e1 (Addr: tcp/127.0.0.1:29994) (DC: dc1)" +31:19.281 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.281 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 +31:19.284 [INFO] TestAgent.server: Updating LAN server: server="Node-82730898-43ad-a510-7761-43d2da8b35e1 (Addr: tcp/127.0.0.1:29994) (DC: dc1)" +31:19.284 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-82730898-43ad-a510-7761-43d2da8b35e1.dc1 area=wan +31:19.299 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.303 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.304 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.353 [DEBUG] TestAgent.dns: request served from client: name=foo.service.consul. type=A class=IN latency=393.852µs client=127.0.0.1:35967 client_network=udp +31:19.353 [INFO] TestAgent: Requesting shutdown +31:19.354 [INFO] TestAgent.server: shutting down server +31:19.354 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:19.354 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:19.354 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.354 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:19.354 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:19.364 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:19.365 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.365 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.365 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.365 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.365 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.365 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-82730898-43ad-a510-7761-43d2da8b35e1 +31:19.365 [INFO] TestAgent.server: member joined, marking health alive: member=Node-82730898-43ad-a510-7761-43d2da8b35e1 +2021-10-05T16:31:19.366Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.366Z [INFO] agent.router.manager: shutting down +31:19.384 [INFO] TestAgent: consul server down +31:19.384 [INFO] TestAgent: shutdown complete +31:19.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29989 network=tcp +31:19.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29989 network=udp +31:19.384 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29990 network=tcp +31:19.384 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-82730898-43ad-a510-7761-43d2da8b35e1 127.0.0.1 29992 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:82730898-43ad-a510-7761-43d2da8b35e1 port:29994 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29993] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:19.384 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.384 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.384 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.384 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.384 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.384 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.384 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.884 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.884 [INFO] TestAgent: Endpoints down +=== RUN TestDNS_ServiceLookup_FilterACL/ACLToken_==_anonymous +2021-10-05T16:31:18.826Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.868Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.960 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8c4a7d6f-d645-c88d-8589-0eb1e340567d Address:127.0.0.1:29980}]" +31:18.960 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d.dc1 127.0.0.1 +31:18.961 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d 127.0.0.1 +2021-10-05T16:31:18.961Z [INFO] agent.router: Initializing LAN area manager +31:18.961 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29975 network=udp +31:18.961 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29980 [Follower]" leader= +31:18.962 [INFO] TestAgent.server: Adding LAN server: server="Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d (Addr: tcp/127.0.0.1:29980) (DC: dc1)" +31:18.962 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d.dc1 area=wan +31:18.962 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29975 network=tcp +31:18.962 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29976 network=tcp +31:18.962 [INFO] TestAgent: started state syncer +31:18.962 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29995 network=tcp +31:19.016 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.016 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29980 [Candidate]" term=2 +31:19.038 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.038 [DEBUG] TestAgent.server.raft: vote granted: from=8c4a7d6f-d645-c88d-8589-0eb1e340567d term=2 tally=1 +31:19.039 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.039 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29980 [Leader]" +31:19.039 [INFO] TestAgent.server: cluster leadership acquired +31:19.039 [INFO] TestAgent.server: New leader elected: payload=Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d +31:19.054 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29980 +31:19.089 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.137 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.137 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.137 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.137 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.137 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.138 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d +31:19.138 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8c4a7d6f-d645-c88d-8589-0eb1e340567d +31:19.164 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.176 [INFO] TestAgent: Synced node info +31:19.177 [DEBUG] TestAgent: Node info in sync +31:19.392 [DEBUG] TestAgent.dns: request served from client: name=2607002040050808000000000000200e.addr.consul. type=SRV class=IN latency=63.751µs client=127.0.0.1:57859 client_network=udp +31:19.392 [DEBUG] TestAgent.dns: request served from client: name=2607112040051808ffffffffffff200e.addr.consul. type=SRV class=IN latency=18.161µs client=127.0.0.1:38408 client_network=udp +31:19.392 [INFO] TestAgent: Requesting shutdown +31:19.392 [INFO] TestAgent.server: shutting down server +31:19.392 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.392 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.392 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.392 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.392 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.392 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.392 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.392 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.392 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.392 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.400 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.400Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.400Z [INFO] agent.router.manager: shutting down +31:19.400 [INFO] TestAgent: consul server down +31:19.400 [INFO] TestAgent: shutdown complete +31:19.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29975 network=tcp +31:19.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29975 network=udp +31:19.400 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29976 network=tcp +31:19.902 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.902 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_AddressLookupIPV6 (1.10s) === CONT TestDNS_ServiceLookup_AnswerLimits === RUN TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} === PAUSE TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} @@ -28355,9111 +27015,3760 @@ === RUN TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} === PAUSE TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} === CONT TestDNS_ServiceLookup_LargeResponses +2021-10-05T16:31:19.164Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.198Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.344 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b63fbff-472a-5973-8717-b4b924dc31a5 Address:127.0.0.1:30022}]" +31:19.345 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30022 [Follower]" leader= +31:19.345 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b63fbff-472a-5973-8717-b4b924dc31a5.dc1 127.0.0.1 +31:19.345 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b63fbff-472a-5973-8717-b4b924dc31a5 127.0.0.1 +2021-10-05T16:31:19.345Z [INFO] agent.router: Initializing LAN area manager +31:19.346 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30017 network=udp +31:19.346 [INFO] TestAgent.server: Adding LAN server: server="Node-4b63fbff-472a-5973-8717-b4b924dc31a5 (Addr: tcp/127.0.0.1:30022) (DC: dc1)" +31:19.346 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b63fbff-472a-5973-8717-b4b924dc31a5.dc1 area=wan +31:19.346 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30017 network=tcp +31:19.347 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30018 network=tcp +31:19.347 [INFO] TestAgent: started state syncer +31:19.347 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30009 network=tcp +31:19.384 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.384 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30022 [Candidate]" term=2 +31:19.410 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.410 [DEBUG] TestAgent.server.raft: vote granted: from=4b63fbff-472a-5973-8717-b4b924dc31a5 term=2 tally=1 +31:19.410 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.410 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30022 [Leader]" +31:19.411 [INFO] TestAgent.server: cluster leadership acquired +31:19.411 [INFO] TestAgent.server: New leader elected: payload=Node-4b63fbff-472a-5973-8717-b4b924dc31a5 +31:19.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30022 +31:19.434 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.470 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.470 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.470 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.470 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.470 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.470 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b63fbff-472a-5973-8717-b4b924dc31a5 +31:19.470 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b63fbff-472a-5973-8717-b4b924dc31a5 +31:19.482 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.508 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.service.dc1.consul. type=SRV class=IN latency=135.849µs client=127.0.0.1:52744 client_network=udp +31:19.509 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.service.consul. type=SRV class=IN latency=99.968µs client=127.0.0.1:43645 client_network=udp +31:19.509 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.dc1.consul. type=SRV class=IN latency=81.416µs client=127.0.0.1:43918 client_network=udp +31:19.509 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.consul. type=SRV class=IN latency=78.541µs client=127.0.0.1:57714 client_network=udp +31:19.509 [INFO] TestAgent: Requesting shutdown +31:19.509 [INFO] TestAgent.server: shutting down server +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.509 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.509 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.509 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.520 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.520Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.520Z [INFO] agent.router.manager: shutting down +31:19.520 [INFO] TestAgent: consul server down +31:19.520 [INFO] TestAgent: shutdown complete +31:19.520 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30017 network=tcp +31:19.520 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30017 network=udp +31:19.520 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30018 network=tcp +31:20.020 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.020 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_SRV_RFC (0.90s) +=== CONT TestDNS_ServiceLookup_Truncate +2021-10-05T16:31:19.066Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.134Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.217 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6ddf76c0-d92c-846b-18da-61924bb9f012 Address:127.0.0.1:30008}]" +31:19.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6ddf76c0-d92c-846b-18da-61924bb9f012.dc1 127.0.0.1 +31:19.218 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6ddf76c0-d92c-846b-18da-61924bb9f012 127.0.0.1 +2021-10-05T16:31:19.218Z [INFO] agent.router: Initializing LAN area manager +31:19.218 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30003 network=udp +31:19.219 [INFO] TestAgent.server: Adding LAN server: server="Node-6ddf76c0-d92c-846b-18da-61924bb9f012 (Addr: tcp/127.0.0.1:30008) (DC: dc1)" +31:19.219 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6ddf76c0-d92c-846b-18da-61924bb9f012.dc1 area=wan +31:19.220 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30003 network=tcp +31:19.220 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30008 [Follower]" leader= +31:19.220 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30004 network=tcp +31:19.220 [INFO] TestAgent: started state syncer +31:19.220 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29981 network=tcp +31:19.288 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.288 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30008 [Candidate]" term=2 +31:19.339 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.339 [DEBUG] TestAgent.server.raft: vote granted: from=6ddf76c0-d92c-846b-18da-61924bb9f012 term=2 tally=1 +31:19.339 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.339 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30008 [Leader]" +31:19.339 [INFO] TestAgent.server: cluster leadership acquired +31:19.339 [INFO] TestAgent.server: New leader elected: payload=Node-6ddf76c0-d92c-846b-18da-61924bb9f012 +31:19.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30008 +31:19.386 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.420 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.420 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.420 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.420 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.420 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.420 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6ddf76c0-d92c-846b-18da-61924bb9f012 +31:19.420 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6ddf76c0-d92c-846b-18da-61924bb9f012 +31:19.484 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.484 [INFO] TestAgent: Synced node info +31:19.484 [DEBUG] TestAgent: Node info in sync +31:19.504 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.service.dc1.consul. type=SRV class=IN latency=236.438µs client=127.0.0.1:43496 client_network=udp +31:19.508 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.service.consul. type=SRV class=IN latency=179.963µs client=127.0.0.1:48628 client_network=udp +31:19.508 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.dc1.consul. type=SRV class=IN latency=84.858µs client=127.0.0.1:51942 client_network=udp +31:19.509 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.consul. type=SRV class=IN latency=92.89µs client=127.0.0.1:56488 client_network=udp +31:19.509 [INFO] TestAgent: Requesting shutdown +31:19.509 [INFO] TestAgent.server: shutting down server +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.509 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.509 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.509 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.509 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.524 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.528Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.528Z [INFO] agent.router.manager: shutting down +31:19.528 [INFO] TestAgent: consul server down +31:19.528 [INFO] TestAgent: shutdown complete +31:19.528 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30003 network=tcp +31:19.528 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30003 network=udp +31:19.528 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30004 network=tcp +31:20.028 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.028 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_SRV_RFC_TCP_Default (0.99s) +=== CONT TestDNS_TCP_and_UDP_Truncate +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_ServiceLookup_TTL/db.service.consul. === RUN TestDNS_ServiceLookup_TTL/dblb.service.consul. === RUN TestDNS_ServiceLookup_TTL/dk.service.consul. === RUN TestDNS_ServiceLookup_TTL/api.service.consul. -2022-11-07T22:46:04.058Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.070Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.111 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e32dc91a-db45-4fbc-9471-613cf171bab8 Address:127.0.0.1:11721}]" -46:04.111 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11721 [Follower]" leader= -46:04.112 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e32dc91a-db45-4fbc-9471-613cf171bab8.dc1 127.0.0.1 -46:04.112 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e32dc91a-db45-4fbc-9471-613cf171bab8 127.0.0.1 -2022-11-07T22:46:04.112Z [INFO] agent.router: Initializing LAN area manager -46:04.113 [INFO] TestAgent.server: Adding LAN server: server="Node-e32dc91a-db45-4fbc-9471-613cf171bab8 (Addr: tcp/127.0.0.1:11721) (DC: dc1)" -46:04.113 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e32dc91a-db45-4fbc-9471-613cf171bab8.dc1 area=wan -46:04.113 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11716 network=udp -46:04.113 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11716 network=tcp -46:04.113 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11717 network=tcp -46:04.113 [INFO] TestAgent: started state syncer -46:04.113 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11736 network=tcp -46:04.152 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.152 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11721 [Candidate]" term=2 -46:04.162 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.162 [DEBUG] TestAgent.server.raft: vote granted: from=e32dc91a-db45-4fbc-9471-613cf171bab8 term=2 tally=1 -46:04.162 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.162 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11721 [Leader]" -46:04.162 [INFO] TestAgent.server: cluster leadership acquired -46:04.162 [INFO] TestAgent.server: New leader elected: payload=Node-e32dc91a-db45-4fbc-9471-613cf171bab8 -46:04.173 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11721 -46:04.200 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.242 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.242 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.242 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.242 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.242 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.242 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e32dc91a-db45-4fbc-9471-613cf171bab8 -46:04.242 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e32dc91a-db45-4fbc-9471-613cf171bab8 -46:04.262 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.325 [INFO] TestAgent: Requesting shutdown -46:04.325 [INFO] TestAgent.server: shutting down server -46:04.325 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.325 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.325 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.325 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.325 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.325 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.325 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.325 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.325 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.325 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.325 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.332 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.336Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.336Z [INFO] agent.router.manager: shutting down -46:04.336 [INFO] TestAgent: consul server down -46:04.336 [INFO] TestAgent: shutdown complete -46:04.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11716 network=tcp -46:04.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11716 network=udp -46:04.336 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11717 network=tcp -46:04.836 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.836 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsSpecificDelete_good (0.79s) -=== CONT TestDNS_ServiceLookup_Truncate -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:04.104Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.112Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.155 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 Address:127.0.0.1:11756}]" -46:04.155 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11756 [Follower]" leader= -46:04.155 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4.dc1 127.0.0.1 -46:04.156 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 127.0.0.1 -2022-11-07T22:46:04.156Z [INFO] agent.router: Initializing LAN area manager -46:04.156 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4.dc1 area=wan -46:04.156 [INFO] TestAgent.server: Adding LAN server: server="Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 (Addr: tcp/127.0.0.1:11756) (DC: dc1)" -46:04.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11751 network=udp -46:04.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11751 network=tcp -46:04.156 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11752 network=tcp -46:04.156 [INFO] TestAgent: started state syncer -46:04.156 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11743 network=tcp -46:04.200 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.200 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11756 [Candidate]" term=2 -46:04.213 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.213 [DEBUG] TestAgent.server.raft: vote granted: from=e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 term=2 tally=1 -46:04.213 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.213 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11756 [Leader]" -46:04.214 [INFO] TestAgent.server: cluster leadership acquired -46:04.214 [INFO] TestAgent.server: New leader elected: payload=Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 -46:04.223 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11756 -46:04.248 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.278 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.278 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.278 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.278 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.278 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.278 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 -46:04.278 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e2bc59b6-374a-a20a-eb61-b7d2ed3c03c4 -46:04.346 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Health.ServiceNodes -46:04.346 [ERROR] TestAgent.dns: rpc error: error="No path to datacenter" -46:04.346 [DEBUG] TestAgent.dns: request served from client: name=consul.service.dc2.consul. type=ANY class=IN latency=82.479µs client=127.0.0.1:42351 client_network=udp -46:04.346 [INFO] TestAgent: Requesting shutdown -46:04.346 [INFO] TestAgent.server: shutting down server -46:04.346 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.346 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.346 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.346 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.346 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -46:04.346 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.346 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.346 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.346 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.346 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.346 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.351 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.359Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.359Z [INFO] agent.router.manager: shutting down -46:04.359 [INFO] TestAgent: consul server down -46:04.359 [INFO] TestAgent: shutdown complete -46:04.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11751 network=tcp -46:04.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11751 network=udp -46:04.359 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11752 network=tcp -46:04.860 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.860 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NonExistentDC_Server (0.77s) -=== CONT TestDNS_TCP_and_UDP_Truncate -2022-11-07T22:46:04.075Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.093Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.134 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ff611c6d-c8a0-72b2-e229-fb74e2ffe254 Address:127.0.0.1:11742}]" -46:04.134 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11742 [Follower]" leader= -46:04.134 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254.dc1 127.0.0.1 -46:04.135 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254 127.0.0.1 -2022-11-07T22:46:04.135Z [INFO] agent.router: Initializing LAN area manager -46:04.135 [INFO] TestAgent.server: Adding LAN server: server="Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254 (Addr: tcp/127.0.0.1:11742) (DC: dc1)" -46:04.135 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254.dc1 area=wan -46:04.135 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11737 network=udp -46:04.135 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11737 network=tcp -46:04.136 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11738 network=tcp -46:04.136 [INFO] TestAgent: started state syncer -46:04.136 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11729 network=tcp -46:04.187 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.187 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11742 [Candidate]" term=2 -46:04.200 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.200 [DEBUG] TestAgent.server.raft: vote granted: from=ff611c6d-c8a0-72b2-e229-fb74e2ffe254 term=2 tally=1 -46:04.200 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.200 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11742 [Leader]" -46:04.200 [INFO] TestAgent.server: cluster leadership acquired -46:04.200 [INFO] TestAgent.server: New leader elected: payload=Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254 -46:04.219 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11742 -46:04.242 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.273 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.273 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.273 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.273 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.273 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.273 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254 -46:04.273 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ff611c6d-c8a0-72b2-e229-fb74e2ffe254 -46:04.291 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.343 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.347 [INFO] TestAgent: Synced node info -46:04.347 [WARN] TestAgent.dns: QName invalid: qname=nonexisting. -46:04.347 [DEBUG] TestAgent.dns: request served from client: name=nonexisting.consul. type=ANY class=IN latency=40.423µs client=127.0.0.1:45489 client_network=udp -46:04.347 [INFO] TestAgent: Requesting shutdown -46:04.347 [INFO] TestAgent.server: shutting down server -46:04.347 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.347 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.347 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.347 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.347 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.347 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.355 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.362Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.362Z [INFO] agent.router.manager: shutting down -46:04.362 [INFO] TestAgent: consul server down -46:04.362 [INFO] TestAgent: shutdown complete -46:04.362 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11737 network=tcp -46:04.362 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11737 network=udp -46:04.362 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11738 network=tcp -46:04.863 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.863 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NonExistingLookup (0.80s) +=== RUN TestDNS_PreparedQuery_TTL/db.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/db-ttl.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/dblb.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/dblb-ttl.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/dk.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/dk-ttl.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/api.query.consul. +=== RUN TestDNS_PreparedQuery_TTL/api-ttl.query.consul. +31:19.163 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +2021-10-05T16:31:19.163Z [INFO] agent.router: Initializing LAN area manager +31:19.166 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29982 network=udp +31:19.166 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29982 network=tcp +31:19.166 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29983 network=tcp +31:19.166 [INFO] TestAgent: started state syncer +31:19.167 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30016 network=tcp +2021-10-05T16:31:19.167Z [WARN] agent.router.manager: No servers available +31:19.167 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:19.169 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:29957] +31:19.169 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:29957 +31:19.172 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: test-server 127.0.0.1 +31:19.172 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:19.172 [DEBUG] TestAgent: systemd notify failed: error="No socket" +2021-10-05T16:31:19.172Z [WARN] agent.router.manager: No servers available +31:19.172 [INFO] TestAgent.client: adding server: server="test-server (Addr: tcp/127.0.0.1:29959) (DC: dc1)" +31:19.209 [ERROR] TestAgent.client: RPC failed to server: method=Health.ServiceNodes server=127.0.0.1:29959 error="rpc error making call: No path to datacenter" +31:19.209 [ERROR] TestAgent.dns: rpc error: error="rpc error making call: No path to datacenter" +31:19.209 [DEBUG] TestAgent.dns: request served from client: name=consul.service.dc2.consul. type=ANY class=IN latency=334.425µs client=127.0.0.1:53684 client_network=udp +31:19.209 [INFO] TestAgent: Requesting shutdown +31:19.209 [INFO] TestAgent.client: shutting down client +31:19.209 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:19.248 [INFO] TestAgent: consul client down +31:19.248 [INFO] TestAgent: shutdown complete +31:19.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29982 network=tcp +31:19.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29982 network=udp +31:19.248 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29983 network=tcp +31:19.748 [INFO] TestAgent: Waiting for endpoints to shut down +31:19.748 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:18.773Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:18.787Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:18.896 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eaff7bc0-90cf-8add-5e63-087ec86d3dfb Address:127.0.0.1:29959}]" +31:18.896 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29959 [Follower]" leader= +31:18.897 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-server.dc1 127.0.0.1 +31:18.897 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-server 127.0.0.1 +2021-10-05T16:31:18.897Z [INFO] agent.router: Initializing LAN area manager +31:18.897 [INFO] TestAgent.server: Adding LAN server: server="test-server (Addr: tcp/127.0.0.1:29959) (DC: dc1)" +31:18.897 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29954 network=udp +31:18.898 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-server.dc1 area=wan +31:18.898 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29954 network=tcp +31:18.898 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29955 network=tcp +31:18.898 [INFO] TestAgent: started state syncer +31:18.898 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29967 network=tcp +31:18.963 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:18.963 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29959 [Candidate]" term=2 +31:19.002 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.002 [DEBUG] TestAgent.server.raft: vote granted: from=eaff7bc0-90cf-8add-5e63-087ec86d3dfb term=2 tally=1 +31:19.002 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.002 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29959 [Leader]" +31:19.003 [INFO] TestAgent.server: cluster leadership acquired +31:19.003 [INFO] TestAgent.server: New leader elected: payload=test-server +31:19.026 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29959 +31:19.056 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.128 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.128 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.128 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.128 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.128 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.128 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-server +31:19.128 [INFO] TestAgent.server: member joined, marking health alive: member=test-server +31:19.167 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.170 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:54074 +31:19.170 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 +31:19.170 [INFO] TestAgent.server: member joined, marking health alive: member=test-client +31:19.208 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Health.ServiceNodes +31:19.265 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:19.280 [INFO] TestAgent: Synced node info +31:19.280 [DEBUG] TestAgent: Node info in sync +31:19.748 [INFO] TestAgent: Requesting shutdown +31:19.748 [INFO] TestAgent.server: shutting down server +31:19.748 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:19.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:19.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:19.748 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:19.748 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:19.748 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:19.748 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:19.749 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:19.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:19.749 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:19.757 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:19.757Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:19.757Z [INFO] agent.router.manager: shutting down +31:19.757 [INFO] TestAgent: consul server down +31:19.757 [INFO] TestAgent: shutdown complete +31:19.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29954 network=tcp +31:19.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29954 network=udp +31:19.757 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29955 network=tcp +31:20.257 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.257 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NonExistentDC_RPC (1.52s) === CONT TestBinarySearch === RUN TestBinarySearch/binarySearch_12 === RUN TestBinarySearch/binarySearch_256 === RUN TestBinarySearch/binarySearch_512 === RUN TestBinarySearch/binarySearch_8192 === RUN TestBinarySearch/binarySearch_65535 +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestBinarySearch/binarySearch_12#01 === RUN TestBinarySearch/binarySearch_256#01 === RUN TestBinarySearch/binarySearch_512#01 === RUN TestBinarySearch/binarySearch_8192#01 -2022-11-07T22:46:04.180Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.180Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.191Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.191Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.240 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d6a00251-155b-efd4-20ce-7d5457f1fe5f Address:127.0.0.1:11770}]" -46:04.240 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11770 [Follower]" leader= -46:04.241 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f.dc1 127.0.0.1 -46:04.241 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f 127.0.0.1 -2022-11-07T22:46:04.241Z [INFO] agent.router: Initializing LAN area manager -46:04.241 [INFO] TestAgent.server: Adding LAN server: server="Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f (Addr: tcp/127.0.0.1:11770) (DC: dc1)" -46:04.241 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f.dc1 area=wan -46:04.241 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11765 network=udp -46:04.241 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11765 network=tcp -46:04.242 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11766 network=tcp -46:04.242 [INFO] TestAgent: started state syncer -46:04.242 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11757 network=tcp -46:04.291 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:04.300 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.300 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11770 [Candidate]" term=2 -46:04.313 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.313 [DEBUG] TestAgent.server.raft: vote granted: from=d6a00251-155b-efd4-20ce-7d5457f1fe5f term=2 tally=1 -46:04.313 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.313 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11770 [Leader]" -46:04.313 [INFO] TestAgent.server: cluster leadership acquired -46:04.313 [INFO] TestAgent.server: New leader elected: payload=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.322 [INFO] TestAgent.server: initializing acls -46:04.327 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:04.327 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:04.334 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:04.346 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:04.346 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:04.346 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:04.346 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.346 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f.dc1 -46:04.346 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f.dc1 area=wan -46:04.346 [INFO] TestAgent.server: Updating LAN server: server="Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f (Addr: tcp/127.0.0.1:11770) (DC: dc1)" -46:04.358 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.380 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.380 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.380 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.380 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.380 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.380 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.380 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.385 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.390 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.431 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.431 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-d6a00251-155b-efd4-20ce-7d5457f1fe5f -46:04.438 [DEBUG] TestAgent.dns: request served from client: name=foo.service.consul. type=A class=IN latency=151.004µs client=127.0.0.1:50610 client_network=udp -46:04.438 [INFO] TestAgent: Requesting shutdown -46:04.438 [INFO] TestAgent.server: shutting down server -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.438 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:04.438 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.438 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.438 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.438 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:04.442 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.445Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.445Z [INFO] agent.router.manager: shutting down -46:04.445 [INFO] TestAgent: consul server down -46:04.445 [INFO] TestAgent: shutdown complete -46:04.445 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11765 network=tcp -46:04.445 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11765 network=udp -46:04.445 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11766 network=tcp -46:04.946 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.946 [INFO] TestAgent: Endpoints down -=== RUN TestDNS_ServiceLookup_FilterACL/ACLToken_==_anonymous === RUN TestBinarySearch/binarySearch_65535#01 -2022-11-07T22:46:04.193Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.204Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.247 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:10a0da3b-99db-5f1c-a2eb-f195d98105a8 Address:127.0.0.1:11763}]" -46:04.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11763 [Follower]" leader= -46:04.248 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8.dc1 127.0.0.1 -46:04.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8 127.0.0.1 -2022-11-07T22:46:04.249Z [INFO] agent.router: Initializing LAN area manager -46:04.249 [INFO] TestAgent.server: Adding LAN server: server="Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8 (Addr: tcp/127.0.0.1:11763) (DC: dc1)" -46:04.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11758 network=udp -46:04.249 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8.dc1 area=wan -46:04.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11758 network=tcp -46:04.250 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11759 network=tcp -46:04.250 [INFO] TestAgent: started state syncer -46:04.250 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11771 network=tcp -46:04.284 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.284 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11763 [Candidate]" term=2 -46:04.295 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.295 [DEBUG] TestAgent.server.raft: vote granted: from=10a0da3b-99db-5f1c-a2eb-f195d98105a8 term=2 tally=1 -46:04.295 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11763 [Leader]" -46:04.295 [INFO] TestAgent.server: cluster leadership acquired -46:04.295 [INFO] TestAgent.server: New leader elected: payload=Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8 -46:04.306 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11763 -46:04.328 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.361 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.361 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.361 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.361 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.361 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.361 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8 -46:04.361 [INFO] TestAgent.server: member joined, marking health alive: member=Node-10a0da3b-99db-5f1c-a2eb-f195d98105a8 -46:04.368 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.442 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.service.dc1.consul. type=SRV class=IN latency=127.977µs client=127.0.0.1:33318 client_network=udp -46:04.443 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.service.consul. type=SRV class=IN latency=126.137µs client=127.0.0.1:33024 client_network=udp -46:04.443 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.dc1.consul. type=SRV class=IN latency=91.513µs client=127.0.0.1:53050 client_network=udp -46:04.443 [DEBUG] TestAgent.dns: request served from client: name=_db._primary.consul. type=SRV class=IN latency=90.426µs client=127.0.0.1:38069 client_network=udp -46:04.443 [INFO] TestAgent: Requesting shutdown -46:04.443 [INFO] TestAgent.server: shutting down server -46:04.443 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.443 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.443 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.443 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.443 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.443 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.443 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.443 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.443 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.443 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.443 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.447 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.454Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.454Z [INFO] agent.router.manager: shutting down -46:04.454 [INFO] TestAgent: consul server down -46:04.454 [INFO] TestAgent: shutdown complete -46:04.454 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11758 network=tcp -46:04.454 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11758 network=udp -46:04.454 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11759 network=tcp -46:04.954 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.954 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_SRV_RFC (0.77s) -=== CONT TestAgent_AddCheck_Alias_userToken ---- PASS: TestBinarySearch (0.09s) +--- PASS: TestBinarySearch (0.13s) --- PASS: TestBinarySearch/binarySearch_12 (0.01s) --- PASS: TestBinarySearch/binarySearch_256 (0.01s) --- PASS: TestBinarySearch/binarySearch_512 (0.01s) --- PASS: TestBinarySearch/binarySearch_8192 (0.01s) - --- PASS: TestBinarySearch/binarySearch_65535 (0.01s) + --- PASS: TestBinarySearch/binarySearch_65535 (0.02s) --- PASS: TestBinarySearch/binarySearch_12#01 (0.01s) --- PASS: TestBinarySearch/binarySearch_256#01 (0.01s) --- PASS: TestBinarySearch/binarySearch_512#01 (0.01s) --- PASS: TestBinarySearch/binarySearch_8192#01 (0.01s) - --- PASS: TestBinarySearch/binarySearch_65535#01 (0.01s) -=== CONT TestIntentionsCheck_noDestination -2022-11-07T22:46:04.155Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.164Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 Address:127.0.0.1:11749}]" -46:04.212 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11749 [Follower]" leader= -46:04.212 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5.dc1 127.0.0.1 -46:04.213 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 127.0.0.1 -2022-11-07T22:46:04.213Z [INFO] agent.router: Initializing LAN area manager -46:04.213 [INFO] TestAgent.server: Adding LAN server: server="Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 (Addr: tcp/127.0.0.1:11749) (DC: dc1)" -46:04.213 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5.dc1 area=wan -46:04.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11744 network=udp -46:04.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11744 network=tcp -46:04.213 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11745 network=tcp -46:04.213 [INFO] TestAgent: started state syncer -46:04.213 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11764 network=tcp -46:04.252 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.252 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11749 [Candidate]" term=2 -46:04.264 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.264 [DEBUG] TestAgent.server.raft: vote granted: from=b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 term=2 tally=1 -46:04.264 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.264 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11749 [Leader]" -46:04.265 [INFO] TestAgent.server: cluster leadership acquired -46:04.265 [INFO] TestAgent.server: New leader elected: payload=Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 -46:04.273 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11749 -46:04.292 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.328 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.328 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.328 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.328 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.328 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.328 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 -46:04.328 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b1c7b4f0-4c1b-6479-da9e-cb4eee50dbc5 -46:04.341 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.468 [DEBUG] TestAgent.dns: request served from client: name=7f000001.addr.dc1.consul. type=SRV class=IN latency=29.814µs client=127.0.0.1:37458 client_network=udp -46:04.468 [INFO] TestAgent: Requesting shutdown -46:04.468 [INFO] TestAgent.server: shutting down server -46:04.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.468 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.468 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.468 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.468 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.468 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.468 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.468 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.473 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.479Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.479Z [INFO] agent.router.manager: shutting down -46:04.479 [INFO] TestAgent: consul server down -46:04.479 [INFO] TestAgent: shutdown complete -46:04.479 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11744 network=tcp -46:04.479 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11744 network=udp -46:04.479 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11745 network=tcp -46:04.979 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.979 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_AddressLookup (0.83s) -=== CONT TestIntentionsCreate_noBody -2022-11-07T22:46:04.032Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.040Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.082 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ce649fbd-34c1-5303-3796-d0ed3736ccbb Address:127.0.0.1:11707}]" -46:04.083 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11707 [Follower]" leader= -46:04.084 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb.dc1 127.0.0.1 -46:04.085 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb 127.0.0.1 -2022-11-07T22:46:04.085Z [INFO] agent.router: Initializing LAN area manager -46:04.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11702 network=udp -46:04.087 [INFO] TestAgent.server: Adding LAN server: server="Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb (Addr: tcp/127.0.0.1:11707) (DC: dc1)" -46:04.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11702 network=tcp -46:04.088 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb.dc1 area=wan -46:04.089 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11703 network=tcp -46:04.089 [INFO] TestAgent: started state syncer -46:04.090 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11708 network=tcp -46:04.123 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.123 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11707 [Candidate]" term=2 -46:04.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.137 [DEBUG] TestAgent.server.raft: vote granted: from=ce649fbd-34c1-5303-3796-d0ed3736ccbb term=2 tally=1 -46:04.137 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11707 [Leader]" -46:04.137 [INFO] TestAgent.server: cluster leadership acquired -46:04.138 [INFO] TestAgent.server: New leader elected: payload=Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb -46:04.145 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11707 -46:04.159 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.193 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.193 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.193 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.193 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.193 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.193 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb -46:04.193 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ce649fbd-34c1-5303-3796-d0ed3736ccbb -46:04.224 [INFO] TestAgent: Synced node info -46:04.224 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.470 [DEBUG] TestAgent.dns: request served from client: name=webv4.service.consul. type=AAAA class=IN latency=128.861µs client=127.0.0.1:41877 client_network=udp -46:04.470 [DEBUG] TestAgent.dns: request served from client: name=webv4.query.consul. type=AAAA class=IN latency=118.08µs client=127.0.0.1:51761 client_network=udp -46:04.471 [DEBUG] TestAgent.dns: request served from client: name=webv6.service.consul. type=A class=IN latency=93.348µs client=127.0.0.1:33470 client_network=udp -46:04.471 [DEBUG] TestAgent.dns: request served from client: name=webv6.query.consul. type=A class=IN latency=113.307µs client=127.0.0.1:47121 client_network=udp -46:04.471 [INFO] TestAgent: Requesting shutdown -46:04.471 [INFO] TestAgent.server: shutting down server -46:04.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.471 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.471 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.471 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.471 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.471 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.471 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.476 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.481Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.481Z [INFO] agent.router.manager: shutting down -46:04.482 [INFO] TestAgent: consul server down -46:04.482 [INFO] TestAgent: shutdown complete -46:04.482 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11702 network=tcp -46:04.482 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11702 network=udp -46:04.482 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11703 network=tcp -46:04.982 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.982 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NonExistingLookupEmptyAorAAAA (0.96s) -=== CONT TestIntentionsMatch_byInvalid -2022-11-07T22:46:04.079Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.099Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.137 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:572b3c9f-941a-e985-36cc-dd3fba25db7e Address:127.0.0.1:11735}]" -46:04.137 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11735 [Follower]" leader= -46:04.137 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-572b3c9f-941a-e985-36cc-dd3fba25db7e.dc1 127.0.0.1 -46:04.137 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-572b3c9f-941a-e985-36cc-dd3fba25db7e 127.0.0.1 -2022-11-07T22:46:04.137Z [INFO] agent.router: Initializing LAN area manager -46:04.137 [INFO] TestAgent.server: Adding LAN server: server="Node-572b3c9f-941a-e985-36cc-dd3fba25db7e (Addr: tcp/127.0.0.1:11735) (DC: dc1)" -46:04.137 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-572b3c9f-941a-e985-36cc-dd3fba25db7e.dc1 area=wan -46:04.138 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11730 network=udp -46:04.138 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11730 network=tcp -46:04.138 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11731 network=tcp -46:04.138 [INFO] TestAgent: started state syncer -46:04.138 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11750 network=tcp -46:04.179 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.179 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11735 [Candidate]" term=2 -46:04.196 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.196 [DEBUG] TestAgent.server.raft: vote granted: from=572b3c9f-941a-e985-36cc-dd3fba25db7e term=2 tally=1 -46:04.196 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.196 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11735 [Leader]" -46:04.196 [INFO] TestAgent.server: cluster leadership acquired -46:04.196 [INFO] TestAgent.server: New leader elected: payload=Node-572b3c9f-941a-e985-36cc-dd3fba25db7e -46:04.211 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11735 -46:04.232 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.270 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.270 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.270 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.270 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.270 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.270 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-572b3c9f-941a-e985-36cc-dd3fba25db7e -46:04.270 [INFO] TestAgent.server: member joined, marking health alive: member=Node-572b3c9f-941a-e985-36cc-dd3fba25db7e -46:04.284 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.371 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.374 [INFO] TestAgent: Synced node info -46:04.374 [DEBUG] TestAgent: Node info in sync -46:04.492 [DEBUG] TestAgent.dns: request served from client: name=2607002040050808000000000000200e.addr.consul. type=SRV class=IN latency=37.935µs client=127.0.0.1:33648 client_network=udp -46:04.492 [DEBUG] TestAgent.dns: request served from client: name=2607112040051808ffffffffffff200e.addr.consul. type=SRV class=IN latency=52.847µs client=127.0.0.1:49309 client_network=udp -46:04.492 [INFO] TestAgent: Requesting shutdown -46:04.492 [INFO] TestAgent.server: shutting down server -46:04.492 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.492 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.492 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.492 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.492 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.492 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.492 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.492 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.492 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.495 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.501Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.501Z [INFO] agent.router.manager: shutting down -46:04.501 [INFO] TestAgent: consul server down -46:04.501 [INFO] TestAgent: shutdown complete -46:04.501 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11730 network=tcp -46:04.501 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11730 network=udp -46:04.501 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11731 network=tcp -46:05.001 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.001 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_AddressLookupIPV6 (0.93s) -=== CONT TestIntentionsCreate_good -2022-11-07T22:46:04.303Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.313Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.342 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f21570ab-e586-650e-70b5-909b77b1eed4 Address:127.0.0.1:11798}]" -46:04.342 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11798 [Follower]" leader= -46:04.342 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f21570ab-e586-650e-70b5-909b77b1eed4.dc1 127.0.0.1 -46:04.343 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f21570ab-e586-650e-70b5-909b77b1eed4 127.0.0.1 -2022-11-07T22:46:04.343Z [INFO] agent.router: Initializing LAN area manager -46:04.343 [INFO] TestAgent.server: Adding LAN server: server="Node-f21570ab-e586-650e-70b5-909b77b1eed4 (Addr: tcp/127.0.0.1:11798) (DC: dc1)" -46:04.343 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11793 network=udp -46:04.343 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f21570ab-e586-650e-70b5-909b77b1eed4.dc1 area=wan -46:04.343 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11793 network=tcp -46:04.343 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11794 network=tcp -46:04.343 [INFO] TestAgent: started state syncer -46:04.343 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11785 network=tcp -46:04.404 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.404 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11798 [Candidate]" term=2 -46:04.414 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.414 [DEBUG] TestAgent.server.raft: vote granted: from=f21570ab-e586-650e-70b5-909b77b1eed4 term=2 tally=1 -46:04.414 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.414 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11798 [Leader]" -46:04.414 [INFO] TestAgent.server: cluster leadership acquired -46:04.414 [INFO] TestAgent.server: New leader elected: payload=Node-f21570ab-e586-650e-70b5-909b77b1eed4 -46:04.418 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11798 -46:04.430 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.447 [INFO] TestAgent: Synced node info -46:04.452 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.452 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.452 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.452 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.452 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.452 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f21570ab-e586-650e-70b5-909b77b1eed4 -46:04.452 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f21570ab-e586-650e-70b5-909b77b1eed4 -46:04.459 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.557 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.service.dc1.consul. type=SRV class=IN latency=124.515µs client=127.0.0.1:33539 client_network=udp -46:04.558 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.service.consul. type=SRV class=IN latency=101.733µs client=127.0.0.1:50957 client_network=udp -46:04.558 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.dc1.consul. type=SRV class=IN latency=99.709µs client=127.0.0.1:48039 client_network=udp -46:04.558 [DEBUG] TestAgent.dns: request served from client: name=_db._tcp.consul. type=SRV class=IN latency=97.94µs client=127.0.0.1:41712 client_network=udp -46:04.558 [INFO] TestAgent: Requesting shutdown -46:04.558 [INFO] TestAgent.server: shutting down server -46:04.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.558 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.558 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.558 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.558 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.558 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.558 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.562 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.565Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.565Z [INFO] agent.router.manager: shutting down -46:04.565 [INFO] TestAgent: consul server down -46:04.565 [INFO] TestAgent: shutdown complete -46:04.565 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11793 network=tcp -46:04.565 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11793 network=udp -46:04.565 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11794 network=tcp -46:05.065 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.065 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_SRV_RFC_TCP_Default (0.77s) -=== CONT TestIntentionsSpecificGet_invalidId -[WARN] freeport: 10 out of 66 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:04.487Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.497Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.521 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:062a990e-9f8b-dfd2-4dd6-29c3f70c83bb Address:127.0.0.1:11812}]" -46:04.521 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11812 [Follower]" leader= -46:04.522 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb.dc1 127.0.0.1 -46:04.522 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb 127.0.0.1 -2022-11-07T22:46:04.522Z [INFO] agent.router: Initializing LAN area manager -46:04.522 [INFO] TestAgent.server: Adding LAN server: server="Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb (Addr: tcp/127.0.0.1:11812) (DC: dc1)" -46:04.522 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb.dc1 area=wan -46:04.522 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11807 network=udp -46:04.523 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11807 network=tcp -46:04.523 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11808 network=tcp -46:04.523 [INFO] TestAgent: started state syncer -46:04.523 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11799 network=tcp -46:04.583 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.583 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11812 [Candidate]" term=2 -46:04.591 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.591 [DEBUG] TestAgent.server.raft: vote granted: from=062a990e-9f8b-dfd2-4dd6-29c3f70c83bb term=2 tally=1 -46:04.592 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.592 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11812 [Leader]" -46:04.592 [INFO] TestAgent.server: cluster leadership acquired -46:04.592 [INFO] TestAgent.server: New leader elected: payload=Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb -46:04.597 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11812 -46:04.619 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.648 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.648 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.648 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.648 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.648 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.648 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb -46:04.648 [INFO] TestAgent.server: member joined, marking health alive: member=Node-062a990e-9f8b-dfd2-4dd6-29c3f70c83bb -46:04.657 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.725 [DEBUG] TestAgent.dns: request served from client: name=db.query.consul. type=SRV class=IN latency=135.402µs client=127.0.0.1:37365 client_network=udp -46:04.725 [DEBUG] TestAgent.dns: request served from client: name=db-ttl.query.consul. type=SRV class=IN latency=142.222µs client=127.0.0.1:54569 client_network=udp -46:04.726 [DEBUG] TestAgent.dns: request served from client: name=dblb.query.consul. type=SRV class=IN latency=85.006µs client=127.0.0.1:40234 client_network=udp -46:04.726 [DEBUG] TestAgent.dns: request served from client: name=dblb-ttl.query.consul. type=SRV class=IN latency=116.807µs client=127.0.0.1:46211 client_network=udp -46:04.726 [DEBUG] TestAgent.dns: request served from client: name=dk.query.consul. type=SRV class=IN latency=149.877µs client=127.0.0.1:38388 client_network=udp -46:04.727 [DEBUG] TestAgent.dns: request served from client: name=dk-ttl.query.consul. type=SRV class=IN latency=156.766µs client=127.0.0.1:60455 client_network=udp -46:04.727 [DEBUG] TestAgent.dns: request served from client: name=api.query.consul. type=SRV class=IN latency=105.818µs client=127.0.0.1:51843 client_network=udp -46:04.727 [DEBUG] TestAgent.dns: request served from client: name=api-ttl.query.consul. type=SRV class=IN latency=130.181µs client=127.0.0.1:38223 client_network=udp -46:04.727 [INFO] TestAgent: Requesting shutdown -46:04.727 [INFO] TestAgent.server: shutting down server -46:04.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.727 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.727 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.727 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.727 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.727 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.727 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.727 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.732 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.736Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.736Z [INFO] agent.router.manager: shutting down -46:04.736 [INFO] TestAgent: consul server down -46:04.736 [INFO] TestAgent: shutdown complete -46:04.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11807 network=tcp -46:04.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11807 network=udp -46:04.736 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11808 network=tcp -46:05.236 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.236 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQuery_TTL (0.76s) - --- PASS: TestDNS_PreparedQuery_TTL/db.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/db-ttl.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/dblb.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/dblb-ttl.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/dk.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/dk-ttl.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/api.query.consul. (0.00s) - --- PASS: TestDNS_PreparedQuery_TTL/api-ttl.query.consul. (0.00s) -=== CONT TestIntentionsCheck_basic + --- PASS: TestBinarySearch/binarySearch_65535#01 (0.02s) +=== CONT TestDNS_ServiceLookup_Randomize === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=true === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=true#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=false#01 -46:04.274 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -2022-11-07T22:46:04.274Z [INFO] agent.router: Initializing LAN area manager -46:04.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11772 network=udp -46:04.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11772 network=tcp -46:04.275 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11773 network=tcp -46:04.275 [INFO] TestAgent: started state syncer -46:04.275 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11712] -46:04.276 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11778 network=tcp -2022-11-07T22:46:04.276Z [WARN] agent.router.manager: No servers available -46:04.276 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:04.276 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11712 -46:04.276 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: test-server 127.0.0.1 -46:04.276 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:04.276 [DEBUG] TestAgent: systemd notify failed: error="No socket" -2022-11-07T22:46:04.276Z [WARN] agent.router.manager: No servers available -46:04.276 [INFO] TestAgent.client: adding server: server="test-server (Addr: tcp/127.0.0.1:11714) (DC: dc1)" -46:04.302 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:04.303 [ERROR] TestAgent.client: RPC failed to server: method=Health.ServiceNodes server=127.0.0.1:11714 error="rpc error making call: No path to datacenter" -46:04.303 [ERROR] TestAgent.dns: rpc error: error="rpc error making call: No path to datacenter" -46:04.303 [DEBUG] TestAgent.dns: request served from client: name=consul.service.dc2.consul. type=ANY class=IN latency=275.741µs client=127.0.0.1:40781 client_network=udp -46:04.303 [INFO] TestAgent: Requesting shutdown -46:04.303 [INFO] TestAgent.client: shutting down client -46:04.303 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:04.314 [INFO] TestAgent: consul client down -46:04.314 [INFO] TestAgent: shutdown complete -46:04.314 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11772 network=tcp -46:04.314 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11772 network=udp -46:04.314 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11773 network=tcp -46:04.814 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.814 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:04.046Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.056Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.100 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:322a7a7f-d577-110f-0b94-6bb48fa52198 Address:127.0.0.1:11714}]" -46:04.100 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11714 [Follower]" leader= -46:04.101 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-server.dc1 127.0.0.1 -46:04.101 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-server 127.0.0.1 -2022-11-07T22:46:04.101Z [INFO] agent.router: Initializing LAN area manager -46:04.101 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-server.dc1 area=wan -46:04.101 [INFO] TestAgent.server: Adding LAN server: server="test-server (Addr: tcp/127.0.0.1:11714) (DC: dc1)" -46:04.101 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11709 network=udp -46:04.101 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11709 network=tcp -46:04.102 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11710 network=tcp -46:04.102 [INFO] TestAgent: started state syncer -46:04.102 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11722 network=tcp -46:04.152 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.152 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11714 [Candidate]" term=2 -46:04.162 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.162 [DEBUG] TestAgent.server.raft: vote granted: from=322a7a7f-d577-110f-0b94-6bb48fa52198 term=2 tally=1 -46:04.162 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.162 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11714 [Leader]" -46:04.162 [INFO] TestAgent.server: cluster leadership acquired -46:04.163 [INFO] TestAgent.server: New leader elected: payload=test-server -46:04.169 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11714 -46:04.191 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.238 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.238 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.238 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.238 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.238 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.238 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-server -46:04.238 [INFO] TestAgent.server: member joined, marking health alive: member=test-server -46:04.257 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.276 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:45250 -46:04.276 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-client 127.0.0.1 -46:04.276 [INFO] TestAgent.server: member joined, marking health alive: member=test-client -46:04.303 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc2 method=Health.ServiceNodes -46:04.433 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.435 [INFO] TestAgent: Synced node info -46:04.814 [INFO] TestAgent: Requesting shutdown -46:04.814 [INFO] TestAgent.server: shutting down server -46:04.814 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.814 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.814 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.814 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.814 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.814 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.814 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.814 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.814 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.814 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.819 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.827Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.827Z [INFO] agent.router.manager: shutting down -46:04.827 [INFO] TestAgent: consul server down -46:04.827 [INFO] TestAgent: shutdown complete -46:04.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11709 network=tcp -46:04.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11709 network=udp -46:04.827 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11710 network=tcp -46:05.327 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.327 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NonExistentDC_RPC (1.29s) -=== CONT TestIntentionsSpecificGet_good -2022-11-07T22:46:04.681Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.690Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.719 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:18e50704-053e-4e70-93b6-1feb0e71069f Address:127.0.0.1:11830}]" -46:04.719 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11830 [Follower]" leader= -46:04.720 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-18e50704-053e-4e70-93b6-1feb0e71069f.dc1 127.0.0.1 -46:04.720 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-18e50704-053e-4e70-93b6-1feb0e71069f 127.0.0.1 -2022-11-07T22:46:04.720Z [INFO] agent.router: Initializing LAN area manager -46:04.720 [INFO] TestAgent.server: Adding LAN server: server="Node-18e50704-053e-4e70-93b6-1feb0e71069f (Addr: tcp/127.0.0.1:11830) (DC: dc1)" -46:04.720 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11838 network=udp -46:04.720 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-18e50704-053e-4e70-93b6-1feb0e71069f.dc1 area=wan -46:04.720 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11838 network=tcp -46:04.721 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11839 network=tcp -46:04.721 [INFO] TestAgent: started state syncer -46:04.721 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11831 network=tcp -46:04.766 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.766 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11830 [Candidate]" term=2 -46:04.778 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.778 [DEBUG] TestAgent.server.raft: vote granted: from=18e50704-053e-4e70-93b6-1feb0e71069f term=2 tally=1 -46:04.778 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.778 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11830 [Leader]" -46:04.778 [INFO] TestAgent.server: cluster leadership acquired -46:04.779 [INFO] TestAgent.server: New leader elected: payload=Node-18e50704-053e-4e70-93b6-1feb0e71069f -46:04.785 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11830 -46:04.803 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.816 [INFO] TestAgent: Requesting shutdown -46:04.816 [INFO] TestAgent.server: shutting down server -46:04.817 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.817 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.822 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:04.829 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.829 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.829 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.829 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.829 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.829 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-18e50704-053e-4e70-93b6-1feb0e71069f -46:04.829 [INFO] TestAgent.server: member joined, marking health alive: member=Node-18e50704-053e-4e70-93b6-1feb0e71069f -2022-11-07T22:46:04.830Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.830Z [INFO] agent.router.manager: shutting down -46:04.830 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-18e50704-053e-4e70-93b6-1feb0e71069f 127.0.0.1 11828 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:18e50704-053e-4e70-93b6-1feb0e71069f port:11830 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11829] alive 1 5 2 2 5 4}" error="raft is already shutdown" -46:04.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.830 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.830 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.830 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.830 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.834 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:04.834 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.834 [INFO] TestAgent: consul server down -46:04.834 [INFO] TestAgent: shutdown complete -46:04.834 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11838 network=tcp -46:04.834 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11838 network=udp -46:04.834 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11839 network=tcp -46:05.334 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.334 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_LoadKeyrings/client_with_keys -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:04.646Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.655Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.687 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bddf5fb6-ec2e-d285-3353-5cd4f5d54141 Address:127.0.0.1:11826}]" -46:04.687 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11826 [Follower]" leader= -46:04.688 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141.dc1 127.0.0.1 -46:04.688 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141 127.0.0.1 -2022-11-07T22:46:04.688Z [INFO] agent.router: Initializing LAN area manager -46:04.688 [INFO] TestAgent.server: Adding LAN server: server="Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141 (Addr: tcp/127.0.0.1:11826) (DC: dc1)" -46:04.688 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141.dc1 area=wan -46:04.688 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11821 network=udp -46:04.689 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11821 network=tcp -46:04.689 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11822 network=tcp -46:04.689 [INFO] TestAgent: started state syncer -46:04.689 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11813 network=tcp -46:04.741 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.741 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11826 [Candidate]" term=2 -46:04.750 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.750 [DEBUG] TestAgent.server.raft: vote granted: from=bddf5fb6-ec2e-d285-3353-5cd4f5d54141 term=2 tally=1 -46:04.750 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.750 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11826 [Leader]" -46:04.750 [INFO] TestAgent.server: cluster leadership acquired -46:04.751 [INFO] TestAgent.server: New leader elected: payload=Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141 -46:04.758 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11826 -46:04.771 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.800 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.800 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.800 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.800 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.800 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.800 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141 -46:04.800 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bddf5fb6-ec2e-d285-3353-5cd4f5d54141 -46:04.817 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.835 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=112.827µs client=127.0.0.1:42418 client_network=udp -46:04.835 [DEBUG] TestAgent.dns: request served from client: name=dblb.service.consul. type=SRV class=IN latency=125.77µs client=127.0.0.1:47924 client_network=udp -46:04.836 [DEBUG] TestAgent.dns: request served from client: name=dk.service.consul. type=SRV class=IN latency=112.369µs client=127.0.0.1:36934 client_network=udp -46:04.836 [DEBUG] TestAgent.dns: request served from client: name=api.service.consul. type=SRV class=IN latency=105.277µs client=127.0.0.1:48317 client_network=udp -46:04.836 [INFO] TestAgent: Requesting shutdown -46:04.836 [INFO] TestAgent.server: shutting down server -46:04.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.836 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.836 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.836 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.836 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.836 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.836 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.836 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.839 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.842Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.842Z [INFO] agent.router.manager: shutting down -46:04.842 [INFO] TestAgent: consul server down -46:04.842 [INFO] TestAgent: shutdown complete -46:04.842 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11821 network=tcp -46:04.842 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11821 network=udp -46:04.842 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11822 network=tcp -46:05.342 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.342 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_TTL (0.71s) - --- PASS: TestDNS_ServiceLookup_TTL/db.service.consul. (0.00s) +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=false#01 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:19.601Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.635Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.720 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:48ba11c9-96e2-5ef4-dd78-25238dcc9c8b Address:127.0.0.1:30029}]" +31:19.721 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b.dc1 127.0.0.1 +31:19.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b 127.0.0.1 +2021-10-05T16:31:19.721Z [INFO] agent.router: Initializing LAN area manager +31:19.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30024 network=udp +31:19.721 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30029 [Follower]" leader= +31:19.722 [INFO] TestAgent.server: Adding LAN server: server="Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b (Addr: tcp/127.0.0.1:30029) (DC: dc1)" +31:19.722 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b.dc1 area=wan +31:19.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30024 network=tcp +31:19.723 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30025 network=tcp +31:19.723 [INFO] TestAgent: started state syncer +31:19.723 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30051 network=tcp +31:19.788 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.788 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30029 [Candidate]" term=2 +31:19.825 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.825 [DEBUG] TestAgent.server.raft: vote granted: from=48ba11c9-96e2-5ef4-dd78-25238dcc9c8b term=2 tally=1 +31:19.825 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.825 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30029 [Leader]" +31:19.830 [INFO] TestAgent.server: cluster leadership acquired +31:19.830 [INFO] TestAgent.server: New leader elected: payload=Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b +31:19.862 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30029 +31:19.901 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.000 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.000 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.000 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.000 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.000 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.000 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b +31:20.000 [INFO] TestAgent.server: member joined, marking health alive: member=Node-48ba11c9-96e2-5ef4-dd78-25238dcc9c8b +31:20.001 [INFO] TestAgent: Synced node info +31:20.040 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.098 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=1.008966ms client=127.0.0.1:42137 client_network=udp +31:20.098 [DEBUG] TestAgent.dns: request served from client: name=dblb.service.consul. type=SRV class=IN latency=292.002µs client=127.0.0.1:53503 client_network=udp +31:20.099 [DEBUG] TestAgent.dns: request served from client: name=dk.service.consul. type=SRV class=IN latency=118.506µs client=127.0.0.1:43107 client_network=udp +31:20.099 [DEBUG] TestAgent.dns: request served from client: name=api.service.consul. type=SRV class=IN latency=184.829µs client=127.0.0.1:50487 client_network=udp +31:20.100 [INFO] TestAgent: Requesting shutdown +31:20.100 [INFO] TestAgent.server: shutting down server +31:20.100 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.100 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.100 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.100 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.100 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.100 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.100 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.100 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.100 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.100 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.107 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.107Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.107Z [INFO] agent.router.manager: shutting down +31:20.107 [INFO] TestAgent: consul server down +31:20.107 [INFO] TestAgent: shutdown complete +31:20.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30024 network=tcp +31:20.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30024 network=udp +31:20.107 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30025 network=tcp +31:20.608 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.608 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_TTL (1.04s) + --- PASS: TestDNS_ServiceLookup_TTL/db.service.consul. (0.01s) --- PASS: TestDNS_ServiceLookup_TTL/dblb.service.consul. (0.00s) --- PASS: TestDNS_ServiceLookup_TTL/dk.service.consul. (0.00s) --- PASS: TestDNS_ServiceLookup_TTL/api.service.consul. (0.00s) -=== CONT TestSessionCreate_NodeChecks -2022-11-07T22:46:04.804Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.813Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.842 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:23f792ba-9dc3-7536-e4fc-a9209af7e186 Address:127.0.0.1:11851}]" -46:04.842 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11851 [Follower]" leader= -46:04.842 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-23f792ba-9dc3-7536-e4fc-a9209af7e186.dc1 127.0.0.1 -46:04.843 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-23f792ba-9dc3-7536-e4fc-a9209af7e186 127.0.0.1 -2022-11-07T22:46:04.843Z [INFO] agent.router: Initializing LAN area manager -46:04.843 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-23f792ba-9dc3-7536-e4fc-a9209af7e186.dc1 area=wan -46:04.843 [INFO] TestAgent.server: Adding LAN server: server="Node-23f792ba-9dc3-7536-e4fc-a9209af7e186 (Addr: tcp/127.0.0.1:11851) (DC: dc1)" -46:04.843 [DEBUG] TestAgent.dns: recursor enabled -46:04.843 [DEBUG] TestAgent.dns: recursor enabled -46:04.843 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11832 network=tcp -46:04.843 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11832 network=udp -46:04.843 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11833 network=tcp -46:04.843 [INFO] TestAgent: started state syncer -46:04.843 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11852 network=tcp -46:04.897 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.897 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11851 [Candidate]" term=2 -46:04.904 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.904 [DEBUG] TestAgent.server.raft: vote granted: from=23f792ba-9dc3-7536-e4fc-a9209af7e186 term=2 tally=1 -46:04.904 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.904 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11851 [Leader]" -46:04.904 [INFO] TestAgent.server: cluster leadership acquired -46:04.904 [INFO] TestAgent.server: New leader elected: payload=Node-23f792ba-9dc3-7536-e4fc-a9209af7e186 -46:04.912 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11851 -46:04.924 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.948 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.948 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.948 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.948 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.948 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.948 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-23f792ba-9dc3-7536-e4fc-a9209af7e186 -46:04.948 [INFO] TestAgent.server: member joined, marking health alive: member=Node-23f792ba-9dc3-7536-e4fc-a9209af7e186 -46:04.958 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.972 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=143.98µs -46:04.972 [DEBUG] TestAgent.dns: request served from client: name=search.service.consul. type=ANY class=IN latency=410.159µs client=127.0.0.1:36763 client_network=udp -46:04.972 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=68.077µs -46:04.973 [DEBUG] TestAgent.dns: request served from client: name=a2f63b8d-f1e6-5a31-e9e5-8c7ad88dfab5.query.consul. type=ANY class=IN latency=254.132µs client=127.0.0.1:47969 client_network=udp -46:04.973 [INFO] TestAgent: Requesting shutdown -46:04.973 [INFO] TestAgent.server: shutting down server -46:04.973 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.973 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.973 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.973 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.973 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.973 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.973 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.973 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.973 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.976 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.987Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.987Z [INFO] agent.router.manager: shutting down -46:04.988 [INFO] TestAgent: consul server down -46:04.988 [INFO] TestAgent: shutdown complete -46:04.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11832 network=tcp -46:04.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11832 network=udp -46:04.988 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11833 network=tcp -46:05.488 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.488 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_ServiceAddress_CNAME (0.70s) -=== CONT TestIntentionsMatch_noName -2022-11-07T22:46:04.660Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.669Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.709 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:703d1015-4856-d47b-cb94-6ce36faa75f6 Address:127.0.0.1:11819}]" -46:04.709 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11819 [Follower]" leader= -46:04.710 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-703d1015-4856-d47b-cb94-6ce36faa75f6.dc1 127.0.0.1 -46:04.710 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-703d1015-4856-d47b-cb94-6ce36faa75f6 127.0.0.1 -2022-11-07T22:46:04.710Z [INFO] agent.router: Initializing LAN area manager -46:04.710 [DEBUG] TestAgent.dns: recursor enabled -46:04.710 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11814 network=udp -46:04.710 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-703d1015-4856-d47b-cb94-6ce36faa75f6.dc1 area=wan -46:04.710 [INFO] TestAgent.server: Adding LAN server: server="Node-703d1015-4856-d47b-cb94-6ce36faa75f6 (Addr: tcp/127.0.0.1:11819) (DC: dc1)" -46:04.711 [DEBUG] TestAgent.dns: recursor enabled -46:04.711 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11814 network=tcp -46:04.711 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11815 network=tcp -46:04.711 [INFO] TestAgent: started state syncer -46:04.711 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11836 network=tcp -46:04.753 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.753 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11819 [Candidate]" term=2 -46:04.763 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.763 [DEBUG] TestAgent.server.raft: vote granted: from=703d1015-4856-d47b-cb94-6ce36faa75f6 term=2 tally=1 -46:04.763 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.763 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11819 [Leader]" -46:04.764 [INFO] TestAgent.server: cluster leadership acquired -46:04.764 [INFO] TestAgent.server: New leader elected: payload=Node-703d1015-4856-d47b-cb94-6ce36faa75f6 -46:04.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11819 -46:04.791 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.822 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.822 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.822 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.822 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.822 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.822 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-703d1015-4856-d47b-cb94-6ce36faa75f6 -46:04.822 [INFO] TestAgent.server: member joined, marking health alive: member=Node-703d1015-4856-d47b-cb94-6ce36faa75f6 -46:04.831 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.984 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=ANY class=IN latency=107.409µs client=127.0.0.1:46981 client_network=udp -46:04.988 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=77.676µs client=127.0.0.1:36862 client_network=udp -46:04.993 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=123.08µs -46:04.993 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=ANY class=IN latency=266.731µs client=127.0.0.1:37330 client_network=udp -46:04.993 [INFO] TestAgent: Requesting shutdown -46:04.993 [INFO] TestAgent.server: shutting down server -46:04.993 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.993 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.993 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.993 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.993 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.993 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.993 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.993 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.993 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.993 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.993 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.002 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.009Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.009Z [INFO] agent.router.manager: shutting down -46:05.010 [INFO] TestAgent: consul server down -46:05.010 [INFO] TestAgent: shutdown complete -46:05.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11814 network=tcp -46:05.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11814 network=udp -46:05.010 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11815 network=tcp -46:05.510 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.510 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_TTL (0.86s) -=== CONT TestIntentionsMatch_basic -2022-11-07T22:46:04.836Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.845Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.867 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e116d981-57eb-4130-1933-8e3d266859ab Address:127.0.0.1:11844}]" -46:04.867 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11844 [Follower]" leader= -46:04.868 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e116d981-57eb-4130-1933-8e3d266859ab.dc1 127.0.0.1 -46:04.868 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e116d981-57eb-4130-1933-8e3d266859ab 127.0.0.1 -2022-11-07T22:46:04.868Z [INFO] agent.router: Initializing LAN area manager -46:04.868 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e116d981-57eb-4130-1933-8e3d266859ab.dc1 area=wan -46:04.868 [INFO] TestAgent.server: Adding LAN server: server="Node-e116d981-57eb-4130-1933-8e3d266859ab (Addr: tcp/127.0.0.1:11844) (DC: dc1)" -46:04.868 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11853 network=udp -46:04.868 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11853 network=tcp -46:04.869 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11854 network=tcp -46:04.869 [INFO] TestAgent: started state syncer -46:04.869 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11845 network=tcp -46:04.906 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.906 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11844 [Candidate]" term=2 -46:04.914 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.914 [DEBUG] TestAgent.server.raft: vote granted: from=e116d981-57eb-4130-1933-8e3d266859ab term=2 tally=1 -46:04.914 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.914 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11844 [Leader]" -46:04.914 [INFO] TestAgent.server: cluster leadership acquired -46:04.914 [INFO] TestAgent.server: New leader elected: payload=Node-e116d981-57eb-4130-1933-8e3d266859ab -46:04.920 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11844 -46:04.931 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.953 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.953 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.953 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.953 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.953 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.953 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e116d981-57eb-4130-1933-8e3d266859ab -46:04.953 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e116d981-57eb-4130-1933-8e3d266859ab -46:04.965 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.013 [DEBUG] TestAgent.dns: request served from client: name=_this-is-a-very-very-very-very-very-long-name-for-a-service._primary.service.consul. type=SRV class=IN latency=222.943µs client=127.0.0.1:42219 client_network=udp -46:05.013 [DEBUG] TestAgent.dns: request served from client: name=this-is-a-very-very-very-very-very-long-name-for-a-service.query.consul. type=SRV class=IN latency=142.491µs client=127.0.0.1:33861 client_network=udp -46:05.013 [INFO] TestAgent: Requesting shutdown -46:05.013 [INFO] TestAgent.server: shutting down server -46:05.013 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.013 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.013 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.013 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.013 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.013 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.013 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.013 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.013 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.013 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.013 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.019 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.027Z [INFO] agent.router.manager: shutting down -46:05.027 [INFO] TestAgent: consul server down -46:05.027 [INFO] TestAgent: shutdown complete -46:05.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11853 network=tcp -46:05.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11853 network=udp -46:05.027 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11854 network=tcp -46:05.527 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.527 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_LargeResponses (0.71s) -=== CONT TestShouldProcessUserEvent -2022-11-07T22:46:04.595Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.611Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.650 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e9abc855-5e19-bd30-b480-04dedd2bc917 Address:127.0.0.1:11805}]" -46:04.650 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11805 [Follower]" leader= -46:04.650 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e9abc855-5e19-bd30-b480-04dedd2bc917.dc1 127.0.0.1 -46:04.651 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e9abc855-5e19-bd30-b480-04dedd2bc917 127.0.0.1 -2022-11-07T22:46:04.651Z [INFO] agent.router: Initializing LAN area manager -46:04.651 [INFO] TestAgent.server: Adding LAN server: server="Node-e9abc855-5e19-bd30-b480-04dedd2bc917 (Addr: tcp/127.0.0.1:11805) (DC: dc1)" -46:04.651 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e9abc855-5e19-bd30-b480-04dedd2bc917.dc1 area=wan -46:04.651 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11800 network=tcp -46:04.651 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11800 network=udp -46:04.651 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11801 network=tcp -46:04.651 [INFO] TestAgent: started state syncer -46:04.651 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11820 network=tcp -46:04.707 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.707 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11805 [Candidate]" term=2 -46:04.720 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.720 [DEBUG] TestAgent.server.raft: vote granted: from=e9abc855-5e19-bd30-b480-04dedd2bc917 term=2 tally=1 -46:04.720 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.720 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11805 [Leader]" -46:04.720 [INFO] TestAgent.server: cluster leadership acquired -46:04.720 [INFO] TestAgent.server: New leader elected: payload=Node-e9abc855-5e19-bd30-b480-04dedd2bc917 -46:04.727 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11805 -46:04.740 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.759 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.759 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.759 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.759 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.759 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.759 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e9abc855-5e19-bd30-b480-04dedd2bc917 -46:04.759 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e9abc855-5e19-bd30-b480-04dedd2bc917 -46:04.774 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.872 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.878 [INFO] TestAgent: Synced node info -46:05.058 [INFO] TestAgent: Requesting shutdown -46:05.058 [INFO] TestAgent.server: shutting down server -46:05.058 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.058 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.058 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.058 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.058 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.058 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.058 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.058 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.058 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.058 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.066 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.072Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.072Z [INFO] agent.router.manager: shutting down -46:05.072 [INFO] TestAgent: consul server down -46:05.072 [INFO] TestAgent: shutdown complete -46:05.072 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11800 network=tcp -46:05.072 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11800 network=udp -46:05.072 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11801 network=tcp -46:05.573 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.573 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_InmemKeyrings/client_with_keys -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:04.418Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.418Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.429Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.429Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.459 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf Address:127.0.0.1:11791}]" -46:04.459 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11791 [Follower]" leader= -46:04.459 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 127.0.0.1 -46:04.460 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf 127.0.0.1 -2022-11-07T22:46:04.460Z [INFO] agent.router: Initializing LAN area manager -46:04.460 [INFO] TestAgent.server: Adding LAN server: server="Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf (Addr: tcp/127.0.0.1:11791) (DC: dc2)" -46:04.460 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 area=wan -46:04.460 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11786 network=udp -46:04.460 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11786 network=tcp -46:04.460 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11787 network=tcp -46:04.460 [INFO] TestAgent: started state syncer -46:04.460 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11806 network=tcp -46:04.505 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.505 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11791 [Candidate]" term=2 -46:04.510 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:04.514 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.514 [DEBUG] TestAgent.server.raft: vote granted: from=30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf term=2 tally=1 -46:04.514 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.514 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11791 [Leader]" -46:04.514 [INFO] TestAgent.server: cluster leadership acquired -46:04.514 [INFO] TestAgent.server: New leader elected: payload=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf -46:04.521 [INFO] TestAgent.server: initializing acls -46:04.523 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:04.529 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:04.529 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:04.529 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:04.529 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf -46:04.529 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 -46:04.529 [INFO] TestAgent.server: Updating LAN server: server="Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf (Addr: tcp/127.0.0.1:11791) (DC: dc2)" -46:04.529 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 area=wan -46:04.542 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.555 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.555 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.555 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.555 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.555 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.555 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf -46:04.555 [INFO] TestAgent.server: member joined, marking health alive: member=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf -46:04.562 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf -46:04.566 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.662 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11783] -46:04.662 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11783 -46:04.662 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 127.0.0.1 -46:04.663 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:04.663 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 area=wan -46:04.674 [INFO] TestAgent: Requesting shutdown -46:04.674 [INFO] TestAgent.server: shutting down server -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.674 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.674 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.674 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.674 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:04.674 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.682 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.691Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.691Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.691Z [INFO] agent.router.manager: shutting down -46:04.691 [INFO] TestAgent: consul server down -46:04.691 [INFO] TestAgent: shutdown complete -46:04.691 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11786 network=tcp -46:04.691 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11786 network=udp -46:04.691 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11787 network=tcp -46:05.192 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.192 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:04.268Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.268Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.277Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.277Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.316 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:77a725a7-57c4-33eb-9867-3902951ccc9a Address:127.0.0.1:11784}]" -46:04.316 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11784 [Follower]" leader= -46:04.316 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 127.0.0.1 -46:04.316 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-77a725a7-57c4-33eb-9867-3902951ccc9a 127.0.0.1 -2022-11-07T22:46:04.316Z [INFO] agent.router: Initializing LAN area manager -46:04.316 [INFO] TestAgent.server: Adding LAN server: server="Node-77a725a7-57c4-33eb-9867-3902951ccc9a (Addr: tcp/127.0.0.1:11784) (DC: dc1)" -46:04.316 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 area=wan -46:04.316 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11779 network=udp -46:04.316 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11779 network=tcp -46:04.317 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11780 network=tcp -46:04.317 [INFO] TestAgent: started state syncer -46:04.317 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11792 network=tcp -46:04.353 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.353 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11784 [Candidate]" term=2 -46:04.363 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.363 [DEBUG] TestAgent.server.raft: vote granted: from=77a725a7-57c4-33eb-9867-3902951ccc9a term=2 tally=1 -46:04.363 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.363 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11784 [Leader]" -46:04.363 [INFO] TestAgent.server: cluster leadership acquired -46:04.363 [INFO] TestAgent.server: New leader elected: payload=Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.367 [INFO] TestAgent.server: initializing acls -46:04.370 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:04.370 [INFO] TestAgent.server: initializing acls -46:04.387 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:04.387 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:04.387 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:04.387 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:04.387 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:04.387 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.387 [INFO] TestAgent.server: Updating LAN server: server="Node-77a725a7-57c4-33eb-9867-3902951ccc9a (Addr: tcp/127.0.0.1:11784) (DC: dc1)" -46:04.387 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 -46:04.387 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 area=wan -46:04.387 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.387 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 -46:04.387 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-77a725a7-57c4-33eb-9867-3902951ccc9a.dc1 area=wan -46:04.387 [INFO] TestAgent.server: Updating LAN server: server="Node-77a725a7-57c4-33eb-9867-3902951ccc9a (Addr: tcp/127.0.0.1:11784) (DC: dc1)" -46:04.396 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.412 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.412 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.412 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.412 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.412 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.412 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.412 [INFO] TestAgent.server: member joined, marking health alive: member=Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.417 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.418 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-77a725a7-57c4-33eb-9867-3902951ccc9a -46:04.422 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.493 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.501 [INFO] TestAgent: Synced node info -46:04.662 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:41080 -46:04.662 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 127.0.0.1 -46:04.663 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-30d0ecfd-59b3-c2c6-efe9-27d2d2768bbf.dc2 area=wan -46:04.674 [DEBUG] TestAgent.dns: request served from client: name=my-query.query.consul. type=SRV class=IN latency=961.703µs client=127.0.0.1:57148 client_network=udp -46:04.921 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:04.921 [DEBUG] TestAgent: Node info in sync -46:04.921 [DEBUG] TestAgent: Node info in sync -46:05.192 [INFO] TestAgent: Requesting shutdown -46:05.192 [INFO] TestAgent.server: shutting down server -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.192 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.192 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.195 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.201Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.201Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.201Z [INFO] agent.router.manager: shutting down -46:05.201 [INFO] TestAgent: consul server down -46:05.201 [INFO] TestAgent: shutdown complete -46:05.201 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11779 network=tcp -46:05.201 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11779 network=udp -46:05.202 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11780 network=tcp -46:05.702 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.702 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQuery_Failover (1.45s) -=== CONT TestTxnEndpoint_Bad_Size_Item -=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_default_limits -2022-11-07T22:46:04.956Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.956Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.966Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:04.966Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.007 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 Address:127.0.0.1:11872}]" -46:05.008 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1.dc1 127.0.0.1 -46:05.009 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 127.0.0.1 -2022-11-07T22:46:05.009Z [INFO] agent.router: Initializing LAN area manager -46:05.009 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11867 network=udp -46:05.009 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11872 [Follower]" leader= -46:05.009 [INFO] TestAgent.server: Adding LAN server: server="Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 (Addr: tcp/127.0.0.1:11872) (DC: dc1)" -46:05.009 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1.dc1 area=wan -46:05.009 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11867 network=tcp -46:05.010 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11868 network=tcp -46:05.010 [INFO] TestAgent: started state syncer -46:05.010 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11873 network=tcp -46:05.055 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.055 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11872 [Candidate]" term=2 -46:05.059 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:05.068 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.068 [DEBUG] TestAgent.server.raft: vote granted: from=e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 term=2 tally=1 -46:05.068 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.068 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11872 [Leader]" -46:05.068 [INFO] TestAgent.server: cluster leadership acquired -46:05.068 [INFO] TestAgent.server: New leader elected: payload=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.077 [INFO] TestAgent.server: initializing acls -46:05.082 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:05.082 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:05.091 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:05.109 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:05.109 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:05.109 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:05.109 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.109 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1.dc1 -46:05.109 [INFO] TestAgent.server: Updating LAN server: server="Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 (Addr: tcp/127.0.0.1:11872) (DC: dc1)" -46:05.109 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1.dc1 area=wan -46:05.128 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.158 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.158 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.158 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.158 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.158 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.158 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.158 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.164 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.169 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.186 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e2f0f8bd-e701-4d7f-3015-9dc3ca12eee1 -46:05.195 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=foo -46:05.195 [DEBUG] TestAgent.dns: request served from client: name=foo.service.consul. type=A class=IN latency=143.3µs client=127.0.0.1:51425 client_network=udp -46:05.195 [INFO] TestAgent: Requesting shutdown -46:05.195 [INFO] TestAgent.server: shutting down server -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.195 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.195 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.195 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.195 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.195 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.199 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.207Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.207Z [INFO] agent.router.manager: shutting down -46:05.208 [INFO] TestAgent: consul server down -46:05.208 [INFO] TestAgent: shutdown complete -46:05.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11867 network=tcp -46:05.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11867 network=udp -46:05.208 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11868 network=tcp -46:05.708 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.708 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_FilterACL (1.54s) - --- PASS: TestDNS_ServiceLookup_FilterACL/ACLToken_==_root (0.78s) - --- PASS: TestDNS_ServiceLookup_FilterACL/ACLToken_==_anonymous (0.76s) -=== CONT TestIntentionsMatch_noBy -2022-11-07T22:46:05.018Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.027Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.056 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97a31f7a-7bd2-feea-5feb-5acad42f43d2 Address:127.0.0.1:11907}]" -46:05.056 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11907 [Follower]" leader= -46:05.057 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2.dc1 127.0.0.1 -46:05.057 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2 127.0.0.1 -2022-11-07T22:46:05.057Z [INFO] agent.router: Initializing LAN area manager -46:05.057 [INFO] TestAgent.server: Adding LAN server: server="Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2 (Addr: tcp/127.0.0.1:11907) (DC: dc1)" -46:05.057 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2.dc1 area=wan -46:05.057 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11902 network=udp -46:05.057 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11902 network=tcp -46:05.058 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11903 network=tcp -46:05.058 [INFO] TestAgent: started state syncer -46:05.058 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11908 network=tcp -46:05.117 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.117 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11907 [Candidate]" term=2 -46:05.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.137 [DEBUG] TestAgent.server.raft: vote granted: from=97a31f7a-7bd2-feea-5feb-5acad42f43d2 term=2 tally=1 -46:05.137 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11907 [Leader]" -46:05.137 [INFO] TestAgent.server: cluster leadership acquired -46:05.137 [INFO] TestAgent.server: New leader elected: payload=Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2 -46:05.147 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11907 -46:05.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.184 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.184 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.184 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.184 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.184 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.184 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2 -46:05.184 [INFO] TestAgent.server: member joined, marking health alive: member=Node-97a31f7a-7bd2-feea-5feb-5acad42f43d2 -46:05.196 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.259 [INFO] TestAgent: Requesting shutdown -46:05.259 [INFO] TestAgent.server: shutting down server -46:05.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.259 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.259 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.259 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.259 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.259 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.259 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.259 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.266 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.273Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.273Z [INFO] agent.router.manager: shutting down -46:05.273 [INFO] TestAgent: consul server down -46:05.273 [INFO] TestAgent: shutdown complete -46:05.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11902 network=tcp -46:05.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11902 network=udp -46:05.273 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11903 network=tcp -46:05.773 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.773 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsCreate_good (0.77s) -=== CONT TestValidateUserEventParams ---- PASS: TestValidateUserEventParams (0.00s) -=== CONT TestTxnEndpoint_Bad_JSON -2022-11-07T22:46:04.989Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.998Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.041 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:090e2c18-9e2f-4866-58d4-13180da9ef26 Address:127.0.0.1:11893}]" -46:05.041 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11893 [Follower]" leader= -46:05.042 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-090e2c18-9e2f-4866-58d4-13180da9ef26.dc1 127.0.0.1 -46:05.042 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-090e2c18-9e2f-4866-58d4-13180da9ef26 127.0.0.1 -2022-11-07T22:46:05.042Z [INFO] agent.router: Initializing LAN area manager -46:05.042 [INFO] TestAgent.server: Adding LAN server: server="Node-090e2c18-9e2f-4866-58d4-13180da9ef26 (Addr: tcp/127.0.0.1:11893) (DC: dc1)" -46:05.042 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-090e2c18-9e2f-4866-58d4-13180da9ef26.dc1 area=wan -46:05.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11888 network=udp -46:05.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11888 network=tcp -46:05.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11889 network=tcp -46:05.043 [INFO] TestAgent: started state syncer -46:05.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11894 network=tcp -46:05.088 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.088 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11893 [Candidate]" term=2 -46:05.103 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.103 [DEBUG] TestAgent.server.raft: vote granted: from=090e2c18-9e2f-4866-58d4-13180da9ef26 term=2 tally=1 -46:05.103 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.103 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11893 [Leader]" -46:05.103 [INFO] TestAgent.server: cluster leadership acquired -46:05.103 [INFO] TestAgent.server: New leader elected: payload=Node-090e2c18-9e2f-4866-58d4-13180da9ef26 -46:05.113 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11893 -46:05.133 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.162 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.162 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.162 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.162 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.162 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.162 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-090e2c18-9e2f-4866-58d4-13180da9ef26 -46:05.162 [INFO] TestAgent.server: member joined, marking health alive: member=Node-090e2c18-9e2f-4866-58d4-13180da9ef26 -46:05.172 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.189 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.194 [INFO] TestAgent: Synced node info -46:05.194 [DEBUG] TestAgent: Node info in sync -46:05.303 [INFO] TestAgent: Requesting shutdown -46:05.303 [INFO] TestAgent.server: shutting down server -46:05.303 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.303 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.303 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.303 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.303 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.303 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.303 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.303 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.303 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.303 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.307 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.316Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.316Z [INFO] agent.router.manager: shutting down -46:05.316 [INFO] TestAgent: consul server down -46:05.316 [INFO] TestAgent: shutdown complete -46:05.316 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11888 network=tcp -46:05.316 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11888 network=udp -46:05.316 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11889 network=tcp -46:05.816 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.816 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsCreate_noBody (0.84s) -=== CONT TestUIGatewayServiceNodes_Ingress -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -46:05.368 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-4858b16b-4b7f-f3ae-ce65-f5c5e0183a24 127.0.0.1 -2022-11-07T22:46:05.368Z [INFO] agent.router: Initializing LAN area manager -46:05.369 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11930 network=tcp -46:05.369 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11930 network=udp -46:05.369 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11931 network=tcp -46:05.369 [INFO] TestAgent: started state syncer -46:05.369 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11936 network=tcp -2022-11-07T22:46:05.369Z [WARN] agent.router.manager: No servers available -46:05.369 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:05.370 [INFO] TestAgent: Requesting shutdown -46:05.370 [INFO] TestAgent.client: shutting down client -46:05.370 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:05.375 [INFO] TestAgent: consul client down -46:05.375 [INFO] TestAgent: shutdown complete -46:05.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11930 network=tcp -46:05.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11930 network=udp -46:05.375 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11931 network=tcp -46:05.875 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.875 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_LoadKeyrings (1.88s) - --- PASS: TestAgent_LoadKeyrings/no_keys (0.68s) - --- PASS: TestAgent_LoadKeyrings/server_with_keys (0.66s) - --- PASS: TestAgent_LoadKeyrings/client_with_keys (0.54s) -=== CONT TestStatusPeersSecondary -2022-11-07T22:46:04.965Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.974Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.015 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e31807ff-3f00-a9f3-8309-b0c779e48429 Address:127.0.0.1:11879}]" -46:05.015 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11879 [Follower]" leader= -46:05.015 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e31807ff-3f00-a9f3-8309-b0c779e48429.dc1 127.0.0.1 -46:05.016 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e31807ff-3f00-a9f3-8309-b0c779e48429 127.0.0.1 -2022-11-07T22:46:05.016Z [INFO] agent.router: Initializing LAN area manager -46:05.016 [INFO] TestAgent.server: Adding LAN server: server="Node-e31807ff-3f00-a9f3-8309-b0c779e48429 (Addr: tcp/127.0.0.1:11879) (DC: dc1)" -46:05.016 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e31807ff-3f00-a9f3-8309-b0c779e48429.dc1 area=wan -46:05.016 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11874 network=tcp -46:05.016 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11874 network=udp -46:05.016 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11875 network=tcp -46:05.016 [INFO] TestAgent: started state syncer -46:05.016 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11880 network=tcp -46:05.052 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.052 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11879 [Candidate]" term=2 -46:05.065 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.065 [DEBUG] TestAgent.server.raft: vote granted: from=e31807ff-3f00-a9f3-8309-b0c779e48429 term=2 tally=1 -46:05.065 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.065 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11879 [Leader]" -46:05.065 [INFO] TestAgent.server: cluster leadership acquired -46:05.065 [INFO] TestAgent.server: New leader elected: payload=Node-e31807ff-3f00-a9f3-8309-b0c779e48429 -46:05.074 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11879 -46:05.094 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.128 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.128 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.128 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.128 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.128 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.128 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e31807ff-3f00-a9f3-8309-b0c779e48429 -46:05.128 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e31807ff-3f00-a9f3-8309-b0c779e48429 -46:05.145 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.375 [INFO] TestAgent: Requesting shutdown -46:05.375 [INFO] TestAgent.server: shutting down server -46:05.375 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.375 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.375 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.375 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.375 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.375 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.375 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.375 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.375 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.375 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.375 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.382 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.391Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.391Z [INFO] agent.router.manager: shutting down -46:05.391 [INFO] TestAgent: consul server down -46:05.391 [INFO] TestAgent: shutdown complete -46:05.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11874 network=tcp -46:05.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11874 network=udp -46:05.391 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11875 network=tcp -46:05.892 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.892 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:05.076Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.085Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.129 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ce9f968-39e8-2861-335f-0beb14c61d79 Address:127.0.0.1:11914}]" -46:05.129 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11914 [Follower]" leader= -46:05.130 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4ce9f968-39e8-2861-335f-0beb14c61d79.dc1 127.0.0.1 -46:05.130 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4ce9f968-39e8-2861-335f-0beb14c61d79 127.0.0.1 -2022-11-07T22:46:05.130Z [INFO] agent.router: Initializing LAN area manager -46:05.130 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4ce9f968-39e8-2861-335f-0beb14c61d79.dc1 area=wan -46:05.130 [INFO] TestAgent.server: Adding LAN server: server="Node-4ce9f968-39e8-2861-335f-0beb14c61d79 (Addr: tcp/127.0.0.1:11914) (DC: dc1)" -46:05.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11909 network=udp -46:05.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11909 network=tcp -46:05.131 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11910 network=tcp -46:05.131 [INFO] TestAgent: started state syncer -46:05.131 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11915 network=tcp -46:05.190 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.190 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11914 [Candidate]" term=2 -46:05.201 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.201 [DEBUG] TestAgent.server.raft: vote granted: from=4ce9f968-39e8-2861-335f-0beb14c61d79 term=2 tally=1 -46:05.201 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.201 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11914 [Leader]" -46:05.201 [INFO] TestAgent.server: cluster leadership acquired -46:05.201 [INFO] TestAgent.server: New leader elected: payload=Node-4ce9f968-39e8-2861-335f-0beb14c61d79 -46:05.211 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11914 -46:05.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.245 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.245 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.245 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.245 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.245 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.245 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4ce9f968-39e8-2861-335f-0beb14c61d79 -46:05.245 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4ce9f968-39e8-2861-335f-0beb14c61d79 -46:05.251 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.257 [INFO] TestAgent: Synced node info -46:05.377 [INFO] TestAgent: Requesting shutdown -46:05.377 [INFO] TestAgent.server: shutting down server -46:05.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.377 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.377 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.377 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.377 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.377 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.377 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.382 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.391Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.391Z [INFO] agent.router.manager: shutting down -46:05.391 [INFO] TestAgent: consul server down -46:05.391 [INFO] TestAgent: shutdown complete -46:05.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11909 network=tcp -46:05.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11909 network=udp -46:05.391 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11910 network=tcp -46:05.892 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.892 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsSpecificGet_invalidId (0.83s) -=== CONT TestAgent_AddCheck_Alias_userAndSetToken -=== CONT TestUIGatewayServiceNodes_Terminating ---- PASS: TestAgent_AddCheck_Alias_userToken (0.94s) -2022-11-07T22:46:04.968Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.979Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.021 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a26d2622-9e7a-6525-b283-3dc050ec4180 Address:127.0.0.1:11886}]" -46:05.021 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11886 [Follower]" leader= -46:05.021 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a26d2622-9e7a-6525-b283-3dc050ec4180.dc1 127.0.0.1 -46:05.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a26d2622-9e7a-6525-b283-3dc050ec4180 127.0.0.1 -2022-11-07T22:46:05.021Z [INFO] agent.router: Initializing LAN area manager -46:05.022 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a26d2622-9e7a-6525-b283-3dc050ec4180.dc1 area=wan -46:05.022 [INFO] TestAgent.server: Adding LAN server: server="Node-a26d2622-9e7a-6525-b283-3dc050ec4180 (Addr: tcp/127.0.0.1:11886) (DC: dc1)" -46:05.022 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11881 network=udp -46:05.022 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11881 network=tcp -46:05.022 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11882 network=tcp -46:05.022 [INFO] TestAgent: started state syncer -46:05.022 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11887 network=tcp -46:05.069 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.069 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11886 [Candidate]" term=2 -46:05.082 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.083 [DEBUG] TestAgent.server.raft: vote granted: from=a26d2622-9e7a-6525-b283-3dc050ec4180 term=2 tally=1 -46:05.083 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.083 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11886 [Leader]" -46:05.083 [INFO] TestAgent.server: cluster leadership acquired -46:05.083 [INFO] TestAgent.server: New leader elected: payload=Node-a26d2622-9e7a-6525-b283-3dc050ec4180 -46:05.091 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11886 -46:05.108 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.128 [INFO] TestAgent: Synced node info -46:05.146 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.146 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.146 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.146 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.146 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.146 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a26d2622-9e7a-6525-b283-3dc050ec4180 -46:05.146 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a26d2622-9e7a-6525-b283-3dc050ec4180 -46:05.160 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.380 [INFO] TestAgent: Requesting shutdown -46:05.380 [INFO] TestAgent.server: shutting down server -46:05.380 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.380 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.380 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.380 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.380 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.380 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.380 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.380 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.380 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.380 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.388 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.394Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.394Z [INFO] agent.router.manager: shutting down -46:05.394 [INFO] TestAgent: consul server down -46:05.394 [INFO] TestAgent: shutdown complete -46:05.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11881 network=tcp -46:05.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11881 network=udp -46:05.394 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11882 network=tcp -46:05.895 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.895 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsCheck_noDestination (0.94s) +=== CONT TestDNS_ServiceLookup_OnlyPassing +2021-10-05T16:31:19.823Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.836Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.903 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:569a3bbe-9ed5-1568-83b7-783bd6d2bf02 Address:127.0.0.1:30057}]" +31:19.904 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30057 [Follower]" leader= +31:19.905 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02.dc1 127.0.0.1 +31:19.905 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02 127.0.0.1 +2021-10-05T16:31:19.905Z [INFO] agent.router: Initializing LAN area manager +31:19.905 [INFO] TestAgent.server: Adding LAN server: server="Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02 (Addr: tcp/127.0.0.1:30057) (DC: dc1)" +31:19.905 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02.dc1 area=wan +31:19.906 [DEBUG] TestAgent.dns: recursor enabled +31:19.906 [DEBUG] TestAgent.dns: recursor enabled +31:19.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30052 network=udp +31:19.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30052 network=tcp +31:19.925 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30053 network=tcp +31:19.925 [INFO] TestAgent: started state syncer +31:19.926 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30037 network=tcp +31:19.959 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.959 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30057 [Candidate]" term=2 +31:19.972 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.972 [DEBUG] TestAgent.server.raft: vote granted: from=569a3bbe-9ed5-1568-83b7-783bd6d2bf02 term=2 tally=1 +31:19.972 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.972 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30057 [Leader]" +31:19.975 [INFO] TestAgent.server: cluster leadership acquired +31:19.975 [INFO] TestAgent.server: New leader elected: payload=Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02 +31:19.985 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30057 +31:20.017 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.080 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.080 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.080 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.080 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.080 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.080 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02 +31:20.080 [INFO] TestAgent.server: member joined, marking health alive: member=Node-569a3bbe-9ed5-1568-83b7-783bd6d2bf02 +31:20.108 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.144 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=ANY class=IN latency=91.707µs client=127.0.0.1:40600 client_network=udp +31:20.157 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=ANY class=IN latency=162.401µs client=127.0.0.1:38417 client_network=udp +31:20.168 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=235.134µs +31:20.169 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=ANY class=IN latency=691.994µs client=127.0.0.1:53657 client_network=udp +31:20.169 [INFO] TestAgent: Requesting shutdown +31:20.169 [INFO] TestAgent.server: shutting down server +31:20.169 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.169 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.169 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.169 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.169 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.169 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.170 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:20.170 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.170 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.182 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.191Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.191Z [INFO] agent.router.manager: shutting down +31:20.191 [INFO] TestAgent: consul server down +31:20.191 [INFO] TestAgent: shutdown complete +31:20.191 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30052 network=tcp +31:20.191 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30052 network=udp +31:20.191 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30053 network=tcp +31:20.692 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.692 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_TTL (0.90s) +=== CONT TestDNS_ServiceLookup_OnlyFailing +2021-10-05T16:31:19.870Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.899Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.995 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d2d91ece-dc82-79c7-e9ca-ddc559509516 Address:127.0.0.1:30043}]" +31:19.995 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30043 [Follower]" leader= +31:19.996 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d2d91ece-dc82-79c7-e9ca-ddc559509516.dc1 127.0.0.1 +31:19.996 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d2d91ece-dc82-79c7-e9ca-ddc559509516 127.0.0.1 +2021-10-05T16:31:19.996Z [INFO] agent.router: Initializing LAN area manager +31:19.997 [DEBUG] TestAgent.dns: recursor enabled +31:19.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30038 network=udp +31:19.997 [INFO] TestAgent.server: Adding LAN server: server="Node-d2d91ece-dc82-79c7-e9ca-ddc559509516 (Addr: tcp/127.0.0.1:30043) (DC: dc1)" +31:19.997 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d2d91ece-dc82-79c7-e9ca-ddc559509516.dc1 area=wan +31:19.997 [DEBUG] TestAgent.dns: recursor enabled +31:19.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30038 network=tcp +31:19.997 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30039 network=tcp +31:19.997 [INFO] TestAgent: started state syncer +31:19.998 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30058 network=tcp +31:20.040 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.040 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30043 [Candidate]" term=2 +31:20.069 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.069 [DEBUG] TestAgent.server.raft: vote granted: from=d2d91ece-dc82-79c7-e9ca-ddc559509516 term=2 tally=1 +31:20.069 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.069 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30043 [Leader]" +31:20.071 [INFO] TestAgent.server: cluster leadership acquired +31:20.071 [INFO] TestAgent.server: New leader elected: payload=Node-d2d91ece-dc82-79c7-e9ca-ddc559509516 +31:20.084 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30043 +31:20.112 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.138 [INFO] TestAgent: Synced node info +31:20.182 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.182 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.182 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.182 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.182 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.182 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d2d91ece-dc82-79c7-e9ca-ddc559509516 +31:20.182 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d2d91ece-dc82-79c7-e9ca-ddc559509516 +31:20.206 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=208.24µs +31:20.206 [DEBUG] TestAgent.dns: request served from client: name=search.service.consul. type=ANY class=IN latency=519.582µs client=127.0.0.1:35927 client_network=udp +31:20.207 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=277.25µs +31:20.207 [DEBUG] TestAgent.dns: request served from client: name=d11d1265-44a7-fe34-91f8-26a38d42e4c0.query.consul. type=ANY class=IN latency=514.316µs client=127.0.0.1:38917 client_network=udp +31:20.207 [INFO] TestAgent: Requesting shutdown +31:20.207 [INFO] TestAgent.server: shutting down server +31:20.207 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.207 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.207 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.208 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.208 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.208 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.208 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.218 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:20.224 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.224 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:31:20.226Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.226Z [INFO] agent.router.manager: shutting down +31:20.226 [INFO] TestAgent: consul server down +31:20.226 [INFO] TestAgent: shutdown complete +31:20.226 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30038 network=tcp +31:20.227 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30038 network=udp +31:20.227 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30039 network=tcp +31:20.727 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.727 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_ServiceAddress_CNAME (0.88s) +=== CONT TestDNS_ServiceLookup_FilterCritical +2021-10-05T16:31:19.564Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.595Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.664 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dab25e5e-d952-7287-5cdd-12bd119a05b6 Address:127.0.0.1:30036}]" +31:19.665 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dab25e5e-d952-7287-5cdd-12bd119a05b6.dc1 127.0.0.1 +31:19.665 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dab25e5e-d952-7287-5cdd-12bd119a05b6 127.0.0.1 +2021-10-05T16:31:19.665Z [INFO] agent.router: Initializing LAN area manager +31:19.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30036 [Follower]" leader= +31:19.669 [INFO] TestAgent.server: Adding LAN server: server="Node-dab25e5e-d952-7287-5cdd-12bd119a05b6 (Addr: tcp/127.0.0.1:30036) (DC: dc1)" +31:19.669 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dab25e5e-d952-7287-5cdd-12bd119a05b6.dc1 area=wan +31:19.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30031 network=udp +31:19.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30031 network=tcp +31:19.676 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30032 network=tcp +31:19.676 [INFO] TestAgent: started state syncer +31:19.676 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30023 network=tcp +31:19.732 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.732 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30036 [Candidate]" term=2 +31:19.776 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.776 [DEBUG] TestAgent.server.raft: vote granted: from=dab25e5e-d952-7287-5cdd-12bd119a05b6 term=2 tally=1 +31:19.776 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.776 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30036 [Leader]" +31:19.782 [INFO] TestAgent.server: cluster leadership acquired +31:19.783 [INFO] TestAgent.server: New leader elected: payload=Node-dab25e5e-d952-7287-5cdd-12bd119a05b6 +31:19.795 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30036 +31:19.810 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.864 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.864 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.864 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.864 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.864 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.864 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dab25e5e-d952-7287-5cdd-12bd119a05b6 +31:19.864 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dab25e5e-d952-7287-5cdd-12bd119a05b6 +31:19.902 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.071 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.080 [INFO] TestAgent: Synced node info +31:20.239 [DEBUG] TestAgent.dns: request served from client: name=db.query.consul. type=SRV class=IN latency=508.851µs client=127.0.0.1:47294 client_network=udp +31:20.240 [DEBUG] TestAgent.dns: request served from client: name=db-ttl.query.consul. type=SRV class=IN latency=142.613µs client=127.0.0.1:39129 client_network=udp +31:20.241 [DEBUG] TestAgent.dns: request served from client: name=dblb.query.consul. type=SRV class=IN latency=156.957µs client=127.0.0.1:46507 client_network=udp +31:20.241 [DEBUG] TestAgent.dns: request served from client: name=dblb-ttl.query.consul. type=SRV class=IN latency=131.16µs client=127.0.0.1:48113 client_network=udp +31:20.241 [DEBUG] TestAgent.dns: request served from client: name=dk.query.consul. type=SRV class=IN latency=83.894µs client=127.0.0.1:40490 client_network=udp +31:20.242 [DEBUG] TestAgent.dns: request served from client: name=dk-ttl.query.consul. type=SRV class=IN latency=129.365µs client=127.0.0.1:44179 client_network=udp +31:20.242 [DEBUG] TestAgent.dns: request served from client: name=api.query.consul. type=SRV class=IN latency=96.38µs client=127.0.0.1:42786 client_network=udp +31:20.242 [DEBUG] TestAgent.dns: request served from client: name=api-ttl.query.consul. type=SRV class=IN latency=90.479µs client=127.0.0.1:42082 client_network=udp +31:20.242 [INFO] TestAgent: Requesting shutdown +31:20.242 [INFO] TestAgent.server: shutting down server +31:20.242 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.242 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.242 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.242 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.242 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.243 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.243 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.243 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.253 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.260Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.260Z [INFO] agent.router.manager: shutting down +31:20.260 [INFO] TestAgent: consul server down +31:20.260 [INFO] TestAgent: shutdown complete +31:20.260 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30031 network=tcp +31:20.260 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30031 network=udp +31:20.260 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30032 network=tcp +31:20.761 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.761 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQuery_TTL (1.21s) + --- PASS: TestDNS_PreparedQuery_TTL/db.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/db-ttl.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/dblb.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/dblb-ttl.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/dk.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/dk-ttl.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/api.query.consul. (0.00s) + --- PASS: TestDNS_PreparedQuery_TTL/api-ttl.query.consul. (0.00s) === CONT TestDNS_RecursorTimeout -2022-11-07T22:46:04.995Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.003Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.040 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2da6f227-dc32-e473-abc2-9b65338ec0a9 Address:127.0.0.1:11900}]" -46:05.040 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11900 [Follower]" leader= -46:05.040 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2da6f227-dc32-e473-abc2-9b65338ec0a9.dc1 127.0.0.1 -46:05.041 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2da6f227-dc32-e473-abc2-9b65338ec0a9 127.0.0.1 -2022-11-07T22:46:05.041Z [INFO] agent.router: Initializing LAN area manager -46:05.042 [INFO] TestAgent.server: Adding LAN server: server="Node-2da6f227-dc32-e473-abc2-9b65338ec0a9 (Addr: tcp/127.0.0.1:11900) (DC: dc1)" -46:05.042 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2da6f227-dc32-e473-abc2-9b65338ec0a9.dc1 area=wan -46:05.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11895 network=tcp -46:05.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11895 network=udp -46:05.042 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11896 network=tcp -46:05.042 [INFO] TestAgent: started state syncer -46:05.042 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11901 network=tcp -46:05.094 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.094 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11900 [Candidate]" term=2 -46:05.105 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.105 [DEBUG] TestAgent.server.raft: vote granted: from=2da6f227-dc32-e473-abc2-9b65338ec0a9 term=2 tally=1 -46:05.105 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.105 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11900 [Leader]" -46:05.106 [INFO] TestAgent.server: cluster leadership acquired -46:05.106 [INFO] TestAgent.server: New leader elected: payload=Node-2da6f227-dc32-e473-abc2-9b65338ec0a9 -46:05.118 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11900 -46:05.137 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.164 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.164 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.164 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.164 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.164 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.164 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2da6f227-dc32-e473-abc2-9b65338ec0a9 -46:05.164 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2da6f227-dc32-e473-abc2-9b65338ec0a9 -46:05.178 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.267 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.270 [INFO] TestAgent: Synced node info -46:05.270 [DEBUG] TestAgent: Node info in sync -46:05.392 [INFO] TestAgent: Requesting shutdown -46:05.392 [INFO] TestAgent.server: shutting down server -46:05.392 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.392 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.392 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.392 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.392 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.392 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.392 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.392 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.392 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.392 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.399 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.409Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.409Z [INFO] agent.router.manager: shutting down -46:05.409 [INFO] TestAgent: consul server down -46:05.409 [INFO] TestAgent: shutdown complete -46:05.409 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11895 network=tcp -46:05.409 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11895 network=udp -46:05.409 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11896 network=tcp -46:05.909 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.909 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsMatch_byInvalid (0.93s) -=== CONT TestStatusPeers -2022-11-07T22:46:05.246Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.257Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.289 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c226e9aa-4b82-8721-e410-8b988693360d Address:127.0.0.1:11921}]" -46:05.289 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11921 [Follower]" leader= -46:05.289 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c226e9aa-4b82-8721-e410-8b988693360d.dc1 127.0.0.1 -46:05.290 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c226e9aa-4b82-8721-e410-8b988693360d 127.0.0.1 -2022-11-07T22:46:05.290Z [INFO] agent.router: Initializing LAN area manager -46:05.290 [INFO] TestAgent.server: Adding LAN server: server="Node-c226e9aa-4b82-8721-e410-8b988693360d (Addr: tcp/127.0.0.1:11921) (DC: dc1)" -46:05.290 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11916 network=udp -46:05.290 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c226e9aa-4b82-8721-e410-8b988693360d.dc1 area=wan -46:05.290 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11916 network=tcp -46:05.291 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11917 network=tcp -46:05.291 [INFO] TestAgent: started state syncer -46:05.291 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11922 network=tcp -46:05.327 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.327 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11921 [Candidate]" term=2 -46:05.337 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.337 [DEBUG] TestAgent.server.raft: vote granted: from=c226e9aa-4b82-8721-e410-8b988693360d term=2 tally=1 -46:05.337 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.337 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11921 [Leader]" -46:05.337 [INFO] TestAgent.server: cluster leadership acquired -46:05.337 [INFO] TestAgent.server: New leader elected: payload=Node-c226e9aa-4b82-8721-e410-8b988693360d -46:05.344 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11921 -46:05.376 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.406 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.406 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.406 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.406 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.406 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.406 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c226e9aa-4b82-8721-e410-8b988693360d -46:05.406 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c226e9aa-4b82-8721-e410-8b988693360d -46:05.420 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.446 [INFO] TestAgent: Requesting shutdown -46:05.446 [INFO] TestAgent.server: shutting down server -46:05.446 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.446 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.446 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.446 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.446 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.446 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.446 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.446 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.446 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.449 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.459Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.459Z [INFO] agent.router.manager: shutting down -46:05.459 [INFO] TestAgent: consul server down -46:05.459 [INFO] TestAgent: shutdown complete -46:05.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11916 network=tcp -46:05.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11916 network=udp -46:05.459 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11917 network=tcp -46:05.959 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.959 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsCheck_basic (0.72s) -=== CONT TestUiServices -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -46:05.593 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-4e4bcaae-4ed1-3fbc-020e-d4840ba04393 127.0.0.1 -2022-11-07T22:46:05.593Z [INFO] agent.router: Initializing LAN area manager -46:05.593 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11965 network=udp -46:05.593 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11965 network=tcp -46:05.594 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11966 network=tcp -46:05.594 [INFO] TestAgent: started state syncer -46:05.594 [INFO] TestAgent: Requesting shutdown -46:05.594 [INFO] TestAgent.client: shutting down client -46:05.594 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:05.594 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11971 network=tcp -46:05.594 [ERROR] TestAgent: gRPC server failed: error="grpc: the server has been stopped" -2022-11-07T22:46:05.594Z [WARN] agent.router.manager: No servers available -46:05.594 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:05.601 [INFO] TestAgent: consul client down -46:05.601 [INFO] TestAgent: shutdown complete -46:05.601 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11965 network=tcp -46:05.601 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11965 network=udp -46:05.601 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11966 network=tcp -46:06.101 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.101 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_InmemKeyrings/ignore_files -2022-11-07T22:46:05.364Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.373Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.409 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 Address:127.0.0.1:11942}]" -46:05.409 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11942 [Follower]" leader= -46:05.410 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00.dc1 127.0.0.1 -46:05.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 127.0.0.1 -2022-11-07T22:46:05.411Z [INFO] agent.router: Initializing LAN area manager -46:05.411 [INFO] TestAgent.server: Adding LAN server: server="Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 (Addr: tcp/127.0.0.1:11942) (DC: dc1)" -46:05.411 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00.dc1 area=wan -46:05.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11937 network=udp -46:05.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11937 network=tcp -46:05.411 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11938 network=tcp -46:05.412 [INFO] TestAgent: started state syncer -46:05.412 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11943 network=tcp -46:05.460 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.460 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11942 [Candidate]" term=2 -46:05.471 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.471 [DEBUG] TestAgent.server.raft: vote granted: from=d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 term=2 tally=1 -46:05.471 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.471 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11942 [Leader]" -46:05.471 [INFO] TestAgent.server: cluster leadership acquired -46:05.471 [INFO] TestAgent.server: New leader elected: payload=Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 -46:05.478 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11942 -46:05.495 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.519 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.519 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.519 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.519 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.519 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.519 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 -46:05.519 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d8cb7bcb-cd61-0d32-79c8-b6e7cb127f00 -46:05.533 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.608 [INFO] TestAgent: Requesting shutdown -46:05.608 [INFO] TestAgent.server: shutting down server -46:05.608 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.608 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.608 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.608 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.608 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.608 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.608 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.608 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.608 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.608 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.608 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.611 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.618Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.618Z [INFO] agent.router.manager: shutting down -46:05.618 [INFO] TestAgent: consul server down -46:05.618 [INFO] TestAgent: shutdown complete -46:05.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11937 network=tcp -46:05.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11937 network=udp -46:05.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11938 network=tcp -46:06.118 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.118 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCreate_NodeChecks (0.78s) -=== CONT TestUiNodeInfo -2022-11-07T22:46:05.498Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.507Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.539 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:abe0544c-0c20-96e3-0567-8ccf5271f736 Address:127.0.0.1:11949}]" -46:05.539 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11949 [Follower]" leader= -46:05.540 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-abe0544c-0c20-96e3-0567-8ccf5271f736.dc1 127.0.0.1 -46:05.540 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-abe0544c-0c20-96e3-0567-8ccf5271f736 127.0.0.1 -2022-11-07T22:46:05.540Z [INFO] agent.router: Initializing LAN area manager -46:05.541 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11944 network=udp -46:05.541 [INFO] TestAgent.server: Adding LAN server: server="Node-abe0544c-0c20-96e3-0567-8ccf5271f736 (Addr: tcp/127.0.0.1:11949) (DC: dc1)" -46:05.541 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-abe0544c-0c20-96e3-0567-8ccf5271f736.dc1 area=wan -46:05.541 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11944 network=tcp -46:05.541 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11945 network=tcp -46:05.541 [INFO] TestAgent: started state syncer -46:05.541 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11950 network=tcp -46:05.579 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.579 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11949 [Candidate]" term=2 -46:05.589 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.589 [DEBUG] TestAgent.server.raft: vote granted: from=abe0544c-0c20-96e3-0567-8ccf5271f736 term=2 tally=1 -46:05.589 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.589 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11949 [Leader]" -46:05.589 [INFO] TestAgent.server: cluster leadership acquired -46:05.589 [INFO] TestAgent.server: New leader elected: payload=Node-abe0544c-0c20-96e3-0567-8ccf5271f736 -46:05.597 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11949 -46:05.610 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.613 [INFO] TestAgent: Requesting shutdown -46:05.613 [INFO] TestAgent.server: shutting down server -46:05.613 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.614 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:05.622 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.627Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.627Z [INFO] agent.router.manager: shutting down -46:05.627 [INFO] TestAgent: consul server down -46:05.627 [INFO] TestAgent: shutdown complete -46:05.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11944 network=tcp -46:05.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11944 network=udp -46:05.627 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11945 network=tcp -46:05.627 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" -46:05.628 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:05.628 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" -46:06.127 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.127 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsMatch_noName (0.64s) +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:19.931Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:19.931Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.979Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:19.979Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.058 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bcdf67dc-1767-71ed-4364-299e1843df77 Address:127.0.0.1:30064}]" +31:20.058 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30064 [Follower]" leader= +31:20.059 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 127.0.0.1 +31:20.059 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bcdf67dc-1767-71ed-4364-299e1843df77 127.0.0.1 +2021-10-05T16:31:20.059Z [INFO] agent.router: Initializing LAN area manager +31:20.060 [INFO] TestAgent.server: Adding LAN server: server="Node-bcdf67dc-1767-71ed-4364-299e1843df77 (Addr: tcp/127.0.0.1:30064) (DC: dc1)" +31:20.060 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30059 network=udp +31:20.060 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 area=wan +31:20.060 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30059 network=tcp +31:20.061 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30060 network=tcp +31:20.061 [INFO] TestAgent: started state syncer +31:20.062 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30044 network=tcp +31:20.104 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.104 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30064 [Candidate]" term=2 +31:20.110 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.144 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.144 [DEBUG] TestAgent.server.raft: vote granted: from=bcdf67dc-1767-71ed-4364-299e1843df77 term=2 tally=1 +31:20.144 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.144 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30064 [Leader]" +31:20.145 [INFO] TestAgent.server: cluster leadership acquired +31:20.145 [INFO] TestAgent.server: New leader elected: payload=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.164 [INFO] TestAgent.server: initializing acls +31:20.173 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.173 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.190 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:20.210 [INFO] TestAgent.server: initializing acls +31:20.210 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.232 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.232 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.232 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.232 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:20.232 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 +31:20.233 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 area=wan +31:20.233 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.233 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 +31:20.234 [INFO] TestAgent.server: Updating LAN server: server="Node-bcdf67dc-1767-71ed-4364-299e1843df77 (Addr: tcp/127.0.0.1:30064) (DC: dc1)" +31:20.234 [INFO] TestAgent.server: Updating LAN server: server="Node-bcdf67dc-1767-71ed-4364-299e1843df77 (Addr: tcp/127.0.0.1:30064) (DC: dc1)" +31:20.234 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bcdf67dc-1767-71ed-4364-299e1843df77.dc1 area=wan +31:20.271 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.308 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.308 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.308 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.308 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.308 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.308 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.308 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.324 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.324 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.331 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bcdf67dc-1767-71ed-4364-299e1843df77 +31:20.334 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.348 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=foo +31:20.348 [DEBUG] TestAgent.dns: request served from client: name=foo.service.consul. type=A class=IN latency=261.527µs client=127.0.0.1:40927 client_network=udp +31:20.348 [INFO] TestAgent: Requesting shutdown +31:20.348 [INFO] TestAgent.server: shutting down server +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.348 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.348 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.348 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.348 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.348 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.364 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.374Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.374Z [INFO] agent.router.manager: shutting down +31:20.375 [INFO] TestAgent: consul server down +31:20.375 [INFO] TestAgent: shutdown complete +31:20.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30059 network=tcp +31:20.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30059 network=udp +31:20.375 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30060 network=tcp +31:20.875 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.875 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_FilterACL (1.97s) + --- PASS: TestDNS_ServiceLookup_FilterACL/ACLToken_==_root (0.98s) + --- PASS: TestDNS_ServiceLookup_FilterACL/ACLToken_==_anonymous (0.99s) === CONT TestDNS_Recurse_Truncation -2022-11-07T22:46:04.847Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.857Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.882 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 Address:127.0.0.1:11858}]" -46:04.882 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11858 [Follower]" leader= -46:04.882 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36.dc1 127.0.0.1 -46:04.882 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 127.0.0.1 -2022-11-07T22:46:04.882Z [INFO] agent.router: Initializing LAN area manager -46:04.883 [INFO] TestAgent.server: Adding LAN server: server="Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 (Addr: tcp/127.0.0.1:11858) (DC: dc1)" -46:04.883 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36.dc1 area=wan -46:04.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11846 network=udp -46:04.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11846 network=tcp -46:04.883 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11847 network=tcp -46:04.883 [INFO] TestAgent: started state syncer -46:04.883 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11859 network=tcp -46:04.934 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.934 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11858 [Candidate]" term=2 -46:04.944 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.944 [DEBUG] TestAgent.server.raft: vote granted: from=d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 term=2 tally=1 -46:04.944 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.944 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11858 [Leader]" -46:04.944 [INFO] TestAgent.server: cluster leadership acquired -46:04.944 [INFO] TestAgent.server: New leader elected: payload=Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 -46:04.951 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11858 -46:04.964 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.994 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.994 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.994 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.994 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.994 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.994 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 -46:04.994 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d0d9e3a4-90c1-ef6b-4c09-9d30c901ba36 -46:05.004 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.128 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.136 [INFO] TestAgent: Synced node info -46:05.136 [DEBUG] TestAgent: Node info in sync -46:05.693 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.113993ms client=127.0.0.1:49347 client_network=udp -46:05.694 [DEBUG] TestAgent.dns: request served from client: name=6297cdb2-130d-f5a9-5232-b4adfbea2262.query.consul. type=ANY class=IN latency=805.641µs client=127.0.0.1:49693 client_network=udp -46:05.694 [INFO] TestAgent: Requesting shutdown -46:05.694 [INFO] TestAgent.server: shutting down server -46:05.694 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.694 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.694 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.694 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.694 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.694 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.694 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.694 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.694 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.694 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.700 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.706Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.706Z [INFO] agent.router.manager: shutting down -46:05.706 [INFO] TestAgent: consul server down -46:05.706 [INFO] TestAgent: shutdown complete -46:05.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11846 network=tcp -46:05.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11846 network=udp -46:05.706 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11847 network=tcp -46:06.207 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.207 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_Truncate (1.37s) -=== CONT TestStatusLeaderSecondary -=== RUN TestUiServices/No_Filter -=== PAUSE TestUiServices/No_Filter -=== RUN TestUiServices/Filtered -2022-11-07T22:46:05.538Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.549Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.584 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:36db344c-6b8c-48c2-522c-d3f4d8778cb2 Address:127.0.0.1:11963}]" -46:05.584 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11963 [Follower]" leader= -46:05.584 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2.dc1 127.0.0.1 -46:05.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2 127.0.0.1 -2022-11-07T22:46:05.585Z [INFO] agent.router: Initializing LAN area manager -46:05.585 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2.dc1 area=wan -46:05.585 [INFO] TestAgent.server: Adding LAN server: server="Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2 (Addr: tcp/127.0.0.1:11963) (DC: dc1)" -46:05.585 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11958 network=udp -46:05.585 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11958 network=tcp -46:05.585 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11959 network=tcp -46:05.585 [INFO] TestAgent: started state syncer -46:05.585 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11964 network=tcp -46:05.637 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.637 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11963 [Candidate]" term=2 -46:05.651 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.651 [DEBUG] TestAgent.server.raft: vote granted: from=36db344c-6b8c-48c2-522c-d3f4d8778cb2 term=2 tally=1 -46:05.651 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.651 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11963 [Leader]" -46:05.651 [INFO] TestAgent.server: cluster leadership acquired -46:05.652 [INFO] TestAgent.server: New leader elected: payload=Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2 -46:05.665 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11963 -46:05.665 [INFO] TestAgent: Synced node info -46:05.665 [DEBUG] TestAgent: Node info in sync -46:05.694 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.711 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.711 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.711 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.711 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.711 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.711 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2 -46:05.711 [INFO] TestAgent.server: member joined, marking health alive: member=Node-36db344c-6b8c-48c2-522c-d3f4d8778cb2 -46:05.727 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.749 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.749 [DEBUG] TestAgent: Node info in sync -46:05.799 [INFO] TestAgent: Requesting shutdown -46:05.799 [DEBUG] TestAgent: Node info in sync -46:05.800 [INFO] TestAgent.server: shutting down server -46:05.800 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.800 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.800 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.800 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.800 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.800 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.800 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.800 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.800 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.800 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.803 [INFO] TestAgent: Synced service: service=mysql -46:05.806 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.812Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.812Z [INFO] agent.router.manager: shutting down -46:05.812 [INFO] TestAgent: consul server down -46:05.812 [INFO] TestAgent: shutdown complete -46:05.812 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11958 network=tcp -46:05.812 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11958 network=udp -46:05.812 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11959 network=tcp -46:06.313 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.313 [INFO] TestAgent: Endpoints down ---- PASS: TestShouldProcessUserEvent (0.78s) -=== CONT TestUiNodes_Filter -2022-11-07T22:46:05.337Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.347Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.389 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef4dafd0-5789-bb55-f46a-895c3d4e4233 Address:127.0.0.1:11928}]" -46:05.389 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11928 [Follower]" leader= -46:05.389 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233.dc1 127.0.0.1 -46:05.390 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233 127.0.0.1 -2022-11-07T22:46:05.390Z [INFO] agent.router: Initializing LAN area manager -46:05.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11923 network=udp -46:05.390 [INFO] TestAgent.server: Adding LAN server: server="Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233 (Addr: tcp/127.0.0.1:11928) (DC: dc1)" -46:05.390 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233.dc1 area=wan -46:05.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11923 network=tcp -46:05.390 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11924 network=tcp -46:05.390 [INFO] TestAgent: started state syncer -46:05.390 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11929 network=tcp -46:05.439 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.439 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11928 [Candidate]" term=2 -46:05.451 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.451 [DEBUG] TestAgent.server.raft: vote granted: from=ef4dafd0-5789-bb55-f46a-895c3d4e4233 term=2 tally=1 -46:05.451 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.451 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11928 [Leader]" -46:05.451 [INFO] TestAgent.server: cluster leadership acquired -46:05.451 [INFO] TestAgent.server: New leader elected: payload=Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233 -46:05.458 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11928 -46:05.476 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.503 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.503 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.503 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233 -46:05.503 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef4dafd0-5789-bb55-f46a-895c3d4e4233 -46:05.519 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.672 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.675 [INFO] TestAgent: Synced node info -46:05.819 [INFO] TestAgent: Requesting shutdown -46:05.819 [INFO] TestAgent.server: shutting down server -46:05.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.819 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.819 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.819 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.819 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.819 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.819 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.819 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.819 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.825 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.831Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.831Z [INFO] agent.router.manager: shutting down -46:05.831 [INFO] TestAgent: consul server down -46:05.831 [INFO] TestAgent: shutdown complete -46:05.831 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11923 network=tcp -46:05.831 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11923 network=udp -46:05.831 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11924 network=tcp -46:06.331 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.331 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsSpecificGet_good (1.00s) +2021-10-05T16:31:19.979Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.007Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.109 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:335be017-af5d-921f-e6ff-9e1ffaf82854 Address:127.0.0.1:30050}]" +31:20.109 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30050 [Follower]" leader= +31:20.109 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-335be017-af5d-921f-e6ff-9e1ffaf82854.dc1 127.0.0.1 +31:20.110 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-335be017-af5d-921f-e6ff-9e1ffaf82854 127.0.0.1 +2021-10-05T16:31:20.110Z [INFO] agent.router: Initializing LAN area manager +31:20.110 [INFO] TestAgent.server: Adding LAN server: server="Node-335be017-af5d-921f-e6ff-9e1ffaf82854 (Addr: tcp/127.0.0.1:30050) (DC: dc1)" +31:20.110 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-335be017-af5d-921f-e6ff-9e1ffaf82854.dc1 area=wan +31:20.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30045 network=udp +31:20.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30045 network=tcp +31:20.111 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30046 network=tcp +31:20.111 [INFO] TestAgent: started state syncer +31:20.111 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30065 network=tcp +31:20.178 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.178 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30050 [Candidate]" term=2 +31:20.211 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.211 [DEBUG] TestAgent.server.raft: vote granted: from=335be017-af5d-921f-e6ff-9e1ffaf82854 term=2 tally=1 +31:20.211 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.211 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30050 [Leader]" +31:20.212 [INFO] TestAgent.server: cluster leadership acquired +31:20.212 [INFO] TestAgent.server: New leader elected: payload=Node-335be017-af5d-921f-e6ff-9e1ffaf82854 +31:20.233 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30050 +31:20.266 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.321 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.321 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.321 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.321 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.321 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.321 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-335be017-af5d-921f-e6ff-9e1ffaf82854 +31:20.321 [INFO] TestAgent.server: member joined, marking health alive: member=Node-335be017-af5d-921f-e6ff-9e1ffaf82854 +31:20.353 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.423 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.433 [INFO] TestAgent: Synced node info +31:20.444 [DEBUG] TestAgent.dns: request served from client: name=_this-is-a-very-very-very-very-very-long-name-for-a-service._primary.service.consul. type=SRV class=IN latency=253.469µs client=127.0.0.1:36942 client_network=udp +31:20.448 [DEBUG] TestAgent.dns: request served from client: name=this-is-a-very-very-very-very-very-long-name-for-a-service.query.consul. type=SRV class=IN latency=229.921µs client=127.0.0.1:43071 client_network=udp +31:20.448 [INFO] TestAgent: Requesting shutdown +31:20.448 [INFO] TestAgent.server: shutting down server +31:20.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.448 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.448 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.448 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.448 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.448 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.448 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.448 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.448 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.464 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.476Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.476Z [INFO] agent.router.manager: shutting down +31:20.476 [INFO] TestAgent: consul server down +31:20.476 [INFO] TestAgent: shutdown complete +31:20.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30045 network=tcp +31:20.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30045 network=udp +31:20.476 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30046 network=tcp +31:20.978 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.978 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_LargeResponses (1.08s) === CONT TestDNS_Recurse -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:05.521Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.530Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.568 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c05423b3-6320-7cd8-5434-093125802437 Address:127.0.0.1:11956}]" -46:05.568 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11956 [Follower]" leader= -46:05.569 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c05423b3-6320-7cd8-5434-093125802437.dc1 127.0.0.1 -46:05.570 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c05423b3-6320-7cd8-5434-093125802437 127.0.0.1 -2022-11-07T22:46:05.570Z [INFO] agent.router: Initializing LAN area manager -46:05.570 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c05423b3-6320-7cd8-5434-093125802437.dc1 area=wan -46:05.570 [INFO] TestAgent.server: Adding LAN server: server="Node-c05423b3-6320-7cd8-5434-093125802437 (Addr: tcp/127.0.0.1:11956) (DC: dc1)" -46:05.570 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11951 network=udp -46:05.570 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11951 network=tcp -46:05.570 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11952 network=tcp -46:05.570 [INFO] TestAgent: started state syncer -46:05.570 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11957 network=tcp -46:05.634 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.634 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11956 [Candidate]" term=2 -46:05.651 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.651 [DEBUG] TestAgent.server.raft: vote granted: from=c05423b3-6320-7cd8-5434-093125802437 term=2 tally=1 -46:05.651 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.651 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11956 [Leader]" -46:05.651 [INFO] TestAgent.server: cluster leadership acquired -46:05.651 [INFO] TestAgent.server: New leader elected: payload=Node-c05423b3-6320-7cd8-5434-093125802437 -46:05.661 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11956 -46:05.681 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.707 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.707 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.707 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.707 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.707 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.707 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c05423b3-6320-7cd8-5434-093125802437 -46:05.707 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c05423b3-6320-7cd8-5434-093125802437 -46:05.722 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.851 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.855 [INFO] TestAgent: Synced node info -46:06.010 [INFO] TestAgent: Requesting shutdown -46:06.010 [INFO] TestAgent.server: shutting down server -46:06.010 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.010 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.010 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.010 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.010 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.011 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.010 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.011 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.011 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.018 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.022Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.022Z [INFO] agent.router.manager: shutting down -46:06.022 [INFO] TestAgent: consul server down -46:06.022 [INFO] TestAgent: shutdown complete -46:06.022 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11951 network=tcp -46:06.022 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11951 network=udp -46:06.022 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11952 network=tcp -46:06.522 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.522 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsMatch_basic (1.01s) -=== CONT TestStatusLeader -2022-11-07T22:46:05.907Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.929Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.972 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fbecbc6-d438-9ef2-200a-eec15689a139 Address:127.0.0.1:12012}]" -46:05.972 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12012 [Follower]" leader= -46:05.972 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fbecbc6-d438-9ef2-200a-eec15689a139.dc1 127.0.0.1 -46:05.972 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fbecbc6-d438-9ef2-200a-eec15689a139 127.0.0.1 -2022-11-07T22:46:05.972Z [INFO] agent.router: Initializing LAN area manager -46:05.973 [INFO] TestAgent.server: Adding LAN server: server="Node-2fbecbc6-d438-9ef2-200a-eec15689a139 (Addr: tcp/127.0.0.1:12012) (DC: dc1)" -46:05.973 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fbecbc6-d438-9ef2-200a-eec15689a139.dc1 area=wan -46:05.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12007 network=udp -46:05.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12007 network=tcp -46:05.973 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12008 network=tcp -46:05.973 [INFO] TestAgent: started state syncer -46:05.973 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12013 network=tcp -46:06.010 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.010 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12012 [Candidate]" term=2 -46:06.025 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.025 [DEBUG] TestAgent.server.raft: vote granted: from=2fbecbc6-d438-9ef2-200a-eec15689a139 term=2 tally=1 -46:06.025 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.025 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12012 [Leader]" -46:06.025 [INFO] TestAgent.server: cluster leadership acquired -46:06.025 [INFO] TestAgent.server: New leader elected: payload=Node-2fbecbc6-d438-9ef2-200a-eec15689a139 -46:06.033 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12012 -46:06.048 [INFO] TestAgent: Requesting shutdown -46:06.048 [INFO] TestAgent.server: shutting down server -46:06.048 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.048 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.056 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.058 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.068Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.068Z [INFO] agent.router.manager: shutting down -46:06.073 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:06.073 [INFO] TestAgent: consul server down -46:06.073 [INFO] TestAgent: shutdown complete -46:06.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12007 network=tcp -46:06.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12007 network=udp -46:06.073 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12008 network=tcp -46:06.574 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.574 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_Alias_userAndSetToken (0.68s) -=== CONT TestUiNodes -2022-11-07T22:46:05.723Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.737Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.767 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fbc1451e-392d-5f42-7d3f-7aaebda7231c Address:127.0.0.1:11977}]" -46:05.767 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11977 [Follower]" leader= -46:05.768 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c.dc1 127.0.0.1 -46:05.769 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c 127.0.0.1 -2022-11-07T22:46:05.769Z [INFO] agent.router: Initializing LAN area manager -46:05.769 [INFO] TestAgent.server: Adding LAN server: server="Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c (Addr: tcp/127.0.0.1:11977) (DC: dc1)" -46:05.769 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c.dc1 area=wan -46:05.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11972 network=udp -46:05.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11972 network=tcp -46:05.769 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11973 network=tcp -46:05.769 [INFO] TestAgent: started state syncer -46:05.769 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11978 network=tcp -46:05.811 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.811 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11977 [Candidate]" term=2 -46:05.824 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.824 [DEBUG] TestAgent.server.raft: vote granted: from=fbc1451e-392d-5f42-7d3f-7aaebda7231c term=2 tally=1 -46:05.824 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.824 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11977 [Leader]" -46:05.824 [INFO] TestAgent.server: cluster leadership acquired -46:05.824 [INFO] TestAgent.server: New leader elected: payload=Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c -46:05.832 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11977 -46:05.847 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.873 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.873 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.873 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.873 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.873 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.873 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c -46:05.873 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fbc1451e-392d-5f42-7d3f-7aaebda7231c -46:05.888 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.060 [INFO] TestAgent: Requesting shutdown -46:06.060 [INFO] TestAgent.server: shutting down server -46:06.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.060 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.060 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.060 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.061 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.061 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.061 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.061 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.068 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.075Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.075Z [INFO] agent.router.manager: shutting down -46:06.075 [INFO] TestAgent: consul server down -46:06.075 [INFO] TestAgent: shutdown complete -46:06.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11972 network=tcp -46:06.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11972 network=udp -46:06.075 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11973 network=tcp -46:06.576 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.576 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_configured_max_txn_len -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:05.784Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.793Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.823 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a31c6de7-e9e0-4063-1980-83156facb4c5 Address:127.0.0.1:11991}]" -46:05.823 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11991 [Follower]" leader= -46:05.824 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a31c6de7-e9e0-4063-1980-83156facb4c5.dc1 127.0.0.1 -46:05.825 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a31c6de7-e9e0-4063-1980-83156facb4c5 127.0.0.1 -2022-11-07T22:46:05.825Z [INFO] agent.router: Initializing LAN area manager -46:05.825 [INFO] TestAgent.server: Adding LAN server: server="Node-a31c6de7-e9e0-4063-1980-83156facb4c5 (Addr: tcp/127.0.0.1:11991) (DC: dc1)" -46:05.825 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a31c6de7-e9e0-4063-1980-83156facb4c5.dc1 area=wan -46:05.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11986 network=udp -46:05.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11986 network=tcp -46:05.825 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11987 network=tcp -46:05.825 [INFO] TestAgent: started state syncer -46:05.825 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11992 network=tcp -46:05.868 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.868 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11991 [Candidate]" term=2 -46:05.879 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.879 [DEBUG] TestAgent.server.raft: vote granted: from=a31c6de7-e9e0-4063-1980-83156facb4c5 term=2 tally=1 -46:05.879 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.879 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11991 [Leader]" -46:05.879 [INFO] TestAgent.server: cluster leadership acquired -46:05.879 [INFO] TestAgent.server: New leader elected: payload=Node-a31c6de7-e9e0-4063-1980-83156facb4c5 -46:05.887 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11991 -46:05.901 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.928 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.928 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.928 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.928 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.928 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.928 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a31c6de7-e9e0-4063-1980-83156facb4c5 -46:05.928 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a31c6de7-e9e0-4063-1980-83156facb4c5 -46:05.940 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.108 [INFO] TestAgent: Requesting shutdown -46:06.108 [INFO] TestAgent.server: shutting down server -46:06.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.108 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.108 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.108 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.108 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.108 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.108 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.108 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.114 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.121Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.121Z [INFO] agent.router.manager: shutting down -46:06.121 [INFO] TestAgent: consul server down -46:06.121 [INFO] TestAgent: shutdown complete -46:06.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11986 network=tcp -46:06.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11986 network=udp -46:06.121 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11987 network=tcp -46:06.622 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.622 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_Bad_JSON (0.85s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:20.624Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.641Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.681 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2570725a-7bab-6d94-13ee-4abc6d098f0b Address:127.0.0.1:30106}]" +31:20.681 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30106 [Follower]" leader= +31:20.682 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2570725a-7bab-6d94-13ee-4abc6d098f0b.dc1 127.0.0.1 +31:20.682 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2570725a-7bab-6d94-13ee-4abc6d098f0b 127.0.0.1 +2021-10-05T16:31:20.682Z [INFO] agent.router: Initializing LAN area manager +31:20.683 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30101 network=udp +31:20.683 [INFO] TestAgent.server: Adding LAN server: server="Node-2570725a-7bab-6d94-13ee-4abc6d098f0b (Addr: tcp/127.0.0.1:30106) (DC: dc1)" +31:20.683 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30101 network=tcp +31:20.683 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2570725a-7bab-6d94-13ee-4abc6d098f0b.dc1 area=wan +31:20.683 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30102 network=tcp +31:20.684 [INFO] TestAgent: started state syncer +31:20.684 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30093 network=tcp +31:20.732 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.732 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30106 [Candidate]" term=2 +31:20.751 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.751 [DEBUG] TestAgent.server.raft: vote granted: from=2570725a-7bab-6d94-13ee-4abc6d098f0b term=2 tally=1 +31:20.751 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.751 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30106 [Leader]" +31:20.752 [INFO] TestAgent.server: cluster leadership acquired +31:20.752 [INFO] TestAgent.server: New leader elected: payload=Node-2570725a-7bab-6d94-13ee-4abc6d098f0b +31:20.764 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30106 +31:20.788 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.826 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.826 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.826 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.826 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.826 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.826 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2570725a-7bab-6d94-13ee-4abc6d098f0b +31:20.826 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2570725a-7bab-6d94-13ee-4abc6d098f0b +31:20.854 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.916 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=269.435µs client=127.0.0.1:50794 client_network=udp +31:20.917 [DEBUG] TestAgent.dns: request served from client: name=783e7916-97bb-478f-138e-b38ca08201ed.query.consul. type=ANY class=IN latency=160.544µs client=127.0.0.1:54379 client_network=udp +31:20.917 [INFO] TestAgent: Requesting shutdown +31:20.917 [INFO] TestAgent.server: shutting down server +31:20.917 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.917 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.917 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:20.927 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.935Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.935Z [INFO] agent.router.manager: shutting down +31:20.935 [INFO] TestAgent: consul server down +31:20.935 [INFO] TestAgent: shutdown complete +31:20.935 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30101 network=tcp +31:20.935 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30101 network=udp +31:20.935 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30102 network=tcp +31:21.436 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.436 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_OnlyPassing (0.83s) === CONT TestDNS_ServiceLookup_Dedup_SRV -2022-11-07T22:46:05.827Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.836Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.867 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d8600a99-b51e-5aa3-b18d-ffdea17f951d Address:127.0.0.1:11998}]" -46:05.867 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11998 [Follower]" leader= -46:05.868 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d.dc1 127.0.0.1 -46:05.868 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d 127.0.0.1 -2022-11-07T22:46:05.868Z [INFO] agent.router: Initializing LAN area manager -46:05.868 [INFO] TestAgent.server: Adding LAN server: server="Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d (Addr: tcp/127.0.0.1:11998) (DC: dc1)" -46:05.868 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d.dc1 area=wan -46:05.869 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11993 network=udp -46:05.869 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11993 network=tcp -46:05.869 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11994 network=tcp -46:05.869 [INFO] TestAgent: started state syncer -46:05.869 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11999 network=tcp -46:05.929 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.929 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11998 [Candidate]" term=2 -46:05.941 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.941 [DEBUG] TestAgent.server.raft: vote granted: from=d8600a99-b51e-5aa3-b18d-ffdea17f951d term=2 tally=1 -46:05.941 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.941 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11998 [Leader]" -46:05.941 [INFO] TestAgent.server: cluster leadership acquired -46:05.941 [INFO] TestAgent.server: New leader elected: payload=Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d -46:05.949 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11998 -46:05.965 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.001 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.001 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.001 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.001 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.001 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.001 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d -46:06.001 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d8600a99-b51e-5aa3-b18d-ffdea17f951d -46:06.012 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.118 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.121 [INFO] TestAgent: Requesting shutdown -46:06.121 [INFO] TestAgent.server: shutting down server -46:06.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.121 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.121 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.121 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.121 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.121 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.121 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.121 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.121 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.125 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:06.125 [INFO] TestAgent: Synced node info -46:06.125 [DEBUG] TestAgent: Node info in sync -2022-11-07T22:46:06.136Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.136Z [INFO] agent.router.manager: shutting down -46:06.136 [INFO] TestAgent: consul server down -46:06.136 [INFO] TestAgent: shutdown complete -46:06.136 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11993 network=tcp -46:06.136 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11993 network=udp -46:06.136 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11994 network=tcp -46:06.636 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.636 [INFO] TestAgent: Endpoints down ---- PASS: TestUIGatewayServiceNodes_Ingress (0.82s) -=== CONT TestSnapshot_Options -=== RUN TestSnapshot_Options/GET -2022-11-07T22:46:05.726Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.735Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.760 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:99c94178-ef65-5e6e-dcc8-840aaf2e630f Address:127.0.0.1:11984}]" -46:05.760 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11984 [Follower]" leader= -46:05.761 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f.dc1 127.0.0.1 -46:05.761 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f 127.0.0.1 -2022-11-07T22:46:05.761Z [INFO] agent.router: Initializing LAN area manager -46:05.761 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f.dc1 area=wan -46:05.761 [INFO] TestAgent.server: Adding LAN server: server="Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f (Addr: tcp/127.0.0.1:11984) (DC: dc1)" -46:05.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11979 network=udp -46:05.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11979 network=tcp -46:05.762 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11980 network=tcp -46:05.762 [INFO] TestAgent: started state syncer -46:05.762 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11985 network=tcp -46:05.813 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.813 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11984 [Candidate]" term=2 -46:05.825 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.825 [DEBUG] TestAgent.server.raft: vote granted: from=99c94178-ef65-5e6e-dcc8-840aaf2e630f term=2 tally=1 -46:05.825 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.825 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11984 [Leader]" -46:05.825 [INFO] TestAgent.server: cluster leadership acquired -46:05.826 [INFO] TestAgent.server: New leader elected: payload=Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f -46:05.834 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11984 -46:05.847 [INFO] TestAgent: Synced node info -46:05.847 [DEBUG] TestAgent: Node info in sync -46:05.861 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.877 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.878 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.878 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.878 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.878 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.878 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f -46:05.878 [INFO] TestAgent.server: member joined, marking health alive: member=Node-99c94178-ef65-5e6e-dcc8-840aaf2e630f -46:05.892 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.192 [INFO] TestAgent: Requesting shutdown -46:06.192 [INFO] TestAgent.server: shutting down server -46:06.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.192 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.192 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.192 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.197 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.200Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.200Z [INFO] agent.router.manager: shutting down -46:06.200 [INFO] TestAgent: consul server down -46:06.200 [INFO] TestAgent: shutdown complete -46:06.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11979 network=tcp -46:06.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11979 network=udp -46:06.200 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11980 network=tcp -46:06.700 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.700 [INFO] TestAgent: Endpoints down ---- PASS: TestIntentionsMatch_noBy (0.99s) -=== CONT TestUiIndex -2022-11-07T22:46:05.906Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.922Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.970 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:616a5780-18dc-ebbd-775f-df67e4b6f348 Address:127.0.0.1:12019}]" -46:05.970 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12019 [Follower]" leader= -46:05.971 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-616a5780-18dc-ebbd-775f-df67e4b6f348.dc1 127.0.0.1 -46:05.971 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-616a5780-18dc-ebbd-775f-df67e4b6f348 127.0.0.1 -2022-11-07T22:46:05.971Z [INFO] agent.router: Initializing LAN area manager -46:05.971 [INFO] TestAgent.server: Adding LAN server: server="Node-616a5780-18dc-ebbd-775f-df67e4b6f348 (Addr: tcp/127.0.0.1:12019) (DC: dc1)" -46:05.971 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-616a5780-18dc-ebbd-775f-df67e4b6f348.dc1 area=wan -46:05.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12014 network=udp -46:05.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12014 network=tcp -46:05.972 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12015 network=tcp -46:05.972 [INFO] TestAgent: started state syncer -46:05.972 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12020 network=tcp -46:06.023 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.023 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12019 [Candidate]" term=2 -46:06.040 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.040 [DEBUG] TestAgent.server.raft: vote granted: from=616a5780-18dc-ebbd-775f-df67e4b6f348 term=2 tally=1 -46:06.040 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.040 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12019 [Leader]" -46:06.040 [INFO] TestAgent.server: cluster leadership acquired -46:06.040 [INFO] TestAgent.server: New leader elected: payload=Node-616a5780-18dc-ebbd-775f-df67e4b6f348 -46:06.056 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12019 -46:06.079 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.108 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.108 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.108 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.108 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.108 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.108 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-616a5780-18dc-ebbd-775f-df67e4b6f348 -46:06.108 [INFO] TestAgent.server: member joined, marking health alive: member=Node-616a5780-18dc-ebbd-775f-df67e4b6f348 -46:06.124 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.200 [INFO] TestAgent: Requesting shutdown -46:06.200 [INFO] TestAgent.server: shutting down server -46:06.200 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.200 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.200 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.200 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.200 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.200 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.200 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.200 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.200 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.200 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.200 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.205 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.213Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.213Z [INFO] agent.router.manager: shutting down -46:06.213 [INFO] TestAgent: consul server down -46:06.213 [INFO] TestAgent: shutdown complete -46:06.213 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12014 network=tcp -46:06.213 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12014 network=udp -46:06.213 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12015 network=tcp -46:06.714 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.714 [INFO] TestAgent: Endpoints down ---- PASS: TestUIGatewayServiceNodes_Terminating (0.82s) +2021-10-05T16:31:19.996Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:19.996Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.023Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:20.023Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.136 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a15c335e-06b1-34d9-3c62-e536e966914f Address:127.0.0.1:30071}]" +31:20.137 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 127.0.0.1 +31:20.137 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a15c335e-06b1-34d9-3c62-e536e966914f 127.0.0.1 +2021-10-05T16:31:20.137Z [INFO] agent.router: Initializing LAN area manager +31:20.138 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30071 [Follower]" leader= +31:20.138 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30066 network=udp +31:20.138 [INFO] TestAgent.server: Adding LAN server: server="Node-a15c335e-06b1-34d9-3c62-e536e966914f (Addr: tcp/127.0.0.1:30071) (DC: dc2)" +31:20.138 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 area=wan +31:20.138 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30066 network=tcp +31:20.138 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30067 network=tcp +31:20.139 [INFO] TestAgent: started state syncer +31:20.139 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30072 network=tcp +31:20.188 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.198 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.199 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30071 [Candidate]" term=2 +31:20.235 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.235 [DEBUG] TestAgent.server.raft: vote granted: from=a15c335e-06b1-34d9-3c62-e536e966914f term=2 tally=1 +31:20.235 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.235 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30071 [Leader]" +31:20.236 [INFO] TestAgent.server: cluster leadership acquired +31:20.236 [INFO] TestAgent.server: New leader elected: payload=Node-a15c335e-06b1-34d9-3c62-e536e966914f +31:20.256 [INFO] TestAgent.server: initializing acls +31:20.268 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.283 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.283 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.283 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.284 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a15c335e-06b1-34d9-3c62-e536e966914f +31:20.284 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 +31:20.284 [INFO] TestAgent.server: Updating LAN server: server="Node-a15c335e-06b1-34d9-3c62-e536e966914f (Addr: tcp/127.0.0.1:30071) (DC: dc2)" +31:20.284 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 area=wan +31:20.320 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.358 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.358 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.358 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.358 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.358 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.358 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a15c335e-06b1-34d9-3c62-e536e966914f +31:20.358 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a15c335e-06b1-34d9-3c62-e536e966914f +31:20.374 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a15c335e-06b1-34d9-3c62-e536e966914f +31:20.383 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:30014] +31:20.384 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:30014 +31:20.386 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf57440c-b456-9911-f589-d018e667a573.dc1 127.0.0.1 +31:20.386 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:20.386 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf57440c-b456-9911-f589-d018e667a573.dc1 area=wan +31:20.405 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.415 [INFO] TestAgent: Requesting shutdown +31:20.415 [INFO] TestAgent.server: shutting down server +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.415 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.415 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.415 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.415 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.420 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.429Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.429Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.429Z [INFO] agent.router.manager: shutting down +31:20.429 [INFO] TestAgent: consul server down +31:20.429 [INFO] TestAgent: shutdown complete +31:20.429 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30066 network=tcp +31:20.429 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30066 network=udp +31:20.429 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30067 network=tcp +31:20.929 [INFO] TestAgent: Waiting for endpoints to shut down +31:20.929 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:19.404Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:19.404Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:19.423Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:19.423Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:19.513 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf57440c-b456-9911-f589-d018e667a573 Address:127.0.0.1:30015}]" +31:19.514 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf57440c-b456-9911-f589-d018e667a573.dc1 127.0.0.1 +31:19.514 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf57440c-b456-9911-f589-d018e667a573 127.0.0.1 +2021-10-05T16:31:19.514Z [INFO] agent.router: Initializing LAN area manager +31:19.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30010 network=udp +31:19.514 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30015 [Follower]" leader= +31:19.515 [INFO] TestAgent.server: Adding LAN server: server="Node-cf57440c-b456-9911-f589-d018e667a573 (Addr: tcp/127.0.0.1:30015) (DC: dc1)" +31:19.515 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf57440c-b456-9911-f589-d018e667a573.dc1 area=wan +31:19.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30010 network=tcp +31:19.516 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30011 network=tcp +31:19.516 [INFO] TestAgent: started state syncer +31:19.516 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30030 network=tcp +31:19.568 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:19.584 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:19.584 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30015 [Candidate]" term=2 +31:19.615 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:19.615 [DEBUG] TestAgent.server.raft: vote granted: from=cf57440c-b456-9911-f589-d018e667a573 term=2 tally=1 +31:19.615 [INFO] TestAgent.server.raft: election won: tally=1 +31:19.615 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30015 [Leader]" +31:19.616 [INFO] TestAgent.server: cluster leadership acquired +31:19.616 [INFO] TestAgent.server: New leader elected: payload=Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.644 [INFO] TestAgent.server: initializing acls +31:19.661 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:19.668 [INFO] TestAgent.server: initializing acls +31:19.680 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:19.680 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:19.680 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:19.681 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cf57440c-b456-9911-f589-d018e667a573.dc1 +31:19.681 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cf57440c-b456-9911-f589-d018e667a573.dc1 area=wan +31:19.681 [INFO] TestAgent.server: Updating LAN server: server="Node-cf57440c-b456-9911-f589-d018e667a573 (Addr: tcp/127.0.0.1:30015) (DC: dc1)" +31:19.696 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:19.696 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:19.696 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cf57440c-b456-9911-f589-d018e667a573.dc1 +31:19.696 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cf57440c-b456-9911-f589-d018e667a573.dc1 area=wan +31:19.696 [INFO] TestAgent.server: Updating LAN server: server="Node-cf57440c-b456-9911-f589-d018e667a573 (Addr: tcp/127.0.0.1:30015) (DC: dc1)" +31:19.702 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:19.750 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:19.750 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:19.750 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:19.750 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:19.750 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:19.750 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.750 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.782 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.782 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf57440c-b456-9911-f589-d018e667a573 +31:19.786 [INFO] TestAgent.server: federation state anti-entropy synced +31:19.804 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:19.821 [INFO] TestAgent: Synced node info +31:19.821 [DEBUG] TestAgent: Node info in sync +31:20.384 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:56474 +31:20.385 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 127.0.0.1 +31:20.385 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a15c335e-06b1-34d9-3c62-e536e966914f.dc2 area=wan +31:20.414 [DEBUG] TestAgent.dns: request served from client: name=my-query.query.consul. type=SRV class=IN latency=1.173061ms client=127.0.0.1:46449 client_network=udp +31:20.688 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.688 [DEBUG] TestAgent: Node info in sync +31:20.929 [INFO] TestAgent: Requesting shutdown +31:20.929 [INFO] TestAgent.server: shutting down server +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.929 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.929 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.930 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.930 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.935 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.939Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.939Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.939Z [INFO] agent.router.manager: shutting down +31:20.939 [INFO] TestAgent: consul server down +31:20.939 [INFO] TestAgent: shutdown complete +31:20.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30010 network=tcp +31:20.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30010 network=udp +31:20.939 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30011 network=tcp +31:21.440 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.440 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQuery_Failover (2.06s) === CONT TestDNS_ServiceLookup_Dedup -2022-11-07T22:46:05.933Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.948Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.991 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca0272de-9dad-3806-9296-30df3a797f11 Address:127.0.0.1:12033}]" -46:05.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12033 [Follower]" leader= -46:05.991 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ca0272de-9dad-3806-9296-30df3a797f11.dc1 127.0.0.1 -46:05.992 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ca0272de-9dad-3806-9296-30df3a797f11 127.0.0.1 -2022-11-07T22:46:05.992Z [INFO] agent.router: Initializing LAN area manager -46:05.992 [INFO] TestAgent.server: Adding LAN server: server="Node-ca0272de-9dad-3806-9296-30df3a797f11 (Addr: tcp/127.0.0.1:12033) (DC: dc1)" -46:05.992 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ca0272de-9dad-3806-9296-30df3a797f11.dc1 area=wan -46:05.992 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12028 network=udp -46:05.992 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12028 network=tcp -46:05.992 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12029 network=tcp -46:05.992 [INFO] TestAgent: started state syncer -46:05.992 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12034 network=tcp -46:06.027 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.027 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12033 [Candidate]" term=2 -46:06.042 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.042 [DEBUG] TestAgent.server.raft: vote granted: from=ca0272de-9dad-3806-9296-30df3a797f11 term=2 tally=1 -46:06.042 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.042 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12033 [Leader]" -46:06.042 [INFO] TestAgent.server: cluster leadership acquired -46:06.042 [INFO] TestAgent.server: New leader elected: payload=Node-ca0272de-9dad-3806-9296-30df3a797f11 -46:06.057 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12033 -46:06.076 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.104 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.104 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.104 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.104 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.104 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.104 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca0272de-9dad-3806-9296-30df3a797f11 -46:06.104 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ca0272de-9dad-3806-9296-30df3a797f11 -46:06.119 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.241 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.252 [INFO] TestAgent: Synced node info -46:06.252 [DEBUG] TestAgent: Node info in sync -46:06.255 [INFO] TestAgent: Requesting shutdown -46:06.255 [INFO] TestAgent.server: shutting down server -46:06.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.255 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.255 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.255 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.255 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.255 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.256 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.256 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.256 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.266 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.273Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.273Z [INFO] agent.router.manager: shutting down -46:06.273 [INFO] TestAgent: consul server down -46:06.273 [INFO] TestAgent: shutdown complete -46:06.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12028 network=tcp -46:06.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12028 network=udp -46:06.273 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12029 network=tcp -46:06.773 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.773 [INFO] TestAgent: Endpoints down ---- PASS: TestStatusPeers (0.86s) -=== CONT TestTxnEndpoint_UpdateCheck -=== CONT TestSessionDeleteDestroy -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:06.139Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.148Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.192 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8e07ae72-fb74-d284-5c68-2fbaa3901008 Address:127.0.0.1:11118}]" -46:06.192 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11118 [Follower]" leader= -46:06.192 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8e07ae72-fb74-d284-5c68-2fbaa3901008.dc1 127.0.0.1 -46:06.193 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8e07ae72-fb74-d284-5c68-2fbaa3901008 127.0.0.1 -2022-11-07T22:46:06.193Z [INFO] agent.router: Initializing LAN area manager -46:06.193 [DEBUG] TestAgent.dns: recursor enabled -46:06.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11092 network=udp -46:06.193 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8e07ae72-fb74-d284-5c68-2fbaa3901008.dc1 area=wan -46:06.193 [INFO] TestAgent.server: Adding LAN server: server="Node-8e07ae72-fb74-d284-5c68-2fbaa3901008 (Addr: tcp/127.0.0.1:11118) (DC: dc1)" -46:06.193 [DEBUG] TestAgent.dns: recursor enabled -46:06.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11092 network=tcp -46:06.193 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11093 network=tcp -46:06.193 [INFO] TestAgent: started state syncer -46:06.193 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11119 network=tcp -46:06.229 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.229 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11118 [Candidate]" term=2 -46:06.250 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.250 [DEBUG] TestAgent.server.raft: vote granted: from=8e07ae72-fb74-d284-5c68-2fbaa3901008 term=2 tally=1 -46:06.250 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.250 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11118 [Leader]" -46:06.250 [INFO] TestAgent.server: cluster leadership acquired -46:06.250 [INFO] TestAgent.server: New leader elected: payload=Node-8e07ae72-fb74-d284-5c68-2fbaa3901008 -46:06.261 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11118 -46:06.281 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.311 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.311 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.311 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.311 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.311 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.311 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e07ae72-fb74-d284-5c68-2fbaa3901008 -46:06.311 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8e07ae72-fb74-d284-5c68-2fbaa3901008 -46:06.317 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.334 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=120.368µs recursor=127.0.0.1:36567 -46:06.334 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=228.297µs client=127.0.0.1:55458 client_network=udp -46:06.334 [INFO] TestAgent: Requesting shutdown -46:06.334 [INFO] TestAgent.server: shutting down server -46:06.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.334 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.334 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.334 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.334 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.334 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.334 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.334 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.340 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.345Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.345Z [INFO] agent.router.manager: shutting down -46:06.345 [INFO] TestAgent: consul server down -46:06.345 [INFO] TestAgent: shutdown complete -46:06.345 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11092 network=tcp -46:06.346 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11092 network=udp -46:06.346 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11093 network=tcp -46:06.846 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.846 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Recurse_Truncation (0.72s) -=== CONT TestDNS_ServiceLookup_PreparedQueryNamePeriod -2022-11-07T22:46:06.128Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.139Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.182 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 Address:127.0.0.1:11034}]" -46:06.182 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11034 [Follower]" leader= -46:06.182 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6.dc1 127.0.0.1 -46:06.183 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 127.0.0.1 -2022-11-07T22:46:06.183Z [INFO] agent.router: Initializing LAN area manager -46:06.183 [INFO] TestAgent.server: Adding LAN server: server="Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" -46:06.183 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6.dc1 area=wan -46:06.183 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=udp -46:06.183 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=tcp -46:06.183 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11030 network=tcp -46:06.183 [INFO] TestAgent: started state syncer -46:06.183 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11035 network=tcp -46:06.245 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.245 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11034 [Candidate]" term=2 -46:06.262 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.262 [DEBUG] TestAgent.server.raft: vote granted: from=b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 term=2 tally=1 -46:06.262 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.262 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11034 [Leader]" -46:06.262 [INFO] TestAgent.server: cluster leadership acquired -46:06.262 [INFO] TestAgent.server: New leader elected: payload=Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 -46:06.274 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11034 -46:06.294 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.319 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.319 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.319 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.319 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.319 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.319 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 -46:06.319 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b1ba82e6-43d7-9684-b16f-5a4cdc5cd6f6 -46:06.333 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.469 [INFO] TestAgent: Requesting shutdown -46:06.469 [INFO] TestAgent.server: shutting down server -46:06.469 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.469 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.469 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.469 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.469 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.470 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.470 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.470 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.470 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.470 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.470 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.473 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.480Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.480Z [INFO] agent.router.manager: shutting down -46:06.480 [INFO] TestAgent: consul server down -46:06.480 [INFO] TestAgent: shutdown complete -46:06.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=tcp -46:06.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=udp -46:06.480 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11030 network=tcp -46:06.981 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.981 [INFO] TestAgent: Endpoints down ---- PASS: TestUiNodeInfo (0.86s) -=== CONT TestTxnEndpoint_KV_Actions -=== RUN TestTxnEndpoint_KV_Actions/#00 -2022-11-07T22:46:06.342Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.353Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.383 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 Address:127.0.0.1:11111}]" -46:06.383 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11111 [Follower]" leader= -46:06.383 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377.dc1 127.0.0.1 -46:06.384 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 127.0.0.1 -2022-11-07T22:46:06.384Z [INFO] agent.router: Initializing LAN area manager -46:06.384 [INFO] TestAgent.server: Adding LAN server: server="Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 (Addr: tcp/127.0.0.1:11111) (DC: dc1)" -46:06.384 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377.dc1 area=wan -46:06.384 [DEBUG] TestAgent.dns: recursor enabled -46:06.384 [DEBUG] TestAgent.dns: recursor enabled -46:06.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11134 network=tcp -46:06.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11134 network=udp -46:06.384 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11135 network=tcp -46:06.384 [INFO] TestAgent: started state syncer -46:06.384 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11112 network=tcp -46:06.425 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.425 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11111 [Candidate]" term=2 -46:06.433 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.433 [DEBUG] TestAgent.server.raft: vote granted: from=0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 term=2 tally=1 -46:06.433 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.433 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11111 [Leader]" -46:06.433 [INFO] TestAgent.server: cluster leadership acquired -46:06.433 [INFO] TestAgent.server: New leader elected: payload=Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 -46:06.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11111 -46:06.460 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.484 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.484 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.484 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.484 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.484 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.484 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 -46:06.484 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0a1e331d-5c5e-36c6-af35-e8d1c5ba9377 -46:06.498 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.526 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=89.301µs recursor=127.0.0.1:43223 -46:06.526 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=173.571µs client=127.0.0.1:60090 client_network=udp -46:06.526 [INFO] TestAgent: Requesting shutdown -46:06.526 [INFO] TestAgent.server: shutting down server -46:06.526 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.527 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.527 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.527 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.532 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.535Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.535Z [INFO] agent.router.manager: shutting down -46:06.535 [INFO] TestAgent: consul server down -46:06.535 [INFO] TestAgent: shutdown complete -46:06.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11134 network=tcp -46:06.535 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11134 network=udp -46:06.535 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11135 network=tcp -46:07.035 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.035 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Recurse (0.70s) -=== CONT TestSessionGet -=== RUN TestSessionGet/#00 [WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:06.111Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.121Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.155 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 Address:127.0.0.1:11027}]" -46:06.155 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11027 [Follower]" leader= -46:06.156 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9.dc1 127.0.0.1 -46:06.156 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 127.0.0.1 -2022-11-07T22:46:06.156Z [INFO] agent.router: Initializing LAN area manager -46:06.156 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9.dc1 area=wan -46:06.156 [INFO] TestAgent.server: Adding LAN server: server="Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" -46:06.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=udp -46:06.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=tcp -46:06.156 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11023 network=tcp -46:06.156 [INFO] TestAgent: started state syncer -46:06.157 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11028 network=tcp -46:06.201 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.201 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11027 [Candidate]" term=2 -46:06.213 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.213 [DEBUG] TestAgent.server.raft: vote granted: from=f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 term=2 tally=1 -46:06.213 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.213 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11027 [Leader]" -46:06.213 [INFO] TestAgent.server: cluster leadership acquired -46:06.213 [INFO] TestAgent.server: New leader elected: payload=Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 -46:06.219 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11027 -46:06.230 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.279 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.279 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.279 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.279 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.279 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.279 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 -46:06.279 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8e9f175-ab24-fc68-aed1-a42cb87bd9c9 -46:06.291 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.556 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.560 [INFO] TestAgent: Synced node info -46:06.606 [INFO] TestAgent: Requesting shutdown -46:06.606 [INFO] TestAgent.server: shutting down server -46:06.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.606 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.606 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.606 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.606 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.606 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.606 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.610 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.616Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.616Z [INFO] agent.router.manager: shutting down -46:06.616 [INFO] TestAgent: consul server down -46:06.616 [INFO] TestAgent: shutdown complete -46:06.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=tcp -46:06.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=udp -46:06.616 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11023 network=tcp -46:07.116 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.116 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_InmemKeyrings (3.52s) - --- PASS: TestAgent_InmemKeyrings/no_keys (0.99s) - --- PASS: TestAgent_InmemKeyrings/server_with_keys (0.99s) - --- PASS: TestAgent_InmemKeyrings/client_with_keys (0.53s) - --- PASS: TestAgent_InmemKeyrings/ignore_files (1.02s) -=== CONT TestDNS_PreparedQueryNearIP -Added 3 service nodes -2022-11-07T22:46:06.323Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.332Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.370 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67eb1b58-1270-4388-7b22-c5e09f457c26 Address:127.0.0.1:11132}]" -46:06.370 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11132 [Follower]" leader= -46:06.370 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67eb1b58-1270-4388-7b22-c5e09f457c26.dc1 127.0.0.1 -46:06.370 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67eb1b58-1270-4388-7b22-c5e09f457c26 127.0.0.1 -2022-11-07T22:46:06.370Z [INFO] agent.router: Initializing LAN area manager -46:06.370 [INFO] TestAgent.server: Adding LAN server: server="Node-67eb1b58-1270-4388-7b22-c5e09f457c26 (Addr: tcp/127.0.0.1:11132) (DC: dc1)" -46:06.370 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67eb1b58-1270-4388-7b22-c5e09f457c26.dc1 area=wan -46:06.371 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11078 network=udp -46:06.371 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11078 network=tcp -46:06.371 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11079 network=tcp -46:06.371 [INFO] TestAgent: started state syncer -46:06.371 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11133 network=tcp -46:06.438 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.438 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11132 [Candidate]" term=2 -46:06.450 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.450 [DEBUG] TestAgent.server.raft: vote granted: from=67eb1b58-1270-4388-7b22-c5e09f457c26 term=2 tally=1 -46:06.450 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.450 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11132 [Leader]" -46:06.450 [INFO] TestAgent.server: cluster leadership acquired -46:06.450 [INFO] TestAgent.server: New leader elected: payload=Node-67eb1b58-1270-4388-7b22-c5e09f457c26 -46:06.458 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11132 -46:06.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.498 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.498 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.498 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.498 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.498 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.498 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67eb1b58-1270-4388-7b22-c5e09f457c26 -46:06.498 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67eb1b58-1270-4388-7b22-c5e09f457c26 -46:06.510 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.785 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.793 [INFO] TestAgent: Synced node info -46:06.816 [INFO] TestAgent: Requesting shutdown -46:06.816 [INFO] TestAgent.server: shutting down server -46:06.816 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.816 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.816 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.816 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.816 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.816 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.816 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.816 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.825 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.832Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.832Z [INFO] agent.router.manager: shutting down -46:06.832 [INFO] TestAgent: consul server down -46:06.832 [INFO] TestAgent: shutdown complete -46:06.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11078 network=tcp -46:06.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11078 network=udp -46:06.832 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11079 network=tcp -46:07.333 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.333 [INFO] TestAgent: Endpoints down ---- PASS: TestUiNodes_Filter (1.02s) -=== CONT TestTxnEndpoint_Bad_Size_Ops +2021-10-05T16:31:20.741Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.754Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.837 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:30d3d4ed-e198-876b-1eef-73f3e1b216dc Address:127.0.0.1:30113}]" +31:20.838 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc.dc1 127.0.0.1 +31:20.838 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc 127.0.0.1 +2021-10-05T16:31:20.838Z [INFO] agent.router: Initializing LAN area manager +31:20.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30108 network=udp +31:20.839 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30113 [Follower]" leader= +31:20.839 [INFO] TestAgent.server: Adding LAN server: server="Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc (Addr: tcp/127.0.0.1:30113) (DC: dc1)" +31:20.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc.dc1 area=wan +31:20.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30108 network=tcp +31:20.840 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30109 network=tcp +31:20.840 [INFO] TestAgent: started state syncer +31:20.840 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30114 network=tcp +31:20.901 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.901 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30113 [Candidate]" term=2 +31:20.929 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.929 [DEBUG] TestAgent.server.raft: vote granted: from=30d3d4ed-e198-876b-1eef-73f3e1b216dc term=2 tally=1 +31:20.929 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.929 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30113 [Leader]" +31:20.930 [INFO] TestAgent.server: cluster leadership acquired +31:20.930 [INFO] TestAgent.server: New leader elected: payload=Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc +31:20.942 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30113 +31:20.967 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc +31:21.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-30d3d4ed-e198-876b-1eef-73f3e1b216dc +31:21.082 [INFO] TestAgent: Synced node info +31:21.082 [DEBUG] TestAgent: Node info in sync +31:21.090 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.159 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=182.422µs client=127.0.0.1:53681 client_network=udp +31:21.159 [DEBUG] TestAgent.dns: request served from client: name=4883eabc-7ad3-49f7-fbeb-9d9e64892dfa.query.consul. type=ANY class=IN latency=153.95µs client=127.0.0.1:42192 client_network=udp +31:21.159 [INFO] TestAgent: Requesting shutdown +31:21.159 [INFO] TestAgent.server: shutting down server +31:21.159 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.159 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.159 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.159 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.159 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.159 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.159 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.159 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.159 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.159 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.172 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.184Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.184Z [INFO] agent.router.manager: shutting down +31:21.184 [INFO] TestAgent: consul server down +31:21.184 [INFO] TestAgent: shutdown complete +31:21.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30108 network=tcp +31:21.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30108 network=udp +31:21.184 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30109 network=tcp +31:21.685 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.685 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_FilterCritical (0.96s) +=== CONT TestDNS_ServiceLookup_TagPeriod +2021-10-05T16:31:20.707Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.721Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.772 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e78d696-93d1-eeab-e674-0678bb85487c Address:127.0.0.1:30099}]" +31:20.772 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30099 [Follower]" leader= +31:20.773 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2e78d696-93d1-eeab-e674-0678bb85487c.dc1 127.0.0.1 +31:20.773 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2e78d696-93d1-eeab-e674-0678bb85487c 127.0.0.1 +2021-10-05T16:31:20.773Z [INFO] agent.router: Initializing LAN area manager +31:20.773 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30094 network=udp +31:20.774 [INFO] TestAgent.server: Adding LAN server: server="Node-2e78d696-93d1-eeab-e674-0678bb85487c (Addr: tcp/127.0.0.1:30099) (DC: dc1)" +31:20.774 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2e78d696-93d1-eeab-e674-0678bb85487c.dc1 area=wan +31:20.774 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30094 network=tcp +31:20.774 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30095 network=tcp +31:20.774 [INFO] TestAgent: started state syncer +31:20.774 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30107 network=tcp +31:20.848 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.848 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30099 [Candidate]" term=2 +31:20.890 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.890 [DEBUG] TestAgent.server.raft: vote granted: from=2e78d696-93d1-eeab-e674-0678bb85487c term=2 tally=1 +31:20.890 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.890 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30099 [Leader]" +31:20.891 [INFO] TestAgent.server: cluster leadership acquired +31:20.891 [INFO] TestAgent.server: New leader elected: payload=Node-2e78d696-93d1-eeab-e674-0678bb85487c +31:20.911 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30099 +31:20.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.056 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.056 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.056 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.056 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.056 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.056 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2e78d696-93d1-eeab-e674-0678bb85487c +31:21.056 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2e78d696-93d1-eeab-e674-0678bb85487c +31:21.100 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:21.108 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.131 [INFO] TestAgent: Synced node info +31:21.131 [DEBUG] TestAgent: Node info in sync +31:21.168 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=164.518µs client=127.0.0.1:52273 client_network=udp +31:21.168 [DEBUG] TestAgent.dns: request served from client: name=d371bf74-dbed-922c-e7a4-bfa659779db3.query.consul. type=ANY class=IN latency=155.257µs client=127.0.0.1:33021 client_network=udp +31:21.168 [INFO] TestAgent: Requesting shutdown +31:21.168 [INFO] TestAgent.server: shutting down server +31:21.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.168 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.168 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.168 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.168 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.168 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.168 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.180 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.192Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.192Z [INFO] agent.router.manager: shutting down +31:21.192 [INFO] TestAgent: consul server down +31:21.192 [INFO] TestAgent: shutdown complete +31:21.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30094 network=tcp +31:21.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30094 network=udp +31:21.192 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30095 network=tcp +31:21.693 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.693 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_OnlyFailing (1.00s) +=== CONT TestDNS_ServiceLookup_PreparedQueryNamePeriod [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:06.592Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.606Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.642 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cea4e174-bade-fa45-7a04-5f18bff11984 Address:127.0.0.1:11125}]" -46:06.642 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11125 [Follower]" leader= -46:06.643 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cea4e174-bade-fa45-7a04-5f18bff11984.dc1 127.0.0.1 -46:06.643 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cea4e174-bade-fa45-7a04-5f18bff11984 127.0.0.1 -2022-11-07T22:46:06.643Z [INFO] agent.router: Initializing LAN area manager -46:06.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11141 network=udp -46:06.643 [INFO] TestAgent.server: Adding LAN server: server="Node-cea4e174-bade-fa45-7a04-5f18bff11984 (Addr: tcp/127.0.0.1:11125) (DC: dc1)" -46:06.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11141 network=tcp -46:06.644 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cea4e174-bade-fa45-7a04-5f18bff11984.dc1 area=wan -46:06.644 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11142 network=tcp -46:06.644 [INFO] TestAgent: started state syncer -46:06.644 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11126 network=tcp -46:06.687 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.687 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11125 [Candidate]" term=2 -46:06.697 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.697 [DEBUG] TestAgent.server.raft: vote granted: from=cea4e174-bade-fa45-7a04-5f18bff11984 term=2 tally=1 -46:06.697 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.697 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11125 [Leader]" -46:06.697 [INFO] TestAgent.server: cluster leadership acquired -46:06.698 [INFO] TestAgent.server: New leader elected: payload=Node-cea4e174-bade-fa45-7a04-5f18bff11984 -46:06.704 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11125 -46:06.721 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.741 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.741 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.741 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.741 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.741 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.741 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cea4e174-bade-fa45-7a04-5f18bff11984 -46:06.741 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cea4e174-bade-fa45-7a04-5f18bff11984 -46:06.752 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.854 [INFO] TestAgent: Requesting shutdown -46:06.854 [INFO] TestAgent.server: shutting down server -46:06.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.854 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.854 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.854 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.854 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.854 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.854 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.854 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.854 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.854 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.861 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.867Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.867Z [INFO] agent.router.manager: shutting down -46:06.867 [INFO] TestAgent: consul server down -46:06.867 [INFO] TestAgent: shutdown complete -46:06.867 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11141 network=tcp -46:06.867 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11141 network=udp -46:06.867 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11142 network=tcp -46:07.368 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.368 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_default_max_kv_value_size -2022-11-07T22:46:06.637Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.649Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.681 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e5dc10d7-d33d-0127-1a69-5f6e7119514c Address:127.0.0.1:11097}]" -46:06.681 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11097 [Follower]" leader= -46:06.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c.dc1 127.0.0.1 -46:06.682 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c 127.0.0.1 -2022-11-07T22:46:06.682Z [INFO] agent.router: Initializing LAN area manager -46:06.682 [INFO] TestAgent.server: Adding LAN server: server="Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c (Addr: tcp/127.0.0.1:11097) (DC: dc1)" -46:06.682 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c.dc1 area=wan -46:06.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11113 network=udp -46:06.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11113 network=tcp -46:06.683 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11114 network=tcp -46:06.683 [INFO] TestAgent: started state syncer -46:06.683 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11098 network=tcp -46:06.749 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.749 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11097 [Candidate]" term=2 -46:06.760 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.760 [DEBUG] TestAgent.server.raft: vote granted: from=e5dc10d7-d33d-0127-1a69-5f6e7119514c term=2 tally=1 -46:06.760 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.760 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11097 [Leader]" -46:06.760 [INFO] TestAgent.server: cluster leadership acquired -46:06.760 [INFO] TestAgent.server: New leader elected: payload=Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c -46:06.773 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11097 -46:06.788 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.826 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.826 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.826 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.826 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.826 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.826 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c -46:06.826 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e5dc10d7-d33d-0127-1a69-5f6e7119514c -46:06.854 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.854 [INFO] TestAgent: Synced node info -46:06.899 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=162.175µs client=127.0.0.1:34249 client_network=udp -46:06.899 [DEBUG] TestAgent.dns: request served from client: name=cf73f1dc-d08f-7f57-b653-0e5bb9121823.query.consul. type=SRV class=IN latency=152.618µs client=127.0.0.1:45939 client_network=udp -46:06.900 [INFO] TestAgent: Requesting shutdown -46:06.900 [INFO] TestAgent.server: shutting down server -46:06.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.900 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.900 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.900 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.900 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.900 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.900 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.909 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.917Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.917Z [INFO] agent.router.manager: shutting down -46:06.917 [INFO] TestAgent: consul server down -46:06.917 [INFO] TestAgent: shutdown complete -46:06.917 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11113 network=tcp -46:06.917 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11113 network=udp -46:06.917 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11114 network=tcp -46:07.418 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.418 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_Dedup_SRV (0.80s) -=== CONT TestSessionCreate -2022-11-07T22:46:06.650Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:06.650Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.660Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:06.660Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.695 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 Address:127.0.0.1:11104}]" -46:06.695 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11104 [Follower]" leader= -46:06.695 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8.dc1 127.0.0.1 -46:06.696 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 127.0.0.1 -2022-11-07T22:46:06.696Z [INFO] agent.router: Initializing LAN area manager -46:06.696 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8.dc1 area=wan -46:06.696 [INFO] TestAgent.server: Adding LAN server: server="Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 (Addr: tcp/127.0.0.1:11104) (DC: dc1)" -46:06.696 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11071 network=udp -46:06.696 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11071 network=tcp -46:06.697 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11072 network=tcp -46:06.697 [INFO] TestAgent: started state syncer -46:06.697 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11105 network=tcp -46:06.746 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:06.765 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.765 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11104 [Candidate]" term=2 -46:06.783 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.783 [DEBUG] TestAgent.server.raft: vote granted: from=72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 term=2 tally=1 -46:06.783 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.783 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11104 [Leader]" -46:06.784 [INFO] TestAgent.server: cluster leadership acquired -46:06.784 [INFO] TestAgent.server: New leader elected: payload=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.797 [INFO] TestAgent.server: initializing acls -46:06.802 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:06.802 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:06.812 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:06.832 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:06.832 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:06.832 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:06.832 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.832 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8.dc1 -46:06.833 [INFO] TestAgent.server: Updating LAN server: server="Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 (Addr: tcp/127.0.0.1:11104) (DC: dc1)" -46:06.833 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8.dc1 area=wan -46:06.856 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.883 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.883 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.883 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.883 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.883 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.883 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.883 [INFO] TestAgent.server: member joined, marking health alive: member=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.888 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.897 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.973 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-72a259f6-78c2-d4a5-0f89-b1e499a9b7d8 -46:06.973 [INFO] TestAgent: Requesting shutdown -46:06.973 [INFO] TestAgent.server: shutting down server -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:06.973 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.973 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.973 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.973 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:06.973 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:06.977 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.982Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.982Z [INFO] agent.router.manager: shutting down -46:06.982 [INFO] TestAgent: consul server down -46:06.982 [INFO] TestAgent: shutdown complete -46:06.982 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11071 network=tcp -46:06.982 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11071 network=udp -46:06.982 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11072 network=tcp -46:07.482 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.482 [INFO] TestAgent: Endpoints down -=== RUN TestSnapshot_Options/GET#01 -2022-11-07T22:46:06.531Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.540Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.562 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba8bc033-618f-c29f-e457-8d85f2c29c92 Address:127.0.0.1:11041}]" -46:06.563 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11041 [Follower]" leader= -46:06.563 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba8bc033-618f-c29f-e457-8d85f2c29c92.dc1 127.0.0.1 -46:06.563 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba8bc033-618f-c29f-e457-8d85f2c29c92 127.0.0.1 -2022-11-07T22:46:06.563Z [INFO] agent.router: Initializing LAN area manager -46:06.563 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba8bc033-618f-c29f-e457-8d85f2c29c92.dc1 area=wan -46:06.563 [INFO] TestAgent.server: Adding LAN server: server="Node-ba8bc033-618f-c29f-e457-8d85f2c29c92 (Addr: tcp/127.0.0.1:11041) (DC: dc1)" -46:06.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11120 network=udp -46:06.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11120 network=tcp -46:06.564 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11121 network=tcp -46:06.564 [INFO] TestAgent: started state syncer -46:06.564 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11042 network=tcp -46:06.618 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.618 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11041 [Candidate]" term=2 -46:06.630 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.630 [DEBUG] TestAgent.server.raft: vote granted: from=ba8bc033-618f-c29f-e457-8d85f2c29c92 term=2 tally=1 -46:06.630 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.630 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11041 [Leader]" -46:06.630 [INFO] TestAgent.server: cluster leadership acquired -46:06.630 [INFO] TestAgent.server: New leader elected: payload=Node-ba8bc033-618f-c29f-e457-8d85f2c29c92 -46:06.637 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11041 -46:06.651 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.669 [INFO] TestAgent: Synced node info -46:06.678 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.678 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.678 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.678 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.678 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.678 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ba8bc033-618f-c29f-e457-8d85f2c29c92 -46:06.679 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ba8bc033-618f-c29f-e457-8d85f2c29c92 -46:06.694 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.990 [INFO] TestAgent: Requesting shutdown -46:06.990 [INFO] TestAgent.server: shutting down server -46:06.990 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.990 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.990 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.990 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.990 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.990 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.995 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.005Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.005Z [INFO] agent.router.manager: shutting down -46:07.005 [INFO] TestAgent: consul server down -46:07.005 [INFO] TestAgent: shutdown complete -46:07.005 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11120 network=tcp -46:07.005 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11120 network=udp -46:07.005 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11121 network=tcp -46:07.506 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.506 [INFO] TestAgent: Endpoints down ---- PASS: TestStatusLeader (0.98s) -=== CONT TestSessionCustomTTL -2022-11-07T22:46:06.731Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.740Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.778 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f1cada67-6668-03e5-b4c5-1a4a071c11ac Address:127.0.0.1:11048}]" -46:06.778 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11048 [Follower]" leader= -46:06.779 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac.dc1 127.0.0.1 -46:06.779 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac 127.0.0.1 -2022-11-07T22:46:06.779Z [INFO] agent.router: Initializing LAN area manager -46:06.779 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11162 network=udp -46:06.779 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac.dc1 area=wan -46:06.779 [INFO] TestAgent.server: Adding LAN server: server="Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac (Addr: tcp/127.0.0.1:11048) (DC: dc1)" -46:06.780 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11162 network=tcp -46:06.780 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11163 network=tcp -46:06.780 [INFO] TestAgent: started state syncer -46:06.780 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11049 network=tcp -46:06.843 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.843 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11048 [Candidate]" term=2 -46:06.857 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.857 [DEBUG] TestAgent.server.raft: vote granted: from=f1cada67-6668-03e5-b4c5-1a4a071c11ac term=2 tally=1 -46:06.857 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.857 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11048 [Leader]" -46:06.857 [INFO] TestAgent.server: cluster leadership acquired -46:06.857 [INFO] TestAgent.server: New leader elected: payload=Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac -46:06.866 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11048 -46:06.886 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.924 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.924 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.924 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.924 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.924 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.924 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac -46:06.924 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f1cada67-6668-03e5-b4c5-1a4a071c11ac -46:06.941 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.014 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=129.362µs client=127.0.0.1:33372 client_network=udp -46:07.015 [DEBUG] TestAgent.dns: request served from client: name=49592b9a-af92-a0c9-e39f-440b5bd26875.query.consul. type=ANY class=IN latency=133.172µs client=127.0.0.1:50468 client_network=udp -46:07.015 [INFO] TestAgent: Requesting shutdown -46:07.015 [INFO] TestAgent.server: shutting down server -46:07.015 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.015 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.015 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.015 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.015 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.015 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.015 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.015 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.015 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.015 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.015 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.019 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.027Z [INFO] agent.router.manager: shutting down -46:07.027 [INFO] TestAgent: consul server down -46:07.027 [INFO] TestAgent: shutdown complete -46:07.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11162 network=tcp -46:07.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11162 network=udp -46:07.027 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11163 network=tcp -46:07.527 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.527 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_Dedup (0.81s) +2021-10-05T16:31:20.892Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.907Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.972 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:19bb75f0-20cf-30d6-bb73-61c9cc13ee6a Address:127.0.0.1:30127}]" +31:20.972 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30127 [Follower]" leader= +31:20.972 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a.dc1 127.0.0.1 +31:20.973 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a 127.0.0.1 +2021-10-05T16:31:20.973Z [INFO] agent.router: Initializing LAN area manager +31:20.973 [DEBUG] TestAgent.dns: recursor enabled +31:20.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30122 network=udp +31:20.973 [INFO] TestAgent.server: Adding LAN server: server="Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a (Addr: tcp/127.0.0.1:30127) (DC: dc1)" +31:20.973 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a.dc1 area=wan +31:20.973 [DEBUG] TestAgent.dns: recursor enabled +31:20.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30122 network=tcp +31:20.974 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30123 network=tcp +31:20.974 [INFO] TestAgent: started state syncer +31:20.974 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30128 network=tcp +31:21.015 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.015 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30127 [Candidate]" term=2 +31:21.072 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.072 [DEBUG] TestAgent.server.raft: vote granted: from=19bb75f0-20cf-30d6-bb73-61c9cc13ee6a term=2 tally=1 +31:21.072 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.072 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30127 [Leader]" +31:21.079 [INFO] TestAgent.server: cluster leadership acquired +31:21.079 [INFO] TestAgent.server: New leader elected: payload=Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a +31:21.105 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30127 +31:21.142 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.204 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.204 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.204 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.204 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.204 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.204 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a +31:21.204 [INFO] TestAgent.server: member joined, marking health alive: member=Node-19bb75f0-20cf-30d6-bb73-61c9cc13ee6a +31:21.230 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.242 [INFO] TestAgent: Synced node info +31:21.411 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=101.278µs recursor=127.0.0.1:45890 +31:21.411 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=212.951µs client=127.0.0.1:39350 client_network=udp +31:21.411 [INFO] TestAgent: Requesting shutdown +31:21.411 [INFO] TestAgent.server: shutting down server +31:21.411 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.411 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.411 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.411 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.411 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.411 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.412 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.412 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.418 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.428Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.428Z [INFO] agent.router.manager: shutting down +31:21.428 [INFO] TestAgent: consul server down +31:21.428 [INFO] TestAgent: shutdown complete +31:21.428 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30122 network=tcp +31:21.428 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30122 network=udp +31:21.428 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30123 network=tcp +31:21.932 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.932 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Recurse_Truncation (1.06s) +=== CONT TestDNS_PreparedQueryNearIP +2021-10-05T16:31:20.998Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:21.012Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:21.116 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f Address:127.0.0.1:30134}]" +31:21.117 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f.dc1 127.0.0.1 +31:21.117 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f 127.0.0.1 +2021-10-05T16:31:21.117Z [INFO] agent.router: Initializing LAN area manager +31:21.117 [DEBUG] TestAgent.dns: recursor enabled +31:21.117 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30129 network=udp +31:21.117 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30134 [Follower]" leader= +31:21.118 [INFO] TestAgent.server: Adding LAN server: server="Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f (Addr: tcp/127.0.0.1:30134) (DC: dc1)" +31:21.118 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f.dc1 area=wan +31:21.118 [DEBUG] TestAgent.dns: recursor enabled +31:21.118 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30129 network=tcp +31:21.119 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30130 network=tcp +31:21.119 [INFO] TestAgent: started state syncer +31:21.119 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30142 network=tcp +31:21.184 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.184 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30134 [Candidate]" term=2 +31:21.228 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.228 [DEBUG] TestAgent.server.raft: vote granted: from=8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f term=2 tally=1 +31:21.228 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.228 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30134 [Leader]" +31:21.229 [INFO] TestAgent.server: cluster leadership acquired +31:21.229 [INFO] TestAgent.server: New leader elected: payload=Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f +31:21.240 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30134 +31:21.271 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.336 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.336 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.336 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.336 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.336 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.336 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f +31:21.336 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8d0c015c-d3ef-2fbd-f11a-fda41f5fb24f +31:21.368 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.477 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:21.484 [INFO] TestAgent: Synced node info +31:21.484 [DEBUG] TestAgent: Node info in sync +31:21.493 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{apple.com. 255 1}" rtt=1.178502ms recursor=127.0.0.1:56456 +31:21.493 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=1.34386ms client=127.0.0.1:46774 client_network=udp +31:21.493 [INFO] TestAgent: Requesting shutdown +31:21.493 [INFO] TestAgent.server: shutting down server +31:21.493 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.493 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.493 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.493 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.493 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.493 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.493 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.493 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.493 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.493 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.501 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.508Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.508Z [INFO] agent.router.manager: shutting down +31:21.508 [INFO] TestAgent: consul server down +31:21.508 [INFO] TestAgent: shutdown complete +31:21.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30129 network=tcp +31:21.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30129 network=udp +31:21.508 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30130 network=tcp +31:22.012 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.012 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Recurse (1.03s) === CONT TestDNS_PreparedQueryNearIPEDNS -2022-11-07T22:46:06.136Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.144Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.186 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:be6da189-40b3-e362-1517-2ee4cd50a42b Address:127.0.0.1:11090}]" -46:06.186 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11090 [Follower]" leader= -46:06.186 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-be6da189-40b3-e362-1517-2ee4cd50a42b.secondary 127.0.0.1 -46:06.187 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-be6da189-40b3-e362-1517-2ee4cd50a42b 127.0.0.1 -2022-11-07T22:46:06.187Z [INFO] agent.router: Initializing LAN area manager -46:06.187 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-be6da189-40b3-e362-1517-2ee4cd50a42b.secondary area=wan -46:06.187 [INFO] TestAgent.server: Adding LAN server: server="Node-be6da189-40b3-e362-1517-2ee4cd50a42b (Addr: tcp/127.0.0.1:11090) (DC: secondary)" -46:06.187 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=tcp -46:06.187 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=udp -46:06.188 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11037 network=tcp -46:06.188 [INFO] TestAgent: started state syncer -46:06.188 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11091 network=tcp -46:06.231 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.231 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11090 [Candidate]" term=2 -46:06.255 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.255 [DEBUG] TestAgent.server.raft: vote granted: from=be6da189-40b3-e362-1517-2ee4cd50a42b term=2 tally=1 -46:06.255 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.255 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11090 [Leader]" -46:06.255 [INFO] TestAgent.server: cluster leadership acquired -46:06.255 [INFO] TestAgent.server: New leader elected: payload=Node-be6da189-40b3-e362-1517-2ee4cd50a42b -46:06.266 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11090 -46:06.291 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.314 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.314 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.314 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.314 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.314 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.314 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-be6da189-40b3-e362-1517-2ee4cd50a42b -46:06.314 [INFO] TestAgent.server: member joined, marking health alive: member=Node-be6da189-40b3-e362-1517-2ee4cd50a42b -46:06.328 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.454 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.460 [INFO] TestAgent: Synced node info -46:06.460 [DEBUG] TestAgent: Node info in sync -46:06.520 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:12004] -46:06.520 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:12004 -46:06.520 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9fa07102-538c-7f27-9fc9-de3f0989d674.primary 127.0.0.1 -46:06.521 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:06.521 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9fa07102-538c-7f27-9fc9-de3f0989d674.primary area=wan -46:06.523 [INFO] TestAgent: Requesting shutdown -46:06.523 [INFO] TestAgent.server: shutting down server -46:06.524 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.524 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.524 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.524 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.524 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.524 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.524 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.524 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.524 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.524 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.527 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.532Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.532Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.532Z [INFO] agent.router.manager: shutting down -46:06.532 [INFO] TestAgent: consul server down -46:06.532 [INFO] TestAgent: shutdown complete -46:06.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=tcp -46:06.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=udp -46:06.532 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11037 network=tcp -46:07.032 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.032 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:05.889Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.903Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.935 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9fa07102-538c-7f27-9fc9-de3f0989d674 Address:127.0.0.1:12005}]" -46:05.935 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12005 [Follower]" leader= -46:05.935 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9fa07102-538c-7f27-9fc9-de3f0989d674.primary 127.0.0.1 -46:05.936 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9fa07102-538c-7f27-9fc9-de3f0989d674 127.0.0.1 -2022-11-07T22:46:05.936Z [INFO] agent.router: Initializing LAN area manager -46:05.936 [INFO] TestAgent.server: Adding LAN server: server="Node-9fa07102-538c-7f27-9fc9-de3f0989d674 (Addr: tcp/127.0.0.1:12005) (DC: primary)" -46:05.936 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9fa07102-538c-7f27-9fc9-de3f0989d674.primary area=wan -46:05.936 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12000 network=udp -46:05.936 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12000 network=tcp -46:05.936 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12001 network=tcp -46:05.936 [INFO] TestAgent: started state syncer -46:05.936 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12006 network=tcp -46:05.979 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:05.979 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12005 [Candidate]" term=2 -46:05.992 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:05.992 [DEBUG] TestAgent.server.raft: vote granted: from=9fa07102-538c-7f27-9fc9-de3f0989d674 term=2 tally=1 -46:05.992 [INFO] TestAgent.server.raft: election won: tally=1 -46:05.992 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12005 [Leader]" -46:05.992 [INFO] TestAgent.server: cluster leadership acquired -46:05.992 [INFO] TestAgent.server: New leader elected: payload=Node-9fa07102-538c-7f27-9fc9-de3f0989d674 -46:06.000 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12005 -46:06.020 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.050 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.050 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.050 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.050 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.050 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.051 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9fa07102-538c-7f27-9fc9-de3f0989d674 -46:06.051 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9fa07102-538c-7f27-9fc9-de3f0989d674 -46:06.064 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.235 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.241 [INFO] TestAgent: Synced node info -46:06.520 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:52512 -46:06.520 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-be6da189-40b3-e362-1517-2ee4cd50a42b.secondary 127.0.0.1 -46:06.521 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-be6da189-40b3-e362-1517-2ee4cd50a42b.secondary area=wan -46:07.032 [INFO] TestAgent: Requesting shutdown -46:07.032 [INFO] TestAgent.server: shutting down server -46:07.032 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.032 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.032 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.032 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.032 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.032 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.037 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.043Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.043Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.043Z [INFO] agent.router.manager: shutting down -46:07.043 [INFO] TestAgent: consul server down -46:07.043 [INFO] TestAgent: shutdown complete -46:07.043 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12000 network=tcp -46:07.043 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12000 network=udp -46:07.043 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12001 network=tcp -46:07.544 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.544 [INFO] TestAgent: Endpoints down ---- PASS: TestStatusPeersSecondary (1.67s) -=== CONT TestTxnEndpoint_Bad_Size_Net -=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_default_limits -2022-11-07T22:46:06.810Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.818Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.868 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a59f0b80-8250-cb8a-38c4-087d5142969e Address:127.0.0.1:11146}]" -46:06.868 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11146 [Follower]" leader= -46:06.869 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a59f0b80-8250-cb8a-38c4-087d5142969e.dc1 127.0.0.1 -46:06.869 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a59f0b80-8250-cb8a-38c4-087d5142969e 127.0.0.1 -2022-11-07T22:46:06.869Z [INFO] agent.router: Initializing LAN area manager -46:06.869 [INFO] TestAgent.server: Adding LAN server: server="Node-a59f0b80-8250-cb8a-38c4-087d5142969e (Addr: tcp/127.0.0.1:11146) (DC: dc1)" -46:06.869 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a59f0b80-8250-cb8a-38c4-087d5142969e.dc1 area=wan -46:06.869 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11085 network=udp -46:06.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11085 network=tcp -46:06.870 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11086 network=tcp -46:06.870 [INFO] TestAgent: started state syncer -46:06.870 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11147 network=tcp -46:06.921 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.921 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11146 [Candidate]" term=2 -46:06.931 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.931 [DEBUG] TestAgent.server.raft: vote granted: from=a59f0b80-8250-cb8a-38c4-087d5142969e term=2 tally=1 -46:06.931 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.931 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11146 [Leader]" -46:06.932 [INFO] TestAgent.server: cluster leadership acquired -46:06.932 [INFO] TestAgent.server: New leader elected: payload=Node-a59f0b80-8250-cb8a-38c4-087d5142969e -46:06.940 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11146 -46:06.955 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.986 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.986 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.986 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.986 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.986 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.986 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a59f0b80-8250-cb8a-38c4-087d5142969e -46:06.987 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a59f0b80-8250-cb8a-38c4-087d5142969e -46:07.002 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.036 [INFO] TestAgent: Requesting shutdown -46:07.036 [INFO] TestAgent.server: shutting down server -46:07.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.036 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.036 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.036 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.036 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.036 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.036 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.036 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.036 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.041 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.046Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.046Z [INFO] agent.router.manager: shutting down -46:07.046 [INFO] TestAgent: consul server down -46:07.046 [INFO] TestAgent: shutdown complete -46:07.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11085 network=tcp -46:07.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11085 network=udp -46:07.046 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11086 network=tcp -46:07.546 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.546 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:06.711Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.728Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.760 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:57743aa4-d198-9623-7612-1307150de42f Address:127.0.0.1:11160}]" -46:06.760 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11160 [Follower]" leader= -46:06.761 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-57743aa4-d198-9623-7612-1307150de42f.dc1 127.0.0.1 -46:06.762 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-57743aa4-d198-9623-7612-1307150de42f 127.0.0.1 -2022-11-07T22:46:06.762Z [INFO] agent.router: Initializing LAN area manager -46:06.762 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-57743aa4-d198-9623-7612-1307150de42f.dc1 area=wan -46:06.762 [INFO] TestAgent.server: Adding LAN server: server="Node-57743aa4-d198-9623-7612-1307150de42f (Addr: tcp/127.0.0.1:11160) (DC: dc1)" -46:06.762 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11106 network=udp -46:06.762 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11106 network=tcp -46:06.762 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11107 network=tcp -46:06.763 [INFO] TestAgent: started state syncer -46:06.763 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11161 network=tcp -46:06.818 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.818 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11160 [Candidate]" term=2 -46:06.841 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.841 [DEBUG] TestAgent.server.raft: vote granted: from=57743aa4-d198-9623-7612-1307150de42f term=2 tally=1 -46:06.841 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.841 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11160 [Leader]" -46:06.841 [INFO] TestAgent.server: cluster leadership acquired -46:06.841 [INFO] TestAgent.server: New leader elected: payload=Node-57743aa4-d198-9623-7612-1307150de42f -46:06.856 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11160 -46:06.873 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.911 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.911 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.911 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.911 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.911 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.911 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-57743aa4-d198-9623-7612-1307150de42f -46:06.911 [INFO] TestAgent.server: member joined, marking health alive: member=Node-57743aa4-d198-9623-7612-1307150de42f -46:06.919 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.034 [INFO] TestAgent: Requesting shutdown -46:07.034 [INFO] TestAgent.server: shutting down server -46:07.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.034 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.034 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.034 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.034 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.034 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.034 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.034 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.041 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.046Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.046Z [INFO] agent.router.manager: shutting down -46:07.046 [INFO] TestAgent: consul server down -46:07.046 [INFO] TestAgent: shutdown complete -46:07.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11106 network=tcp -46:07.046 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11106 network=udp -46:07.046 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11107 network=tcp -46:07.546 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.546 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionDeleteDestroy (0.75s) -=== CONT TestServiceManager_PersistService_ConfigFiles ---- PASS: TestUiIndex (0.85s) -=== CONT TestSessionDestroy -2022-11-07T22:46:06.588Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.603Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.633 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8ff8043-a860-e1d7-50bf-f776b25200a6 Address:127.0.0.1:11139}]" -46:06.633 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11139 [Follower]" leader= -46:06.633 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8ff8043-a860-e1d7-50bf-f776b25200a6.dc1 127.0.0.1 -46:06.633 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8ff8043-a860-e1d7-50bf-f776b25200a6 127.0.0.1 -2022-11-07T22:46:06.633Z [INFO] agent.router: Initializing LAN area manager -46:06.634 [INFO] TestAgent.server: Adding LAN server: server="Node-f8ff8043-a860-e1d7-50bf-f776b25200a6 (Addr: tcp/127.0.0.1:11139) (DC: dc1)" -46:06.634 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=udp -46:06.634 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8ff8043-a860-e1d7-50bf-f776b25200a6.dc1 area=wan -46:06.634 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=tcp -46:06.634 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11044 network=tcp -46:06.634 [INFO] TestAgent: started state syncer -46:06.634 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11140 network=tcp -46:06.702 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.702 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11139 [Candidate]" term=2 -46:06.715 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.715 [DEBUG] TestAgent.server.raft: vote granted: from=f8ff8043-a860-e1d7-50bf-f776b25200a6 term=2 tally=1 -46:06.715 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.715 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11139 [Leader]" -46:06.716 [INFO] TestAgent.server: cluster leadership acquired -46:06.716 [INFO] TestAgent.server: New leader elected: payload=Node-f8ff8043-a860-e1d7-50bf-f776b25200a6 -46:06.726 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11139 -46:06.743 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.775 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.775 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.775 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.775 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.775 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.775 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8ff8043-a860-e1d7-50bf-f776b25200a6 -46:06.775 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8ff8043-a860-e1d7-50bf-f776b25200a6 -46:06.796 [INFO] TestAgent: Synced node info -46:06.796 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.060 [INFO] TestAgent: Requesting shutdown -46:07.060 [INFO] TestAgent.server: shutting down server -46:07.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.060 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.060 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.060 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.060 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.061 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.061 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.066 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.073Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.073Z [INFO] agent.router.manager: shutting down -46:07.073 [INFO] TestAgent: consul server down -46:07.073 [INFO] TestAgent: shutdown complete -46:07.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=tcp -46:07.073 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=udp -46:07.073 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11044 network=tcp -46:07.573 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.573 [INFO] TestAgent: Endpoints down ---- PASS: TestUiNodes (1.00s) -=== CONT TestDNS_ServiceLookup_TagPeriod -[WARN] freeport: 10 out of 80 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:06.784Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.793Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.840 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e7e838f6-8b7d-d693-1324-ff2c246a4096 Address:127.0.0.1:11083}]" -46:06.840 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11083 [Follower]" leader= -46:06.841 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e7e838f6-8b7d-d693-1324-ff2c246a4096.dc1 127.0.0.1 -46:06.841 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e7e838f6-8b7d-d693-1324-ff2c246a4096 127.0.0.1 -2022-11-07T22:46:06.841Z [INFO] agent.router: Initializing LAN area manager -46:06.842 [INFO] TestAgent.server: Adding LAN server: server="Node-e7e838f6-8b7d-d693-1324-ff2c246a4096 (Addr: tcp/127.0.0.1:11083) (DC: dc1)" -46:06.842 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e7e838f6-8b7d-d693-1324-ff2c246a4096.dc1 area=wan -46:06.842 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=udp -46:06.842 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=tcp -46:06.842 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11051 network=tcp -46:06.842 [INFO] TestAgent: started state syncer -46:06.842 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11084 network=tcp -46:06.882 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.882 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11083 [Candidate]" term=2 -46:06.898 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.898 [DEBUG] TestAgent.server.raft: vote granted: from=e7e838f6-8b7d-d693-1324-ff2c246a4096 term=2 tally=1 -46:06.898 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.898 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11083 [Leader]" -46:06.898 [INFO] TestAgent.server: cluster leadership acquired -46:06.898 [INFO] TestAgent.server: New leader elected: payload=Node-e7e838f6-8b7d-d693-1324-ff2c246a4096 -46:06.905 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11083 -46:06.919 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.939 [INFO] TestAgent: Synced node info -46:06.951 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.951 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.951 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.951 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.951 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.951 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e7e838f6-8b7d-d693-1324-ff2c246a4096 -46:06.951 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e7e838f6-8b7d-d693-1324-ff2c246a4096 -46:06.967 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.109 [INFO] TestAgent: Requesting shutdown -46:07.109 [INFO] TestAgent.server: shutting down server -46:07.109 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.109 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.109 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.109 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.109 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.109 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.109 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.109 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.109 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.109 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.113 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.118Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.118Z [INFO] agent.router.manager: shutting down -46:07.119 [INFO] TestAgent: consul server down -46:07.119 [INFO] TestAgent: shutdown complete -46:07.119 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=tcp -46:07.119 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=udp -46:07.119 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11051 network=tcp -46:07.620 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.620 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_UpdateCheck (0.85s) -=== CONT TestConfig_Delete -2022-11-07T22:46:06.991Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.002Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.031 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:01e3f52b-35ad-fbfe-6614-3b16104d071e Address:127.0.0.1:11153}]" -46:07.031 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11153 [Follower]" leader= -46:07.031 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-01e3f52b-35ad-fbfe-6614-3b16104d071e.dc1 127.0.0.1 -46:07.032 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-01e3f52b-35ad-fbfe-6614-3b16104d071e 127.0.0.1 -2022-11-07T22:46:07.032Z [INFO] agent.router: Initializing LAN area manager -46:07.032 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-01e3f52b-35ad-fbfe-6614-3b16104d071e.dc1 area=wan -46:07.032 [INFO] TestAgent.server: Adding LAN server: server="Node-01e3f52b-35ad-fbfe-6614-3b16104d071e (Addr: tcp/127.0.0.1:11153) (DC: dc1)" -46:07.032 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=udp -46:07.032 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=tcp -46:07.033 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11058 network=tcp -46:07.033 [INFO] TestAgent: started state syncer -46:07.033 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11154 network=tcp -46:07.068 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.068 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11153 [Candidate]" term=2 -46:07.082 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.082 [DEBUG] TestAgent.server.raft: vote granted: from=01e3f52b-35ad-fbfe-6614-3b16104d071e term=2 tally=1 -46:07.082 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.082 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11153 [Leader]" -46:07.082 [INFO] TestAgent.server: cluster leadership acquired -46:07.082 [INFO] TestAgent.server: New leader elected: payload=Node-01e3f52b-35ad-fbfe-6614-3b16104d071e -46:07.088 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11153 -46:07.098 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.123 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.123 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.123 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.123 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.123 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.123 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-01e3f52b-35ad-fbfe-6614-3b16104d071e -46:07.123 [INFO] TestAgent.server: member joined, marking health alive: member=Node-01e3f52b-35ad-fbfe-6614-3b16104d071e -46:07.133 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.156 [INFO] TestAgent: Requesting shutdown -46:07.156 [INFO] TestAgent.server: shutting down server -46:07.156 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.156 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.156 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.156 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.156 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.156 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.156 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.156 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.156 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.156 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.156 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.162 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.170Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.170Z [INFO] agent.router.manager: shutting down -46:07.170 [INFO] TestAgent: consul server down -46:07.170 [INFO] TestAgent: shutdown complete -46:07.170 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=tcp -46:07.170 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=udp -46:07.170 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11058 network=tcp -46:07.670 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.670 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_KV_Actions/#01 -Added 3 service nodes -2022-11-07T22:46:06.862Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.871Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.914 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:26908e0e-1705-548a-3e73-96a44c880e06 Address:127.0.0.1:11069}]" -46:06.914 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11069 [Follower]" leader= -46:06.914 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-26908e0e-1705-548a-3e73-96a44c880e06.dc1 127.0.0.1 -46:06.915 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-26908e0e-1705-548a-3e73-96a44c880e06 127.0.0.1 -2022-11-07T22:46:06.915Z [INFO] agent.router: Initializing LAN area manager -46:06.915 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11148 network=udp -46:06.915 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11148 network=tcp -46:06.915 [INFO] TestAgent.server: Adding LAN server: server="Node-26908e0e-1705-548a-3e73-96a44c880e06 (Addr: tcp/127.0.0.1:11069) (DC: dc1)" -46:06.915 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-26908e0e-1705-548a-3e73-96a44c880e06.dc1 area=wan -46:06.915 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11149 network=tcp -46:06.916 [INFO] TestAgent: started state syncer -46:06.916 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11070 network=tcp -46:06.954 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.954 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11069 [Candidate]" term=2 -46:06.968 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.968 [DEBUG] TestAgent.server.raft: vote granted: from=26908e0e-1705-548a-3e73-96a44c880e06 term=2 tally=1 -46:06.968 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.968 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11069 [Leader]" -46:06.969 [INFO] TestAgent.server: cluster leadership acquired -46:06.969 [INFO] TestAgent.server: New leader elected: payload=Node-26908e0e-1705-548a-3e73-96a44c880e06 -46:06.975 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11069 -46:06.993 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.024 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.024 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.024 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.024 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.024 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.024 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-26908e0e-1705-548a-3e73-96a44c880e06 -46:07.024 [INFO] TestAgent.server: member joined, marking health alive: member=Node-26908e0e-1705-548a-3e73-96a44c880e06 -46:07.032 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.253 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=SRV class=IN latency=123.007µs client=127.0.0.1:38811 client_network=udp -46:07.253 [INFO] TestAgent: Requesting shutdown -46:07.253 [INFO] TestAgent.server: shutting down server -46:07.253 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.253 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.253 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.253 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.253 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.253 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.253 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.253 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.253 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.253 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.253 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.256 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.261Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.261Z [INFO] agent.router.manager: shutting down -46:07.261 [INFO] TestAgent: consul server down -46:07.261 [INFO] TestAgent: shutdown complete -46:07.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11148 network=tcp -46:07.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11148 network=udp -46:07.261 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11149 network=tcp -46:07.761 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.761 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_PreparedQueryNamePeriod (0.91s) -=== CONT TestHandleRemoteExecFailed -2022-11-07T22:46:07.127Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.136Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.177 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab0e5ba2-d63e-b895-14af-f921f31b4ae8 Address:127.0.0.1:11209}]" -46:07.177 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11209 [Follower]" leader= -46:07.178 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8.dc1 127.0.0.1 -46:07.178 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8 127.0.0.1 -2022-11-07T22:46:07.178Z [INFO] agent.router: Initializing LAN area manager -46:07.178 [INFO] TestAgent.server: Adding LAN server: server="Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8 (Addr: tcp/127.0.0.1:11209) (DC: dc1)" -46:07.178 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8.dc1 area=wan -46:07.178 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11064 network=udp -46:07.178 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11064 network=tcp -46:07.179 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11065 network=tcp -46:07.179 [INFO] TestAgent: started state syncer -46:07.179 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11210 network=tcp -46:07.223 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.223 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11209 [Candidate]" term=2 -46:07.232 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.232 [DEBUG] TestAgent.server.raft: vote granted: from=ab0e5ba2-d63e-b895-14af-f921f31b4ae8 term=2 tally=1 -46:07.232 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.232 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11209 [Leader]" -46:07.232 [INFO] TestAgent.server: cluster leadership acquired -46:07.232 [INFO] TestAgent.server: New leader elected: payload=Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8 -46:07.238 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11209 -46:07.253 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.274 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.274 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.274 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.274 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.274 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.274 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8 -46:07.274 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab0e5ba2-d63e-b895-14af-f921f31b4ae8 -46:07.284 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.319 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=A class=IN latency=159.3µs client=127.0.0.1:37309 client_network=udp -46:07.319 [INFO] TestAgent: Requesting shutdown -46:07.319 [INFO] TestAgent.server: shutting down server -46:07.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.320 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.320 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.320 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.320 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.320 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.320 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.323 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.327Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.327Z [INFO] agent.router.manager: shutting down -46:07.327 [INFO] TestAgent: consul server down -46:07.327 [INFO] TestAgent: shutdown complete -46:07.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11064 network=tcp -46:07.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11064 network=udp -46:07.327 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11065 network=tcp -46:07.827 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.827 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQueryNearIP (0.71s) -=== CONT TestHandleRemoteExec -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:06.641Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.652Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.690 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:508a9611-baf4-803d-2821-f606d8c2d6cf Address:127.0.0.1:11062}]" -46:06.690 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11062 [Follower]" leader= -46:06.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-508a9611-baf4-803d-2821-f606d8c2d6cf.secondary 127.0.0.1 -46:06.691 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-508a9611-baf4-803d-2821-f606d8c2d6cf 127.0.0.1 -2022-11-07T22:46:06.691Z [INFO] agent.router: Initializing LAN area manager -46:06.692 [INFO] TestAgent.server: Adding LAN server: server="Node-508a9611-baf4-803d-2821-f606d8c2d6cf (Addr: tcp/127.0.0.1:11062) (DC: secondary)" -46:06.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-508a9611-baf4-803d-2821-f606d8c2d6cf.secondary area=wan -46:06.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11099 network=tcp -46:06.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11099 network=udp -46:06.692 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11100 network=tcp -46:06.692 [INFO] TestAgent: started state syncer -46:06.692 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11063 network=tcp -46:06.757 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.757 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11062 [Candidate]" term=2 -46:06.771 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.771 [DEBUG] TestAgent.server.raft: vote granted: from=508a9611-baf4-803d-2821-f606d8c2d6cf term=2 tally=1 -46:06.771 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.771 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11062 [Leader]" -46:06.771 [INFO] TestAgent.server: cluster leadership acquired -46:06.771 [INFO] TestAgent.server: New leader elected: payload=Node-508a9611-baf4-803d-2821-f606d8c2d6cf -46:06.784 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11062 -46:06.808 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.856 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.856 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.856 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.856 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.856 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.856 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-508a9611-baf4-803d-2821-f606d8c2d6cf -46:06.856 [INFO] TestAgent.server: member joined, marking health alive: member=Node-508a9611-baf4-803d-2821-f606d8c2d6cf -46:06.866 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.890 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11075] -46:06.891 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11075 -46:06.891 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4.primary 127.0.0.1 -46:06.891 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:06.891 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4.primary area=wan -46:06.893 [INFO] TestAgent: Requesting shutdown -46:06.893 [INFO] TestAgent.server: shutting down server -46:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.893 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.893 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.893 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.893 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.894 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.899 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.909Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.909Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.909Z [INFO] agent.router.manager: shutting down -46:06.909 [INFO] TestAgent: consul server down -46:06.909 [INFO] TestAgent: shutdown complete -46:06.909 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11099 network=tcp -46:06.909 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11099 network=udp -46:06.909 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11100 network=tcp -46:07.409 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.409 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:06.218Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:06.227Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.282 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 Address:127.0.0.1:11076}]" -46:06.282 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11076 [Follower]" leader= -46:06.282 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4.primary 127.0.0.1 -46:06.283 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 127.0.0.1 -2022-11-07T22:46:06.283Z [INFO] agent.router: Initializing LAN area manager -46:06.283 [INFO] TestAgent.server: Adding LAN server: server="Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 (Addr: tcp/127.0.0.1:11076) (DC: primary)" -46:06.283 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4.primary area=wan -46:06.283 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11127 network=udp -46:06.283 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11127 network=tcp -46:06.283 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11128 network=tcp -46:06.283 [INFO] TestAgent: started state syncer -46:06.283 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11077 network=tcp -46:06.345 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.345 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11076 [Candidate]" term=2 -46:06.358 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.358 [DEBUG] TestAgent.server.raft: vote granted: from=5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 term=2 tally=1 -46:06.358 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.358 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11076 [Leader]" -46:06.359 [INFO] TestAgent.server: cluster leadership acquired -46:06.359 [INFO] TestAgent.server: New leader elected: payload=Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 -46:06.368 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11076 -46:06.381 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.402 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.402 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.402 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.402 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.402 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.402 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 -46:06.402 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5a1275f6-8c88-a03e-0cc3-c07b6757d2c4 -46:06.417 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.418 [INFO] TestAgent: Synced node info -46:06.891 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:54180 -46:06.891 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-508a9611-baf4-803d-2821-f606d8c2d6cf.secondary 127.0.0.1 -46:06.891 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-508a9611-baf4-803d-2821-f606d8c2d6cf.secondary area=wan -46:07.409 [INFO] TestAgent: Requesting shutdown -46:07.409 [INFO] TestAgent.server: shutting down server -46:07.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.409 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.409 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.409 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.409 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.409 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.409 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.413 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.416Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.416Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.416Z [INFO] agent.router.manager: shutting down -46:07.417 [INFO] TestAgent: consul server down -46:07.417 [INFO] TestAgent: shutdown complete -46:07.417 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11127 network=tcp -46:07.417 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11127 network=udp -46:07.417 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11128 network=tcp -46:07.917 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.917 [INFO] TestAgent: Endpoints down ---- PASS: TestStatusLeaderSecondary (1.71s) -=== CONT TestSessionCreate_NoCheck -2022-11-07T22:46:07.046Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.055Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.083 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7ef1575-b9bb-7d12-d399-65805fef6b9f Address:127.0.0.1:11055}]" -46:07.083 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11055 [Follower]" leader= -46:07.084 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f.dc1 127.0.0.1 -46:07.085 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f 127.0.0.1 -2022-11-07T22:46:07.085Z [INFO] agent.router: Initializing LAN area manager -46:07.085 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f.dc1 area=wan -46:07.085 [INFO] TestAgent.server: Adding LAN server: server="Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f (Addr: tcp/127.0.0.1:11055) (DC: dc1)" -46:07.085 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11155 network=udp -46:07.085 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11155 network=tcp -46:07.085 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11156 network=tcp -46:07.085 [INFO] TestAgent: started state syncer -46:07.085 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11056 network=tcp -46:07.143 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.143 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11055 [Candidate]" term=2 -46:07.166 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.166 [DEBUG] TestAgent.server.raft: vote granted: from=c7ef1575-b9bb-7d12-d399-65805fef6b9f term=2 tally=1 -46:07.166 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.166 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11055 [Leader]" -46:07.166 [INFO] TestAgent.server: cluster leadership acquired -46:07.166 [INFO] TestAgent.server: New leader elected: payload=Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f -46:07.176 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11055 -46:07.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.208 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.208 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.208 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.208 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.208 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.208 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f -46:07.208 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7ef1575-b9bb-7d12-d399-65805fef6b9f -46:07.213 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.301 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.307 [INFO] TestAgent: Synced node info -46:07.307 [DEBUG] TestAgent: Node info in sync -46:07.482 [INFO] TestAgent: Requesting shutdown -46:07.482 [INFO] TestAgent.server: shutting down server -46:07.482 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.482 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.482 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.482 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.482 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.482 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.482 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.482 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.482 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.482 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.486 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.490Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.490Z [INFO] agent.router.manager: shutting down -46:07.490 [INFO] TestAgent: consul server down -46:07.490 [INFO] TestAgent: shutdown complete -46:07.490 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11155 network=tcp -46:07.490 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11155 network=udp -46:07.490 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11156 network=tcp -46:07.991 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.991 [INFO] TestAgent: Endpoints down -=== RUN TestSessionGet/#01 -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestSessionCreate_NoCheck/no_check_fields_should_yield_default_serfHealth -=== RUN TestSessionCreate_NoCheck/overwrite_nodechecks_to_associate_with_no_checks -=== RUN TestSessionCreate_NoCheck/overwrite_checks_to_associate_with_no_checks -2022-11-07T22:46:07.429Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.437Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.470 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef88622b-bf2f-dbcc-90da-84f72b9b763e Address:127.0.0.1:11195}]" -46:07.470 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11195 [Follower]" leader= -46:07.470 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e.dc1 127.0.0.1 -46:07.471 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e 127.0.0.1 -2022-11-07T22:46:07.471Z [INFO] agent.router: Initializing LAN area manager -46:07.471 [INFO] TestAgent.server: Adding LAN server: server="Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e (Addr: tcp/127.0.0.1:11195) (DC: dc1)" -46:07.471 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e.dc1 area=wan -46:07.471 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11176 network=tcp -46:07.471 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11176 network=udp -46:07.471 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11177 network=tcp -46:07.471 [INFO] TestAgent: started state syncer -46:07.471 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11196 network=tcp -46:07.506 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.506 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11195 [Candidate]" term=2 -46:07.516 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.516 [DEBUG] TestAgent.server.raft: vote granted: from=ef88622b-bf2f-dbcc-90da-84f72b9b763e term=2 tally=1 -46:07.516 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.516 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11195 [Leader]" -46:07.516 [INFO] TestAgent.server: cluster leadership acquired -46:07.516 [INFO] TestAgent.server: New leader elected: payload=Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e -46:07.523 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11195 -46:07.542 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.572 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.572 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.572 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.572 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.572 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.572 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e -46:07.572 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef88622b-bf2f-dbcc-90da-84f72b9b763e -46:07.586 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.662 [INFO] TestAgent: Requesting shutdown -46:07.663 [INFO] TestAgent.server: shutting down server -46:07.663 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.663 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.663 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.663 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.663 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.663 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.663 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.663 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.663 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.663 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.663 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.670 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.678Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.678Z [INFO] agent.router.manager: shutting down -46:07.678 [INFO] TestAgent: consul server down -46:07.678 [INFO] TestAgent: shutdown complete -46:07.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11176 network=tcp -46:07.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11176 network=udp -46:07.678 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11177 network=tcp -46:08.179 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.179 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCreate (0.76s) -=== CONT TestConfig_Get -2022-11-07T22:46:07.501Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:07.501Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.512Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:07.512Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.546 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0fb0cc88-d93f-706d-eb5f-40e5171ffd7f Address:127.0.0.1:11188}]" -46:07.546 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11188 [Follower]" leader= -46:07.547 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f.dc1 127.0.0.1 -46:07.547 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f 127.0.0.1 -2022-11-07T22:46:07.547Z [INFO] agent.router: Initializing LAN area manager -46:07.548 [INFO] TestAgent.server: Adding LAN server: server="Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f (Addr: tcp/127.0.0.1:11188) (DC: dc1)" -46:07.548 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f.dc1 area=wan -46:07.548 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11197 network=udp -46:07.548 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11197 network=tcp -46:07.548 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11198 network=tcp -46:07.548 [INFO] TestAgent: started state syncer -46:07.548 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11189 network=tcp -46:07.586 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.586 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11188 [Candidate]" term=2 -46:07.598 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:07.604 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.604 [DEBUG] TestAgent.server.raft: vote granted: from=0fb0cc88-d93f-706d-eb5f-40e5171ffd7f term=2 tally=1 -46:07.604 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.604 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11188 [Leader]" -46:07.604 [INFO] TestAgent.server: cluster leadership acquired -46:07.605 [INFO] TestAgent.server: New leader elected: payload=Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f -46:07.616 [INFO] TestAgent.server: initializing acls -46:07.621 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:07.621 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:07.628 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:07.638 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:07.638 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:07.638 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:07.638 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f -46:07.638 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f.dc1 -46:07.638 [INFO] TestAgent.server: Updating LAN server: server="Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f (Addr: tcp/127.0.0.1:11188) (DC: dc1)" -46:07.638 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0fb0cc88-d93f-706d-eb5f-40e5171ffd7f.dc1 area=wan -46:07.648 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.650 [INFO] TestAgent: Requesting shutdown -46:07.651 [INFO] TestAgent.server: shutting down server -46:07.651 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:07.651 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:07.651 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.651 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.651 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:07.651 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:07.661 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.670Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.670Z [INFO] agent.router.manager: shutting down -46:07.679 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" -46:07.679 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" -46:07.679 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:07.679 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" -46:07.679 [INFO] TestAgent: consul server down -46:07.679 [INFO] TestAgent: shutdown complete -46:07.679 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11197 network=tcp -46:07.679 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11197 network=udp -46:07.679 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11198 network=tcp -46:08.182 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.182 [INFO] TestAgent: Endpoints down -=== RUN TestSnapshot_Options/GET#02 -2022-11-07T22:46:07.343Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.352Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.376 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cfaa456c-75b7-ce9a-f7b1-083837ea05c2 Address:127.0.0.1:11181}]" -46:07.376 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11181 [Follower]" leader= -46:07.376 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2.dc1 127.0.0.1 -46:07.377 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2 127.0.0.1 -2022-11-07T22:46:07.377Z [INFO] agent.router: Initializing LAN area manager -46:07.377 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11211 network=udp -46:07.377 [INFO] TestAgent.server: Adding LAN server: server="Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2 (Addr: tcp/127.0.0.1:11181) (DC: dc1)" -46:07.377 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2.dc1 area=wan -46:07.377 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11211 network=tcp -46:07.378 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11212 network=tcp -46:07.378 [INFO] TestAgent: started state syncer -46:07.378 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11182 network=tcp -46:07.416 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.416 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11181 [Candidate]" term=2 -46:07.428 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.428 [DEBUG] TestAgent.server.raft: vote granted: from=cfaa456c-75b7-ce9a-f7b1-083837ea05c2 term=2 tally=1 -46:07.428 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11181 [Leader]" -46:07.428 [INFO] TestAgent.server: cluster leadership acquired -46:07.428 [INFO] TestAgent.server: New leader elected: payload=Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2 -46:07.434 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11181 -46:07.446 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.473 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.473 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.473 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.473 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.473 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.473 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2 -46:07.473 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cfaa456c-75b7-ce9a-f7b1-083837ea05c2 -46:07.490 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.624 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.628 [INFO] TestAgent: Synced node info -46:07.628 [DEBUG] TestAgent: Node info in sync -46:07.686 [INFO] TestAgent: Requesting shutdown -46:07.686 [INFO] TestAgent.server: shutting down server -46:07.686 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.686 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.686 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.686 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.686 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.686 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.686 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.686 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.686 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.686 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.692 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.702Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.702Z [INFO] agent.router.manager: shutting down -46:07.702 [INFO] TestAgent: consul server down -46:07.702 [INFO] TestAgent: shutdown complete -46:07.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11211 network=tcp -46:07.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11211 network=udp -46:07.702 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11212 network=tcp -46:08.202 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.202 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_Bad_Size_Ops (0.87s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:20.041Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.084Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.142 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb Address:127.0.0.1:30078}]" +31:20.142 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30078 [Follower]" leader= +31:20.143 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb.dc1 127.0.0.1 +31:20.144 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb 127.0.0.1 +2021-10-05T16:31:20.144Z [INFO] agent.router: Initializing LAN area manager +31:20.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30073 network=udp +31:20.144 [INFO] TestAgent.server: Adding LAN server: server="Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb (Addr: tcp/127.0.0.1:30078) (DC: dc1)" +31:20.145 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb.dc1 area=wan +31:20.145 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30073 network=tcp +31:20.145 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30074 network=tcp +31:20.145 [INFO] TestAgent: started state syncer +31:20.145 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30086 network=tcp +31:20.180 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.180 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30078 [Candidate]" term=2 +31:20.213 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.213 [DEBUG] TestAgent.server.raft: vote granted: from=5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb term=2 tally=1 +31:20.213 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.213 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30078 [Leader]" +31:20.214 [INFO] TestAgent.server: cluster leadership acquired +31:20.214 [INFO] TestAgent.server: New leader elected: payload=Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb +31:20.237 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30078 +31:20.272 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.330 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.330 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.330 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.330 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.330 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.331 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb +31:20.331 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5a0dc513-8a3a-f4cc-fdf0-b126cc0408bb +31:20.367 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.529 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.536 [INFO] TestAgent: Synced node info +31:20.536 [DEBUG] TestAgent: Node info in sync +31:21.737 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.443149ms client=127.0.0.1:48639 client_network=udp +31:21.746 [DEBUG] TestAgent.dns: request served from client: name=30adc3a5-45f1-47c4-068a-98d5c2c08872.query.consul. type=ANY class=IN latency=1.299808ms client=127.0.0.1:54926 client_network=udp +31:21.746 [INFO] TestAgent: Requesting shutdown +31:21.746 [INFO] TestAgent.server: shutting down server +31:21.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.746 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.746 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.746 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.746 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.746 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.746 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.757 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.759Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.759Z [INFO] agent.router.manager: shutting down +31:21.759 [INFO] TestAgent: consul server down +31:21.759 [INFO] TestAgent: shutdown complete +31:21.759 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30073 network=tcp +31:21.759 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30073 network=udp +31:21.759 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30074 network=tcp +31:22.259 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.259 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_Truncate (2.24s) === CONT TestDNS_CaseInsensitiveServiceLookup -2022-11-07T22:46:07.538Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.577 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2dcd13b1-11e8-8ad5-f155-e2c411aa5073 Address:127.0.0.1:11230}]" -46:07.577 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11230 [Follower]" leader= -46:07.577 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073.dc1 127.0.0.1 -46:07.578 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073 127.0.0.1 -2022-11-07T22:46:07.578Z [INFO] agent.router: Initializing LAN area manager -46:07.578 [INFO] TestAgent.server: Adding LAN server: server="Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073 (Addr: tcp/127.0.0.1:11230) (DC: dc1)" -46:07.578 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073.dc1 area=wan -46:07.578 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11225 network=udp -46:07.578 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11225 network=tcp -46:07.578 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11226 network=tcp -46:07.578 [INFO] TestAgent: started state syncer -46:07.578 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11231 network=tcp -46:07.616 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.616 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11230 [Candidate]" term=2 -46:07.629 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.629 [DEBUG] TestAgent.server.raft: vote granted: from=2dcd13b1-11e8-8ad5-f155-e2c411aa5073 term=2 tally=1 -46:07.629 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.629 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11230 [Leader]" -46:07.629 [INFO] TestAgent.server: cluster leadership acquired -46:07.629 [INFO] TestAgent.server: New leader elected: payload=Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073 -46:07.637 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11230 -46:07.654 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.696 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.696 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.696 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.696 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.696 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.696 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073 -46:07.696 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2dcd13b1-11e8-8ad5-f155-e2c411aa5073 -46:07.722 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.772 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=A class=IN latency=202.963µs client=127.0.0.1:58380 client_network=udp -46:07.772 [INFO] TestAgent: Requesting shutdown -46:07.772 [INFO] TestAgent.server: shutting down server -46:07.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.772 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.772 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.772 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.772 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.772 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.772 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.772 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.772 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.772 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.782 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.789Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.789Z [INFO] agent.router.manager: shutting down -46:07.789 [INFO] TestAgent: consul server down -46:07.789 [INFO] TestAgent: shutdown complete -46:07.789 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11225 network=tcp -46:07.789 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11225 network=udp -46:07.789 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11226 network=tcp -46:08.290 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.290 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_PreparedQueryNearIPEDNS (0.76s) -=== CONT TestRemoteExecWrites_ACLDeny -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:07.690Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.704Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.756 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c130130e-9d31-c7aa-fde3-f8c448087043 Address:127.0.0.1:11251}]" -46:07.756 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11251 [Follower]" leader= -46:07.757 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c130130e-9d31-c7aa-fde3-f8c448087043.dc1 127.0.0.1 -46:07.757 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c130130e-9d31-c7aa-fde3-f8c448087043 127.0.0.1 -2022-11-07T22:46:07.757Z [INFO] agent.router: Initializing LAN area manager -46:07.757 [INFO] TestAgent.server: Adding LAN server: server="Node-c130130e-9d31-c7aa-fde3-f8c448087043 (Addr: tcp/127.0.0.1:11251) (DC: dc1)" -46:07.757 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c130130e-9d31-c7aa-fde3-f8c448087043.dc1 area=wan -46:07.757 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11274 network=tcp -46:07.757 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11274 network=udp -46:07.758 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11275 network=tcp -46:07.758 [INFO] TestAgent: started state syncer -46:07.758 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11252 network=tcp -46:07.805 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.805 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11251 [Candidate]" term=2 -46:07.826 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.826 [DEBUG] TestAgent.server.raft: vote granted: from=c130130e-9d31-c7aa-fde3-f8c448087043 term=2 tally=1 -46:07.826 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.826 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11251 [Leader]" -46:07.826 [INFO] TestAgent.server: cluster leadership acquired -46:07.826 [INFO] TestAgent.server: New leader elected: payload=Node-c130130e-9d31-c7aa-fde3-f8c448087043 -46:07.833 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11251 -46:07.840 [INFO] TestAgent: Requesting shutdown -46:07.840 [INFO] TestAgent.server: shutting down server -46:07.840 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.840 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.843 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.848Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.848Z [INFO] agent.router.manager: shutting down -46:07.855 [INFO] TestAgent: consul server down -46:07.855 [INFO] TestAgent: shutdown complete -46:07.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11274 network=tcp -46:07.855 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" -46:07.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11274 network=udp -46:07.855 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11275 network=tcp -46:08.356 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.356 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_KV_Actions (1.38s) - --- PASS: TestTxnEndpoint_KV_Actions/#00 (0.69s) - --- PASS: TestTxnEndpoint_KV_Actions/#01 (0.69s) -=== CONT TestSessionCreate_DefaultCheck -2022-11-07T22:46:07.383Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.397Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.424 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:26451189-d2b5-0e11-ec51-b807dbd05f72 Address:127.0.0.1:11167}]" -46:07.424 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-26451189-d2b5-0e11-ec51-b807dbd05f72.dc1 127.0.0.1 -46:07.425 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-26451189-d2b5-0e11-ec51-b807dbd05f72 127.0.0.1 -2022-11-07T22:46:07.425Z [INFO] agent.router: Initializing LAN area manager -46:07.425 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11167 [Follower]" leader= -46:07.425 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-26451189-d2b5-0e11-ec51-b807dbd05f72.dc1 area=wan -46:07.425 [INFO] TestAgent.server: Adding LAN server: server="Node-26451189-d2b5-0e11-ec51-b807dbd05f72 (Addr: tcp/127.0.0.1:11167) (DC: dc1)" -46:07.425 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11169 network=tcp -46:07.425 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11169 network=udp -46:07.426 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11170 network=tcp -46:07.426 [INFO] TestAgent: started state syncer -46:07.426 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11168 network=tcp -46:07.464 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.464 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11167 [Candidate]" term=2 -46:07.473 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.473 [DEBUG] TestAgent.server.raft: vote granted: from=26451189-d2b5-0e11-ec51-b807dbd05f72 term=2 tally=1 -46:07.473 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.473 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11167 [Leader]" -46:07.473 [INFO] TestAgent.server: cluster leadership acquired -46:07.473 [INFO] TestAgent.server: New leader elected: payload=Node-26451189-d2b5-0e11-ec51-b807dbd05f72 -46:07.480 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11167 -46:07.499 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.522 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.522 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.522 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.522 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.522 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.522 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-26451189-d2b5-0e11-ec51-b807dbd05f72 -46:07.522 [INFO] TestAgent.server: member joined, marking health alive: member=Node-26451189-d2b5-0e11-ec51-b807dbd05f72 -46:07.537 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.674 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.682 [INFO] TestAgent: Synced node info -46:07.710 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.710 [DEBUG] TestAgent: Node info in sync -46:07.710 [DEBUG] TestAgent: Node info in sync -46:07.845 [INFO] TestAgent: Requesting shutdown -46:07.845 [INFO] TestAgent.server: shutting down server -46:07.845 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.845 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.845 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.845 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.845 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.846 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.846 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.846 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.846 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.846 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.851 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.856Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.856Z [INFO] agent.router.manager: shutting down -46:07.857 [INFO] TestAgent: consul server down -46:07.857 [INFO] TestAgent: shutdown complete -46:07.857 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11169 network=tcp -46:07.857 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11169 network=udp -46:07.857 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11170 network=tcp -46:08.357 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.357 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Item/allowed -2022-11-07T22:46:07.556Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.565Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.604 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f63a2e54-1684-b5d4-df69-8dcb76663eb0 Address:127.0.0.1:11174}]" -46:07.605 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11174 [Follower]" leader= -46:07.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0.dc1 127.0.0.1 -46:07.606 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0 127.0.0.1 -2022-11-07T22:46:07.606Z [INFO] agent.router: Initializing LAN area manager -46:07.606 [INFO] TestAgent.server: Adding LAN server: server="Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0 (Addr: tcp/127.0.0.1:11174) (DC: dc1)" -46:07.606 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0.dc1 area=wan -46:07.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11267 network=udp -46:07.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11267 network=tcp -46:07.608 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11268 network=tcp -46:07.608 [INFO] TestAgent: started state syncer -46:07.608 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11175 network=tcp -46:07.647 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.647 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11174 [Candidate]" term=2 -46:07.658 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.658 [DEBUG] TestAgent.server.raft: vote granted: from=f63a2e54-1684-b5d4-df69-8dcb76663eb0 term=2 tally=1 -46:07.658 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.658 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11174 [Leader]" -46:07.658 [INFO] TestAgent.server: cluster leadership acquired -46:07.658 [INFO] TestAgent.server: New leader elected: payload=Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0 -46:07.665 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11174 -46:07.687 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.727 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.727 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.727 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.727 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.727 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.727 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0 -46:07.727 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f63a2e54-1684-b5d4-df69-8dcb76663eb0 -46:07.737 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.908 [INFO] TestAgent: Requesting shutdown -46:07.908 [INFO] TestAgent.server: shutting down server -46:07.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.908 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.908 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.908 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.908 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:07.908 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.908 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.908 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.913 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.918Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.918Z [INFO] agent.router.manager: shutting down -46:07.918 [INFO] TestAgent: consul server down -46:07.918 [INFO] TestAgent: shutdown complete -46:07.918 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11267 network=tcp -46:07.918 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11267 network=udp -46:07.918 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11268 network=tcp -46:08.419 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.419 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionDestroy (0.87s) -=== CONT TestCatalog_GatewayServices_Ingress -2022-11-07T22:46:07.634Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.647Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.695 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:077cafdd-785f-f7a8-9530-e68b2e68a325 Address:127.0.0.1:11265}]" -46:07.695 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11265 [Follower]" leader= -46:07.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-077cafdd-785f-f7a8-9530-e68b2e68a325.dc1 127.0.0.1 -46:07.697 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-077cafdd-785f-f7a8-9530-e68b2e68a325 127.0.0.1 -2022-11-07T22:46:07.697Z [INFO] agent.router: Initializing LAN area manager -46:07.697 [INFO] TestAgent.server: Adding LAN server: server="Node-077cafdd-785f-f7a8-9530-e68b2e68a325 (Addr: tcp/127.0.0.1:11265) (DC: dc1)" -46:07.697 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-077cafdd-785f-f7a8-9530-e68b2e68a325.dc1 area=wan -46:07.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11218 network=udp -46:07.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11218 network=tcp -46:07.697 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11219 network=tcp -46:07.697 [INFO] TestAgent: started state syncer -46:07.697 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11266 network=tcp -46:07.738 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.738 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11265 [Candidate]" term=2 -46:07.757 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.757 [DEBUG] TestAgent.server.raft: vote granted: from=077cafdd-785f-f7a8-9530-e68b2e68a325 term=2 tally=1 -46:07.757 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.757 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11265 [Leader]" -46:07.757 [INFO] TestAgent.server: cluster leadership acquired -46:07.757 [INFO] TestAgent.server: New leader elected: payload=Node-077cafdd-785f-f7a8-9530-e68b2e68a325 -46:07.765 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11265 -46:07.765 [INFO] TestAgent: Synced node info -46:07.765 [DEBUG] TestAgent: Node info in sync -46:07.788 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.806 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.806 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.806 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.806 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.806 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.806 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-077cafdd-785f-f7a8-9530-e68b2e68a325 -46:07.806 [INFO] TestAgent.server: member joined, marking health alive: member=Node-077cafdd-785f-f7a8-9530-e68b2e68a325 -46:07.825 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.954 [INFO] TestAgent: Requesting shutdown -46:07.955 [INFO] TestAgent.server: shutting down server -46:07.955 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:07.955 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:07.955 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:07.955 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:07.955 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:07.955 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:07.955 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:07.955 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:07.955 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:07.955 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:07.962 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:07.970Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:07.970Z [INFO] agent.router.manager: shutting down -46:07.970 [INFO] TestAgent: consul server down -46:07.970 [INFO] TestAgent: shutdown complete -46:07.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11218 network=tcp -46:07.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11218 network=udp -46:07.970 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11219 network=tcp -46:08.470 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.470 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Delete (0.85s) -=== CONT TestRemoteExecWrites_ACLAgentToken -2022-11-07T22:46:07.582Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.591Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.626 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:da273515-9b05-14e6-9aea-7774f66e6816 Address:127.0.0.1:11216}]" -46:07.626 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11216 [Follower]" leader= -46:07.626 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-da273515-9b05-14e6-9aea-7774f66e6816.dc1 127.0.0.1 -46:07.626 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-da273515-9b05-14e6-9aea-7774f66e6816 127.0.0.1 -2022-11-07T22:46:07.627Z [INFO] agent.router: Initializing LAN area manager -46:07.627 [INFO] TestAgent.server: Adding LAN server: server="Node-da273515-9b05-14e6-9aea-7774f66e6816 (Addr: tcp/127.0.0.1:11216) (DC: dc1)" -46:07.627 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-da273515-9b05-14e6-9aea-7774f66e6816.dc1 area=wan -46:07.627 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11183 network=udp -46:07.627 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11183 network=tcp -46:07.627 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11184 network=tcp -46:07.627 [INFO] TestAgent: started state syncer -46:07.627 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11217 network=tcp -46:07.694 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.694 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11216 [Candidate]" term=2 -46:07.705 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.705 [DEBUG] TestAgent.server.raft: vote granted: from=da273515-9b05-14e6-9aea-7774f66e6816 term=2 tally=1 -46:07.705 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.705 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11216 [Leader]" -46:07.706 [INFO] TestAgent.server: cluster leadership acquired -46:07.706 [INFO] TestAgent.server: New leader elected: payload=Node-da273515-9b05-14e6-9aea-7774f66e6816 -46:07.714 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11216 -46:07.738 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.774 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.774 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.774 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.774 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.774 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.774 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-da273515-9b05-14e6-9aea-7774f66e6816 -46:07.774 [INFO] TestAgent.server: member joined, marking health alive: member=Node-da273515-9b05-14e6-9aea-7774f66e6816 -46:07.784 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.052 [DEBUG] TestAgent.dns: request served from client: name=v1.primary2.db.service.consul. type=SRV class=IN latency=101.195µs client=127.0.0.1:51214 client_network=udp -46:08.052 [DEBUG] TestAgent.dns: request served from client: name=v1.primary.db.service.consul. type=SRV class=IN latency=93.573µs client=127.0.0.1:38686 client_network=udp -46:08.052 [INFO] TestAgent: Requesting shutdown -46:08.053 [INFO] TestAgent.server: shutting down server -46:08.053 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.053 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.053 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.053 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.053 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.053 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.053 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.053 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.053 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.053 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.053 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.056 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.061Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.061Z [INFO] agent.router.manager: shutting down -46:08.061 [INFO] TestAgent: consul server down -46:08.061 [INFO] TestAgent: shutdown complete -46:08.061 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11183 network=tcp -46:08.061 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11183 network=udp -46:08.061 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11184 network=tcp -46:08.561 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.561 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_TagPeriod (0.99s) +Added 3 service nodes +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:21.464Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:21.479Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:21.558 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:65140595-d1a6-a3be-abdb-02c3db443646 Address:127.0.0.1:30141}]" +31:21.559 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-65140595-d1a6-a3be-abdb-02c3db443646.dc1 127.0.0.1 +31:21.560 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-65140595-d1a6-a3be-abdb-02c3db443646 127.0.0.1 +2021-10-05T16:31:21.560Z [INFO] agent.router: Initializing LAN area manager +31:21.560 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30141 [Follower]" leader= +31:21.560 [INFO] TestAgent.server: Adding LAN server: server="Node-65140595-d1a6-a3be-abdb-02c3db443646 (Addr: tcp/127.0.0.1:30141) (DC: dc1)" +31:21.560 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-65140595-d1a6-a3be-abdb-02c3db443646.dc1 area=wan +31:21.560 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30136 network=udp +31:21.560 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30136 network=tcp +31:21.561 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30137 network=tcp +31:21.561 [INFO] TestAgent: started state syncer +31:21.561 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30149 network=tcp +31:21.616 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.616 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30141 [Candidate]" term=2 +31:21.633 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.633 [DEBUG] TestAgent.server.raft: vote granted: from=65140595-d1a6-a3be-abdb-02c3db443646 term=2 tally=1 +31:21.633 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.633 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30141 [Leader]" +31:21.634 [INFO] TestAgent.server: cluster leadership acquired +31:21.634 [INFO] TestAgent.server: New leader elected: payload=Node-65140595-d1a6-a3be-abdb-02c3db443646 +31:21.649 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30141 +31:21.680 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.734 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.734 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.734 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.734 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.734 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.734 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-65140595-d1a6-a3be-abdb-02c3db443646 +31:21.734 [INFO] TestAgent.server: member joined, marking health alive: member=Node-65140595-d1a6-a3be-abdb-02c3db443646 +31:21.747 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.881 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=212.298µs client=127.0.0.1:54761 client_network=udp +31:21.881 [DEBUG] TestAgent.dns: request served from client: name=ac2cba8a-cdb1-7ae7-7c60-0637de8a0079.query.consul. type=ANY class=IN latency=122.248µs client=127.0.0.1:45664 client_network=udp +31:21.881 [INFO] TestAgent: Requesting shutdown +31:21.881 [INFO] TestAgent.server: shutting down server +31:21.881 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.881 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.881 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.881 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.881 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.881 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.881 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.881 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:21.881 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.881 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.881 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.887 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.888Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.888Z [INFO] agent.router.manager: shutting down +31:21.888 [INFO] TestAgent: consul server down +31:21.888 [INFO] TestAgent: shutdown complete +31:21.888 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30136 network=tcp +31:21.888 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30136 network=udp +31:21.888 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30137 network=tcp +31:22.393 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.393 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_Dedup (0.95s) === CONT TestDNS_Lookup_TaggedIPAddresses -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:07.558Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.570Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.609 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 Address:127.0.0.1:11202}]" -46:07.610 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11202 [Follower]" leader= -46:07.610 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7.dc1 127.0.0.1 -46:07.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 127.0.0.1 -2022-11-07T22:46:07.611Z [INFO] agent.router: Initializing LAN area manager -46:07.611 [INFO] TestAgent.server: Adding LAN server: server="Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 (Addr: tcp/127.0.0.1:11202) (DC: dc1)" -46:07.611 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7.dc1 area=wan -46:07.611 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11232 network=udp -46:07.611 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11232 network=tcp -46:07.612 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11233 network=tcp -46:07.612 [INFO] TestAgent: started state syncer -46:07.612 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11203 network=tcp -46:07.668 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.668 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11202 [Candidate]" term=2 -46:07.681 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.681 [DEBUG] TestAgent.server.raft: vote granted: from=bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 term=2 tally=1 -46:07.681 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.681 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11202 [Leader]" -46:07.682 [INFO] TestAgent.server: cluster leadership acquired -46:07.682 [INFO] TestAgent.server: New leader elected: payload=Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 -46:07.692 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11202 -46:07.722 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.742 [INFO] TestAgent: Synced node info -46:07.743 [DEBUG] TestAgent: Node info in sync -46:07.766 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.766 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.766 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.766 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.766 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.766 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 -46:07.766 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc4a4af3-6a60-e553-8cdf-ab1f354fe8f7 -46:07.783 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.092 [INFO] TestAgent: Requesting shutdown -46:08.092 [INFO] TestAgent.server: shutting down server -46:08.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.093 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.093 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.093 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.093 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.093 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.093 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.093 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.093 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.099 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.107Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.107Z [INFO] agent.router.manager: shutting down -46:08.107 [INFO] TestAgent: consul server down -46:08.107 [INFO] TestAgent: shutdown complete -46:08.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11232 network=tcp -46:08.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11232 network=udp -46:08.107 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11233 network=tcp -46:08.607 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.607 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_configured_max_txn_len -2022-11-07T22:46:07.927Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.937Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.969 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f24e73ae-be9b-ec63-8a29-443c9e55a38c Address:127.0.0.1:11237}]" -46:07.969 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11237 [Follower]" leader= -46:07.969 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c.dc1 127.0.0.1 -46:07.970 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c 127.0.0.1 -2022-11-07T22:46:07.970Z [INFO] agent.router: Initializing LAN area manager -46:07.970 [INFO] TestAgent.server: Adding LAN server: server="Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c (Addr: tcp/127.0.0.1:11237) (DC: dc1)" -46:07.970 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c.dc1 area=wan -46:07.970 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11260 network=udp -46:07.970 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11260 network=tcp -46:07.970 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11261 network=tcp -46:07.971 [INFO] TestAgent: started state syncer -46:07.971 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11238 network=tcp -46:08.034 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.034 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11237 [Candidate]" term=2 -46:08.044 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.044 [DEBUG] TestAgent.server.raft: vote granted: from=f24e73ae-be9b-ec63-8a29-443c9e55a38c term=2 tally=1 -46:08.044 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.044 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11237 [Leader]" -46:08.044 [INFO] TestAgent.server: cluster leadership acquired -46:08.044 [INFO] TestAgent.server: New leader elected: payload=Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c -46:08.050 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11237 -46:08.063 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.084 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.084 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.084 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.084 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.084 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.084 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c -46:08.084 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f24e73ae-be9b-ec63-8a29-443c9e55a38c -46:08.109 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.118 [INFO] TestAgent: Requesting shutdown -46:08.118 [INFO] TestAgent.server: shutting down server -46:08.118 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.118 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.118 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.118 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.118 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.118 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.118 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.118 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.118 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.118 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.118 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.125 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.131Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.131Z [INFO] agent.router.manager: shutting down -46:08.131 [INFO] TestAgent: consul server down -46:08.131 [INFO] TestAgent: shutdown complete -46:08.131 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11260 network=tcp -46:08.131 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11260 network=udp -46:08.131 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11261 network=tcp -46:08.632 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.632 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCreate_NoCheck (0.71s) - --- PASS: TestSessionCreate_NoCheck/no_check_fields_should_yield_default_serfHealth (0.01s) - --- PASS: TestSessionCreate_NoCheck/overwrite_nodechecks_to_associate_with_no_checks (0.00s) - --- PASS: TestSessionCreate_NoCheck/overwrite_checks_to_associate_with_no_checks (0.00s) -=== CONT TestCatalog_GatewayServices_Terminating -=== RUN TestConfig_Get/get_a_single_service_entry -=== RUN TestConfig_Get/list_both_service_entries -=== RUN TestConfig_Get/get_global_proxy_config -=== RUN TestConfig_Get/error_on_no_arguments -2022-11-07T22:46:07.837Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.846Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.871 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d Address:127.0.0.1:11258}]" -46:07.871 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11258 [Follower]" leader= -46:07.871 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d.dc1 127.0.0.1 -46:07.872 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d 127.0.0.1 -2022-11-07T22:46:07.872Z [INFO] agent.router: Initializing LAN area manager -46:07.872 [INFO] TestAgent.server: Adding LAN server: server="Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d (Addr: tcp/127.0.0.1:11258) (DC: dc1)" -46:07.872 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d.dc1 area=wan -46:07.872 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11288 network=udp -46:07.872 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11288 network=tcp -46:07.872 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11289 network=tcp -46:07.873 [INFO] TestAgent: started state syncer -46:07.873 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11259 network=tcp -46:07.924 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.924 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11258 [Candidate]" term=2 -46:07.934 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.934 [DEBUG] TestAgent.server.raft: vote granted: from=53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d term=2 tally=1 -46:07.934 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.934 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11258 [Leader]" -46:07.934 [INFO] TestAgent.server: cluster leadership acquired -46:07.935 [INFO] TestAgent.server: New leader elected: payload=Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d -46:07.940 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11258 -46:07.960 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.984 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.984 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.984 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.984 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.984 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.984 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d -46:07.984 [INFO] TestAgent.server: member joined, marking health alive: member=Node-53c03d79-eb8d-5d99-fb3c-c42bdd7ad14d -46:07.994 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.155 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.158 [INFO] TestAgent: Synced node info -46:08.230 [DEBUG] TestAgent: received remote exec event: id=964bf07a-d0cf-cff2-6261-aba812636748 -46:08.235 [INFO] TestAgent: remote exec script: script=uptime -46:08.255 [INFO] TestAgent: Requesting shutdown -46:08.255 [INFO] TestAgent.server: shutting down server -46:08.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.255 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.255 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.255 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.255 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.255 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.255 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.255 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.255 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.255 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.258 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.262Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.262Z [INFO] agent.router.manager: shutting down -46:08.262 [INFO] TestAgent: consul server down -46:08.262 [INFO] TestAgent: shutdown complete -46:08.262 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11288 network=tcp -46:08.262 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11288 network=udp -46:08.262 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11289 network=tcp -46:08.762 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.762 [INFO] TestAgent: Endpoints down ---- PASS: TestHandleRemoteExec (0.93s) -=== CONT TestRemoteExecWrites_ACLToken -2022-11-07T22:46:07.773Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.793Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.827 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1785f7e9-e4a7-3835-82f3-5e984ac77308 Address:127.0.0.1:11244}]" -46:07.827 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11244 [Follower]" leader= -46:07.827 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1785f7e9-e4a7-3835-82f3-5e984ac77308.dc1 127.0.0.1 -46:07.828 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1785f7e9-e4a7-3835-82f3-5e984ac77308 127.0.0.1 -2022-11-07T22:46:07.828Z [INFO] agent.router: Initializing LAN area manager -46:07.828 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1785f7e9-e4a7-3835-82f3-5e984ac77308.dc1 area=wan -46:07.828 [INFO] TestAgent.server: Adding LAN server: server="Node-1785f7e9-e4a7-3835-82f3-5e984ac77308 (Addr: tcp/127.0.0.1:11244) (DC: dc1)" -46:07.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11253 network=tcp -46:07.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11253 network=udp -46:07.828 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11254 network=tcp -46:07.829 [INFO] TestAgent: started state syncer -46:07.829 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11245 network=tcp -46:07.888 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11244 [Candidate]" term=2 -46:07.896 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.896 [DEBUG] TestAgent.server.raft: vote granted: from=1785f7e9-e4a7-3835-82f3-5e984ac77308 term=2 tally=1 -46:07.896 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.896 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11244 [Leader]" -46:07.897 [INFO] TestAgent.server: cluster leadership acquired -46:07.897 [INFO] TestAgent.server: New leader elected: payload=Node-1785f7e9-e4a7-3835-82f3-5e984ac77308 -46:07.903 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11244 -46:07.917 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.936 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.936 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.936 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.936 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.936 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.936 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1785f7e9-e4a7-3835-82f3-5e984ac77308 -46:07.936 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1785f7e9-e4a7-3835-82f3-5e984ac77308 -46:07.948 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.951 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.954 [INFO] TestAgent: Synced node info -46:08.267 [DEBUG] TestAgent: received remote exec event: id=66660071-ed64-9f3f-1c14-259259ebf798 -46:08.270 [INFO] TestAgent: remote exec script: script="echo failing;exit 2" -46:08.291 [INFO] TestAgent: Requesting shutdown -46:08.291 [INFO] TestAgent.server: shutting down server -46:08.291 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.292 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.292 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.292 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.292 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.292 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.292 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.292 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.292 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.292 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.297 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.302Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.302Z [INFO] agent.router.manager: shutting down -46:08.302 [INFO] TestAgent: consul server down -46:08.302 [INFO] TestAgent: shutdown complete -46:08.302 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11253 network=tcp -46:08.302 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11253 network=udp -46:08.302 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11254 network=tcp -46:08.802 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.802 [INFO] TestAgent: Endpoints down ---- PASS: TestHandleRemoteExecFailed (1.04s) -=== CONT TestSessionCreate_Delete -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:08.000Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.008Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.032 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2496dca8-67b8-3d46-56ec-1cbce6d1058b Address:127.0.0.1:11307}]" -46:08.032 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11307 [Follower]" leader= -46:08.032 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b.dc1 127.0.0.1 -46:08.032 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b 127.0.0.1 -2022-11-07T22:46:08.032Z [INFO] agent.router: Initializing LAN area manager -46:08.033 [INFO] TestAgent.server: Adding LAN server: server="Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b (Addr: tcp/127.0.0.1:11307) (DC: dc1)" -46:08.033 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b.dc1 area=wan -46:08.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11239 network=udp -46:08.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11239 network=tcp -46:08.033 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11240 network=tcp -46:08.033 [INFO] TestAgent: started state syncer -46:08.033 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11308 network=tcp -46:08.099 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.099 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11307 [Candidate]" term=2 -46:08.111 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.111 [DEBUG] TestAgent.server.raft: vote granted: from=2496dca8-67b8-3d46-56ec-1cbce6d1058b term=2 tally=1 -46:08.111 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.111 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11307 [Leader]" -46:08.111 [INFO] TestAgent.server: cluster leadership acquired -46:08.111 [INFO] TestAgent.server: New leader elected: payload=Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b -46:08.117 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11307 -46:08.130 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.148 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.148 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.148 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.148 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.148 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.148 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b -46:08.148 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2496dca8-67b8-3d46-56ec-1cbce6d1058b -46:08.160 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.182 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.185 [INFO] TestAgent: Synced node info -46:08.185 [DEBUG] TestAgent: Node info in sync -46:08.366 [INFO] TestAgent: Requesting shutdown -46:08.366 [INFO] TestAgent.server: shutting down server -46:08.366 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.366 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.366 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.366 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.366 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.366 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.366 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.366 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.366 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.366 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.377Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.377Z [INFO] agent.router.manager: shutting down -46:08.377 [INFO] TestAgent: consul server down -46:08.377 [INFO] TestAgent: shutdown complete -46:08.377 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11239 network=tcp -46:08.377 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11239 network=udp -46:08.377 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11240 network=tcp -46:08.877 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.877 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionGet (1.84s) - --- PASS: TestSessionGet/#00 (0.96s) - --- PASS: TestSessionGet/#01 (0.89s) -=== CONT TestRemoteExecWrites +Added 3 service nodes +2021-10-05T16:31:21.478Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:21.523Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:21.596 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 Address:127.0.0.1:30148}]" +31:21.597 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60.dc1 127.0.0.1 +31:21.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 127.0.0.1 +2021-10-05T16:31:21.597Z [INFO] agent.router: Initializing LAN area manager +31:21.598 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30143 network=udp +31:21.598 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30148 [Follower]" leader= +31:21.598 [INFO] TestAgent.server: Adding LAN server: server="Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 (Addr: tcp/127.0.0.1:30148) (DC: dc1)" +31:21.598 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60.dc1 area=wan +31:21.598 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30143 network=tcp +31:21.599 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30144 network=tcp +31:21.599 [INFO] TestAgent: started state syncer +31:21.599 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30135 network=tcp +31:21.668 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.668 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30148 [Candidate]" term=2 +31:21.712 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.712 [DEBUG] TestAgent.server.raft: vote granted: from=c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 term=2 tally=1 +31:21.712 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.712 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30148 [Leader]" +31:21.713 [INFO] TestAgent.server: cluster leadership acquired +31:21.713 [INFO] TestAgent.server: New leader elected: payload=Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 +31:21.748 [INFO] TestAgent: Synced node info +31:21.748 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30148 +31:21.809 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.831 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.831 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.831 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.831 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.831 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.831 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 +31:21.831 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c6c3d66a-2686-d9fb-b6ef-31b3a7200b60 +31:21.866 [INFO] TestAgent.server: federation state anti-entropy synced +31:21.912 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=254.585µs client=127.0.0.1:58524 client_network=udp +31:21.916 [DEBUG] TestAgent.dns: request served from client: name=5fea16fc-900a-b57c-aede-5229c5a3c9ad.query.consul. type=SRV class=IN latency=212.006µs client=127.0.0.1:46799 client_network=udp +31:21.916 [INFO] TestAgent: Requesting shutdown +31:21.916 [INFO] TestAgent.server: shutting down server +31:21.916 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:21.916 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:21.916 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:21.916 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:21.916 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:21.916 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:21.916 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:21.916 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:21.916 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:21.916 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:21.924 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:21.926Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:21.926Z [INFO] agent.router.manager: shutting down +31:21.926 [INFO] TestAgent: consul server down +31:21.926 [INFO] TestAgent: shutdown complete +31:21.926 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30143 network=tcp +31:21.926 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30143 network=udp +31:21.927 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30144 network=tcp +31:22.427 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.427 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_Dedup_SRV (0.99s) +=== CONT TestDNS_ServiceLookup_WanTranslation +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:21.699Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:21.717Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:21.769 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02ba6f28-c223-ad5f-330f-0bc5311a7098 Address:127.0.0.1:30155}]" +31:21.770 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02ba6f28-c223-ad5f-330f-0bc5311a7098.dc1 127.0.0.1 +31:21.770 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02ba6f28-c223-ad5f-330f-0bc5311a7098 127.0.0.1 +2021-10-05T16:31:21.770Z [INFO] agent.router: Initializing LAN area manager +31:21.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30150 network=udp +31:21.771 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30155 [Follower]" leader= +31:21.771 [INFO] TestAgent.server: Adding LAN server: server="Node-02ba6f28-c223-ad5f-330f-0bc5311a7098 (Addr: tcp/127.0.0.1:30155) (DC: dc1)" +31:21.771 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02ba6f28-c223-ad5f-330f-0bc5311a7098.dc1 area=wan +31:21.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30150 network=tcp +31:21.772 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30151 network=tcp +31:21.772 [INFO] TestAgent: started state syncer +31:21.772 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30156 network=tcp +31:21.817 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.817 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30155 [Candidate]" term=2 +31:21.841 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.841 [DEBUG] TestAgent.server.raft: vote granted: from=02ba6f28-c223-ad5f-330f-0bc5311a7098 term=2 tally=1 +31:21.841 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.841 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30155 [Leader]" +31:21.841 [INFO] TestAgent.server: cluster leadership acquired +31:21.841 [INFO] TestAgent.server: New leader elected: payload=Node-02ba6f28-c223-ad5f-330f-0bc5311a7098 +31:21.868 [INFO] TestAgent: Synced node info +31:21.868 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30155 +31:21.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.952 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.952 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.952 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.952 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.952 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.952 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02ba6f28-c223-ad5f-330f-0bc5311a7098 +31:21.952 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02ba6f28-c223-ad5f-330f-0bc5311a7098 +31:21.984 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.026 [DEBUG] TestAgent.dns: request served from client: name=v1.primary2.db.service.consul. type=SRV class=IN latency=154.04µs client=127.0.0.1:57186 client_network=udp +31:22.038 [DEBUG] TestAgent.dns: request served from client: name=v1.primary.db.service.consul. type=SRV class=IN latency=188.811µs client=127.0.0.1:34163 client_network=udp +31:22.038 [INFO] TestAgent: Requesting shutdown +31:22.038 [INFO] TestAgent.server: shutting down server +31:22.038 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.038 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.038 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.038 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.038 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.038 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.038 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.038 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.043 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:22.047 [INFO] TestAgent: consul server down +31:22.047 [INFO] TestAgent: shutdown complete +31:22.047 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30150 network=tcp +2021-10-05T16:31:22.047Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.047Z [INFO] agent.router.manager: shutting down +31:22.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30150 network=udp +31:22.092 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30151 network=tcp +31:22.592 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.592 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_TagPeriod (0.91s) +=== CONT TestDNS_ServiceLookup_ServiceAddressIPV6 +2021-10-05T16:31:20.406Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.425Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.482 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d23f3158-d156-fcae-e549-a6dac38bf3b6 Address:127.0.0.1:30085}]" +31:20.483 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d23f3158-d156-fcae-e549-a6dac38bf3b6.dc1 127.0.0.1 +31:20.483 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d23f3158-d156-fcae-e549-a6dac38bf3b6 127.0.0.1 +2021-10-05T16:31:20.483Z [INFO] agent.router: Initializing LAN area manager +31:20.484 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30085 [Follower]" leader= +31:20.484 [INFO] TestAgent.server: Adding LAN server: server="Node-d23f3158-d156-fcae-e549-a6dac38bf3b6 (Addr: tcp/127.0.0.1:30085) (DC: dc1)" +31:20.484 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d23f3158-d156-fcae-e549-a6dac38bf3b6.dc1 area=wan +31:20.484 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30080 network=tcp +31:20.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30080 network=udp +31:20.486 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30081 network=tcp +31:20.486 [INFO] TestAgent: started state syncer +31:20.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30100 network=tcp +31:20.551 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.551 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30085 [Candidate]" term=2 +31:20.576 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.576 [DEBUG] TestAgent.server.raft: vote granted: from=d23f3158-d156-fcae-e549-a6dac38bf3b6 term=2 tally=1 +31:20.576 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.576 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30085 [Leader]" +31:20.577 [INFO] TestAgent.server: cluster leadership acquired +31:20.577 [INFO] TestAgent.server: New leader elected: payload=Node-d23f3158-d156-fcae-e549-a6dac38bf3b6 +31:20.588 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30085 +31:20.644 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d23f3158-d156-fcae-e549-a6dac38bf3b6 +31:20.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d23f3158-d156-fcae-e549-a6dac38bf3b6 +31:20.706 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.880 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.908 [INFO] TestAgent: Synced node info +31:21.957 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.462271ms client=127.0.0.1:37037 client_network=udp +31:21.959 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.059313ms client=127.0.0.1:50661 client_network=udp +31:21.972 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=13.341911ms client=127.0.0.1:51743 client_network=udp +31:21.977 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.344694ms client=127.0.0.1:46252 client_network=udp +31:21.979 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.202896ms client=127.0.0.1:51374 client_network=udp +31:21.981 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.107572ms client=127.0.0.1:48273 client_network=udp +31:21.985 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.50288ms client=127.0.0.1:33057 client_network=udp +31:22.005 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=16.075438ms client=127.0.0.1:47590 client_network=udp +31:22.007 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=2.00887ms client=127.0.0.1:59495 client_network=udp +31:22.009 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=ANY class=IN latency=1.876095ms client=127.0.0.1:46903 client_network=udp +31:22.019 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=9.73692ms client=127.0.0.1:46111 client_network=udp +31:22.024 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=3.197748ms client=127.0.0.1:60827 client_network=udp +31:22.030 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=3.358319ms client=127.0.0.1:46017 client_network=udp +31:22.087 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=48.744012ms client=127.0.0.1:58217 client_network=udp +31:22.090 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=1.502389ms client=127.0.0.1:60946 client_network=udp +31:22.095 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=4.653404ms client=127.0.0.1:59503 client_network=udp +31:22.098 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=1.195475ms client=127.0.0.1:36550 client_network=udp +31:22.102 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=2.127663ms client=127.0.0.1:59499 client_network=udp +31:22.106 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=1.547409ms client=127.0.0.1:43827 client_network=udp +31:22.113 [DEBUG] TestAgent.dns: request served from client: name=0519b739-1d1a-4cae-588f-ed95d09e9e69.query.consul. type=ANY class=IN latency=1.542701ms client=127.0.0.1:35556 client_network=udp +31:22.121 [INFO] TestAgent: Requesting shutdown +31:22.122 [INFO] TestAgent.server: shutting down server +31:22.122 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.122 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.122 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.122 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.122 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.122 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.122 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.122 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.122 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.122 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.128 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.129Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.129Z [INFO] agent.router.manager: shutting down +31:22.129 [INFO] TestAgent: consul server down +31:22.129 [INFO] TestAgent: shutdown complete +31:22.129 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30080 network=tcp +31:22.129 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30080 network=udp +31:22.129 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30081 network=tcp +31:22.630 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.630 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_Randomize (2.25s) +=== CONT TestDNS_ServiceLookup_ServiceAddress_SRV +2021-10-05T16:31:21.724Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:21.751Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:21.839 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 Address:127.0.0.1:30162}]" +31:21.840 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451.dc1 127.0.0.1 +31:21.840 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30162 [Follower]" leader= +31:21.841 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 127.0.0.1 +2021-10-05T16:31:21.841Z [INFO] agent.router: Initializing LAN area manager +31:21.841 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451.dc1 area=wan +31:21.841 [INFO] TestAgent.server: Adding LAN server: server="Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 (Addr: tcp/127.0.0.1:30162) (DC: dc1)" +31:21.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30157 network=udp +31:21.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30157 network=tcp +31:21.842 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30158 network=tcp +31:21.842 [INFO] TestAgent: started state syncer +31:21.842 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30163 network=tcp +31:21.895 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:21.895 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30162 [Candidate]" term=2 +31:21.914 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:21.914 [DEBUG] TestAgent.server.raft: vote granted: from=d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 term=2 tally=1 +31:21.914 [INFO] TestAgent.server.raft: election won: tally=1 +31:21.914 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30162 [Leader]" +31:21.916 [INFO] TestAgent.server: cluster leadership acquired +31:21.916 [INFO] TestAgent.server: New leader elected: payload=Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 +31:21.925 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30162 +31:21.963 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.056 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.056 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.056 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.056 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.056 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.056 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 +31:22.056 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d1675e8e-fe8a-a92f-1bcc-d6feb8e96451 +31:22.121 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.132 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=SRV class=IN latency=240.295µs client=127.0.0.1:57822 client_network=udp +31:22.132 [INFO] TestAgent: Requesting shutdown +31:22.132 [INFO] TestAgent.server: shutting down server +31:22.132 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.132 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.132 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.132 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.132 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.133 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.133 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.133 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:22.133 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.133 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.133 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.137 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.141Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.141Z [INFO] agent.router.manager: shutting down +31:22.141 [INFO] TestAgent: consul server down +31:22.141 [INFO] TestAgent: shutdown complete +31:22.141 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30157 network=tcp +31:22.141 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30157 network=udp +31:22.142 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30158 network=tcp +31:22.642 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.642 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_PreparedQueryNamePeriod (0.95s) +=== CONT TestDNS_ServiceLookup_ServiceAddress_A === RUN TestDNS_Lookup_TaggedIPAddresses/simple-ipv4 === RUN TestDNS_Lookup_TaggedIPAddresses/simple-ipv6 === RUN TestDNS_Lookup_TaggedIPAddresses/ipv4-with-tagged-ipv6 === RUN TestDNS_Lookup_TaggedIPAddresses/ipv6-with-tagged-ipv4 -2022-11-07T22:46:08.213Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.223Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0edb1fa4-933f-967b-844d-f6d55bc59f64 Address:127.0.0.1:11363}]" -46:08.252 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11363 [Follower]" leader= -46:08.252 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0edb1fa4-933f-967b-844d-f6d55bc59f64.dc1 127.0.0.1 -46:08.253 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0edb1fa4-933f-967b-844d-f6d55bc59f64 127.0.0.1 -2022-11-07T22:46:08.253Z [INFO] agent.router: Initializing LAN area manager -46:08.253 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0edb1fa4-933f-967b-844d-f6d55bc59f64.dc1 area=wan -46:08.253 [INFO] TestAgent.server: Adding LAN server: server="Node-0edb1fa4-933f-967b-844d-f6d55bc59f64 (Addr: tcp/127.0.0.1:11363) (DC: dc1)" -46:08.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11337 network=udp -46:08.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11337 network=tcp -46:08.254 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11338 network=tcp -46:08.254 [INFO] TestAgent: started state syncer -46:08.254 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11364 network=tcp -46:08.293 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.293 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11363 [Candidate]" term=2 -46:08.304 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.304 [DEBUG] TestAgent.server.raft: vote granted: from=0edb1fa4-933f-967b-844d-f6d55bc59f64 term=2 tally=1 -46:08.304 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.304 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11363 [Leader]" -46:08.305 [INFO] TestAgent.server: cluster leadership acquired -46:08.305 [INFO] TestAgent.server: New leader elected: payload=Node-0edb1fa4-933f-967b-844d-f6d55bc59f64 -46:08.311 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11363 -46:08.344 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.344 [INFO] TestAgent: Synced node info -46:08.344 [DEBUG] TestAgent: Node info in sync -46:08.367 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.367 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.367 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.367 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.367 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.367 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0edb1fa4-933f-967b-844d-f6d55bc59f64 -46:08.367 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0edb1fa4-933f-967b-844d-f6d55bc59f64 -46:08.381 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.453 [DEBUG] TestAgent.dns: request served from client: name=primary.db.service.consul. type=SRV class=IN latency=123.857µs client=127.0.0.1:51631 client_network=udp -46:08.453 [DEBUG] TestAgent.dns: request served from client: name=pRIMARY.dB.service.consul. type=SRV class=IN latency=116.472µs client=127.0.0.1:47709 client_network=udp -46:08.454 [DEBUG] TestAgent.dns: request served from client: name=PRIMARY.dB.service.consul. type=SRV class=IN latency=122.321µs client=127.0.0.1:46600 client_network=udp -46:08.454 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=109.438µs client=127.0.0.1:33889 client_network=udp -46:08.454 [DEBUG] TestAgent.dns: request served from client: name=DB.service.consul. type=SRV class=IN latency=108.3µs client=127.0.0.1:36677 client_network=udp -46:08.455 [DEBUG] TestAgent.dns: request served from client: name=Db.service.consul. type=SRV class=IN latency=117.449µs client=127.0.0.1:42933 client_network=udp -46:08.455 [DEBUG] TestAgent.dns: request served from client: name=somequery.query.consul. type=SRV class=IN latency=116.283µs client=127.0.0.1:33134 client_network=udp -46:08.455 [DEBUG] TestAgent.dns: request served from client: name=SomeQuery.query.consul. type=SRV class=IN latency=113.628µs client=127.0.0.1:44304 client_network=udp -46:08.455 [DEBUG] TestAgent.dns: request served from client: name=SOMEQUERY.query.consul. type=SRV class=IN latency=111.76µs client=127.0.0.1:42402 client_network=udp -46:08.455 [INFO] TestAgent: Requesting shutdown -46:08.455 [INFO] TestAgent.server: shutting down server -46:08.455 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.455 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.455 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.455 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.455 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.456 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.456 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.456 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.456 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.456 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.461 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.466Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.466Z [INFO] agent.router.manager: shutting down -46:08.467 [INFO] TestAgent: consul server down -46:08.467 [INFO] TestAgent: shutdown complete -46:08.467 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11337 network=tcp -46:08.467 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11337 network=udp -46:08.467 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11338 network=tcp -46:08.967 [INFO] TestAgent: Waiting for endpoints to shut down -46:08.967 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_CaseInsensitiveServiceLookup (0.76s) -=== CONT TestDNS_ServiceLookup_WanTranslation -2022-11-07T22:46:08.301Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.301Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.311Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.311Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.350 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c4611dc1-4798-8936-19e4-7cab12542049 Address:127.0.0.1:11314}]" -46:08.350 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11314 [Follower]" leader= -46:08.351 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 127.0.0.1 -46:08.351 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c4611dc1-4798-8936-19e4-7cab12542049 127.0.0.1 -2022-11-07T22:46:08.351Z [INFO] agent.router: Initializing LAN area manager -46:08.352 [INFO] TestAgent.server: Adding LAN server: server="Node-c4611dc1-4798-8936-19e4-7cab12542049 (Addr: tcp/127.0.0.1:11314) (DC: dc1)" -46:08.352 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 area=wan -46:08.352 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11365 network=udp -46:08.352 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11365 network=tcp -46:08.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11366 network=tcp -46:08.353 [INFO] TestAgent: started state syncer -46:08.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11315 network=tcp -46:08.387 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.387 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11314 [Candidate]" term=2 -46:08.397 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.397 [DEBUG] TestAgent.server.raft: vote granted: from=c4611dc1-4798-8936-19e4-7cab12542049 term=2 tally=1 -46:08.397 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.397 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11314 [Leader]" -46:08.397 [INFO] TestAgent.server: cluster leadership acquired -46:08.397 [INFO] TestAgent.server: New leader elected: payload=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.402 [INFO] TestAgent.server: initializing acls -46:08.404 [INFO] TestAgent.server: initializing acls -46:08.410 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.410 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.416 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.416 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.429 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.429 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.435 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.435 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:08.435 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:08.435 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.435 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 -46:08.435 [INFO] TestAgent.server: Updating LAN server: server="Node-c4611dc1-4798-8936-19e4-7cab12542049 (Addr: tcp/127.0.0.1:11314) (DC: dc1)" -46:08.435 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 area=wan -46:08.444 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.444 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:08.444 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.444 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 -46:08.444 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c4611dc1-4798-8936-19e4-7cab12542049.dc1 area=wan -46:08.444 [INFO] TestAgent.server: Updating LAN server: server="Node-c4611dc1-4798-8936-19e4-7cab12542049 (Addr: tcp/127.0.0.1:11314) (DC: dc1)" -46:08.460 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.481 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.481 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.481 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.481 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.481 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.481 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.481 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.485 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.494 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.494 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.529 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.529 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c4611dc1-4798-8936-19e4-7cab12542049 -46:08.533 [ERROR] TestAgent: failed to ack remote exec job: error="Permission denied" -46:08.533 [ERROR] TestAgent: failed to write output for remote exec job: error="Permission denied" -46:08.533 [ERROR] TestAgent: failed to write output for remote exec job: error="Permission denied" -46:08.537 [INFO] TestAgent: Requesting shutdown -46:08.537 [INFO] TestAgent.server: shutting down server -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:08.537 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:08.537 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.537 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.537 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.537 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.545 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.549Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.549Z [INFO] agent.router.manager: shutting down -46:08.549 [INFO] TestAgent: consul server down -46:08.549 [INFO] TestAgent: shutdown complete -46:08.549 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11365 network=tcp -46:08.549 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11365 network=udp -46:08.550 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11366 network=tcp -46:09.050 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.050 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecWrites_ACLDeny (0.76s) -=== CONT TestCatalogNodeServices_WanTranslation -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:07.519Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.528Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.566 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:946a84e7-c141-5d2c-dd8a-7b831908d807 Address:127.0.0.1:11223}]" -46:07.566 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11223 [Follower]" leader= -46:07.566 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-946a84e7-c141-5d2c-dd8a-7b831908d807.dc1 127.0.0.1 -46:07.566 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-946a84e7-c141-5d2c-dd8a-7b831908d807 127.0.0.1 -2022-11-07T22:46:07.566Z [INFO] agent.router: Initializing LAN area manager -46:07.567 [INFO] TestAgent.server: Adding LAN server: server="Node-946a84e7-c141-5d2c-dd8a-7b831908d807 (Addr: tcp/127.0.0.1:11223) (DC: dc1)" -46:07.567 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-946a84e7-c141-5d2c-dd8a-7b831908d807.dc1 area=wan -46:07.567 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11190 network=udp -46:07.567 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11190 network=tcp -46:07.567 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11191 network=tcp -46:07.567 [INFO] TestAgent: started state syncer -46:07.567 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11224 network=tcp -46:07.615 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.615 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11223 [Candidate]" term=2 -46:07.631 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.631 [DEBUG] TestAgent.server.raft: vote granted: from=946a84e7-c141-5d2c-dd8a-7b831908d807 term=2 tally=1 -46:07.631 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.631 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11223 [Leader]" -46:07.632 [INFO] TestAgent.server: cluster leadership acquired -46:07.632 [INFO] TestAgent.server: New leader elected: payload=Node-946a84e7-c141-5d2c-dd8a-7b831908d807 -46:07.638 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11223 -46:07.654 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.691 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.691 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.691 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.691 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.691 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.691 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-946a84e7-c141-5d2c-dd8a-7b831908d807 -46:07.691 [INFO] TestAgent.server: member joined, marking health alive: member=Node-946a84e7-c141-5d2c-dd8a-7b831908d807 -46:07.717 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.888 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.891 [INFO] TestAgent: Synced node info -46:08.334 [DEBUG] TestAgent.server: Session TTL expired: session=4542df43-0391-a999-2774-7abe5d096fed -46:08.578 [INFO] TestAgent: Requesting shutdown -46:08.578 [INFO] TestAgent.server: shutting down server -46:08.578 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.578 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.578 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.578 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.578 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.578 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.578 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.578 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.578 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.578 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.582 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.590Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.590Z [INFO] agent.router.manager: shutting down -46:08.590 [INFO] TestAgent: consul server down -46:08.590 [INFO] TestAgent: shutdown complete -46:08.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11190 network=tcp -46:08.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11190 network=udp -46:08.590 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11191 network=tcp -46:09.090 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.090 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCustomTTL (1.58s) -=== CONT TestRemoteExecGetSpec_ACLDeny -2022-11-07T22:46:08.371Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.382Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.415 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d08b4d34-6614-807f-3a84-8478f92bf312 Address:127.0.0.1:11300}]" -46:08.415 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11300 [Follower]" leader= -46:08.415 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d08b4d34-6614-807f-3a84-8478f92bf312.dc1 127.0.0.1 -46:08.416 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d08b4d34-6614-807f-3a84-8478f92bf312 127.0.0.1 -2022-11-07T22:46:08.416Z [INFO] agent.router: Initializing LAN area manager -46:08.416 [INFO] TestAgent.server: Adding LAN server: server="Node-d08b4d34-6614-807f-3a84-8478f92bf312 (Addr: tcp/127.0.0.1:11300) (DC: dc1)" -46:08.416 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d08b4d34-6614-807f-3a84-8478f92bf312.dc1 area=wan -46:08.416 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11316 network=udp -46:08.416 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11316 network=tcp -46:08.416 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11317 network=tcp -46:08.416 [INFO] TestAgent: started state syncer -46:08.416 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11301 network=tcp -46:08.451 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.451 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11300 [Candidate]" term=2 -46:08.463 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.463 [DEBUG] TestAgent.server.raft: vote granted: from=d08b4d34-6614-807f-3a84-8478f92bf312 term=2 tally=1 -46:08.463 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.463 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11300 [Leader]" -46:08.463 [INFO] TestAgent.server: cluster leadership acquired -46:08.463 [INFO] TestAgent.server: New leader elected: payload=Node-d08b4d34-6614-807f-3a84-8478f92bf312 -46:08.470 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11300 -46:08.485 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.513 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.513 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.513 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.513 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.513 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.513 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d08b4d34-6614-807f-3a84-8478f92bf312 -46:08.513 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d08b4d34-6614-807f-3a84-8478f92bf312 -46:08.527 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.635 [INFO] TestAgent: Requesting shutdown -46:08.635 [INFO] TestAgent.server: shutting down server -46:08.635 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.635 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.635 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.635 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.635 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.635 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.635 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.635 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.635 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.635 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.635 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.640 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.646Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.646Z [INFO] agent.router.manager: shutting down -46:08.646 [INFO] TestAgent: consul server down -46:08.646 [INFO] TestAgent: shutdown complete -46:08.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11316 network=tcp -46:08.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11316 network=udp -46:08.646 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11317 network=tcp -46:09.146 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.146 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCreate_DefaultCheck (0.79s) -=== CONT TestPreparedQuery_ExecuteCached -2022-11-07T22:46:08.192Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.192Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.201Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.201Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.237 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:94d6e8dc-1916-2588-60d9-268d3bb6156d Address:127.0.0.1:11335}]" -46:08.237 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11335 [Follower]" leader= -46:08.238 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 127.0.0.1 -46:08.239 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d 127.0.0.1 -2022-11-07T22:46:08.239Z [INFO] agent.router: Initializing LAN area manager -46:08.239 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 area=wan -46:08.239 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11295 network=tcp -46:08.239 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11295 network=udp -46:08.240 [INFO] TestAgent.server: Adding LAN server: server="Node-94d6e8dc-1916-2588-60d9-268d3bb6156d (Addr: tcp/127.0.0.1:11335) (DC: dc1)" -46:08.240 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11296 network=tcp -46:08.240 [INFO] TestAgent: started state syncer -46:08.240 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11336 network=tcp -46:08.274 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.274 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11335 [Candidate]" term=2 -46:08.285 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.285 [DEBUG] TestAgent.server.raft: vote granted: from=94d6e8dc-1916-2588-60d9-268d3bb6156d term=2 tally=1 -46:08.285 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.285 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11335 [Leader]" -46:08.285 [INFO] TestAgent.server: cluster leadership acquired -46:08.285 [INFO] TestAgent.server: New leader elected: payload=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.290 [INFO] TestAgent.server: initializing acls -46:08.292 [INFO] TestAgent.server: initializing acls -46:08.298 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.298 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.308 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.308 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.323 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.323 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.326 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.326 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:08.326 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:08.326 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.326 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 -46:08.326 [INFO] TestAgent.server: Updating LAN server: server="Node-94d6e8dc-1916-2588-60d9-268d3bb6156d (Addr: tcp/127.0.0.1:11335) (DC: dc1)" -46:08.326 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 area=wan -46:08.335 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.335 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:08.336 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.336 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 -46:08.336 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d.dc1 area=wan -46:08.336 [INFO] TestAgent.server: Updating LAN server: server="Node-94d6e8dc-1916-2588-60d9-268d3bb6156d (Addr: tcp/127.0.0.1:11335) (DC: dc1)" -46:08.348 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.367 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.367 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.367 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.367 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.367 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.367 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.367 [INFO] TestAgent.server: member joined, marking health alive: member=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.376 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.376 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.382 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.429 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.435 [INFO] TestAgent: Synced node info -46:08.435 [DEBUG] TestAgent: Node info in sync -46:08.617 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-94d6e8dc-1916-2588-60d9-268d3bb6156d -46:08.618 [INFO] TestAgent.server.fsm: snapshot created: duration=39.188µs -46:08.618 [INFO] TestAgent.server.raft: starting snapshot up to: index=18 -46:08.618 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_Options_GET#02-TestAgent-agent344282465/raft/snapshots/2-18-1667861168618.tmp -46:08.643 [INFO] TestAgent.server.raft: snapshot complete up to: index=18 -46:08.649 [INFO] TestAgent: Requesting shutdown -46:08.649 [INFO] TestAgent.server: shutting down server -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:08.650 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:08.650 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.650 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:08.650 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:08.654 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.660Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.660Z [INFO] agent.router.manager: shutting down -46:08.660 [INFO] TestAgent: consul server down -46:08.660 [INFO] TestAgent: shutdown complete -46:08.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11295 network=tcp -46:08.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11295 network=udp -46:08.660 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11296 network=tcp -46:09.160 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.160 [INFO] TestAgent: Endpoints down -=== RUN TestSnapshot_Options/PUT -2022-11-07T22:46:08.189Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.198Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.224 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e64531f5-5a96-172c-4d88-96043c800d2a Address:127.0.0.1:11293}]" -46:08.224 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11293 [Follower]" leader= -46:08.224 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e64531f5-5a96-172c-4d88-96043c800d2a.dc1 127.0.0.1 -46:08.225 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e64531f5-5a96-172c-4d88-96043c800d2a 127.0.0.1 -2022-11-07T22:46:08.225Z [INFO] agent.router: Initializing LAN area manager -46:08.225 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e64531f5-5a96-172c-4d88-96043c800d2a.dc1 area=wan -46:08.225 [INFO] TestAgent.server: Adding LAN server: server="Node-e64531f5-5a96-172c-4d88-96043c800d2a (Addr: tcp/127.0.0.1:11293) (DC: dc1)" -46:08.225 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11309 network=udp -46:08.225 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11309 network=tcp -46:08.225 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11310 network=tcp -46:08.226 [INFO] TestAgent: started state syncer -46:08.226 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11294 network=tcp -46:08.291 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.291 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11293 [Candidate]" term=2 -46:08.303 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.303 [DEBUG] TestAgent.server.raft: vote granted: from=e64531f5-5a96-172c-4d88-96043c800d2a term=2 tally=1 -46:08.303 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.303 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11293 [Leader]" -46:08.304 [INFO] TestAgent.server: cluster leadership acquired -46:08.304 [INFO] TestAgent.server: New leader elected: payload=Node-e64531f5-5a96-172c-4d88-96043c800d2a -46:08.311 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11293 -46:08.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.358 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.358 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.358 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.358 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.358 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.358 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e64531f5-5a96-172c-4d88-96043c800d2a -46:08.358 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e64531f5-5a96-172c-4d88-96043c800d2a -46:08.369 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.544 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.549 [INFO] TestAgent: Synced node info -46:08.549 [DEBUG] TestAgent: Node info in sync -46:08.654 [INFO] TestAgent: Requesting shutdown -46:08.654 [INFO] TestAgent.server: shutting down server -46:08.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.654 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.654 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.654 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.654 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.654 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.654 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.660 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.666Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.666Z [INFO] agent.router.manager: shutting down -46:08.666 [INFO] TestAgent: consul server down -46:08.666 [INFO] TestAgent: shutdown complete -46:08.666 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11309 network=tcp -46:08.666 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11309 network=udp -46:08.666 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11310 network=tcp -46:09.166 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.166 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Get (0.99s) - --- PASS: TestConfig_Get/get_a_single_service_entry (0.00s) - --- PASS: TestConfig_Get/list_both_service_entries (0.00s) - --- PASS: TestConfig_Get/get_global_proxy_config (0.00s) - --- PASS: TestConfig_Get/error_on_no_arguments (0.00s) -=== CONT TestDNS_ServiceLookup_ServiceAddressIPV6 -2022-11-07T22:46:08.430Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.438Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.474 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f0b0a853-6de3-d2a3-8bdf-4999feb81596 Address:127.0.0.1:11342}]" -46:08.474 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11342 [Follower]" leader= -46:08.475 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596.dc1 127.0.0.1 -46:08.475 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596 127.0.0.1 -2022-11-07T22:46:08.475Z [INFO] agent.router: Initializing LAN area manager -46:08.475 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596.dc1 area=wan -46:08.475 [INFO] TestAgent.server: Adding LAN server: server="Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596 (Addr: tcp/127.0.0.1:11342) (DC: dc1)" -46:08.475 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11358 network=udp -46:08.476 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11358 network=tcp -46:08.476 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11359 network=tcp -46:08.476 [INFO] TestAgent: started state syncer -46:08.476 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11343 network=tcp -46:08.524 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.524 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11342 [Candidate]" term=2 -46:08.534 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.534 [DEBUG] TestAgent.server.raft: vote granted: from=f0b0a853-6de3-d2a3-8bdf-4999feb81596 term=2 tally=1 -46:08.534 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.534 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11342 [Leader]" -46:08.534 [INFO] TestAgent.server: cluster leadership acquired -46:08.534 [INFO] TestAgent.server: New leader elected: payload=Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596 -46:08.541 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11342 -46:08.558 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.589 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.589 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.589 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.589 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.589 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.589 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596 -46:08.589 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f0b0a853-6de3-d2a3-8bdf-4999feb81596 -46:08.596 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.718 [INFO] TestAgent: Requesting shutdown -46:08.718 [INFO] TestAgent.server: shutting down server -46:08.718 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.718 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.718 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.718 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.718 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.718 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.718 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.718 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.718 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.718 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.718 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.723 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.731Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.731Z [INFO] agent.router.manager: shutting down -46:08.732 [INFO] TestAgent: consul server down -46:08.732 [INFO] TestAgent: shutdown complete -46:08.732 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11358 network=tcp -46:08.732 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11358 network=udp -46:08.732 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11359 network=tcp -46:09.232 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.232 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalog_GatewayServices_Ingress (0.81s) -=== CONT TestCatalogNodeServices_ConnectProxy -2022-11-07T22:46:08.481Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.481Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.490Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.490Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.520 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 Address:127.0.0.1:11321}]" -46:08.520 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11321 [Follower]" leader= -46:08.521 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7.dc1 127.0.0.1 -46:08.521 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 127.0.0.1 -2022-11-07T22:46:08.521Z [INFO] agent.router: Initializing LAN area manager -46:08.521 [INFO] TestAgent.server: Adding LAN server: server="Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 (Addr: tcp/127.0.0.1:11321) (DC: dc1)" -46:08.521 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7.dc1 area=wan -46:08.522 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11344 network=tcp -46:08.522 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11344 network=udp -46:08.522 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11345 network=tcp -46:08.522 [INFO] TestAgent: started state syncer -46:08.522 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11322 network=tcp -46:08.568 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.568 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11321 [Candidate]" term=2 -46:08.572 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:08.585 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.585 [DEBUG] TestAgent.server.raft: vote granted: from=8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 term=2 tally=1 -46:08.585 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.585 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11321 [Leader]" -46:08.585 [INFO] TestAgent.server: cluster leadership acquired -46:08.585 [INFO] TestAgent.server: New leader elected: payload=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.593 [INFO] TestAgent.server: initializing acls -46:08.598 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.598 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.606 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.617 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.617 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:08.617 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:08.617 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.617 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7.dc1 -46:08.617 [INFO] TestAgent.server: Updating LAN server: server="Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 (Addr: tcp/127.0.0.1:11321) (DC: dc1)" -46:08.617 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7.dc1 area=wan -46:08.631 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.651 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.651 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.651 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.651 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.651 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.651 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.651 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.664 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.665 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.705 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.710 [INFO] TestAgent: Synced node info -46:08.761 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.761 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8b2693ad-ce9d-fa25-5d4b-8ef25e73dde7 -46:08.788 [INFO] TestAgent: Requesting shutdown -46:08.788 [INFO] TestAgent.server: shutting down server -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:08.788 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:08.788 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:08.793 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.800Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.800Z [INFO] agent.router.manager: shutting down -46:08.800 [INFO] TestAgent: consul server down -46:08.800 [INFO] TestAgent: shutdown complete -46:08.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11344 network=tcp -46:08.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11344 network=udp -46:08.800 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11345 network=tcp -46:09.300 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.300 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecWrites_ACLAgentToken (0.83s) -=== CONT TestPreparedQuery_Execute -=== RUN TestPreparedQuery_Execute/#00 -2022-11-07T22:46:08.368Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.379Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.412 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a561bc43-2efe-3a4b-0c6c-54c7cae43f18 Address:127.0.0.1:11356}]" -46:08.412 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11356 [Follower]" leader= -46:08.413 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18.dc1 127.0.0.1 -46:08.413 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18 127.0.0.1 -2022-11-07T22:46:08.413Z [INFO] agent.router: Initializing LAN area manager -46:08.413 [INFO] TestAgent.server: Adding LAN server: server="Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18 (Addr: tcp/127.0.0.1:11356) (DC: dc1)" -46:08.414 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18.dc1 area=wan -46:08.414 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11302 network=tcp -46:08.414 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11302 network=udp -46:08.414 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11303 network=tcp -46:08.414 [INFO] TestAgent: started state syncer -46:08.414 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11357 network=tcp -46:08.479 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.479 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11356 [Candidate]" term=2 -46:08.490 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.490 [DEBUG] TestAgent.server.raft: vote granted: from=a561bc43-2efe-3a4b-0c6c-54c7cae43f18 term=2 tally=1 -46:08.490 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.490 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11356 [Leader]" -46:08.490 [INFO] TestAgent.server: cluster leadership acquired -46:08.490 [INFO] TestAgent.server: New leader elected: payload=Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18 -46:08.498 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11356 -46:08.516 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.547 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.547 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.547 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.547 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.547 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.547 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18 -46:08.547 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a561bc43-2efe-3a4b-0c6c-54c7cae43f18 -46:08.555 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.740 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.747 [INFO] TestAgent: Synced node info -46:08.747 [DEBUG] TestAgent: Node info in sync -46:08.772 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=1573029 -46:08.788 [INFO] TestAgent: Requesting shutdown -46:08.788 [INFO] TestAgent.server: shutting down server -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.788 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.788 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.788 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.788 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.793 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.800Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.800Z [INFO] agent.router.manager: shutting down -46:08.800 [INFO] TestAgent: consul server down -46:08.800 [INFO] TestAgent: shutdown complete -46:08.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11302 network=tcp -46:08.800 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11302 network=udp -46:08.800 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11303 network=tcp -46:09.300 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.300 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_Bad_Size_Item (3.60s) - --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_default_limits (0.87s) - --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_configured_max_txn_len (0.79s) - --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_default_max_kv_value_size (0.99s) - --- PASS: TestTxnEndpoint_Bad_Size_Item/allowed (0.94s) -=== CONT TestDNS_ServiceLookup_ServiceAddress_SRV -2022-11-07T22:46:08.643Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.652Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.685 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:700170d3-75aa-efc3-1c3f-fa0d895aa916 Address:127.0.0.1:11391}]" -46:08.685 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11391 [Follower]" leader= -46:08.685 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-700170d3-75aa-efc3-1c3f-fa0d895aa916.dc1 127.0.0.1 -46:08.686 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-700170d3-75aa-efc3-1c3f-fa0d895aa916 127.0.0.1 -2022-11-07T22:46:08.686Z [INFO] agent.router: Initializing LAN area manager -46:08.686 [INFO] TestAgent.server: Adding LAN server: server="Node-700170d3-75aa-efc3-1c3f-fa0d895aa916 (Addr: tcp/127.0.0.1:11391) (DC: dc1)" -46:08.686 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-700170d3-75aa-efc3-1c3f-fa0d895aa916.dc1 area=wan -46:08.686 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11386 network=udp -46:08.686 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11386 network=tcp -46:08.687 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11387 network=tcp -46:08.687 [INFO] TestAgent: started state syncer -46:08.687 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11392 network=tcp -46:08.723 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.723 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11391 [Candidate]" term=2 -46:08.738 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.738 [DEBUG] TestAgent.server.raft: vote granted: from=700170d3-75aa-efc3-1c3f-fa0d895aa916 term=2 tally=1 -46:08.738 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.738 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11391 [Leader]" -46:08.738 [INFO] TestAgent.server: cluster leadership acquired -46:08.738 [INFO] TestAgent.server: New leader elected: payload=Node-700170d3-75aa-efc3-1c3f-fa0d895aa916 -46:08.748 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11391 -46:08.759 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.791 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.791 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.791 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.791 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.791 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.791 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-700170d3-75aa-efc3-1c3f-fa0d895aa916 -46:08.791 [INFO] TestAgent.server: member joined, marking health alive: member=Node-700170d3-75aa-efc3-1c3f-fa0d895aa916 -46:08.798 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.810 [INFO] TestAgent: Requesting shutdown -46:08.810 [INFO] TestAgent.server: shutting down server -46:08.810 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.810 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.810 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.810 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.810 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:08.810 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.810 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.816 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.820Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.820Z [INFO] agent.router.manager: shutting down -46:08.820 [INFO] TestAgent: consul server down -46:08.820 [INFO] TestAgent: shutdown complete -46:08.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11386 network=tcp -46:08.820 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11386 network=udp -46:08.820 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11387 network=tcp -46:09.321 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.321 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalog_GatewayServices_Terminating (0.69s) -=== CONT TestRemoteExecGetSpec_ACLAgentToken -[WARN] freeport: 10 out of 66 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:08.575Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.584Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.616 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 Address:127.0.0.1:11349}]" -46:08.616 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11349 [Follower]" leader= -46:08.616 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020.dc1 127.0.0.1 -46:08.617 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 127.0.0.1 -2022-11-07T22:46:08.617Z [INFO] agent.router: Initializing LAN area manager -46:08.617 [INFO] TestAgent.server: Adding LAN server: server="Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 (Addr: tcp/127.0.0.1:11349) (DC: dc1)" -46:08.617 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020.dc1 area=wan -46:08.617 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11323 network=tcp -46:08.617 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11323 network=udp -46:08.617 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11324 network=tcp -46:08.617 [INFO] TestAgent: started state syncer -46:08.617 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11350 network=tcp -46:08.673 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.673 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11349 [Candidate]" term=2 -46:08.684 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.684 [DEBUG] TestAgent.server.raft: vote granted: from=0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 term=2 tally=1 -46:08.684 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.684 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11349 [Leader]" -46:08.684 [INFO] TestAgent.server: cluster leadership acquired -46:08.684 [INFO] TestAgent.server: New leader elected: payload=Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 -46:08.690 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11349 -46:08.706 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.733 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.733 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.733 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.733 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.733 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.733 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 -46:08.733 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0de0bfc7-b8f7-ec5f-3d6d-9d5be437f020 -46:08.749 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.899 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=162.43µs client=127.0.0.1:49408 client_network=udp -46:08.900 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=106.808µs client=127.0.0.1:59613 client_network=udp -46:08.900 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=A class=IN latency=144.94µs client=127.0.0.1:43737 client_network=udp -46:08.901 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=AAAA class=IN latency=97.024µs client=127.0.0.1:38719 client_network=udp -46:08.901 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=64.835µs client=127.0.0.1:54781 client_network=udp -46:08.901 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=58.433µs client=127.0.0.1:57862 client_network=udp -46:08.903 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.911 [INFO] TestAgent: Synced node info -46:08.912 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=118.295µs client=127.0.0.1:43788 client_network=udp -46:08.912 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=132.292µs client=127.0.0.1:57341 client_network=udp -46:08.912 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=A class=IN latency=94.313µs client=127.0.0.1:53991 client_network=udp -46:08.913 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=AAAA class=IN latency=106.532µs client=127.0.0.1:60040 client_network=udp -46:08.913 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=57.522µs client=127.0.0.1:36794 client_network=udp -46:08.913 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=101.17µs client=127.0.0.1:50327 client_network=udp -46:08.919 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=86.617µs client=127.0.0.1:58352 client_network=udp -46:08.920 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=73.67µs client=127.0.0.1:38401 client_network=udp -46:08.920 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=A class=IN latency=87.921µs client=127.0.0.1:52818 client_network=udp -46:08.920 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=AAAA class=IN latency=110.318µs client=127.0.0.1:51441 client_network=udp -46:08.920 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=63.81µs client=127.0.0.1:32940 client_network=udp -46:08.920 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=56.781µs client=127.0.0.1:53251 client_network=udp -46:08.927 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=90.83µs client=127.0.0.1:53937 client_network=udp -46:08.927 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=98.843µs client=127.0.0.1:34626 client_network=udp -46:08.927 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=A class=IN latency=84.16µs client=127.0.0.1:33768 client_network=udp -46:08.928 [DEBUG] TestAgent.dns: request served from client: name=4479d34c-22e9-64a8-17b7-3dda97901fa7.query.consul. type=AAAA class=IN latency=111.541µs client=127.0.0.1:57223 client_network=udp -46:08.928 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=62.581µs client=127.0.0.1:37305 client_network=udp -46:08.928 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=55.441µs client=127.0.0.1:53142 client_network=udp -46:08.928 [INFO] TestAgent: Requesting shutdown -46:08.928 [INFO] TestAgent.server: shutting down server -46:08.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.928 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.928 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.928 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.928 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.928 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.935 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.943Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.943Z [INFO] agent.router.manager: shutting down -46:08.943 [INFO] TestAgent: consul server down -46:08.943 [INFO] TestAgent: shutdown complete -46:08.943 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11323 network=tcp -46:08.943 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11323 network=udp -46:08.943 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11324 network=tcp -46:09.444 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.444 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Lookup_TaggedIPAddresses (0.88s) - --- PASS: TestDNS_Lookup_TaggedIPAddresses/simple-ipv4 (0.01s) - --- PASS: TestDNS_Lookup_TaggedIPAddresses/simple-ipv6 (0.01s) - --- PASS: TestDNS_Lookup_TaggedIPAddresses/ipv4-with-tagged-ipv6 (0.01s) - --- PASS: TestDNS_Lookup_TaggedIPAddresses/ipv6-with-tagged-ipv4 (0.01s) -=== CONT TestCatalogNodeServices_Filter -2022-11-07T22:46:08.778Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.778Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.787Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.787Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.815 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb34a001-7403-97a5-6642-1a64cc22d2b9 Address:127.0.0.1:11405}]" -46:08.815 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11405 [Follower]" leader= -46:08.816 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cb34a001-7403-97a5-6642-1a64cc22d2b9.dc1 127.0.0.1 -46:08.816 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 127.0.0.1 -2022-11-07T22:46:08.816Z [INFO] agent.router: Initializing LAN area manager -46:08.816 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9.dc1 area=wan -46:08.816 [INFO] TestAgent.server: Adding LAN server: server="Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 (Addr: tcp/127.0.0.1:11405) (DC: dc1)" -46:08.817 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11407 network=udp -46:08.817 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11407 network=tcp -46:08.817 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11408 network=tcp -46:08.817 [INFO] TestAgent: started state syncer -46:08.817 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11406 network=tcp -46:08.861 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.861 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11405 [Candidate]" term=2 -46:08.867 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:08.871 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.871 [DEBUG] TestAgent.server.raft: vote granted: from=cb34a001-7403-97a5-6642-1a64cc22d2b9 term=2 tally=1 -46:08.871 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.871 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11405 [Leader]" -46:08.872 [INFO] TestAgent.server: cluster leadership acquired -46:08.872 [INFO] TestAgent.server: New leader elected: payload=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.878 [INFO] TestAgent.server: initializing acls -46:08.881 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:08.881 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:08.887 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:08.896 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:08.896 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:08.896 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:08.896 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.897 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cb34a001-7403-97a5-6642-1a64cc22d2b9.dc1 -46:08.897 [INFO] TestAgent.server: Updating LAN server: server="Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 (Addr: tcp/127.0.0.1:11405) (DC: dc1)" -46:08.897 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9.dc1 area=wan -46:08.909 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.934 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.934 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.934 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.934 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.934 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.934 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.934 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.945 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.945 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.963 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-cb34a001-7403-97a5-6642-1a64cc22d2b9 -46:08.974 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.979 [INFO] TestAgent: Synced node info -46:08.979 [DEBUG] TestAgent: Node info in sync -46:08.988 [INFO] TestAgent: Requesting shutdown -46:08.988 [INFO] TestAgent.server: shutting down server -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:08.988 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:08.988 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:08.988 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:08.988 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:08.992 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:08.999Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:08.999Z [INFO] agent.router.manager: shutting down -46:09.000 [INFO] TestAgent: consul server down -46:09.000 [INFO] TestAgent: shutdown complete -46:09.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11407 network=tcp -46:09.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11407 network=udp -46:09.000 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11408 network=tcp -46:09.500 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.500 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecWrites_ACLToken (0.74s) -=== CONT TestPreparedQuery_List -=== RUN TestPreparedQuery_List/#00 -2022-11-07T22:46:08.618Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.635Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.666 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:81e582ea-1dbe-fff4-3596-a900ef062e78 Address:127.0.0.1:11384}]" -46:08.666 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11384 [Follower]" leader= -46:08.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-81e582ea-1dbe-fff4-3596-a900ef062e78.dc1 127.0.0.1 -46:08.667 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-81e582ea-1dbe-fff4-3596-a900ef062e78 127.0.0.1 -2022-11-07T22:46:08.667Z [INFO] agent.router: Initializing LAN area manager -46:08.668 [INFO] TestAgent.server: Adding LAN server: server="Node-81e582ea-1dbe-fff4-3596-a900ef062e78 (Addr: tcp/127.0.0.1:11384) (DC: dc1)" -46:08.668 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-81e582ea-1dbe-fff4-3596-a900ef062e78.dc1 area=wan -46:08.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11351 network=udp -46:08.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11351 network=tcp -46:08.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11352 network=tcp -46:08.668 [INFO] TestAgent: started state syncer -46:08.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11385 network=tcp -46:08.717 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.717 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11384 [Candidate]" term=2 -46:08.734 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.734 [DEBUG] TestAgent.server.raft: vote granted: from=81e582ea-1dbe-fff4-3596-a900ef062e78 term=2 tally=1 -46:08.734 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.734 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11384 [Leader]" -46:08.734 [INFO] TestAgent.server: cluster leadership acquired -46:08.734 [INFO] TestAgent.server: New leader elected: payload=Node-81e582ea-1dbe-fff4-3596-a900ef062e78 -46:08.744 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11384 -46:08.757 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.784 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.784 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.784 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.784 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.784 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.784 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81e582ea-1dbe-fff4-3596-a900ef062e78 -46:08.784 [INFO] TestAgent.server: member joined, marking health alive: member=Node-81e582ea-1dbe-fff4-3596-a900ef062e78 -46:08.804 [INFO] TestAgent.server: federation state anti-entropy synced -46:08.957 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.961 [INFO] TestAgent: Synced node info -46:08.961 [DEBUG] TestAgent: Node info in sync -46:09.026 [INFO] TestAgent: Requesting shutdown -46:09.027 [INFO] TestAgent.server: shutting down server -46:09.027 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.027 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.027 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.027 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.027 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.027 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.027 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.027 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.027 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.027 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.036 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.040Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.040Z [INFO] agent.router.manager: shutting down -46:09.040 [INFO] TestAgent: consul server down -46:09.040 [INFO] TestAgent: shutdown complete -46:09.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11351 network=tcp -46:09.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11351 network=udp -46:09.041 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11352 network=tcp -46:09.541 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.541 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_default_max_kv_value_size [WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc1 -2022-11-07T22:46:08.813Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.823Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.846 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f720f031-68dd-863f-353a-031eafa2b8d6 Address:127.0.0.1:11370}]" -46:08.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11370 [Follower]" leader= -46:08.847 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f720f031-68dd-863f-353a-031eafa2b8d6.dc1 127.0.0.1 -46:08.847 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f720f031-68dd-863f-353a-031eafa2b8d6 127.0.0.1 -2022-11-07T22:46:08.847Z [INFO] agent.router: Initializing LAN area manager -46:08.848 [INFO] TestAgent.server: Adding LAN server: server="Node-f720f031-68dd-863f-353a-031eafa2b8d6 (Addr: tcp/127.0.0.1:11370) (DC: dc1)" -46:08.848 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f720f031-68dd-863f-353a-031eafa2b8d6.dc1 area=wan -46:08.848 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11421 network=tcp -46:08.848 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11421 network=udp -46:08.848 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11422 network=tcp -46:08.848 [INFO] TestAgent: started state syncer -46:08.848 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11371 network=tcp -46:08.900 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.900 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11370 [Candidate]" term=2 -46:08.912 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.912 [DEBUG] TestAgent.server.raft: vote granted: from=f720f031-68dd-863f-353a-031eafa2b8d6 term=2 tally=1 -46:08.912 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.912 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11370 [Leader]" -46:08.912 [INFO] TestAgent.server: cluster leadership acquired -46:08.912 [INFO] TestAgent.server: New leader elected: payload=Node-f720f031-68dd-863f-353a-031eafa2b8d6 -46:08.919 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11370 -46:08.933 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:08.955 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:08.955 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:08.955 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:08.955 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:08.955 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:08.955 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f720f031-68dd-863f-353a-031eafa2b8d6 -46:08.955 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f720f031-68dd-863f-353a-031eafa2b8d6 -46:08.970 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.118 [INFO] TestAgent: Requesting shutdown -46:09.118 [INFO] TestAgent.server: shutting down server -46:09.118 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.118 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.118 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.118 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.118 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.118 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.118 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.118 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.118 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.118 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.118 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.134 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.150Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.150Z [INFO] agent.router.manager: shutting down -46:09.150 [INFO] TestAgent: consul server down -46:09.150 [INFO] TestAgent: shutdown complete -46:09.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11421 network=tcp -46:09.150 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11421 network=udp -46:09.150 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11422 network=tcp -46:09.650 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.650 [INFO] TestAgent: Endpoints down ---- PASS: TestSessionCreate_Delete (0.85s) -=== CONT TestDNS_ServiceLookup_ServiceAddress_A -=== RUN TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc1 -=== RUN TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc1 +2021-10-05T16:31:21.967Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.033Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.116 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca5599d1-4fe0-03aa-23e6-a2ea501a3944 Address:127.0.0.1:30169}]" +31:22.117 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944.dc1 127.0.0.1 +31:22.117 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944 127.0.0.1 +2021-10-05T16:31:22.117Z [INFO] agent.router: Initializing LAN area manager +31:22.118 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30164 network=udp +31:22.118 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30169 [Follower]" leader= +31:22.118 [INFO] TestAgent.server: Adding LAN server: server="Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944 (Addr: tcp/127.0.0.1:30169) (DC: dc1)" +31:22.118 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944.dc1 area=wan +31:22.118 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30164 network=tcp +31:22.119 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30165 network=tcp +31:22.119 [INFO] TestAgent: started state syncer +31:22.119 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30170 network=tcp +31:22.179 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.180 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30169 [Candidate]" term=2 +31:22.191 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.191 [DEBUG] TestAgent.server.raft: vote granted: from=ca5599d1-4fe0-03aa-23e6-a2ea501a3944 term=2 tally=1 +31:22.191 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.191 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30169 [Leader]" +31:22.192 [INFO] TestAgent.server: cluster leadership acquired +31:22.192 [INFO] TestAgent.server: New leader elected: payload=Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944 +31:22.198 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30169 +31:22.210 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.248 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.248 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.248 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.248 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.248 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944 +31:22.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ca5599d1-4fe0-03aa-23e6-a2ea501a3944 +31:22.256 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.303 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=A class=IN latency=185.99µs client=127.0.0.1:50686 client_network=udp +31:22.328 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=A class=IN latency=193.554µs client=127.0.0.1:39394 client_network=udp +31:22.328 [INFO] TestAgent: Requesting shutdown +31:22.328 [INFO] TestAgent.server: shutting down server +31:22.328 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.328 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.328 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.329 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.329 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.329 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:22.329 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.329 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.329 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.336 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.340Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.340Z [INFO] agent.router.manager: shutting down +31:22.340 [INFO] TestAgent: consul server down +31:22.340 [INFO] TestAgent: shutdown complete +31:22.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30164 network=tcp +31:22.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30164 network=udp +31:22.340 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30165 network=tcp +31:22.840 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.840 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQueryNearIP (0.91s) +=== CONT TestDNS_ExternalServiceToConsulCNAMENestedLookup +2021-10-05T16:31:22.098Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.134Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.227 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4be69e9-f138-bf3d-eb06-42ff2384e2e8 Address:127.0.0.1:30176}]" +31:22.228 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30176 [Follower]" leader= +31:22.228 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8.dc1 127.0.0.1 +31:22.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8 127.0.0.1 +2021-10-05T16:31:22.229Z [INFO] agent.router: Initializing LAN area manager +31:22.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30171 network=udp +31:22.229 [INFO] TestAgent.server: Adding LAN server: server="Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8 (Addr: tcp/127.0.0.1:30176) (DC: dc1)" +31:22.229 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8.dc1 area=wan +31:22.230 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30171 network=tcp +31:22.230 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30172 network=tcp +31:22.230 [INFO] TestAgent: started state syncer +31:22.230 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30184 network=tcp +31:22.294 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.294 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30176 [Candidate]" term=2 +31:22.310 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.310 [DEBUG] TestAgent.server.raft: vote granted: from=e4be69e9-f138-bf3d-eb06-42ff2384e2e8 term=2 tally=1 +31:22.310 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.310 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30176 [Leader]" +31:22.310 [INFO] TestAgent.server: cluster leadership acquired +31:22.310 [INFO] TestAgent.server: New leader elected: payload=Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8 +31:22.325 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30176 +31:22.331 [INFO] TestAgent: Synced node info +31:22.331 [DEBUG] TestAgent: Node info in sync +31:22.349 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.448 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.448 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.448 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.448 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.448 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.448 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8 +31:22.448 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e4be69e9-f138-bf3d-eb06-42ff2384e2e8 +31:22.468 [DEBUG] TestAgent.dns: request served from client: name=some.query.we.like.query.consul. type=A class=IN latency=270.998µs client=127.0.0.1:38880 client_network=udp +31:22.468 [INFO] TestAgent: Requesting shutdown +31:22.468 [INFO] TestAgent.server: shutting down server +31:22.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.468 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.468 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.468 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.468 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.468 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.484 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:22.489 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.489 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:31:22.492Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.492Z [INFO] agent.router.manager: shutting down +31:22.492 [INFO] TestAgent: consul server down +31:22.492 [INFO] TestAgent: shutdown complete +31:22.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30171 network=tcp +31:22.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30171 network=udp +31:22.492 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30172 network=tcp +31:22.992 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.992 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_PreparedQueryNearIPEDNS (0.98s) +=== CONT TestDNS_NSRecords_IPV6 +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc1 === RUN TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc2 === RUN TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc2 === RUN TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc2 === RUN TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc2 -2022-11-07T22:46:08.899Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.907Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:08.939 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6ce9e299-3734-3c18-6e0d-0616ca59d5a1 Address:127.0.0.1:11412}]" -46:08.939 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11412 [Follower]" leader= -46:08.940 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1.dc1 127.0.0.1 -46:08.940 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1 127.0.0.1 -2022-11-07T22:46:08.940Z [INFO] agent.router: Initializing LAN area manager -46:08.940 [INFO] TestAgent.server: Adding LAN server: server="Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1 (Addr: tcp/127.0.0.1:11412) (DC: dc1)" -46:08.940 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1.dc1 area=wan -46:08.940 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11372 network=udp -46:08.940 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11372 network=tcp -46:08.941 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11373 network=tcp -46:08.941 [INFO] TestAgent: started state syncer -46:08.941 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11413 network=tcp -46:08.979 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:08.979 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11412 [Candidate]" term=2 -46:08.990 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:08.990 [DEBUG] TestAgent.server.raft: vote granted: from=6ce9e299-3734-3c18-6e0d-0616ca59d5a1 term=2 tally=1 -46:08.990 [INFO] TestAgent.server.raft: election won: tally=1 -46:08.990 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11412 [Leader]" -46:08.990 [INFO] TestAgent.server: cluster leadership acquired -46:08.990 [INFO] TestAgent.server: New leader elected: payload=Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1 -46:08.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11412 -46:09.012 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.046 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.046 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.046 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.046 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.046 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.046 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1 -46:09.046 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6ce9e299-3734-3c18-6e0d-0616ca59d5a1 -46:09.057 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.113 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.122 [INFO] TestAgent: Synced node info -46:09.122 [DEBUG] TestAgent: Node info in sync -46:09.264 [INFO] TestAgent: Requesting shutdown -46:09.264 [INFO] TestAgent.server: shutting down server -46:09.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.264 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.264 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.264 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.264 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.264 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.264 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.267 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.274Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.274Z [INFO] agent.router.manager: shutting down -46:09.274 [INFO] TestAgent: consul server down -46:09.274 [INFO] TestAgent: shutdown complete -46:09.274 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11372 network=tcp -46:09.274 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11372 network=udp -46:09.274 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11373 network=tcp -46:09.775 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.775 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecWrites (0.90s) -=== CONT TestPreparedQuery_Create -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:05.906Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.925Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:05.965 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a4779b8-8639-70a3-39c9-ef3bb1062f2e Address:127.0.0.1:12026}]" -46:05.965 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12026 [Follower]" leader= -46:05.965 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e.dc1 127.0.0.1 -46:05.966 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e 127.0.0.1 -2022-11-07T22:46:05.966Z [INFO] agent.router: Initializing LAN area manager -46:05.966 [INFO] TestAgent.server: Adding LAN server: server="Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e (Addr: tcp/127.0.0.1:12026) (DC: dc1)" -46:05.966 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e.dc1 area=wan -46:05.966 [DEBUG] TestAgent.dns: recursor enabled -46:05.966 [DEBUG] TestAgent.dns: recursor enabled -46:05.966 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12021 network=udp -46:05.966 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12021 network=tcp -46:05.967 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12022 network=tcp -46:05.967 [INFO] TestAgent: started state syncer -46:05.967 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12027 network=tcp -46:06.019 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.019 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12026 [Candidate]" term=2 -46:06.030 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.030 [DEBUG] TestAgent.server.raft: vote granted: from=1a4779b8-8639-70a3-39c9-ef3bb1062f2e term=2 tally=1 -46:06.030 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.030 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12026 [Leader]" -46:06.030 [INFO] TestAgent.server: cluster leadership acquired -46:06.031 [INFO] TestAgent.server: New leader elected: payload=Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e -46:06.040 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12026 -46:06.065 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.092 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.092 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.092 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.092 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.092 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.092 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e -46:06.092 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a4779b8-8639-70a3-39c9-ef3bb1062f2e -46:06.109 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.207 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.210 [INFO] TestAgent: Synced node info -46:06.210 [DEBUG] TestAgent: Node info in sync -46:08.078 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.078 [DEBUG] TestAgent: Node info in sync -46:09.383 [ERROR] TestAgent.dns: recurse failed: error="read udp 127.0.0.1:32951->127.0.0.1:37332: i/o timeout" -46:09.383 [ERROR] TestAgent.dns: all resolvers failed for question from client: question="{apple.com. 255 1}" client=127.0.0.1:40422 client_network=udp -46:09.383 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=3.000233477s client=127.0.0.1:40422 client_network=udp -46:09.383 [INFO] TestAgent: Requesting shutdown -46:09.383 [INFO] TestAgent.server: shutting down server -46:09.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.383 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.383 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.383 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.383 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.383 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.383 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.393 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.400Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.400Z [INFO] agent.router.manager: shutting down -46:09.400 [INFO] TestAgent: consul server down -46:09.400 [INFO] TestAgent: shutdown complete -46:09.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12021 network=tcp -46:09.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12021 network=udp -46:09.400 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12022 network=tcp -46:09.901 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.901 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_RecursorTimeout (4.01s) -=== CONT TestRemoteExecGetSpec_ACLToken -2022-11-07T22:46:09.101Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.101Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.110Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.110Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.187 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f784e10-6da2-524b-b607-8724a3a93b27 Address:127.0.0.1:11328}]" -46:09.187 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11328 [Follower]" leader= -46:09.187 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4f784e10-6da2-524b-b607-8724a3a93b27.dc1 127.0.0.1 -46:09.188 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4f784e10-6da2-524b-b607-8724a3a93b27 127.0.0.1 -2022-11-07T22:46:09.188Z [INFO] agent.router: Initializing LAN area manager -46:09.188 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11379 network=udp -46:09.188 [INFO] TestAgent.server: Adding LAN server: server="Node-4f784e10-6da2-524b-b607-8724a3a93b27 (Addr: tcp/127.0.0.1:11328) (DC: dc1)" -46:09.188 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4f784e10-6da2-524b-b607-8724a3a93b27.dc1 area=wan -46:09.188 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11379 network=tcp -46:09.189 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11380 network=tcp -46:09.189 [INFO] TestAgent: started state syncer -46:09.189 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11329 network=tcp -46:09.238 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.248 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.248 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11328 [Candidate]" term=2 -46:09.265 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.265 [DEBUG] TestAgent.server.raft: vote granted: from=4f784e10-6da2-524b-b607-8724a3a93b27 term=2 tally=1 -46:09.265 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.265 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11328 [Leader]" -46:09.265 [INFO] TestAgent.server: cluster leadership acquired -46:09.265 [INFO] TestAgent.server: New leader elected: payload=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.274 [INFO] TestAgent.server: initializing acls -46:09.284 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.284 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:09.296 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:09.315 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.315 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.315 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.316 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.316 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4f784e10-6da2-524b-b607-8724a3a93b27.dc1 -46:09.316 [INFO] TestAgent.server: Updating LAN server: server="Node-4f784e10-6da2-524b-b607-8724a3a93b27 (Addr: tcp/127.0.0.1:11328) (DC: dc1)" -46:09.316 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4f784e10-6da2-524b-b607-8724a3a93b27.dc1 area=wan -46:09.331 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.362 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.362 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.362 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.362 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.362 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.362 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.362 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.370 [WARN] TestAgent: Node info update blocked by ACLs: node=4f784e10-6da2-524b-b607-8724a3a93b27 accessorID=00000000-0000-0000-0000-000000000002 -46:09.377 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.378 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.498 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.498 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4f784e10-6da2-524b-b607-8724a3a93b27 -46:09.513 [ERROR] TestAgent: failed to get remote exec job: error="Permission denied" -46:09.520 [INFO] TestAgent: Requesting shutdown -46:09.520 [INFO] TestAgent.server: shutting down server -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.520 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.520 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.520 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:09.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.526 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.533Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.533Z [INFO] agent.router.manager: shutting down -46:09.533 [INFO] TestAgent: consul server down -46:09.533 [INFO] TestAgent: shutdown complete -46:09.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11379 network=tcp -46:09.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11379 network=udp -46:09.533 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11380 network=tcp -46:10.034 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.034 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecGetSpec_ACLDeny (0.94s) -=== CONT TestCatalogNodeServiceList -2022-11-07T22:46:09.244Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.254Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.303 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a836c72e-11d0-f61c-c4b7-7e8bf7421a6a Address:127.0.0.1:11496}]" -46:09.303 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11496 [Follower]" leader= -46:09.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a.dc1 127.0.0.1 -46:09.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a 127.0.0.1 -2022-11-07T22:46:09.303Z [INFO] agent.router: Initializing LAN area manager -46:09.304 [INFO] TestAgent.server: Adding LAN server: server="Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a (Addr: tcp/127.0.0.1:11496) (DC: dc1)" -46:09.304 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a.dc1 area=wan -46:09.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11449 network=udp -46:09.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11449 network=tcp -46:09.304 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11450 network=tcp -46:09.304 [INFO] TestAgent: started state syncer -46:09.304 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11497 network=tcp -46:09.357 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.357 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11496 [Candidate]" term=2 -46:09.375 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.375 [DEBUG] TestAgent.server.raft: vote granted: from=a836c72e-11d0-f61c-c4b7-7e8bf7421a6a term=2 tally=1 -46:09.375 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.375 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11496 [Leader]" -46:09.375 [INFO] TestAgent.server: cluster leadership acquired -46:09.375 [INFO] TestAgent.server: New leader elected: payload=Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a -46:09.383 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11496 -46:09.399 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.420 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.420 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.420 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.420 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.420 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.420 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a -46:09.420 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a836c72e-11d0-f61c-c4b7-7e8bf7421a6a -46:09.435 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.532 [INFO] TestAgent: Requesting shutdown -46:09.532 [INFO] TestAgent.server: shutting down server -46:09.532 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.532 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.532 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.532 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.532 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.532 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.532 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.532 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.532 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.532 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.532 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.537 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.543Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.543Z [INFO] agent.router.manager: shutting down -46:09.543 [INFO] TestAgent: consul server down -46:09.543 [INFO] TestAgent: shutdown complete -46:09.543 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11449 network=tcp -46:09.543 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11449 network=udp -46:09.543 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11450 network=tcp -46:10.043 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.043 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodeServices_ConnectProxy (0.81s) -=== CONT TestRemoteExecGetSpec -2022-11-07T22:46:09.160Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.171Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.218 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cea54ba6-b741-5ff9-55e9-7655b627a7d8 Address:127.0.0.1:11468}]" -46:09.218 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11468 [Follower]" leader= -46:09.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8.dc1 127.0.0.1 -46:09.219 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8 127.0.0.1 -2022-11-07T22:46:09.219Z [INFO] agent.router: Initializing LAN area manager -46:09.219 [INFO] TestAgent.server: Adding LAN server: server="Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8 (Addr: tcp/127.0.0.1:11468) (DC: dc1)" -46:09.219 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8.dc1 area=wan -46:09.219 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11330 network=udp -46:09.219 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11330 network=tcp -46:09.219 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11331 network=tcp -46:09.219 [INFO] TestAgent: started state syncer -46:09.219 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11469 network=tcp -46:09.285 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.285 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11468 [Candidate]" term=2 -46:09.303 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.303 [DEBUG] TestAgent.server.raft: vote granted: from=cea54ba6-b741-5ff9-55e9-7655b627a7d8 term=2 tally=1 -46:09.303 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.303 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11468 [Leader]" -46:09.303 [INFO] TestAgent.server: cluster leadership acquired -46:09.303 [INFO] TestAgent.server: New leader elected: payload=Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8 -46:09.314 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11468 -46:09.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.373 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.373 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.373 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.373 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.373 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.373 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8 -46:09.373 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cea54ba6-b741-5ff9-55e9-7655b627a7d8 -46:09.392 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.542 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:09.542 [INFO] TestAgent: Requesting shutdown -46:09.542 [INFO] TestAgent.server: shutting down server -46:09.542 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.542 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.542 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.542 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.542 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.542 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.542 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.542 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.542 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.542 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.542 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.546 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.550Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.550Z [INFO] agent.router.manager: shutting down -46:09.550 [INFO] TestAgent: consul server down -46:09.550 [INFO] TestAgent: shutdown complete -46:09.550 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11330 network=tcp -46:09.550 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11330 network=udp -46:09.550 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11331 network=tcp -46:10.051 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.051 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_ExecuteCached (0.90s) -=== CONT TestDNS_ExternalServiceToConsulCNAMENestedLookup -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:09.170Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.170Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.181Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.181Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.226 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f366b35f-2698-e27a-a541-73bddbe20511 Address:127.0.0.1:11426}]" -46:09.226 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11426 [Follower]" leader= -46:09.227 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f366b35f-2698-e27a-a541-73bddbe20511.dc1 127.0.0.1 -46:09.227 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f366b35f-2698-e27a-a541-73bddbe20511 127.0.0.1 -2022-11-07T22:46:09.227Z [INFO] agent.router: Initializing LAN area manager -46:09.228 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11470 network=udp -46:09.228 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f366b35f-2698-e27a-a541-73bddbe20511.dc1 area=wan -46:09.228 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11470 network=tcp -46:09.228 [INFO] TestAgent.server: Adding LAN server: server="Node-f366b35f-2698-e27a-a541-73bddbe20511 (Addr: tcp/127.0.0.1:11426) (DC: dc1)" -46:09.228 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11471 network=tcp -46:09.228 [INFO] TestAgent: started state syncer -46:09.228 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11427 network=tcp -46:09.278 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.292 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.292 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11426 [Candidate]" term=2 -46:09.308 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.308 [DEBUG] TestAgent.server.raft: vote granted: from=f366b35f-2698-e27a-a541-73bddbe20511 term=2 tally=1 -46:09.308 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.308 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11426 [Leader]" -46:09.308 [INFO] TestAgent.server: cluster leadership acquired -46:09.308 [INFO] TestAgent.server: New leader elected: payload=Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.319 [INFO] TestAgent.server: initializing acls -46:09.322 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.322 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:09.332 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:09.346 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.346 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.346 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.346 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.346 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f366b35f-2698-e27a-a541-73bddbe20511.dc1 -46:09.346 [INFO] TestAgent.server: Updating LAN server: server="Node-f366b35f-2698-e27a-a541-73bddbe20511 (Addr: tcp/127.0.0.1:11426) (DC: dc1)" -46:09.346 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f366b35f-2698-e27a-a541-73bddbe20511.dc1 area=wan -46:09.362 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.396 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.396 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.396 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.396 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.396 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.396 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.412 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.412 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.449 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.461 [INFO] TestAgent: Synced node info -46:09.461 [DEBUG] TestAgent: Node info in sync -46:09.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f366b35f-2698-e27a-a541-73bddbe20511 -46:09.591 [INFO] TestAgent: Requesting shutdown -46:09.591 [INFO] TestAgent.server: shutting down server -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.591 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.591 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.600 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.610Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.610Z [INFO] agent.router.manager: shutting down -46:09.610 [INFO] TestAgent: consul server down -46:09.610 [INFO] TestAgent: shutdown complete -46:09.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11470 network=tcp -46:09.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11470 network=udp -46:09.610 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11471 network=tcp -46:10.110 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.110 [INFO] TestAgent: Endpoints down -=== RUN TestSnapshot_Options/PUT#01 -2022-11-07T22:46:09.178Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.188Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.238 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca Address:127.0.0.1:11419}]" -46:09.238 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11419 [Follower]" leader= -46:09.238 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca.dc1 127.0.0.1 -46:09.240 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca 127.0.0.1 -2022-11-07T22:46:09.240Z [INFO] agent.router: Initializing LAN area manager -46:09.240 [INFO] TestAgent.server: Adding LAN server: server="Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca (Addr: tcp/127.0.0.1:11419) (DC: dc1)" -46:09.240 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca.dc1 area=wan -46:09.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11456 network=udp -46:09.240 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11456 network=tcp -46:09.240 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11457 network=tcp -46:09.240 [INFO] TestAgent: started state syncer -46:09.240 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11420 network=tcp -46:09.275 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.275 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11419 [Candidate]" term=2 -46:09.299 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.299 [DEBUG] TestAgent.server.raft: vote granted: from=fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca term=2 tally=1 -46:09.299 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.299 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11419 [Leader]" -46:09.299 [INFO] TestAgent.server: cluster leadership acquired -46:09.299 [INFO] TestAgent.server: New leader elected: payload=Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca -46:09.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11419 -46:09.329 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.362 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.362 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.362 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.362 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.362 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.362 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca -46:09.362 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fd9ab2f2-a5ba-5d07-10ff-0ef4476d63ca -46:09.384 [INFO] TestAgent: Synced node info -46:09.384 [DEBUG] TestAgent: Node info in sync -46:09.392 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.598 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=143.873µs client=127.0.0.1:40320 client_network=udp -46:09.598 [DEBUG] TestAgent.dns: request served from client: name=f46b976b-5bc1-63de-5a8d-2a1daa601569.query.consul. type=SRV class=IN latency=125.134µs client=127.0.0.1:45431 client_network=udp -46:09.598 [INFO] TestAgent: Requesting shutdown -46:09.598 [INFO] TestAgent.server: shutting down server -46:09.598 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.598 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.598 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.598 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.598 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.598 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.598 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.598 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.598 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.598 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.605 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.614Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.614Z [INFO] agent.router.manager: shutting down -46:09.614 [INFO] TestAgent: consul server down -46:09.614 [INFO] TestAgent: shutdown complete -46:09.614 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11456 network=tcp -46:09.614 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11456 network=udp -46:09.614 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11457 network=tcp -46:10.114 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.114 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_ServiceAddressIPV6 (0.95s) -=== CONT TestOperator_ServerHealth_Unhealthy -2022-11-07T22:46:09.319Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.328Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.370 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dbf0f890-60fd-1029-1af6-a89d3a69cc4a Address:127.0.0.1:11510}]" -46:09.370 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11510 [Follower]" leader= -46:09.370 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a.dc1 127.0.0.1 -46:09.370 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a 127.0.0.1 -2022-11-07T22:46:09.371Z [INFO] agent.router: Initializing LAN area manager -46:09.371 [INFO] TestAgent.server: Adding LAN server: server="Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a (Addr: tcp/127.0.0.1:11510) (DC: dc1)" -46:09.371 [DEBUG] TestAgent.dns: recursor enabled -46:09.371 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a.dc1 area=wan -46:09.371 [DEBUG] TestAgent.dns: recursor enabled -46:09.371 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11477 network=udp -46:09.371 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11477 network=tcp -46:09.371 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11478 network=tcp -46:09.371 [INFO] TestAgent: started state syncer -46:09.371 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11511 network=tcp -46:09.416 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.416 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11510 [Candidate]" term=2 -46:09.430 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.430 [DEBUG] TestAgent.server.raft: vote granted: from=dbf0f890-60fd-1029-1af6-a89d3a69cc4a term=2 tally=1 -46:09.430 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.430 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11510 [Leader]" -46:09.430 [INFO] TestAgent.server: cluster leadership acquired -46:09.430 [INFO] TestAgent.server: New leader elected: payload=Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a -46:09.446 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11510 -46:09.467 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.499 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.499 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.499 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.499 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.499 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.499 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a -46:09.499 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dbf0f890-60fd-1029-1af6-a89d3a69cc4a -46:09.521 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.611 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=138.114µs -46:09.611 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=340.608µs client=127.0.0.1:43505 client_network=udp -46:09.612 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=83.553µs -46:09.612 [DEBUG] TestAgent.dns: request served from client: name=be0627c9-8590-7db3-2d61-6c0ce95c07d1.query.consul. type=SRV class=IN latency=274.141µs client=127.0.0.1:48107 client_network=udp -46:09.612 [INFO] TestAgent: Requesting shutdown -46:09.612 [INFO] TestAgent.server: shutting down server -46:09.612 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.612 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.612 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.612 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.612 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.612 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.612 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.618 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.623Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.623Z [INFO] agent.router.manager: shutting down -46:09.623 [INFO] TestAgent: consul server down -46:09.623 [INFO] TestAgent: shutdown complete -46:09.623 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11477 network=tcp -46:09.623 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11477 network=udp -46:09.623 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11478 network=tcp -46:10.123 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.123 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_ServiceAddress_SRV (0.82s) -=== CONT TestOperator_ServerHealth -2022-11-07T22:46:09.510Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.519Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.552 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f03b215f-b1d0-2379-f225-a2dd74fd3de4 Address:127.0.0.1:11454}]" -46:09.552 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11454 [Follower]" leader= -46:09.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4.dc1 127.0.0.1 -46:09.553 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4 127.0.0.1 -2022-11-07T22:46:09.553Z [INFO] agent.router: Initializing LAN area manager -46:09.553 [INFO] TestAgent.server: Adding LAN server: server="Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4 (Addr: tcp/127.0.0.1:11454) (DC: dc1)" -46:09.553 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4.dc1 area=wan -46:09.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11435 network=udp -46:09.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11435 network=tcp -46:09.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11436 network=tcp -46:09.554 [INFO] TestAgent: started state syncer -46:09.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11455 network=tcp -46:09.591 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.591 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11454 [Candidate]" term=2 -46:09.610 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.610 [DEBUG] TestAgent.server.raft: vote granted: from=f03b215f-b1d0-2379-f225-a2dd74fd3de4 term=2 tally=1 -46:09.610 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.610 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11454 [Leader]" -46:09.610 [INFO] TestAgent.server: cluster leadership acquired -46:09.610 [INFO] TestAgent.server: New leader elected: payload=Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4 -46:09.619 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11454 -46:09.633 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.661 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.661 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.661 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.661 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.661 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.661 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4 -46:09.661 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f03b215f-b1d0-2379-f225-a2dd74fd3de4 -46:09.670 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.768 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:09.768 [INFO] TestAgent: Requesting shutdown -46:09.768 [INFO] TestAgent.server: shutting down server -46:09.768 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.768 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.768 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.768 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.768 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.768 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.768 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.768 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.768 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.768 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.768 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.773 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.780Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.780Z [INFO] agent.router.manager: shutting down -46:09.780 [INFO] TestAgent: consul server down -46:09.780 [INFO] TestAgent: shutdown complete -46:09.780 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11435 network=tcp -46:09.780 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11435 network=udp -46:09.780 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11436 network=tcp -46:10.280 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.280 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_List/#01 -2022-11-07T22:46:09.315Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.324Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.366 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9b95f3ec-86a0-c9dc-d61b-0fc94f00696a Address:127.0.0.1:11475}]" -46:09.367 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11475 [Follower]" leader= -46:09.367 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a.dc1 127.0.0.1 -46:09.368 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a 127.0.0.1 -2022-11-07T22:46:09.368Z [INFO] agent.router: Initializing LAN area manager -46:09.368 [INFO] TestAgent.server: Adding LAN server: server="Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a (Addr: tcp/127.0.0.1:11475) (DC: dc1)" -46:09.368 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11484 network=udp -46:09.368 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11484 network=tcp -46:09.368 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a.dc1 area=wan -46:09.368 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11485 network=tcp -46:09.368 [INFO] TestAgent: started state syncer -46:09.368 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11476 network=tcp -46:09.414 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.414 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11475 [Candidate]" term=2 -46:09.430 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.430 [DEBUG] TestAgent.server.raft: vote granted: from=9b95f3ec-86a0-c9dc-d61b-0fc94f00696a term=2 tally=1 -46:09.430 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.430 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11475 [Leader]" -46:09.430 [INFO] TestAgent.server: cluster leadership acquired -46:09.430 [INFO] TestAgent.server: New leader elected: payload=Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a -46:09.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11475 -46:09.464 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.493 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.493 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.493 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.493 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.493 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.493 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a -46:09.493 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9b95f3ec-86a0-c9dc-d61b-0fc94f00696a -46:09.514 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.695 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.701 [INFO] TestAgent: Synced node info -46:09.772 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:09.772 [INFO] TestAgent: Requesting shutdown -46:09.772 [INFO] TestAgent.server: shutting down server -46:09.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.772 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.772 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.772 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.772 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.772 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.772 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.772 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.772 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.776 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.786Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.786Z [INFO] agent.router.manager: shutting down -46:09.786 [INFO] TestAgent: consul server down -46:09.786 [INFO] TestAgent: shutdown complete -46:09.786 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11484 network=tcp -46:09.786 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11484 network=udp -46:09.786 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11485 network=tcp -46:10.286 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.286 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#01 -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:09.454Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.463Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.504 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5bb7c677-ec26-44e0-f95d-1a6f751b080a Address:127.0.0.1:11440}]" -46:09.504 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11440 [Follower]" leader= -46:09.504 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a.dc1 127.0.0.1 -46:09.505 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a 127.0.0.1 -2022-11-07T22:46:09.505Z [INFO] agent.router: Initializing LAN area manager -46:09.505 [INFO] TestAgent.server: Adding LAN server: server="Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a (Addr: tcp/127.0.0.1:11440) (DC: dc1)" -46:09.505 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a.dc1 area=wan -46:09.505 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11505 network=udp -46:09.505 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11505 network=tcp -46:09.505 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11506 network=tcp -46:09.505 [INFO] TestAgent: started state syncer -46:09.506 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11441 network=tcp -46:09.563 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.563 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11440 [Candidate]" term=2 -46:09.574 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.574 [DEBUG] TestAgent.server.raft: vote granted: from=5bb7c677-ec26-44e0-f95d-1a6f751b080a term=2 tally=1 -46:09.574 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.574 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11440 [Leader]" -46:09.574 [INFO] TestAgent.server: cluster leadership acquired -46:09.574 [INFO] TestAgent.server: New leader elected: payload=Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a -46:09.584 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11440 -46:09.603 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.631 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.631 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.631 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.631 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.631 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.631 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a -46:09.631 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5bb7c677-ec26-44e0-f95d-1a6f751b080a -46:09.639 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.795 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.800 [INFO] TestAgent: Synced node info -46:09.843 [INFO] TestAgent: Requesting shutdown -46:09.843 [INFO] TestAgent.server: shutting down server -46:09.843 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.843 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.843 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.843 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.843 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.843 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.843 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.843 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.843 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.843 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.846 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.850Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.850Z [INFO] agent.router.manager: shutting down -46:09.850 [INFO] TestAgent: consul server down -46:09.850 [INFO] TestAgent: shutdown complete -46:09.850 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11505 network=tcp -46:09.850 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11505 network=udp -46:09.850 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11506 network=tcp -46:10.350 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.350 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodeServices_Filter (0.91s) -=== CONT TestCatalogNodeServices -2022-11-07T22:46:09.334Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.334Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.349Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.349Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.391 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:69790fc0-1025-371f-c19b-30f1856e441f Address:127.0.0.1:11503}]" -46:09.391 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11503 [Follower]" leader= -46:09.392 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-69790fc0-1025-371f-c19b-30f1856e441f.dc1 127.0.0.1 -46:09.392 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-69790fc0-1025-371f-c19b-30f1856e441f 127.0.0.1 -2022-11-07T22:46:09.392Z [INFO] agent.router: Initializing LAN area manager -46:09.392 [INFO] TestAgent.server: Adding LAN server: server="Node-69790fc0-1025-371f-c19b-30f1856e441f (Addr: tcp/127.0.0.1:11503) (DC: dc1)" -46:09.392 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-69790fc0-1025-371f-c19b-30f1856e441f.dc1 area=wan -46:09.392 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11463 network=tcp -46:09.392 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11463 network=udp -46:09.393 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11464 network=tcp -46:09.393 [INFO] TestAgent: started state syncer -46:09.393 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11504 network=tcp -46:09.430 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.431 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11503 [Candidate]" term=2 -46:09.443 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.445 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.445 [DEBUG] TestAgent.server.raft: vote granted: from=69790fc0-1025-371f-c19b-30f1856e441f term=2 tally=1 -46:09.445 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.445 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11503 [Leader]" -46:09.445 [INFO] TestAgent.server: cluster leadership acquired -46:09.445 [INFO] TestAgent.server: New leader elected: payload=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.456 [INFO] TestAgent.server: initializing acls -46:09.461 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.461 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:09.467 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:09.479 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.479 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.479 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.479 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.480 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-69790fc0-1025-371f-c19b-30f1856e441f.dc1 -46:09.480 [INFO] TestAgent.server: Updating LAN server: server="Node-69790fc0-1025-371f-c19b-30f1856e441f (Addr: tcp/127.0.0.1:11503) (DC: dc1)" -46:09.480 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-69790fc0-1025-371f-c19b-30f1856e441f.dc1 area=wan -46:09.496 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.516 [INFO] TestAgent: Synced node info -46:09.516 [DEBUG] TestAgent: Node info in sync -46:09.525 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.525 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.525 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.525 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.525 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.525 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.525 [INFO] TestAgent.server: member joined, marking health alive: member=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.541 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.856 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.856 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-69790fc0-1025-371f-c19b-30f1856e441f -46:09.866 [INFO] TestAgent: Requesting shutdown -46:09.866 [INFO] TestAgent.server: shutting down server -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.866 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:09.866 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.866 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.866 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:09.870 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.874Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.874Z [INFO] agent.router.manager: shutting down -46:09.874 [INFO] TestAgent: consul server down -46:09.874 [INFO] TestAgent: shutdown complete -46:09.874 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11463 network=tcp -46:09.874 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11463 network=udp -46:09.874 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11464 network=tcp -46:10.375 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.375 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecGetSpec_ACLAgentToken (1.05s) -=== CONT TestPreparedQuery_Integration -2022-11-07T22:46:09.660Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.669Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.698 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6e29c288-6f9f-bcfe-cb9a-873e4178a58c Address:127.0.0.1:11523}]" -46:09.698 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11523 [Follower]" leader= -46:09.699 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c.dc1 127.0.0.1 -46:09.699 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c 127.0.0.1 -2022-11-07T22:46:09.699Z [INFO] agent.router: Initializing LAN area manager -46:09.699 [INFO] TestAgent.server: Adding LAN server: server="Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c (Addr: tcp/127.0.0.1:11523) (DC: dc1)" -46:09.699 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c.dc1 area=wan -46:09.699 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11498 network=udp -46:09.699 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11498 network=tcp -46:09.700 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11499 network=tcp -46:09.700 [INFO] TestAgent: started state syncer -46:09.700 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11524 network=tcp -46:09.758 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.758 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11523 [Candidate]" term=2 -46:09.767 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.767 [DEBUG] TestAgent.server.raft: vote granted: from=6e29c288-6f9f-bcfe-cb9a-873e4178a58c term=2 tally=1 -46:09.767 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.767 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11523 [Leader]" -46:09.767 [INFO] TestAgent.server: cluster leadership acquired -46:09.767 [INFO] TestAgent.server: New leader elected: payload=Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c -46:09.775 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11523 -46:09.793 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.821 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.821 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.821 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.821 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.821 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c -46:09.821 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6e29c288-6f9f-bcfe-cb9a-873e4178a58c -46:09.833 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.960 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=146.691µs client=127.0.0.1:56189 client_network=udp -46:09.961 [DEBUG] TestAgent.dns: request served from client: name=5bc1b894-7b7f-998c-3789-c282c80fb078.query.consul. type=SRV class=IN latency=126.53µs client=127.0.0.1:42848 client_network=udp -46:09.961 [INFO] TestAgent: Requesting shutdown -46:09.961 [INFO] TestAgent.server: shutting down server -46:09.961 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.961 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.961 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.961 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.961 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.961 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.961 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.961 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.961 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.961 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.961 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.964 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:09.970 [INFO] TestAgent: consul server down -46:09.970 [INFO] TestAgent: shutdown complete -46:09.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11498 network=tcp -46:09.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11498 network=udp -46:09.970 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11499 network=tcp -2022-11-07T22:46:09.968Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.968Z [INFO] agent.router.manager: shutting down -46:10.470 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.470 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_ServiceAddress_A (0.82s) -=== CONT TestOperator_AutopilotCASConfiguration -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:09.551Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.561Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.586 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 Address:127.0.0.1:11489}]" -46:09.586 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11489 [Follower]" leader= -46:09.587 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48.dc1 127.0.0.1 -46:09.587 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 127.0.0.1 -2022-11-07T22:46:09.587Z [INFO] agent.router: Initializing LAN area manager -46:09.587 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48.dc1 area=wan -46:09.587 [INFO] TestAgent.server: Adding LAN server: server="Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 (Addr: tcp/127.0.0.1:11489) (DC: dc1)" -46:09.587 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11428 network=udp -46:09.587 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11428 network=tcp -46:09.588 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11429 network=tcp -46:09.588 [INFO] TestAgent: started state syncer -46:09.588 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11490 network=tcp -46:09.646 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.646 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11489 [Candidate]" term=2 -46:09.657 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.657 [DEBUG] TestAgent.server.raft: vote granted: from=c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 term=2 tally=1 -46:09.657 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.657 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11489 [Leader]" -46:09.657 [INFO] TestAgent.server: cluster leadership acquired -46:09.657 [INFO] TestAgent.server: New leader elected: payload=Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 -46:09.666 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11489 -46:09.683 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.711 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.711 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.711 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.711 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.711 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.711 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 -46:09.711 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c8e7d47c-987e-6f0c-09b2-3bd36dcb5d48 -46:09.719 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.960 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.963 [INFO] TestAgent: Synced node info -46:10.097 [INFO] TestAgent: Requesting shutdown -46:10.097 [INFO] TestAgent.server: shutting down server -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.097 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.097 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.102 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.109Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.109Z [INFO] agent.router.manager: shutting down -46:10.110 [INFO] TestAgent: consul server down -46:10.110 [INFO] TestAgent: shutdown complete -46:10.110 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11428 network=tcp -46:10.110 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11428 network=udp -46:10.110 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11429 network=tcp -46:10.610 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.610 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Net/allowed -2022-11-07T22:46:09.784Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.793Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.825 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f11c0d2f-ed48-0d91-f9e9-54fa30275f3f Address:127.0.0.1:11530}]" -46:09.825 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11530 [Follower]" leader= -46:09.826 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f.dc1 127.0.0.1 -46:09.827 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f 127.0.0.1 -2022-11-07T22:46:09.827Z [INFO] agent.router: Initializing LAN area manager -46:09.827 [INFO] TestAgent.server: Adding LAN server: server="Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f (Addr: tcp/127.0.0.1:11530) (DC: dc1)" -46:09.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11525 network=udp -46:09.827 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f.dc1 area=wan -46:09.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11525 network=tcp -46:09.827 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11533 network=tcp -46:09.827 [INFO] TestAgent: started state syncer -46:09.827 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11531 network=tcp -46:09.891 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.891 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11530 [Candidate]" term=2 -46:09.902 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.902 [DEBUG] TestAgent.server.raft: vote granted: from=f11c0d2f-ed48-0d91-f9e9-54fa30275f3f term=2 tally=1 -46:09.902 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.902 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11530 [Leader]" -46:09.902 [INFO] TestAgent.server: cluster leadership acquired -46:09.902 [INFO] TestAgent.server: New leader elected: payload=Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f -46:09.910 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11530 -46:09.924 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.942 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.942 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.942 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.942 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.942 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.942 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f -46:09.942 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f11c0d2f-ed48-0d91-f9e9-54fa30275f3f -46:09.958 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.097 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:10.097 [INFO] TestAgent: Requesting shutdown -46:10.097 [INFO] TestAgent.server: shutting down server -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.097 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.097 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.097 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.097 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.097 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.107 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.112Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.112Z [INFO] agent.router.manager: shutting down -46:10.112 [INFO] TestAgent: consul server down -46:10.112 [INFO] TestAgent: shutdown complete -46:10.112 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11525 network=tcp -46:10.112 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11525 network=udp -46:10.112 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11533 network=tcp -46:10.612 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.612 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Create (0.84s) -=== CONT TestDNS_NSRecords_IPV6 -2022-11-07T22:46:09.314Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.314Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.324Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.324Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.358 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b819c8c-1441-4faa-b1c5-534b9e546b2a Address:127.0.0.1:11447}]" -46:09.358 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11447 [Follower]" leader= -46:09.359 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 127.0.0.1 -46:09.360 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a 127.0.0.1 -2022-11-07T22:46:09.360Z [INFO] agent.router: Initializing LAN area manager -46:09.360 [INFO] TestAgent.server: Adding LAN server: server="Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a (Addr: tcp/127.0.0.1:11447) (DC: dc2)" -46:09.360 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 area=wan -46:09.360 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11514 network=udp -46:09.360 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11514 network=tcp -46:09.361 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11526 network=tcp -46:09.361 [INFO] TestAgent: started state syncer -46:09.361 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11448 network=tcp -46:09.410 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.423 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.423 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11447 [Candidate]" term=2 -46:09.437 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.437 [DEBUG] TestAgent.server.raft: vote granted: from=2b819c8c-1441-4faa-b1c5-534b9e546b2a term=2 tally=1 -46:09.437 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.437 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11447 [Leader]" -46:09.437 [INFO] TestAgent.server: cluster leadership acquired -46:09.438 [INFO] TestAgent.server: New leader elected: payload=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a -46:09.450 [INFO] TestAgent.server: initializing acls -46:09.456 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.464 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.464 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.464 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.464 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a -46:09.464 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 -46:09.464 [INFO] TestAgent.server: Updating LAN server: server="Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a (Addr: tcp/127.0.0.1:11447) (DC: dc2)" -46:09.464 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 area=wan -46:09.489 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.515 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.515 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.515 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.515 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.515 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.515 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a -46:09.515 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a -46:09.526 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.535 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a -46:09.622 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.626 [INFO] TestAgent: Synced node info -46:09.674 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11376] -46:09.674 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11376 -46:09.675 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 127.0.0.1 -46:09.675 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:09.675 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 area=wan -46:09.679 [INFO] TestAgent: Requesting shutdown -46:09.679 [INFO] TestAgent.server: shutting down server -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.679 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.679 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.679 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.679 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.686 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.692Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.692Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.692Z [INFO] agent.router.manager: shutting down -46:09.692 [INFO] TestAgent: consul server down -46:09.692 [INFO] TestAgent: shutdown complete -46:09.692 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11514 network=tcp -46:09.692 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11514 network=udp -46:09.692 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11526 network=tcp -46:10.193 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.193 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:09.061Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.061Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.072Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.072Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.109 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46e39907-f597-9162-c4b7-7c944ecff4b4 Address:127.0.0.1:11377}]" -46:09.109 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11377 [Follower]" leader= -46:09.110 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 127.0.0.1 -46:09.110 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46e39907-f597-9162-c4b7-7c944ecff4b4 127.0.0.1 -2022-11-07T22:46:09.110Z [INFO] agent.router: Initializing LAN area manager -46:09.110 [INFO] TestAgent.server: Adding LAN server: server="Node-46e39907-f597-9162-c4b7-7c944ecff4b4 (Addr: tcp/127.0.0.1:11377) (DC: dc1)" -46:09.110 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 area=wan -46:09.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11281 network=udp -46:09.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11281 network=tcp -46:09.111 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11282 network=tcp -46:09.111 [INFO] TestAgent: started state syncer -46:09.111 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11378 network=tcp -46:09.160 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.172 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.172 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11377 [Candidate]" term=2 -46:09.188 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.188 [DEBUG] TestAgent.server.raft: vote granted: from=46e39907-f597-9162-c4b7-7c944ecff4b4 term=2 tally=1 -46:09.188 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.188 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11377 [Leader]" -46:09.188 [INFO] TestAgent.server: cluster leadership acquired -46:09.188 [INFO] TestAgent.server: New leader elected: payload=Node-46e39907-f597-9162-c4b7-7c944ecff4b4 -46:09.197 [INFO] TestAgent.server: initializing acls -46:09.203 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.218 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.218 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.218 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.218 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-46e39907-f597-9162-c4b7-7c944ecff4b4 -46:09.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 -46:09.218 [INFO] TestAgent.server: Updating LAN server: server="Node-46e39907-f597-9162-c4b7-7c944ecff4b4 (Addr: tcp/127.0.0.1:11377) (DC: dc1)" -46:09.218 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-46e39907-f597-9162-c4b7-7c944ecff4b4.dc1 area=wan -46:09.244 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.261 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.261 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.261 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.261 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.261 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.261 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46e39907-f597-9162-c4b7-7c944ecff4b4 -46:09.261 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46e39907-f597-9162-c4b7-7c944ecff4b4 -46:09.269 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.276 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46e39907-f597-9162-c4b7-7c944ecff4b4 -46:09.396 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.398 [INFO] TestAgent: Synced node info -46:09.398 [DEBUG] TestAgent: Node info in sync -46:09.674 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:39048 -46:09.675 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 127.0.0.1 -46:09.675 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b819c8c-1441-4faa-b1c5-534b9e546b2a.dc2 area=wan -46:09.760 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.760 [DEBUG] TestAgent: Node info in sync -46:10.193 [INFO] TestAgent: Requesting shutdown -46:10.193 [INFO] TestAgent.server: shutting down server -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:10.193 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.193 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.193 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:10.193 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:10.198 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.207Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.207Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.207Z [INFO] agent.router.manager: shutting down -46:10.207 [INFO] TestAgent: consul server down -46:10.207 [INFO] TestAgent: shutdown complete -46:10.207 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11281 network=tcp -46:10.207 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11281 network=udp -46:10.208 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11282 network=tcp -46:10.708 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.708 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodeServices_WanTranslation (1.66s) -=== CONT TestPreparedQuery_parseLimit ---- PASS: TestPreparedQuery_parseLimit (0.00s) -=== CONT TestOperator_AutopilotSetConfiguration -2022-11-07T22:46:09.474Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.474Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.491Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.538 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:51c20902-f8c0-f1b7-6bf1-879812910cb8 Address:127.0.0.1:11433}]" -46:09.538 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11433 [Follower]" leader= -46:09.538 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 127.0.0.1 -46:09.538 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 127.0.0.1 -2022-11-07T22:46:09.538Z [INFO] agent.router: Initializing LAN area manager -46:09.539 [INFO] TestAgent.server: Adding LAN server: server="Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 (Addr: tcp/127.0.0.1:11433) (DC: dc2)" -46:09.539 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 area=wan -46:09.539 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11442 network=udp -46:09.539 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11442 network=tcp -46:09.539 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11443 network=tcp -46:09.539 [INFO] TestAgent: started state syncer -46:09.539 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11434 network=tcp -46:09.579 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.579 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11433 [Candidate]" term=2 -46:09.590 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.591 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.591 [DEBUG] TestAgent.server.raft: vote granted: from=51c20902-f8c0-f1b7-6bf1-879812910cb8 term=2 tally=1 -46:09.591 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.591 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11433 [Leader]" -46:09.591 [INFO] TestAgent.server: cluster leadership acquired -46:09.591 [INFO] TestAgent.server: New leader elected: payload=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 -46:09.604 [INFO] TestAgent.server: initializing acls -46:09.609 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.620 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.620 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.620 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.620 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 -46:09.620 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 -46:09.620 [INFO] TestAgent.server: Updating LAN server: server="Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 (Addr: tcp/127.0.0.1:11433) (DC: dc2)" -46:09.620 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 area=wan -46:09.637 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11271] -46:09.637 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11271 -46:09.638 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 127.0.0.1 -46:09.638 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:09.638 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 area=wan -46:09.646 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.684 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.684 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.684 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.684 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.684 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.684 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 -46:09.684 [INFO] TestAgent.server: member joined, marking health alive: member=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 -46:09.702 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.702 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8 -46:09.716 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=121.839µs client=127.0.0.1:35480 client_network=udp -46:09.717 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=153.429µs client=127.0.0.1:45695 client_network=udp -46:09.717 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=126.771µs client=127.0.0.1:46938 client_network=udp -46:09.717 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=137.129µs client=127.0.0.1:49680 client_network=udp -46:09.721 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=124.861µs client=127.0.0.1:36073 client_network=udp -46:09.721 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=123.649µs client=127.0.0.1:52407 client_network=udp -46:09.722 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=136.08µs client=127.0.0.1:39296 client_network=udp -46:09.722 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=211.394µs client=127.0.0.1:40442 client_network=udp -46:09.726 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=120.393µs client=127.0.0.1:37818 client_network=udp -46:09.726 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=132.536µs client=127.0.0.1:46935 client_network=udp -46:09.726 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=154.409µs client=127.0.0.1:49043 client_network=udp -46:09.727 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=193.194µs client=127.0.0.1:40981 client_network=udp -46:09.734 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=157.262µs client=127.0.0.1:45103 client_network=udp -46:09.734 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=124.693µs client=127.0.0.1:39752 client_network=udp -46:09.735 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=132.91µs client=127.0.0.1:53831 client_network=udp -46:09.735 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=188.927µs client=127.0.0.1:35026 client_network=udp -46:09.735 [INFO] TestAgent: Requesting shutdown -46:09.735 [INFO] TestAgent.server: shutting down server -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.735 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.735 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.735 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.735 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.735 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.739 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.744Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.744Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.744Z [INFO] agent.router.manager: shutting down -46:09.744 [INFO] TestAgent: consul server down -46:09.744 [INFO] TestAgent: shutdown complete -46:09.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11442 network=tcp -46:09.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11442 network=udp -46:09.744 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11443 network=tcp -46:10.245 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.245 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:08.982Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.982Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.997Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:08.997Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.041 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c6846215-480d-48c3-f1f7-ffdae41787da Address:127.0.0.1:11272}]" -46:09.041 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11272 [Follower]" leader= -46:09.042 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 127.0.0.1 -46:09.042 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c6846215-480d-48c3-f1f7-ffdae41787da 127.0.0.1 -2022-11-07T22:46:09.042Z [INFO] agent.router: Initializing LAN area manager -46:09.042 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 area=wan -46:09.042 [INFO] TestAgent.server: Adding LAN server: server="Node-c6846215-480d-48c3-f1f7-ffdae41787da (Addr: tcp/127.0.0.1:11272) (DC: dc1)" -46:09.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11393 network=udp -46:09.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11393 network=tcp -46:09.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11394 network=tcp -46:09.043 [INFO] TestAgent: started state syncer -46:09.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11273 network=tcp -46:09.090 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.090 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11272 [Candidate]" term=2 -46:09.093 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:09.109 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.109 [DEBUG] TestAgent.server.raft: vote granted: from=c6846215-480d-48c3-f1f7-ffdae41787da term=2 tally=1 -46:09.109 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11272 [Leader]" -46:09.109 [INFO] TestAgent.server: cluster leadership acquired -46:09.109 [INFO] TestAgent.server: New leader elected: payload=Node-c6846215-480d-48c3-f1f7-ffdae41787da -46:09.123 [INFO] TestAgent.server: initializing acls -46:09.131 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:09.146 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:09.146 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:09.146 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:09.146 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c6846215-480d-48c3-f1f7-ffdae41787da -46:09.146 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 -46:09.146 [INFO] TestAgent.server: Updating LAN server: server="Node-c6846215-480d-48c3-f1f7-ffdae41787da (Addr: tcp/127.0.0.1:11272) (DC: dc1)" -46:09.146 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c6846215-480d-48c3-f1f7-ffdae41787da.dc1 area=wan -46:09.194 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.218 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.218 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.218 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.218 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.218 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.218 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c6846215-480d-48c3-f1f7-ffdae41787da -46:09.218 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c6846215-480d-48c3-f1f7-ffdae41787da -46:09.228 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.238 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c6846215-480d-48c3-f1f7-ffdae41787da -46:09.405 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.410 [INFO] TestAgent: Synced node info -46:09.410 [DEBUG] TestAgent: Node info in sync -46:09.637 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:50748 -46:09.638 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 127.0.0.1 -46:09.638 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-51c20902-f8c0-f1b7-6bf1-879812910cb8.dc2 area=wan -46:09.655 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=734.458µs client=127.0.0.1:58977 client_network=udp -46:09.655 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=512.708µs client=127.0.0.1:35261 client_network=udp -46:09.656 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=396.338µs client=127.0.0.1:49496 client_network=udp -46:09.656 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=354.764µs client=127.0.0.1:57309 client_network=udp -46:09.674 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=438.783µs client=127.0.0.1:42573 client_network=udp -46:09.675 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=469.549µs client=127.0.0.1:44714 client_network=udp -46:09.676 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=367.684µs client=127.0.0.1:50259 client_network=udp -46:09.676 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=385.401µs client=127.0.0.1:50884 client_network=udp -46:09.702 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=471.59µs client=127.0.0.1:35904 client_network=udp -46:09.703 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=471.76µs client=127.0.0.1:36409 client_network=udp -46:09.704 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=382.696µs client=127.0.0.1:52531 client_network=udp -46:09.705 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=397.413µs client=127.0.0.1:39112 client_network=udp -46:09.709 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=414.597µs client=127.0.0.1:49499 client_network=udp -46:09.710 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=SRV class=IN latency=383.841µs client=127.0.0.1:46951 client_network=udp -46:09.710 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=399.412µs client=127.0.0.1:46292 client_network=udp -46:09.711 [DEBUG] TestAgent.dns: request served from client: name=51fafa17-e87d-2449-16ff-ef4616bc89af.query.dc2.consul. type=A class=IN latency=413.102µs client=127.0.0.1:57932 client_network=udp -46:10.245 [INFO] TestAgent: Requesting shutdown -46:10.245 [INFO] TestAgent.server: shutting down server -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:10.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.245 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.245 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.245 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:10.250 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.255Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.255Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.255Z [INFO] agent.router.manager: shutting down -46:10.255 [INFO] TestAgent: consul server down -46:10.255 [INFO] TestAgent: shutdown complete -46:10.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11393 network=tcp -46:10.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11393 network=udp -46:10.255 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11394 network=tcp -46:10.755 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.755 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_WanTranslation (1.79s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc1 (0.01s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc1 (0.02s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc1 (0.03s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc1 (0.01s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc2 (0.01s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc2 (0.00s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc2 (0.00s) - --- PASS: TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc2 (0.01s) -=== CONT TestCatalogConnectServiceNodes_good -2022-11-07T22:46:10.054Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.068Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.106 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e041fcd-6ccf-9a5f-1764-8109afe43fe4 Address:127.0.0.1:11398}]" -46:10.106 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11398 [Follower]" leader= -46:10.107 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4.dc1 127.0.0.1 -46:10.107 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4 127.0.0.1 -2022-11-07T22:46:10.107Z [INFO] agent.router: Initializing LAN area manager -46:10.107 [INFO] TestAgent.server: Adding LAN server: server="Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4 (Addr: tcp/127.0.0.1:11398) (DC: dc1)" -46:10.107 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4.dc1 area=wan -46:10.107 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11414 network=udp -46:10.108 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11414 network=tcp -46:10.108 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11415 network=tcp -46:10.108 [INFO] TestAgent: started state syncer -46:10.108 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11399 network=tcp -46:10.173 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.173 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11398 [Candidate]" term=2 -46:10.189 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.189 [DEBUG] TestAgent.server.raft: vote granted: from=9e041fcd-6ccf-9a5f-1764-8109afe43fe4 term=2 tally=1 -46:10.189 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.189 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11398 [Leader]" -46:10.189 [INFO] TestAgent.server: cluster leadership acquired -46:10.189 [INFO] TestAgent.server: New leader elected: payload=Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4 -46:10.198 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11398 -46:10.214 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.237 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.237 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.237 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.237 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.237 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.237 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4 -46:10.237 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9e041fcd-6ccf-9a5f-1764-8109afe43fe4 -46:10.249 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.318 [INFO] TestAgent: Requesting shutdown -46:10.319 [INFO] TestAgent.server: shutting down server -46:10.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.319 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.319 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.319 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.319 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.319 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.319 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.319 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.319 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.319 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.324 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.332Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.332Z [INFO] agent.router.manager: shutting down -46:10.332 [INFO] TestAgent: consul server down -46:10.332 [INFO] TestAgent: shutdown complete -46:10.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11414 network=tcp -46:10.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11414 network=udp -46:10.332 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11415 network=tcp -46:10.832 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.832 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecGetSpec (0.79s) -=== CONT TestOperator_AutopilotGetConfiguration -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:09.917Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.917Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.932Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:09.932Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.963 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0cd5ed86-33c6-f01b-d531-d5d1db08f018 Address:127.0.0.1:11482}]" -46:09.963 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11482 [Follower]" leader= -46:09.964 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018.dc1 127.0.0.1 -46:09.964 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 127.0.0.1 -2022-11-07T22:46:09.964Z [INFO] agent.router: Initializing LAN area manager -46:09.964 [INFO] TestAgent.server: Adding LAN server: server="Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 (Addr: tcp/127.0.0.1:11482) (DC: dc1)" -46:09.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11532 network=udp -46:09.964 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018.dc1 area=wan -46:09.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11532 network=tcp -46:09.965 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11519 network=tcp -46:09.965 [INFO] TestAgent: started state syncer -46:09.965 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11483 network=tcp -46:10.014 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:10.021 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.021 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11482 [Candidate]" term=2 -46:10.031 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.031 [DEBUG] TestAgent.server.raft: vote granted: from=0cd5ed86-33c6-f01b-d531-d5d1db08f018 term=2 tally=1 -46:10.031 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.031 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11482 [Leader]" -46:10.031 [INFO] TestAgent.server: cluster leadership acquired -46:10.031 [INFO] TestAgent.server: New leader elected: payload=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.040 [INFO] TestAgent.server: initializing acls -46:10.043 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:10.043 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:10.050 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:10.060 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:10.060 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:10.060 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:10.060 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.060 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018.dc1 -46:10.061 [INFO] TestAgent.server: Updating LAN server: server="Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 (Addr: tcp/127.0.0.1:11482) (DC: dc1)" -46:10.061 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018.dc1 area=wan -46:10.071 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.092 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.092 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.092 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.092 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.092 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.092 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.092 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.098 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.109 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.123 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.131 [INFO] TestAgent: Synced node info -46:10.330 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.330 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0cd5ed86-33c6-f01b-d531-d5d1db08f018 -46:10.344 [INFO] TestAgent: Requesting shutdown -46:10.344 [INFO] TestAgent.server: shutting down server -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.344 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.344 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.344 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.344 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.348 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.351Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.351Z [INFO] agent.router.manager: shutting down -46:10.352 [INFO] TestAgent: consul server down -46:10.352 [INFO] TestAgent: shutdown complete -46:10.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11532 network=tcp -46:10.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11532 network=udp -46:10.352 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11519 network=tcp -46:10.852 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.852 [INFO] TestAgent: Endpoints down ---- PASS: TestRemoteExecGetSpec_ACLToken (0.95s) -=== CONT TestPreparedQuery_Delete -2022-11-07T22:46:10.044Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.052Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.085 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 Address:127.0.0.1:11461}]" -46:10.085 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11461 [Follower]" leader= -46:10.086 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5.dc1 127.0.0.1 -46:10.087 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 127.0.0.1 -2022-11-07T22:46:10.087Z [INFO] agent.router: Initializing LAN area manager -46:10.087 [INFO] TestAgent.server: Adding LAN server: server="Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 (Addr: tcp/127.0.0.1:11461) (DC: dc1)" -46:10.087 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5.dc1 area=wan -46:10.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11491 network=udp -46:10.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11491 network=tcp -46:10.088 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11492 network=tcp -46:10.088 [INFO] TestAgent: started state syncer -46:10.088 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11462 network=tcp -46:10.123 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.123 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11461 [Candidate]" term=2 -46:10.135 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.135 [DEBUG] TestAgent.server.raft: vote granted: from=d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 term=2 tally=1 -46:10.135 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.135 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11461 [Leader]" -46:10.135 [INFO] TestAgent.server: cluster leadership acquired -46:10.135 [INFO] TestAgent.server: New leader elected: payload=Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 -46:10.141 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11461 -46:10.154 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.179 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.179 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.179 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.179 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.179 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.179 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 -46:10.179 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d7bfdeee-ec67-6bc0-bd82-6b4b4e6162f5 -46:10.189 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.411 [INFO] TestAgent: Requesting shutdown -46:10.411 [INFO] TestAgent.server: shutting down server -46:10.411 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.411 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.411 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.411 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.411 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.411 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.411 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.411 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.411 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.411 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.411 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.419 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.425Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.425Z [INFO] agent.router.manager: shutting down -46:10.425 [INFO] TestAgent: consul server down -46:10.425 [INFO] TestAgent: shutdown complete -46:10.425 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11491 network=tcp -46:10.425 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11491 network=udp -46:10.425 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11492 network=tcp -46:10.925 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.925 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodeServiceList (0.89s) +2021-10-05T16:31:22.418Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.436Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.484 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:298d7566-c63b-3907-ae95-2d0f72c9f13b Address:127.0.0.1:30183}]" +31:22.485 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30183 [Follower]" leader= +31:22.485 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-298d7566-c63b-3907-ae95-2d0f72c9f13b.dc1 127.0.0.1 +31:22.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-298d7566-c63b-3907-ae95-2d0f72c9f13b 127.0.0.1 +2021-10-05T16:31:22.486Z [INFO] agent.router: Initializing LAN area manager +31:22.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30178 network=udp +31:22.486 [INFO] TestAgent.server: Adding LAN server: server="Node-298d7566-c63b-3907-ae95-2d0f72c9f13b (Addr: tcp/127.0.0.1:30183) (DC: dc1)" +31:22.486 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-298d7566-c63b-3907-ae95-2d0f72c9f13b.dc1 area=wan +31:22.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30178 network=tcp +31:22.487 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30179 network=tcp +31:22.487 [INFO] TestAgent: started state syncer +31:22.487 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30198 network=tcp +31:22.548 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.548 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30183 [Candidate]" term=2 +31:22.577 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.577 [DEBUG] TestAgent.server.raft: vote granted: from=298d7566-c63b-3907-ae95-2d0f72c9f13b term=2 tally=1 +31:22.577 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.577 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30183 [Leader]" +31:22.580 [INFO] TestAgent.server: cluster leadership acquired +31:22.580 [INFO] TestAgent.server: New leader elected: payload=Node-298d7566-c63b-3907-ae95-2d0f72c9f13b +31:22.587 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30183 +31:22.612 [INFO] TestAgent: Synced node info +31:22.617 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.631 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.631 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.631 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.631 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.631 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.631 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-298d7566-c63b-3907-ae95-2d0f72c9f13b +31:22.631 [INFO] TestAgent.server: member joined, marking health alive: member=Node-298d7566-c63b-3907-ae95-2d0f72c9f13b +31:22.667 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.677 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=280.84µs client=127.0.0.1:41987 client_network=udp +31:22.677 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=215.595µs client=127.0.0.1:42992 client_network=udp +31:22.678 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=A class=IN latency=206.605µs client=127.0.0.1:46814 client_network=udp +31:22.678 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=AAAA class=IN latency=178.474µs client=127.0.0.1:58671 client_network=udp +31:22.679 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=77.497µs client=127.0.0.1:37297 client_network=udp +31:22.682 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=67.693µs client=127.0.0.1:56904 client_network=udp +31:22.697 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=200.03µs client=127.0.0.1:57053 client_network=udp +31:22.697 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=106.761µs client=127.0.0.1:48801 client_network=udp +31:22.698 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=A class=IN latency=192.192µs client=127.0.0.1:60111 client_network=udp +31:22.699 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=AAAA class=IN latency=79.916µs client=127.0.0.1:52435 client_network=udp +31:22.699 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=39.486µs client=127.0.0.1:42575 client_network=udp +31:22.699 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=38.214µs client=127.0.0.1:36831 client_network=udp +31:22.708 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=173.466µs client=127.0.0.1:56956 client_network=udp +31:22.708 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=97.844µs client=127.0.0.1:36301 client_network=udp +31:22.709 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=A class=IN latency=112.421µs client=127.0.0.1:35455 client_network=udp +31:22.709 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=AAAA class=IN latency=115.287µs client=127.0.0.1:60788 client_network=udp +31:22.709 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=64.524µs client=127.0.0.1:55489 client_network=udp +31:22.709 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=50.882µs client=127.0.0.1:60065 client_network=udp +31:22.734 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=170.781µs client=127.0.0.1:40014 client_network=udp +31:22.734 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=AAAA class=IN latency=111.809µs client=127.0.0.1:46018 client_network=udp +31:22.734 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=A class=IN latency=142.561µs client=127.0.0.1:50795 client_network=udp +31:22.735 [DEBUG] TestAgent.dns: request served from client: name=44df1e6e-f2c8-b137-6960-93ef16d0d6a1.query.consul. type=AAAA class=IN latency=140.211µs client=127.0.0.1:35696 client_network=udp +31:22.735 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=65.38µs client=127.0.0.1:32897 client_network=udp +31:22.735 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=AAAA class=IN latency=52.571µs client=127.0.0.1:43614 client_network=udp +31:22.735 [INFO] TestAgent: Requesting shutdown +31:22.735 [INFO] TestAgent.server: shutting down server +31:22.735 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.735 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.736 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.736 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.736 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.736 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.736 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.736 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.744 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.755Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.755Z [INFO] agent.router.manager: shutting down +31:22.756 [INFO] TestAgent: consul server down +31:22.756 [INFO] TestAgent: shutdown complete +31:22.756 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30178 network=tcp +31:22.756 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30178 network=udp +31:22.756 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30179 network=tcp +31:23.256 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.256 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Lookup_TaggedIPAddresses (0.86s) + --- PASS: TestDNS_Lookup_TaggedIPAddresses/simple-ipv4 (0.01s) + --- PASS: TestDNS_Lookup_TaggedIPAddresses/simple-ipv6 (0.02s) + --- PASS: TestDNS_Lookup_TaggedIPAddresses/ipv4-with-tagged-ipv6 (0.01s) + --- PASS: TestDNS_Lookup_TaggedIPAddresses/ipv6-with-tagged-ipv4 (0.03s) === CONT TestDNS_NSRecords -2022-11-07T22:46:10.127Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:10.127Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.142Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:10.142Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.179 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:45b86650-0fbe-7b21-d091-d0ecf93bafa9 Address:127.0.0.1:11551}]" -46:10.179 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11551 [Follower]" leader= -46:10.180 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9.dc1 127.0.0.1 -46:10.180 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 127.0.0.1 -2022-11-07T22:46:10.180Z [INFO] agent.router: Initializing LAN area manager -46:10.180 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9.dc1 area=wan -46:10.180 [INFO] TestAgent.server: Adding LAN server: server="Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 (Addr: tcp/127.0.0.1:11551) (DC: dc1)" -46:10.180 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11539 network=udp -46:10.180 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11539 network=tcp -46:10.181 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11540 network=tcp -46:10.181 [INFO] TestAgent: started state syncer -46:10.181 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11552 network=tcp -46:10.225 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.225 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11551 [Candidate]" term=2 -46:10.230 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:10.240 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.240 [DEBUG] TestAgent.server.raft: vote granted: from=45b86650-0fbe-7b21-d091-d0ecf93bafa9 term=2 tally=1 -46:10.240 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.240 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11551 [Leader]" -46:10.240 [INFO] TestAgent.server: cluster leadership acquired -46:10.240 [INFO] TestAgent.server: New leader elected: payload=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.250 [INFO] TestAgent.server: initializing acls -46:10.255 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:10.255 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:10.263 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:10.279 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:10.279 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:10.279 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:10.279 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.279 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9.dc1 -46:10.280 [INFO] TestAgent.server: Updating LAN server: server="Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 (Addr: tcp/127.0.0.1:11551) (DC: dc1)" -46:10.280 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9.dc1 area=wan -46:10.302 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.302 [INFO] TestAgent: Synced node info -46:10.327 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.327 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.327 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.327 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.327 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.327 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.327 [INFO] TestAgent.server: member joined, marking health alive: member=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.333 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.340 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.495 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-45b86650-0fbe-7b21-d091-d0ecf93bafa9 -46:10.495 [INFO] TestAgent: Requesting shutdown -46:10.495 [INFO] TestAgent.server: shutting down server -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.495 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.495 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:10.495 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.495 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.508 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.516Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.516Z [INFO] agent.router.manager: shutting down -46:10.516 [INFO] TestAgent: consul server down -46:10.516 [INFO] TestAgent: shutdown complete -46:10.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11539 network=tcp -46:10.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11539 network=udp -46:10.516 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11540 network=tcp -46:11.016 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.016 [INFO] TestAgent: Endpoints down -=== RUN TestSnapshot_Options/PUT#02 -2022-11-07T22:46:10.067Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.077Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.111 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b2aaecc8-6a6b-b87c-4522-5e9c3cd58889 Address:127.0.0.1:11537}]" -46:10.111 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11537 [Follower]" leader= -46:10.111 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:10.112 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:10.112Z [INFO] agent.router: Initializing LAN area manager -46:10.112 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11537) (DC: dc1)" -46:10.112 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:10.112 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11400 network=udp -46:10.112 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11400 network=tcp -46:10.112 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11401 network=tcp -46:10.112 [INFO] TestAgent: started state syncer -46:10.112 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11538 network=tcp -46:10.162 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.162 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11537 [Candidate]" term=2 -46:10.179 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.179 [DEBUG] TestAgent.server.raft: vote granted: from=b2aaecc8-6a6b-b87c-4522-5e9c3cd58889 term=2 tally=1 -46:10.179 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.179 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11537 [Leader]" -46:10.179 [INFO] TestAgent.server: cluster leadership acquired -46:10.179 [INFO] TestAgent.server: New leader elected: payload=test-node -46:10.187 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11537 -46:10.206 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.235 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.235 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.235 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.235 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.235 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:10.235 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:10.252 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.548 [DEBUG] TestAgent.dns: request served from client: name=alias2.service.consul. type=SRV class=IN latency=245.134µs client=127.0.0.1:33883 client_network=udp -46:10.548 [INFO] TestAgent: Requesting shutdown -46:10.548 [INFO] TestAgent.server: shutting down server -46:10.548 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.548 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.548 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.548 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.548 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.548 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.548 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.548 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.548 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.552 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.557Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.557Z [INFO] agent.router.manager: shutting down -46:10.557 [INFO] TestAgent: consul server down -46:10.557 [INFO] TestAgent: shutdown complete -46:10.557 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11400 network=tcp -46:10.557 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11400 network=udp -46:10.557 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11401 network=tcp -46:11.057 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.057 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ExternalServiceToConsulCNAMENestedLookup (1.01s) -=== CONT TestPreparedQuery_Update -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:10.296Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.305Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.343 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d220b097-c61b-280b-471f-9ab52166aa1b Address:127.0.0.1:11565}]" -46:10.343 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11565 [Follower]" leader= -46:10.343 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d220b097-c61b-280b-471f-9ab52166aa1b.dc1 127.0.0.1 -46:10.344 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d220b097-c61b-280b-471f-9ab52166aa1b 127.0.0.1 -2022-11-07T22:46:10.344Z [INFO] agent.router: Initializing LAN area manager -46:10.344 [INFO] TestAgent.server: Adding LAN server: server="Node-d220b097-c61b-280b-471f-9ab52166aa1b (Addr: tcp/127.0.0.1:11565) (DC: dc1)" -46:10.344 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11602 network=udp -46:10.344 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d220b097-c61b-280b-471f-9ab52166aa1b.dc1 area=wan -46:10.344 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11602 network=tcp -46:10.345 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11589 network=tcp -46:10.345 [INFO] TestAgent: started state syncer -46:10.345 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11566 network=tcp -46:10.392 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.392 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11565 [Candidate]" term=2 -46:10.409 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.409 [DEBUG] TestAgent.server.raft: vote granted: from=d220b097-c61b-280b-471f-9ab52166aa1b term=2 tally=1 -46:10.409 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.409 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11565 [Leader]" -46:10.410 [INFO] TestAgent.server: cluster leadership acquired -46:10.410 [INFO] TestAgent.server: New leader elected: payload=Node-d220b097-c61b-280b-471f-9ab52166aa1b -46:10.422 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11565 -46:10.444 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.472 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.472 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.472 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.472 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.472 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.472 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d220b097-c61b-280b-471f-9ab52166aa1b -46:10.472 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d220b097-c61b-280b-471f-9ab52166aa1b -46:10.493 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.583 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:10.583 [INFO] TestAgent: Requesting shutdown -46:10.583 [INFO] TestAgent.server: shutting down server -46:10.583 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.583 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.583 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.583 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.583 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.583 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.583 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.583 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.583 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.583 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.583 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.587 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.591Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.591Z [INFO] agent.router.manager: shutting down -46:10.591 [INFO] TestAgent: consul server down -46:10.591 [INFO] TestAgent: shutdown complete -46:10.591 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11602 network=tcp -46:10.591 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11602 network=udp -46:10.591 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11589 network=tcp -46:11.091 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.091 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#02 -2022-11-07T22:46:10.386Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.396Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.439 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:948b672b-4d7e-b4b8-7f35-edfbd019e878 Address:127.0.0.1:11558}]" -46:10.439 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11558 [Follower]" leader= -46:10.439 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-948b672b-4d7e-b4b8-7f35-edfbd019e878.dc1 127.0.0.1 -46:10.440 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-948b672b-4d7e-b4b8-7f35-edfbd019e878 127.0.0.1 -2022-11-07T22:46:10.440Z [INFO] agent.router: Initializing LAN area manager -46:10.440 [INFO] TestAgent.server: Adding LAN server: server="Node-948b672b-4d7e-b4b8-7f35-edfbd019e878 (Addr: tcp/127.0.0.1:11558) (DC: dc1)" -46:10.440 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-948b672b-4d7e-b4b8-7f35-edfbd019e878.dc1 area=wan -46:10.440 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11609 network=udp -46:10.440 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11609 network=tcp -46:10.441 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11638 network=tcp -46:10.441 [INFO] TestAgent: started state syncer -46:10.441 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11559 network=tcp -46:10.480 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.480 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11558 [Candidate]" term=2 -46:10.493 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.493 [DEBUG] TestAgent.server.raft: vote granted: from=948b672b-4d7e-b4b8-7f35-edfbd019e878 term=2 tally=1 -46:10.493 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.493 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11558 [Leader]" -46:10.493 [INFO] TestAgent.server: cluster leadership acquired -46:10.493 [INFO] TestAgent.server: New leader elected: payload=Node-948b672b-4d7e-b4b8-7f35-edfbd019e878 -46:10.508 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11558 -46:10.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.549 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.549 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.549 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.549 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.549 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.549 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-948b672b-4d7e-b4b8-7f35-edfbd019e878 -46:10.549 [INFO] TestAgent.server: member joined, marking health alive: member=Node-948b672b-4d7e-b4b8-7f35-edfbd019e878 -46:10.561 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.602 [INFO] TestAgent: Requesting shutdown -46:10.602 [INFO] TestAgent.server: shutting down server -46:10.602 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.602 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.602 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.602 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.602 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.602 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.602 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.602 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.602 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.602 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.602 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.606 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.612Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.612Z [INFO] agent.router.manager: shutting down -46:10.612 [INFO] TestAgent: consul server down -46:10.612 [INFO] TestAgent: shutdown complete -46:10.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11609 network=tcp -46:10.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11609 network=udp -46:10.612 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11638 network=tcp -46:11.112 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.112 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Integration (0.74s) +=== RUN TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc1 +=== RUN TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc1 +2021-10-05T16:31:22.277Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.293Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.360 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76e26525-3419-fd26-3d07-c98becf0987c Address:127.0.0.1:30190}]" +31:22.360 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76e26525-3419-fd26-3d07-c98becf0987c.dc1 127.0.0.1 +31:22.361 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76e26525-3419-fd26-3d07-c98becf0987c 127.0.0.1 +2021-10-05T16:31:22.361Z [INFO] agent.router: Initializing LAN area manager +31:22.362 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30185 network=udp +31:22.362 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30190 [Follower]" leader= +31:22.362 [INFO] TestAgent.server: Adding LAN server: server="Node-76e26525-3419-fd26-3d07-c98becf0987c (Addr: tcp/127.0.0.1:30190) (DC: dc1)" +31:22.362 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76e26525-3419-fd26-3d07-c98becf0987c.dc1 area=wan +31:22.362 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30185 network=tcp +31:22.363 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30186 network=tcp +31:22.363 [INFO] TestAgent: started state syncer +31:22.363 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30177 network=tcp +31:22.408 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.408 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30190 [Candidate]" term=2 +31:22.456 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.456 [DEBUG] TestAgent.server.raft: vote granted: from=76e26525-3419-fd26-3d07-c98becf0987c term=2 tally=1 +31:22.456 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.456 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30190 [Leader]" +31:22.458 [INFO] TestAgent.server: cluster leadership acquired +31:22.458 [INFO] TestAgent.server: New leader elected: payload=Node-76e26525-3419-fd26-3d07-c98becf0987c +31:22.488 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30190 +31:22.488 [INFO] TestAgent: Synced node info +31:22.488 [DEBUG] TestAgent: Node info in sync +31:22.552 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.582 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.582 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.582 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.582 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.582 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.582 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76e26525-3419-fd26-3d07-c98becf0987c +31:22.582 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76e26525-3419-fd26-3d07-c98becf0987c +31:22.612 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.764 [DEBUG] TestAgent.dns: request served from client: name=primary.db.service.consul. type=SRV class=IN latency=377.611µs client=127.0.0.1:35909 client_network=udp +31:22.765 [DEBUG] TestAgent.dns: request served from client: name=pRIMARY.dB.service.consul. type=SRV class=IN latency=102.259µs client=127.0.0.1:59943 client_network=udp +31:22.765 [DEBUG] TestAgent.dns: request served from client: name=PRIMARY.dB.service.consul. type=SRV class=IN latency=93.548µs client=127.0.0.1:38254 client_network=udp +31:22.767 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=252.026µs client=127.0.0.1:34480 client_network=udp +31:22.767 [DEBUG] TestAgent.dns: request served from client: name=DB.service.consul. type=SRV class=IN latency=132.274µs client=127.0.0.1:46881 client_network=udp +31:22.767 [DEBUG] TestAgent.dns: request served from client: name=Db.service.consul. type=SRV class=IN latency=126.86µs client=127.0.0.1:32861 client_network=udp +31:22.768 [DEBUG] TestAgent.dns: request served from client: name=somequery.query.consul. type=SRV class=IN latency=156.502µs client=127.0.0.1:36774 client_network=udp +31:22.768 [DEBUG] TestAgent.dns: request served from client: name=SomeQuery.query.consul. type=SRV class=IN latency=141.884µs client=127.0.0.1:56637 client_network=udp +31:22.769 [DEBUG] TestAgent.dns: request served from client: name=SOMEQUERY.query.consul. type=SRV class=IN latency=110.789µs client=127.0.0.1:49908 client_network=udp +31:22.769 [INFO] TestAgent: Requesting shutdown +31:22.769 [INFO] TestAgent.server: shutting down server +31:22.769 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.769 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.769 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.769 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.769 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.769 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.769 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.769 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.769 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.769 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.776 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.781Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.781Z [INFO] agent.router.manager: shutting down +31:22.781 [INFO] TestAgent: consul server down +31:22.781 [INFO] TestAgent: shutdown complete +31:22.781 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30185 network=tcp +31:22.781 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30185 network=udp +31:22.781 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30186 network=tcp +31:23.282 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.282 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_CaseInsensitiveServiceLookup (1.02s) === CONT TestDNS_ExternalServiceToConsulCNAMELookup -2022-11-07T22:46:10.291Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.300Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.339 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f0d8170f-c008-4e9e-274a-ddd50bca0ce2 Address:127.0.0.1:11600}]" -46:10.339 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11600 [Follower]" leader= -46:10.340 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2.dc1 127.0.0.1 -46:10.340 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2 127.0.0.1 -2022-11-07T22:46:10.340Z [INFO] agent.router: Initializing LAN area manager -46:10.340 [INFO] TestAgent.server: Adding LAN server: server="Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2 (Addr: tcp/127.0.0.1:11600) (DC: dc1)" -46:10.340 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2.dc1 area=wan -46:10.341 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11595 network=udp -46:10.341 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11595 network=tcp -46:10.341 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11617 network=tcp -46:10.341 [INFO] TestAgent: started state syncer -46:10.341 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11601 network=tcp -46:10.402 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.402 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11600 [Candidate]" term=2 -46:10.422 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.422 [DEBUG] TestAgent.server.raft: vote granted: from=f0d8170f-c008-4e9e-274a-ddd50bca0ce2 term=2 tally=1 -46:10.422 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.422 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11600 [Leader]" -46:10.422 [INFO] TestAgent.server: cluster leadership acquired -46:10.422 [INFO] TestAgent.server: New leader elected: payload=Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2 -46:10.431 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11600 -46:10.448 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.481 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.481 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.481 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.481 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.481 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.481 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2 -46:10.481 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f0d8170f-c008-4e9e-274a-ddd50bca0ce2 -46:10.504 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.563 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.567 [INFO] TestAgent: Synced node info -46:10.669 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:10.669 [INFO] TestAgent: Requesting shutdown -46:10.669 [INFO] TestAgent.server: shutting down server -46:10.669 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.669 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.669 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.669 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.669 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.669 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.669 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.669 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.669 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.669 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.674 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.682Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.682Z [INFO] agent.router.manager: shutting down -46:10.682 [INFO] TestAgent: consul server down -46:10.682 [INFO] TestAgent: shutdown complete -46:10.682 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11595 network=tcp -46:10.682 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11595 network=udp -46:10.682 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11617 network=tcp -46:11.182 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.182 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_List (1.68s) - --- PASS: TestPreparedQuery_List/#00 (0.78s) - --- PASS: TestPreparedQuery_List/#01 (0.90s) -=== CONT TestCatalogServiceNodes_ConnectProxy -2022-11-07T22:46:10.480Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.489Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.535 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:45b5eacc-5d05-278c-c1a7-8700540e4f9c Address:127.0.0.1:11642}]" -46:10.535 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11642 [Follower]" leader= -46:10.536 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c.dc1 127.0.0.1 -46:10.536 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c 127.0.0.1 -2022-11-07T22:46:10.536Z [INFO] agent.router: Initializing LAN area manager -46:10.536 [INFO] TestAgent.server: Adding LAN server: server="Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c (Addr: tcp/127.0.0.1:11642) (DC: dc1)" -46:10.536 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c.dc1 area=wan -46:10.536 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11560 network=udp -46:10.536 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11560 network=tcp -46:10.537 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11561 network=tcp -46:10.537 [INFO] TestAgent: started state syncer -46:10.537 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11643 network=tcp -46:10.579 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.579 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11642 [Candidate]" term=2 -46:10.590 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.590 [DEBUG] TestAgent.server.raft: vote granted: from=45b5eacc-5d05-278c-c1a7-8700540e4f9c term=2 tally=1 -46:10.590 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.590 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11642 [Leader]" -46:10.590 [INFO] TestAgent.server: cluster leadership acquired -46:10.590 [INFO] TestAgent.server: New leader elected: payload=Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c -46:10.597 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11642 -46:10.610 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.631 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.631 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.631 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.631 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.631 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.631 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c -46:10.631 [INFO] TestAgent.server: member joined, marking health alive: member=Node-45b5eacc-5d05-278c-c1a7-8700540e4f9c -46:10.645 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.678 [INFO] TestAgent: Requesting shutdown -46:10.679 [INFO] TestAgent.server: shutting down server -46:10.679 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.679 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.679 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.679 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.679 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.679 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.679 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.679 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.679 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.679 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.679 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.685 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.688Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.688Z [INFO] agent.router.manager: shutting down -46:10.688 [INFO] TestAgent: consul server down -46:10.688 [INFO] TestAgent: shutdown complete -46:10.688 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11560 network=tcp -46:10.688 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11560 network=udp -46:10.688 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11561 network=tcp -46:11.188 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.188 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_AutopilotCASConfiguration (0.72s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc1 +2021-10-05T16:31:22.647Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.669Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.724 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5bfea19e-cf45-5c33-c6df-d58c1a755eb9 Address:127.0.0.1:30208}]" +31:22.724 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30208 [Follower]" leader= +31:22.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9.dc1 127.0.0.1 +31:22.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9 127.0.0.1 +2021-10-05T16:31:22.725Z [INFO] agent.router: Initializing LAN area manager +31:22.725 [DEBUG] TestAgent.dns: recursor enabled +31:22.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30216 network=udp +31:22.732 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9.dc1 area=wan +31:22.733 [INFO] TestAgent.server: Adding LAN server: server="Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9 (Addr: tcp/127.0.0.1:30208) (DC: dc1)" +31:22.735 [DEBUG] TestAgent.dns: recursor enabled +31:22.735 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30216 network=tcp +31:22.737 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30217 network=tcp +31:22.737 [INFO] TestAgent: started state syncer +31:22.737 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30209 network=tcp +31:22.790 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.790 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30208 [Candidate]" term=2 +31:22.839 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.839 [DEBUG] TestAgent.server.raft: vote granted: from=5bfea19e-cf45-5c33-c6df-d58c1a755eb9 term=2 tally=1 +31:22.839 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.839 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30208 [Leader]" +31:22.840 [INFO] TestAgent.server: cluster leadership acquired +31:22.840 [INFO] TestAgent.server: New leader elected: payload=Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9 +31:22.848 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30208 +31:22.888 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.924 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.924 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.924 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.924 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.924 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.924 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9 +31:22.924 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5bfea19e-cf45-5c33-c6df-d58c1a755eb9 +31:22.951 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.989 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=85.24µs +31:22.989 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=307.981µs client=127.0.0.1:39102 client_network=udp +31:22.989 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=68.778µs +31:22.989 [DEBUG] TestAgent.dns: request served from client: name=2bf8e080-1071-1bf1-5861-1132016390a1.query.consul. type=SRV class=IN latency=252.352µs client=127.0.0.1:52474 client_network=udp +31:22.990 [INFO] TestAgent: Requesting shutdown +31:22.990 [INFO] TestAgent.server: shutting down server +31:22.990 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.990 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.990 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.990 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.990 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.990 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:22.990 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.000 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.008Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.008Z [INFO] agent.router.manager: shutting down +31:23.008 [INFO] TestAgent: consul server down +31:23.008 [INFO] TestAgent: shutdown complete +31:23.008 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30216 network=tcp +31:23.008 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30216 network=udp +31:23.008 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30217 network=tcp +31:23.516 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.516 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_ServiceAddress_SRV (0.89s) === CONT TestDNS_InifiniteRecursion -2022-11-07T22:46:10.367Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.382Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.417 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c130c460-434d-7686-739e-682b80d79713 Address:127.0.0.1:11607}]" -46:10.417 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11607 [Follower]" leader= -46:10.417 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c130c460-434d-7686-739e-682b80d79713.dc1 127.0.0.1 -46:10.419 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c130c460-434d-7686-739e-682b80d79713 127.0.0.1 -2022-11-07T22:46:10.419Z [INFO] agent.router: Initializing LAN area manager -46:10.420 [INFO] TestAgent.server: Adding LAN server: server="Node-c130c460-434d-7686-739e-682b80d79713 (Addr: tcp/127.0.0.1:11607) (DC: dc1)" -46:10.420 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11567 network=udp -46:10.420 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11567 network=tcp -46:10.420 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c130c460-434d-7686-739e-682b80d79713.dc1 area=wan -46:10.421 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11568 network=tcp -46:10.421 [INFO] TestAgent: started state syncer -46:10.421 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11608 network=tcp -46:10.452 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.452 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11607 [Candidate]" term=2 -46:10.465 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.465 [DEBUG] TestAgent.server.raft: vote granted: from=c130c460-434d-7686-739e-682b80d79713 term=2 tally=1 -46:10.465 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.465 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11607 [Leader]" -46:10.465 [INFO] TestAgent.server: cluster leadership acquired -46:10.465 [INFO] TestAgent.server: New leader elected: payload=Node-c130c460-434d-7686-739e-682b80d79713 -46:10.474 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11607 -46:10.493 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.524 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.524 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.524 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.524 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.524 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.524 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c130c460-434d-7686-739e-682b80d79713 -46:10.524 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c130c460-434d-7686-739e-682b80d79713 -46:10.541 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.573 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.575 [INFO] TestAgent: Synced node info -46:10.575 [DEBUG] TestAgent: Node info in sync -46:10.807 [INFO] TestAgent: Requesting shutdown -46:10.808 [INFO] TestAgent.server: shutting down server -46:10.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.808 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.808 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.808 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.808 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.808 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.808 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.813 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.816Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.816Z [INFO] agent.router.manager: shutting down -46:10.816 [INFO] TestAgent: consul server down -46:10.816 [INFO] TestAgent: shutdown complete -46:10.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11567 network=tcp -46:10.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11567 network=udp -46:10.816 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11568 network=tcp -46:11.317 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.317 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodeServices (0.97s) -=== CONT TestPreparedQuery_Get -=== RUN TestPreparedQuery_Get/#00 -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:10.623Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.633Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.667 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:26e2a18f-d20e-76bd-6f31-90e45a72cad5 Address:[::1]:11614}]" -46:10.667 [INFO] TestAgent.server.raft: entering follower state: follower="Node at [::1]:11614 [Follower]" leader= -46:10.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: server1.dc1 ::1 -46:10.668 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: server1 ::1 -2022-11-07T22:46:10.668Z [INFO] agent.router: Initializing LAN area manager -46:10.668 [INFO] TestAgent.server: Adding LAN server: server="server1 (Addr: tcp/[::1]:11614) (DC: dc1)" -46:10.668 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=server1.dc1 area=wan -46:10.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11574 network=udp -46:10.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11574 network=tcp -46:10.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11575 network=tcp -46:10.668 [INFO] TestAgent: started state syncer -46:10.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11615 network=tcp -46:10.726 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.726 [INFO] TestAgent.server.raft: entering candidate state: node="Node at [::1]:11614 [Candidate]" term=2 -46:10.743 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.743 [DEBUG] TestAgent.server.raft: vote granted: from=26e2a18f-d20e-76bd-6f31-90e45a72cad5 term=2 tally=1 -46:10.743 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.743 [INFO] TestAgent.server.raft: entering leader state: leader="Node at [::1]:11614 [Leader]" -46:10.744 [INFO] TestAgent.server: cluster leadership acquired -46:10.744 [INFO] TestAgent.server: New leader elected: payload=server1 -46:10.751 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=[::1]:11614 -46:10.768 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.796 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.796 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.796 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.796 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.796 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.796 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=server1 -46:10.796 [INFO] TestAgent.server: member joined, marking health alive: member=server1 -46:10.808 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.923 [DEBUG] TestAgent.dns: request served from client: name=server1.node.dc1.consul. type=NS class=IN latency=126.258µs client=127.0.0.1:44362 client_network=udp -46:10.923 [INFO] TestAgent: Requesting shutdown -46:10.923 [INFO] TestAgent.server: shutting down server -46:10.923 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.923 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.923 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.923 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.923 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.923 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.923 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.927 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.933Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.933Z [INFO] agent.router.manager: shutting down -46:10.933 [INFO] TestAgent: consul server down -46:10.933 [INFO] TestAgent: shutdown complete -46:10.933 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11574 network=tcp -46:10.933 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11574 network=udp -46:10.933 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11575 network=tcp -46:11.434 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.434 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NSRecords_IPV6 (0.82s) -=== CONT TestPreparedQuery_Explain -=== RUN TestPreparedQuery_Explain/#00 -2022-11-07T22:46:10.767Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.776Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.810 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d3f40a48-889b-1293-11f8-1713fdebce89 Address:127.0.0.1:11656}]" -46:10.810 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11656 [Follower]" leader= -46:10.811 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d3f40a48-889b-1293-11f8-1713fdebce89.dc1 127.0.0.1 -46:10.812 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d3f40a48-889b-1293-11f8-1713fdebce89 127.0.0.1 -2022-11-07T22:46:10.812Z [INFO] agent.router: Initializing LAN area manager -46:10.812 [INFO] TestAgent.server: Adding LAN server: server="Node-d3f40a48-889b-1293-11f8-1713fdebce89 (Addr: tcp/127.0.0.1:11656) (DC: dc1)" -46:10.812 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d3f40a48-889b-1293-11f8-1713fdebce89.dc1 area=wan -46:10.812 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11623 network=udp -46:10.812 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11623 network=tcp -46:10.812 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11610 network=tcp -46:10.812 [INFO] TestAgent: started state syncer -46:10.812 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11657 network=tcp -46:10.859 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.859 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11656 [Candidate]" term=2 -46:10.874 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.874 [DEBUG] TestAgent.server.raft: vote granted: from=d3f40a48-889b-1293-11f8-1713fdebce89 term=2 tally=1 -46:10.874 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.874 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11656 [Leader]" -46:10.874 [INFO] TestAgent.server: cluster leadership acquired -46:10.875 [INFO] TestAgent.server: New leader elected: payload=Node-d3f40a48-889b-1293-11f8-1713fdebce89 -46:10.882 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11656 -46:10.898 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.919 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.919 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.919 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.919 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.919 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.919 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d3f40a48-889b-1293-11f8-1713fdebce89 -46:10.919 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d3f40a48-889b-1293-11f8-1713fdebce89 -46:10.931 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.983 [INFO] TestAgent: Requesting shutdown -46:10.983 [INFO] TestAgent.server: shutting down server -46:10.984 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.984 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.984 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.984 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.984 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.984 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.984 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.984 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:10.984 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.984 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.984 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.988 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.993Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.993Z [INFO] agent.router.manager: shutting down -46:10.994 [INFO] TestAgent: consul server down -46:10.994 [INFO] TestAgent: shutdown complete -46:10.994 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11623 network=tcp -46:10.994 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11623 network=udp -46:10.994 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11610 network=tcp -46:11.494 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.494 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogConnectServiceNodes_good (0.74s) -=== CONT TestCatalogServiceNodes_DistanceSort -2022-11-07T22:46:10.874Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.882Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.911 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab8846aa-c567-c615-3bd8-1dfa2a1cc71f Address:127.0.0.1:11586}]" -46:10.911 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11586 [Follower]" leader= -46:10.912 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab8846aa-c567-c615-3bd8-1dfa2a1cc71f.dc1 127.0.0.1 -46:10.912 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab8846aa-c567-c615-3bd8-1dfa2a1cc71f 127.0.0.1 -2022-11-07T22:46:10.912Z [INFO] agent.router: Initializing LAN area manager -46:10.912 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab8846aa-c567-c615-3bd8-1dfa2a1cc71f.dc1 area=wan -46:10.912 [INFO] TestAgent.server: Adding LAN server: server="Node-ab8846aa-c567-c615-3bd8-1dfa2a1cc71f (Addr: tcp/127.0.0.1:11586) (DC: dc1)" -46:10.913 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11581 network=tcp -46:10.913 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11581 network=udp -46:10.913 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11582 network=tcp -46:10.913 [INFO] TestAgent: started state syncer -46:10.913 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11587 network=tcp -46:10.970 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.970 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11586 [Candidate]" term=2 -46:10.981 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.981 [DEBUG] TestAgent.server.raft: vote granted: from=ab8846aa-c567-c615-3bd8-1dfa2a1cc71f term=2 tally=1 -46:10.981 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.981 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11586 [Leader]" -46:10.981 [INFO] TestAgent.server: cluster leadership acquired -46:10.981 [INFO] TestAgent.server: New leader elected: payload=Node-ab8846aa-c567-c615-3bd8-1dfa2a1cc71f -46:10.989 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11586 -46:11.004 [INFO] TestAgent: Synced node info -46:11.004 [DEBUG] TestAgent: Node info in sync -46:11.007 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:11.007 [INFO] TestAgent: Requesting shutdown -46:11.007 [INFO] TestAgent.server: shutting down server -46:11.007 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.013 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:11.016 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:11.019Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.019Z [INFO] agent.router.manager: shutting down -46:11.021 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:11.021 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:11.021 [INFO] TestAgent: consul server down -46:11.021 [INFO] TestAgent: shutdown complete -46:11.021 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11581 network=tcp -46:11.021 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11581 network=udp -46:11.021 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11582 network=tcp -46:11.521 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.521 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Delete (0.67s) +2021-10-05T16:31:22.665Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.692Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.765 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:817cbd1e-1121-7220-88a1-e3e973c38014 Address:127.0.0.1:30229}]" +31:22.766 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-817cbd1e-1121-7220-88a1-e3e973c38014.dc1 127.0.0.1 +31:22.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-817cbd1e-1121-7220-88a1-e3e973c38014 127.0.0.1 +2021-10-05T16:31:22.767Z [INFO] agent.router: Initializing LAN area manager +31:22.767 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30210 network=udp +31:22.767 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30229 [Follower]" leader= +31:22.767 [INFO] TestAgent.server: Adding LAN server: server="Node-817cbd1e-1121-7220-88a1-e3e973c38014 (Addr: tcp/127.0.0.1:30229) (DC: dc1)" +31:22.767 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-817cbd1e-1121-7220-88a1-e3e973c38014.dc1 area=wan +31:22.768 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30210 network=tcp +31:22.772 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30211 network=tcp +31:22.772 [INFO] TestAgent: started state syncer +31:22.772 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30230 network=tcp +31:22.824 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.824 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30229 [Candidate]" term=2 +31:22.852 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.852 [DEBUG] TestAgent.server.raft: vote granted: from=817cbd1e-1121-7220-88a1-e3e973c38014 term=2 tally=1 +31:22.852 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.852 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30229 [Leader]" +31:22.853 [INFO] TestAgent.server: cluster leadership acquired +31:22.854 [INFO] TestAgent.server: New leader elected: payload=Node-817cbd1e-1121-7220-88a1-e3e973c38014 +31:22.867 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30229 +31:22.892 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.931 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.931 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.931 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.931 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.931 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.931 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-817cbd1e-1121-7220-88a1-e3e973c38014 +31:22.931 [INFO] TestAgent.server: member joined, marking health alive: member=Node-817cbd1e-1121-7220-88a1-e3e973c38014 +31:22.940 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.012 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=249.705µs client=127.0.0.1:56459 client_network=udp +31:23.017 [DEBUG] TestAgent.dns: request served from client: name=3e8c71a6-2469-b3cc-3fb9-4500e12ac47c.query.consul. type=SRV class=IN latency=297.373µs client=127.0.0.1:55437 client_network=udp +31:23.017 [INFO] TestAgent: Requesting shutdown +31:23.017 [INFO] TestAgent.server: shutting down server +31:23.017 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.017 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.017 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.017 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.017 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.017 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.017 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.017 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:23.017 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.018 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.018 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.024 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.033Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.033Z [INFO] agent.router.manager: shutting down +31:23.033 [INFO] TestAgent: consul server down +31:23.033 [INFO] TestAgent: shutdown complete +31:23.033 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30210 network=tcp +31:23.033 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30210 network=udp +31:23.033 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30211 network=tcp +31:23.533 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.533 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_ServiceAddress_A (0.89s) === CONT TestDNS_ExternalServiceLookup -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:10.721Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.732Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.766 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:761417d2-d920-a344-528a-1e88458e04ee Address:127.0.0.1:11621}]" -46:10.766 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11621 [Follower]" leader= -46:10.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-761417d2-d920-a344-528a-1e88458e04ee.dc1 127.0.0.1 -46:10.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-761417d2-d920-a344-528a-1e88458e04ee 127.0.0.1 -2022-11-07T22:46:10.767Z [INFO] agent.router: Initializing LAN area manager -46:10.767 [INFO] TestAgent.server: Adding LAN server: server="Node-761417d2-d920-a344-528a-1e88458e04ee (Addr: tcp/127.0.0.1:11621) (DC: dc1)" -46:10.767 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-761417d2-d920-a344-528a-1e88458e04ee.dc1 area=wan -46:10.767 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11616 network=udp -46:10.767 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11616 network=tcp -46:10.768 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11624 network=tcp -46:10.768 [INFO] TestAgent: started state syncer -46:10.768 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11622 network=tcp -46:10.814 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.814 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11621 [Candidate]" term=2 -46:10.822 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.822 [DEBUG] TestAgent.server.raft: vote granted: from=761417d2-d920-a344-528a-1e88458e04ee term=2 tally=1 -46:10.822 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.822 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11621 [Leader]" -46:10.822 [INFO] TestAgent.server: cluster leadership acquired -46:10.822 [INFO] TestAgent.server: New leader elected: payload=Node-761417d2-d920-a344-528a-1e88458e04ee -46:10.828 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11621 -46:10.839 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.869 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.869 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.869 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.869 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.869 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.869 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-761417d2-d920-a344-528a-1e88458e04ee -46:10.869 [INFO] TestAgent.server: member joined, marking health alive: member=Node-761417d2-d920-a344-528a-1e88458e04ee -46:10.882 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.161 [INFO] TestAgent: Requesting shutdown -46:11.161 [INFO] TestAgent.server: shutting down server -46:11.161 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.161 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.161 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.161 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.161 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.161 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.161 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.161 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.161 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.161 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.161 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.168 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.175Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.175Z [INFO] agent.router.manager: shutting down -46:11.175 [INFO] TestAgent: consul server down -46:11.175 [INFO] TestAgent: shutdown complete -46:11.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11616 network=tcp -46:11.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11616 network=udp -46:11.176 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11624 network=tcp -46:11.676 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.676 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_AutopilotSetConfiguration (0.97s) -=== CONT TestCatalogServiceNodes_WanTranslation -2022-11-07T22:46:11.026Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.026Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.035Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.035Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.063 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67fbed38-3bc1-27be-1392-cba596cbf178 Address:127.0.0.1:11628}]" -46:11.063 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11628 [Follower]" leader= -46:11.064 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67fbed38-3bc1-27be-1392-cba596cbf178.dc1 127.0.0.1 -46:11.064 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67fbed38-3bc1-27be-1392-cba596cbf178 127.0.0.1 -2022-11-07T22:46:11.064Z [INFO] agent.router: Initializing LAN area manager -46:11.064 [INFO] TestAgent.server: Adding LAN server: server="Node-67fbed38-3bc1-27be-1392-cba596cbf178 (Addr: tcp/127.0.0.1:11628) (DC: dc1)" -46:11.064 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67fbed38-3bc1-27be-1392-cba596cbf178.dc1 area=wan -46:11.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11651 network=udp -46:11.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11651 network=tcp -46:11.065 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11673 network=tcp -46:11.065 [INFO] TestAgent: started state syncer -46:11.065 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11629 network=tcp -46:11.106 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.106 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11628 [Candidate]" term=2 -46:11.114 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:11.115 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.115 [DEBUG] TestAgent.server.raft: vote granted: from=67fbed38-3bc1-27be-1392-cba596cbf178 term=2 tally=1 -46:11.115 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.115 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11628 [Leader]" -46:11.115 [INFO] TestAgent.server: cluster leadership acquired -46:11.115 [INFO] TestAgent.server: New leader elected: payload=Node-67fbed38-3bc1-27be-1392-cba596cbf178 -46:11.122 [INFO] TestAgent.server: initializing acls -46:11.126 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:11.126 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:11.131 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:11.143 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:11.143 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:11.143 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:11.143 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-67fbed38-3bc1-27be-1392-cba596cbf178 -46:11.143 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-67fbed38-3bc1-27be-1392-cba596cbf178.dc1 -46:11.143 [INFO] TestAgent.server: Updating LAN server: server="Node-67fbed38-3bc1-27be-1392-cba596cbf178 (Addr: tcp/127.0.0.1:11628) (DC: dc1)" -46:11.143 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-67fbed38-3bc1-27be-1392-cba596cbf178.dc1 area=wan -46:11.160 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.178 [INFO] TestAgent: Requesting shutdown -46:11.178 [INFO] TestAgent.server: shutting down server -46:11.178 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:11.178 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:11.178 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.178 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:11.178 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:11.183 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:11.183 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.183 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.183 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.183 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.183 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.183 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67fbed38-3bc1-27be-1392-cba596cbf178 -46:11.183 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67fbed38-3bc1-27be-1392-cba596cbf178 -2022-11-07T22:46:11.187Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.187Z [INFO] agent.router.manager: shutting down -46:11.187 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-67fbed38-3bc1-27be-1392-cba596cbf178 127.0.0.1 11626 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:67fbed38-3bc1-27be-1392-cba596cbf178 port:11628 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11627] alive 1 5 2 2 5 4}" error="raft is already shutdown" -46:11.187 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.187 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.187 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.187 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.187 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.187 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.187 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.188 [WARN] TestAgent: Syncing node info failed.: error="leadership lost while committing log" -46:11.188 [INFO] TestAgent: consul server down -46:11.188 [INFO] TestAgent: shutdown complete -46:11.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11651 network=tcp -46:11.188 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="leadership lost while committing log" -46:11.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11651 network=udp -46:11.189 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11673 network=tcp -46:11.689 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.689 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshot_Options (5.05s) - --- PASS: TestSnapshot_Options/GET (0.85s) - --- PASS: TestSnapshot_Options/GET#01 (0.70s) - --- PASS: TestSnapshot_Options/GET#02 (0.98s) - --- PASS: TestSnapshot_Options/PUT (0.95s) - --- PASS: TestSnapshot_Options/PUT#01 (0.91s) - --- PASS: TestSnapshot_Options/PUT#02 (0.67s) -=== CONT TestDNS_CaseInsensitiveNodeLookup -2022-11-07T22:46:10.844Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.860Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.891 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 Address:127.0.0.1:11579}]" -46:10.891 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11579 [Follower]" leader= -46:10.892 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7.dc1 127.0.0.1 -46:10.892 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 127.0.0.1 -2022-11-07T22:46:10.892Z [INFO] agent.router: Initializing LAN area manager -46:10.892 [INFO] TestAgent.server: Adding LAN server: server="Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 (Addr: tcp/127.0.0.1:11579) (DC: dc1)" -46:10.893 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7.dc1 area=wan -46:10.893 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11658 network=udp -46:10.893 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11658 network=tcp -46:10.893 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11645 network=tcp -46:10.893 [INFO] TestAgent: started state syncer -46:10.893 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11580 network=tcp -46:10.945 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.945 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11579 [Candidate]" term=2 -46:10.953 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.953 [DEBUG] TestAgent.server.raft: vote granted: from=4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 term=2 tally=1 -46:10.953 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.953 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11579 [Leader]" -46:10.953 [INFO] TestAgent.server: cluster leadership acquired -46:10.954 [INFO] TestAgent.server: New leader elected: payload=Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 -46:10.959 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11579 -46:10.974 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.998 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.998 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.998 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.998 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.998 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.998 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 -46:10.998 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4cafcc84-b30c-3b27-6a9d-bc6412c3e7a7 -46:11.005 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.219 [INFO] TestAgent: Requesting shutdown -46:11.219 [INFO] TestAgent.server: shutting down server -46:11.219 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.219 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.219 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.219 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.219 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.220 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.220 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.220 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.220 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.224 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.232Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.232Z [INFO] agent.router.manager: shutting down -46:11.232 [INFO] TestAgent: consul server down -46:11.232 [INFO] TestAgent: shutdown complete -46:11.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11658 network=tcp -46:11.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11658 network=udp -46:11.232 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11645 network=tcp -46:11.733 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.733 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_AutopilotGetConfiguration (0.90s) -=== CONT TestDNS_NodeLookup -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:11.124Z [WARN] agent: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:11.124Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.133Z [WARN] agent.auto_config: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:11.133Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.169 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2cc2dcc8-e645-9f61-1b25-75ff379a4432 Address:127.0.0.1:11677}]" -46:11.169 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11677 [Follower]" leader= -46:11.170 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test node.dc1 127.0.0.1 -46:11.170 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test node 127.0.0.1 -2022-11-07T22:46:11.170Z [INFO] agent.router: Initializing LAN area manager -46:11.171 [INFO] TestAgent.server: Handled event for server in area: event=member-join server="test node.dc1" area=wan -46:11.171 [INFO] TestAgent.server: Adding LAN server: server="test node (Addr: tcp/127.0.0.1:11677) (DC: dc1)" -46:11.171 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11665 network=udp -46:11.171 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11665 network=tcp -46:11.171 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11680 network=tcp -46:11.171 [INFO] TestAgent: started state syncer -46:11.171 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11678 network=tcp -46:11.220 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.220 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11677 [Candidate]" term=2 -46:11.234 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.234 [DEBUG] TestAgent.server.raft: vote granted: from=2cc2dcc8-e645-9f61-1b25-75ff379a4432 term=2 tally=1 -46:11.234 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.234 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11677 [Leader]" -46:11.234 [INFO] TestAgent.server: cluster leadership acquired -46:11.234 [INFO] TestAgent.server: New leader elected: payload="test node" -46:11.243 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11677 -46:11.247 [INFO] TestAgent: Synced node info -46:11.247 [DEBUG] TestAgent: Node info in sync -46:11.268 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.282 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.282 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.282 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.282 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.282 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.282 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node="test node" -46:11.282 [INFO] TestAgent.server: member joined, marking health alive: member="test node" -46:11.290 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.360 [DEBUG] TestAgent.dns: request served from client: name=alias.service.consul. type=SRV class=IN latency=165.186µs client=127.0.0.1:53360 client_network=udp -46:11.361 [DEBUG] TestAgent.dns: request served from client: name=alias.service.CoNsUl. type=SRV class=IN latency=163.776µs client=127.0.0.1:34748 client_network=udp -46:11.361 [INFO] TestAgent: Requesting shutdown -46:11.361 [INFO] TestAgent.server: shutting down server -46:11.361 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.361 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.361 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.361 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.361 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.361 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.361 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.361 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.361 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.361 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.367 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.375Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.375Z [INFO] agent.router.manager: shutting down -46:11.375 [INFO] TestAgent: consul server down -46:11.375 [INFO] TestAgent: shutdown complete -46:11.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11665 network=tcp -46:11.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11665 network=udp -46:11.375 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11680 network=tcp -46:11.875 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.875 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ExternalServiceToConsulCNAMELookup (0.76s) -=== CONT TestCatalogServiceNodes_Filter -2022-11-07T22:46:10.621Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.631Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.665 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eb825ded-73ad-f122-00fb-953bb8f5a0dd Address:127.0.0.1:11572}]" -46:10.665 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11572 [Follower]" leader= -46:10.665 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd.dc1 127.0.0.1 -46:10.666 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd 127.0.0.1 -2022-11-07T22:46:10.666Z [INFO] agent.router: Initializing LAN area manager -46:10.666 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd.dc1 area=wan -46:10.666 [INFO] TestAgent.server: Adding LAN server: server="Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd (Addr: tcp/127.0.0.1:11572) (DC: dc1)" -46:10.666 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11644 network=udp -46:10.666 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11644 network=tcp -46:10.666 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11631 network=tcp -46:10.666 [INFO] TestAgent: started state syncer -46:10.666 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11573 network=tcp -46:10.722 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.722 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11572 [Candidate]" term=2 -46:10.733 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.733 [DEBUG] TestAgent.server.raft: vote granted: from=eb825ded-73ad-f122-00fb-953bb8f5a0dd term=2 tally=1 -46:10.733 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.733 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11572 [Leader]" -46:10.733 [INFO] TestAgent.server: cluster leadership acquired -46:10.733 [INFO] TestAgent.server: New leader elected: payload=Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd -46:10.745 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11572 -46:10.767 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.791 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.791 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.791 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.791 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.791 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.791 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd -46:10.791 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eb825ded-73ad-f122-00fb-953bb8f5a0dd -46:10.812 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.894 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.900 [INFO] TestAgent: Synced node info -46:10.900 [DEBUG] TestAgent: Node info in sync -46:11.085 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.085 [DEBUG] TestAgent: Node info in sync -46:11.339 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=4719032 -46:11.363 [INFO] TestAgent: Requesting shutdown -46:11.363 [INFO] TestAgent.server: shutting down server -46:11.363 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.363 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.363 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.363 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.363 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.363 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.378Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.378Z [INFO] agent.router.manager: shutting down -46:11.378 [INFO] TestAgent: consul server down -46:11.378 [INFO] TestAgent: shutdown complete -46:11.378 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11644 network=tcp -46:11.378 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11644 network=udp -46:11.378 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11631 network=tcp -46:11.879 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.879 [INFO] TestAgent: Endpoints down -=== RUN TestTxnEndpoint_Bad_Size_Net/allowed_kv_max_backward_compatible -2022-11-07T22:46:11.205Z [WARN] agent: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:11.205Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.221Z [WARN] agent.auto_config: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:11.221Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.256 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3a90258e-56fb-900c-df22-0eb2302f1ce2 Address:127.0.0.1:11670}]" -46:11.256 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11670 [Follower]" leader= -46:11.256 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test node.dc1 127.0.0.1 -46:11.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test node 127.0.0.1 -2022-11-07T22:46:11.257Z [INFO] agent.router: Initializing LAN area manager -46:11.257 [INFO] TestAgent.server: Adding LAN server: server="test node (Addr: tcp/127.0.0.1:11670) (DC: dc1)" -46:11.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11686 network=udp -46:11.257 [INFO] TestAgent.server: Handled event for server in area: event=member-join server="test node.dc1" area=wan -46:11.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11686 network=tcp -46:11.258 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11666 network=tcp -46:11.258 [INFO] TestAgent: started state syncer -46:11.258 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11671 network=tcp -46:11.299 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.299 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11670 [Candidate]" term=2 -46:11.309 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.309 [DEBUG] TestAgent.server.raft: vote granted: from=3a90258e-56fb-900c-df22-0eb2302f1ce2 term=2 tally=1 -46:11.309 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.309 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11670 [Leader]" -46:11.309 [INFO] TestAgent.server: cluster leadership acquired -46:11.309 [INFO] TestAgent.server: New leader elected: payload="test node" -46:11.315 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11670 -46:11.329 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.360 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.360 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.360 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.360 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.360 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.360 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node="test node" -46:11.360 [INFO] TestAgent.server: member joined, marking health alive: member="test node" -46:11.369 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.387 [ERROR] TestAgent.dns: Infinite recursion detected for name, won't perform any CNAME resolution.: name=web.service.consul. -46:11.388 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=A class=IN latency=261.507µs client=127.0.0.1:55836 client_network=udp -46:11.388 [INFO] TestAgent: Requesting shutdown -46:11.388 [INFO] TestAgent.server: shutting down server -46:11.388 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.388 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.388 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.388 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.388 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.388 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.388 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.388 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.388 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.392 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.395Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.395Z [INFO] agent.router.manager: shutting down -46:11.395 [INFO] TestAgent: consul server down -46:11.395 [INFO] TestAgent: shutdown complete -46:11.395 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11686 network=tcp -46:11.395 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11686 network=udp -46:11.395 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11666 network=tcp -46:11.895 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.895 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_InifiniteRecursion (0.71s) +2021-10-05T16:31:22.612Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.629Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.704 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4dd6c9d2-5639-ea78-1886-39cc9f0429f3 Address:127.0.0.1:30197}]" +31:22.705 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3.dc1 127.0.0.1 +31:22.705 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3 127.0.0.1 +2021-10-05T16:31:22.705Z [INFO] agent.router: Initializing LAN area manager +31:22.705 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30192 network=udp +31:22.705 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30197 [Follower]" leader= +31:22.706 [INFO] TestAgent.server: Adding LAN server: server="Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3 (Addr: tcp/127.0.0.1:30197) (DC: dc1)" +31:22.706 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3.dc1 area=wan +31:22.706 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30192 network=tcp +31:22.706 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30193 network=tcp +31:22.706 [INFO] TestAgent: started state syncer +31:22.707 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30214 network=tcp +31:22.743 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.743 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30197 [Candidate]" term=2 +31:22.812 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.812 [DEBUG] TestAgent.server.raft: vote granted: from=4dd6c9d2-5639-ea78-1886-39cc9f0429f3 term=2 tally=1 +31:22.812 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30197 [Leader]" +31:22.816 [INFO] TestAgent.server: cluster leadership acquired +31:22.816 [INFO] TestAgent.server: New leader elected: payload=Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3 +31:22.852 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30197 +31:22.896 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.950 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.950 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.950 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.950 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.950 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.950 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3 +31:22.950 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4dd6c9d2-5639-ea78-1886-39cc9f0429f3 +31:22.963 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.032 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.036 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=231.585µs client=127.0.0.1:54461 client_network=udp +31:23.036 [DEBUG] TestAgent.dns: request served from client: name=5111c9bb-224f-a4dc-b41e-6f756795abee.query.consul. type=SRV class=IN latency=131.852µs client=127.0.0.1:42493 client_network=udp +31:23.036 [INFO] TestAgent: Requesting shutdown +31:23.036 [INFO] TestAgent.server: shutting down server +31:23.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.036 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.036 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.036 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.036 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.037 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.037 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.045 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:23.045 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.045 [INFO] TestAgent: Synced node info +31:23.045 [DEBUG] TestAgent: Node info in sync +31:23.048 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.055Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.055Z [INFO] agent.router.manager: shutting down +31:23.056 [INFO] TestAgent: consul server down +31:23.056 [INFO] TestAgent: shutdown complete +31:23.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30192 network=tcp +31:23.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30192 network=udp +31:23.056 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30193 network=tcp +31:23.556 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.556 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_ServiceAddressIPV6 (0.96s) === CONT TestDNS_IngressServiceLookup -2022-11-07T22:46:10.936Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.945Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.972 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a7f15781-5da9-bcb9-5ddd-e246942b1efb Address:127.0.0.1:11649}]" -46:10.972 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11649 [Follower]" leader= -46:10.973 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: server1.dc1 127.0.0.1 -46:10.973 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: server1 127.0.0.1 -2022-11-07T22:46:10.973Z [INFO] agent.router: Initializing LAN area manager -46:10.973 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=server1.dc1 area=wan -46:10.973 [INFO] TestAgent.server: Adding LAN server: server="server1 (Addr: tcp/127.0.0.1:11649) (DC: dc1)" -46:10.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11588 network=udp -46:10.974 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11588 network=tcp -46:10.974 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11596 network=tcp -46:10.974 [INFO] TestAgent: started state syncer -46:10.974 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11650 network=tcp -46:11.015 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.015 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11649 [Candidate]" term=2 -46:11.026 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.026 [DEBUG] TestAgent.server.raft: vote granted: from=a7f15781-5da9-bcb9-5ddd-e246942b1efb term=2 tally=1 -46:11.026 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.026 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11649 [Leader]" -46:11.026 [INFO] TestAgent.server: cluster leadership acquired -46:11.026 [INFO] TestAgent.server: New leader elected: payload=server1 -46:11.032 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11649 -46:11.045 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.066 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.066 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.066 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.066 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.066 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.066 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=server1 -46:11.067 [INFO] TestAgent.server: member joined, marking health alive: member=server1 -46:11.077 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.172 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.181 [INFO] TestAgent: Synced node info -46:11.350 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.350 [DEBUG] TestAgent: Node info in sync -46:11.350 [DEBUG] TestAgent: Node info in sync -46:11.405 [DEBUG] TestAgent.dns: request served from client: name=something.node.consul. type=NS class=IN latency=152.499µs client=127.0.0.1:42122 client_network=udp -46:11.405 [INFO] TestAgent: Requesting shutdown -46:11.405 [INFO] TestAgent.server: shutting down server -46:11.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.405 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.405 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.405 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.405 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.406 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.406 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.406 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.409 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.414Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.414Z [INFO] agent.router.manager: shutting down -46:11.414 [INFO] TestAgent: consul server down -46:11.414 [INFO] TestAgent: shutdown complete -46:11.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11588 network=tcp -46:11.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11588 network=udp -46:11.414 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11596 network=tcp -46:11.914 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.914 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NSRecords (0.99s) -=== CONT TestDNS_EmptyAltDomain -2022-11-07T22:46:11.068Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.077Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.101 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07856037-27bf-a044-70c0-cc650c526b21 Address:127.0.0.1:11635}]" -46:11.101 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11635 [Follower]" leader= -46:11.102 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07856037-27bf-a044-70c0-cc650c526b21.dc1 127.0.0.1 -46:11.102 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07856037-27bf-a044-70c0-cc650c526b21 127.0.0.1 -2022-11-07T22:46:11.102Z [INFO] agent.router: Initializing LAN area manager -46:11.103 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07856037-27bf-a044-70c0-cc650c526b21.dc1 area=wan -46:11.103 [INFO] TestAgent.server: Adding LAN server: server="Node-07856037-27bf-a044-70c0-cc650c526b21 (Addr: tcp/127.0.0.1:11635) (DC: dc1)" -46:11.103 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11630 network=udp -46:11.103 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11630 network=tcp -46:11.103 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11603 network=tcp -46:11.103 [INFO] TestAgent: started state syncer -46:11.103 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11636 network=tcp -46:11.142 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.142 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11635 [Candidate]" term=2 -46:11.166 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.166 [DEBUG] TestAgent.server.raft: vote granted: from=07856037-27bf-a044-70c0-cc650c526b21 term=2 tally=1 -46:11.166 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.166 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11635 [Leader]" -46:11.166 [INFO] TestAgent.server: cluster leadership acquired -46:11.166 [INFO] TestAgent.server: New leader elected: payload=Node-07856037-27bf-a044-70c0-cc650c526b21 -46:11.177 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11635 -46:11.192 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.218 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.218 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.218 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.218 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.218 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.218 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07856037-27bf-a044-70c0-cc650c526b21 -46:11.218 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07856037-27bf-a044-70c0-cc650c526b21 -46:11.227 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.421 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:11.421 [INFO] TestAgent: Requesting shutdown -46:11.421 [INFO] TestAgent.server: shutting down server -46:11.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.421 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.421 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.421 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.421 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.421 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.421 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.421 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.425 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.428Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.428Z [INFO] agent.router.manager: shutting down -46:11.428 [INFO] TestAgent: consul server down -46:11.428 [INFO] TestAgent: shutdown complete -46:11.428 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11630 network=tcp -46:11.428 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11630 network=udp -46:11.428 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11603 network=tcp -46:11.929 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.929 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Update (0.87s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:22.876Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.952 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5a3fd63a-2c9f-41bf-5e0e-525293b1b2c4 Address:127.0.0.1:30236}]" +31:22.953 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +31:22.954 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:31:22.954Z [INFO] agent.router: Initializing LAN area manager +31:22.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30224 network=udp +31:22.954 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30236 [Follower]" leader= +31:22.954 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30236) (DC: dc1)" +31:22.954 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +31:22.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30224 network=tcp +31:22.955 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30225 network=tcp +31:22.955 [INFO] TestAgent: started state syncer +31:22.955 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30237 network=tcp +31:22.990 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.990 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30236 [Candidate]" term=2 +31:23.021 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.021 [DEBUG] TestAgent.server.raft: vote granted: from=5a3fd63a-2c9f-41bf-5e0e-525293b1b2c4 term=2 tally=1 +31:23.021 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.021 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30236 [Leader]" +31:23.024 [INFO] TestAgent.server: cluster leadership acquired +31:23.024 [INFO] TestAgent.server: New leader elected: payload=test-node +31:23.035 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30236 +31:23.057 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.114 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.114 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.114 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.114 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.114 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.114 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:23.114 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +31:23.150 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.331 [DEBUG] TestAgent.dns: request served from client: name=alias2.service.consul. type=SRV class=IN latency=211.975µs client=127.0.0.1:34196 client_network=udp +31:23.332 [INFO] TestAgent: Requesting shutdown +31:23.332 [INFO] TestAgent.server: shutting down server +31:23.332 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.332 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.332 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.332 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.332 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:23.332 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.332 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.343 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.352Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.352Z [INFO] agent.router.manager: shutting down +31:23.352 [INFO] TestAgent: consul server down +31:23.352 [INFO] TestAgent: shutdown complete +31:23.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30224 network=tcp +31:23.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30224 network=udp +31:23.352 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30225 network=tcp +31:23.852 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.852 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ExternalServiceToConsulCNAMENestedLookup (1.01s) === CONT TestDNS_ConnectServiceLookup -2022-11-07T22:46:11.102Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.111Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.135 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b281af01-705c-5c4a-1e54-9d8d90fb298d Address:127.0.0.1:11663}]" -46:11.135 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11663 [Follower]" leader= -46:11.136 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b281af01-705c-5c4a-1e54-9d8d90fb298d.dc1 127.0.0.1 -46:11.136 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b281af01-705c-5c4a-1e54-9d8d90fb298d 127.0.0.1 -2022-11-07T22:46:11.136Z [INFO] agent.router: Initializing LAN area manager -46:11.136 [INFO] TestAgent.server: Adding LAN server: server="Node-b281af01-705c-5c4a-1e54-9d8d90fb298d (Addr: tcp/127.0.0.1:11663) (DC: dc1)" -46:11.136 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b281af01-705c-5c4a-1e54-9d8d90fb298d.dc1 area=wan -46:11.136 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11637 network=udp -46:11.136 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11637 network=tcp -46:11.137 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11652 network=tcp -46:11.137 [INFO] TestAgent: started state syncer -46:11.137 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11664 network=tcp -46:11.188 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.188 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11663 [Candidate]" term=2 -46:11.196 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.196 [DEBUG] TestAgent.server.raft: vote granted: from=b281af01-705c-5c4a-1e54-9d8d90fb298d term=2 tally=1 -46:11.196 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.196 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11663 [Leader]" -46:11.196 [INFO] TestAgent.server: cluster leadership acquired -46:11.197 [INFO] TestAgent.server: New leader elected: payload=Node-b281af01-705c-5c4a-1e54-9d8d90fb298d -46:11.204 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11663 -46:11.217 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.250 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.250 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.250 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.250 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.250 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.250 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b281af01-705c-5c4a-1e54-9d8d90fb298d -46:11.250 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b281af01-705c-5c4a-1e54-9d8d90fb298d -46:11.263 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.292 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.294 [INFO] TestAgent: Synced node info -46:11.523 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:11.523 [INFO] TestAgent: Requesting shutdown -46:11.523 [INFO] TestAgent.server: shutting down server -46:11.523 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.523 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.523 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.523 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.523 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.523 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.523 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.523 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.523 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.523 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.527 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.531Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.531Z [INFO] agent.router.manager: shutting down -46:11.531 [INFO] TestAgent: consul server down -46:11.531 [INFO] TestAgent: shutdown complete -46:11.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11637 network=tcp -46:11.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11637 network=udp -46:11.531 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11652 network=tcp -46:12.031 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.031 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#03 -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:11.326Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.342Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.377 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 Address:127.0.0.1:11698}]" -46:11.377 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11698 [Follower]" leader= -46:11.378 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4.dc1 127.0.0.1 -46:11.378 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 127.0.0.1 -2022-11-07T22:46:11.378Z [INFO] agent.router: Initializing LAN area manager -46:11.378 [INFO] TestAgent.server: Adding LAN server: server="Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 (Addr: tcp/127.0.0.1:11698) (DC: dc1)" -46:11.378 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4.dc1 area=wan -46:11.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11672 network=udp -46:11.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11672 network=tcp -46:11.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11694 network=tcp -46:11.379 [INFO] TestAgent: started state syncer -46:11.379 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11699 network=tcp -46:11.439 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.439 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11698 [Candidate]" term=2 -46:11.449 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.449 [DEBUG] TestAgent.server.raft: vote granted: from=4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 term=2 tally=1 -46:11.449 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.449 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11698 [Leader]" -46:11.449 [INFO] TestAgent.server: cluster leadership acquired -46:11.449 [INFO] TestAgent.server: New leader elected: payload=Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 -46:11.455 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11698 -46:11.466 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.494 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.494 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.494 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.494 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.494 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.495 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 -46:11.495 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4f7b54f2-1dd3-3e6c-d4a6-dbe81384a6b4 -46:11.506 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.603 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:11.603 [INFO] TestAgent: Requesting shutdown -46:11.603 [INFO] TestAgent.server: shutting down server -46:11.603 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.603 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.603 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.603 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.603 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.603 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.603 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.603 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.603 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.603 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.603 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.607 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.611Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.611Z [INFO] agent.router.manager: shutting down -46:11.612 [INFO] TestAgent: consul server down -46:11.612 [INFO] TestAgent: shutdown complete -46:11.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11672 network=tcp -46:11.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11672 network=udp -46:11.612 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11694 network=tcp -46:12.112 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.112 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Get/#01 -2022-11-07T22:46:11.192Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.206Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.245 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:663c1e49-ffbc-4f95-994a-158466ac854d Address:127.0.0.1:11684}]" -46:11.245 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11684 [Follower]" leader= -46:11.246 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-663c1e49-ffbc-4f95-994a-158466ac854d.dc1 127.0.0.1 -46:11.246 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-663c1e49-ffbc-4f95-994a-158466ac854d 127.0.0.1 -2022-11-07T22:46:11.246Z [INFO] agent.router: Initializing LAN area manager -46:11.246 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-663c1e49-ffbc-4f95-994a-158466ac854d.dc1 area=wan -46:11.247 [INFO] TestAgent.server: Adding LAN server: server="Node-663c1e49-ffbc-4f95-994a-158466ac854d (Addr: tcp/127.0.0.1:11684) (DC: dc1)" -46:11.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11679 network=udp -46:11.247 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11679 network=tcp -46:11.247 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11659 network=tcp -46:11.247 [INFO] TestAgent: started state syncer -46:11.247 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11685 network=tcp -46:11.286 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.286 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11684 [Candidate]" term=2 -46:11.296 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.296 [DEBUG] TestAgent.server.raft: vote granted: from=663c1e49-ffbc-4f95-994a-158466ac854d term=2 tally=1 -46:11.296 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.296 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11684 [Leader]" -46:11.296 [INFO] TestAgent.server: cluster leadership acquired -46:11.296 [INFO] TestAgent.server: New leader elected: payload=Node-663c1e49-ffbc-4f95-994a-158466ac854d -46:11.302 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11684 -46:11.316 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.331 [INFO] TestAgent: Synced node info -46:11.331 [DEBUG] TestAgent: Node info in sync -46:11.341 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.341 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.341 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.341 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.341 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.341 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-663c1e49-ffbc-4f95-994a-158466ac854d -46:11.341 [INFO] TestAgent.server: member joined, marking health alive: member=Node-663c1e49-ffbc-4f95-994a-158466ac854d -46:11.359 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.612 [INFO] TestAgent: Requesting shutdown -46:11.612 [INFO] TestAgent.server: shutting down server -46:11.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.612 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.612 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.612 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.612 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.612 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.612 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.616 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.620Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.620Z [INFO] agent.router.manager: shutting down -46:11.620 [INFO] TestAgent: consul server down -46:11.620 [INFO] TestAgent: shutdown complete -46:11.620 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11679 network=tcp -46:11.620 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11679 network=udp -46:11.620 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11659 network=tcp -46:12.120 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.120 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes_ConnectProxy (0.94s) -=== CONT TestCatalogServiceNodes_NodeMetaFilter === RUN TestDNS_IngressServiceLookup/api.ingress.consul. === RUN TestDNS_IngressServiceLookup/api.ingress.dc1.consul. === RUN TestDNS_IngressServiceLookup/db.ingress.consul. === RUN TestDNS_IngressServiceLookup/db.ingress.dc1.consul. -2022-11-07T22:46:11.534Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.543Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.572 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb Address:127.0.0.1:11719}]" -46:11.572 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11719 [Follower]" leader= -46:11.573 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb.dc1 127.0.0.1 -46:11.573 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb 127.0.0.1 -2022-11-07T22:46:11.573Z [INFO] agent.router: Initializing LAN area manager -46:11.573 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb.dc1 area=wan -46:11.573 [INFO] TestAgent.server: Adding LAN server: server="Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb (Addr: tcp/127.0.0.1:11719) (DC: dc1)" -46:11.574 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11693 network=udp -46:11.574 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11693 network=tcp -46:11.574 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11701 network=tcp -46:11.574 [INFO] TestAgent: started state syncer -46:11.574 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11720 network=tcp -46:11.635 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.635 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11719 [Candidate]" term=2 -46:11.647 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.647 [DEBUG] TestAgent.server.raft: vote granted: from=1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb term=2 tally=1 -46:11.647 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.647 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11719 [Leader]" -46:11.647 [INFO] TestAgent.server: cluster leadership acquired -46:11.647 [INFO] TestAgent.server: New leader elected: payload=Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb -46:11.653 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11719 -46:11.665 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb -46:11.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1eabbcc2-4cfe-c43e-3ba4-300cf773e0eb -46:11.697 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.703 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=159.277µs client=127.0.0.1:55789 client_network=udp -46:11.703 [INFO] TestAgent: Requesting shutdown -46:11.703 [INFO] TestAgent.server: shutting down server -46:11.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.703 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.703 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.703 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.703 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:11.703 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.703 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.703 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.710 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.718Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.718Z [INFO] agent.router.manager: shutting down -46:11.722 [INFO] TestAgent: consul server down -46:11.722 [INFO] TestAgent: shutdown complete -46:11.722 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11693 network=tcp -46:11.722 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11693 network=udp -46:11.722 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11701 network=tcp -46:12.222 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.222 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ExternalServiceLookup (0.70s) -=== CONT TestDNS_Over_TCP -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:11.506Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.518Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.541 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d14af925-b27c-7f9c-a281-6299c244b4f5 Address:127.0.0.1:11691}]" -46:11.541 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11691 [Follower]" leader= -46:11.541 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d14af925-b27c-7f9c-a281-6299c244b4f5.dc1 127.0.0.1 -46:11.542 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d14af925-b27c-7f9c-a281-6299c244b4f5 127.0.0.1 -2022-11-07T22:46:11.542Z [INFO] agent.router: Initializing LAN area manager -46:11.542 [INFO] TestAgent.server: Adding LAN server: server="Node-d14af925-b27c-7f9c-a281-6299c244b4f5 (Addr: tcp/127.0.0.1:11691) (DC: dc1)" -46:11.542 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d14af925-b27c-7f9c-a281-6299c244b4f5.dc1 area=wan -46:11.542 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11728 network=udp -46:11.542 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11728 network=tcp -46:11.543 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11715 network=tcp -46:11.543 [INFO] TestAgent: started state syncer -46:11.543 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11692 network=tcp -46:11.609 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.609 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11691 [Candidate]" term=2 -46:11.621 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.621 [DEBUG] TestAgent.server.raft: vote granted: from=d14af925-b27c-7f9c-a281-6299c244b4f5 term=2 tally=1 -46:11.621 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.621 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11691 [Leader]" -46:11.621 [INFO] TestAgent.server: cluster leadership acquired -46:11.621 [INFO] TestAgent.server: New leader elected: payload=Node-d14af925-b27c-7f9c-a281-6299c244b4f5 -46:11.627 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11691 -46:11.638 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.659 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.659 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.659 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.659 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.659 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.659 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d14af925-b27c-7f9c-a281-6299c244b4f5 -46:11.659 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d14af925-b27c-7f9c-a281-6299c244b4f5 -46:11.672 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.776 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.779 [INFO] TestAgent: Synced node info -46:11.779 [DEBUG] TestAgent: Node info in sync -46:11.848 [INFO] TestAgent: Requesting shutdown -46:11.848 [INFO] TestAgent.server: shutting down server -46:11.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.848 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.848 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.848 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.848 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.848 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.848 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.855 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.864Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.864Z [INFO] agent.router.manager: shutting down -46:11.864 [INFO] TestAgent: consul server down -46:11.864 [INFO] TestAgent: shutdown complete -46:11.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11728 network=tcp -46:11.864 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11728 network=udp -46:11.864 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11715 network=tcp -46:12.364 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.364 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes_DistanceSort (0.87s) +2021-10-05T16:31:23.012Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.128 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0714833e-eff5-fdb5-c4ef-f75495bb3447 Address:[::1]:30243}]" +31:23.128 [INFO] TestAgent.server.raft: entering follower state: follower="Node at [::1]:30243 [Follower]" leader= +31:23.129 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: server1.dc1 ::1 +31:23.129 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: server1 ::1 +2021-10-05T16:31:23.129Z [INFO] agent.router: Initializing LAN area manager +31:23.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30238 network=udp +31:23.130 [INFO] TestAgent.server: Adding LAN server: server="server1 (Addr: tcp/[::1]:30243) (DC: dc1)" +31:23.130 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=server1.dc1 area=wan +31:23.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30238 network=tcp +31:23.130 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30239 network=tcp +31:23.130 [INFO] TestAgent: started state syncer +31:23.130 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30244 network=tcp +31:23.186 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.186 [INFO] TestAgent.server.raft: entering candidate state: node="Node at [::1]:30243 [Candidate]" term=2 +31:23.224 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.224 [DEBUG] TestAgent.server.raft: vote granted: from=0714833e-eff5-fdb5-c4ef-f75495bb3447 term=2 tally=1 +31:23.224 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.224 [INFO] TestAgent.server.raft: entering leader state: leader="Node at [::1]:30243 [Leader]" +31:23.224 [INFO] TestAgent.server: cluster leadership acquired +31:23.224 [INFO] TestAgent.server: New leader elected: payload=server1 +31:23.255 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=[::1]:30243 +31:23.312 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.371 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.371 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.371 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.371 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.371 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.371 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=server1 +31:23.371 [INFO] TestAgent.server: member joined, marking health alive: member=server1 +31:23.392 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.414 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.432 [INFO] TestAgent: Synced node info +31:23.536 [DEBUG] TestAgent.dns: request served from client: name=server1.node.dc1.consul. type=NS class=IN latency=202.927µs client=127.0.0.1:57368 client_network=udp +31:23.540 [INFO] TestAgent: Requesting shutdown +31:23.540 [INFO] TestAgent.server: shutting down server +31:23.540 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.540 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.540 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.540 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.540 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.540 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.540 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.540 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.540 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.540 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.547 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.554Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.554Z [INFO] agent.router.manager: shutting down +31:23.554 [INFO] TestAgent: consul server down +31:23.554 [INFO] TestAgent: shutdown complete +31:23.554 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30238 network=tcp +31:23.554 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30238 network=udp +31:23.554 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30239 network=tcp +31:24.054 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.054 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NSRecords_IPV6 (1.06s) === CONT TestDNS_ServiceLookupWithInternalServiceAddress -2022-11-07T22:46:11.444Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.453Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.487 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8253edfa-0fee-b9bf-288e-64f8a200ac91 Address:127.0.0.1:11726}]" -46:11.487 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11726 [Follower]" leader= -46:11.488 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8253edfa-0fee-b9bf-288e-64f8a200ac91.dc1 127.0.0.1 -46:11.488 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8253edfa-0fee-b9bf-288e-64f8a200ac91 127.0.0.1 -2022-11-07T22:46:11.488Z [INFO] agent.router: Initializing LAN area manager -46:11.488 [INFO] TestAgent.server: Adding LAN server: server="Node-8253edfa-0fee-b9bf-288e-64f8a200ac91 (Addr: tcp/127.0.0.1:11726) (DC: dc1)" -46:11.488 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8253edfa-0fee-b9bf-288e-64f8a200ac91.dc1 area=wan -46:11.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11700 network=udp -46:11.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11700 network=tcp -46:11.489 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11687 network=tcp -46:11.489 [INFO] TestAgent: started state syncer -46:11.489 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11727 network=tcp -46:11.545 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.545 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11726 [Candidate]" term=2 -46:11.557 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.557 [DEBUG] TestAgent.server.raft: vote granted: from=8253edfa-0fee-b9bf-288e-64f8a200ac91 term=2 tally=1 -46:11.557 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.557 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11726 [Leader]" -46:11.558 [INFO] TestAgent.server: cluster leadership acquired -46:11.558 [INFO] TestAgent.server: New leader elected: payload=Node-8253edfa-0fee-b9bf-288e-64f8a200ac91 -46:11.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11726 -46:11.579 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.598 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.598 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.598 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.598 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.598 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.598 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8253edfa-0fee-b9bf-288e-64f8a200ac91 -46:11.598 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8253edfa-0fee-b9bf-288e-64f8a200ac91 -46:11.609 [INFO] TestAgent.server: federation state anti-entropy synced -46:11.772 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:11.776 [INFO] TestAgent: Synced node info -46:11.776 [DEBUG] TestAgent: Node info in sync -46:11.901 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:11.901 [INFO] TestAgent: Requesting shutdown -46:11.901 [INFO] TestAgent.server: shutting down server -46:11.901 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.901 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.901 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.901 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.901 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.902 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.902 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.902 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.906 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.911Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.911Z [INFO] agent.router.manager: shutting down -46:11.911 [INFO] TestAgent: consul server down -46:11.911 [INFO] TestAgent: shutdown complete -46:11.911 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11700 network=tcp -46:11.911 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11700 network=udp -46:11.912 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11687 network=tcp -46:12.412 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.412 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Explain/#01 [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:11.927Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.940Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.976 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9c6759e2-76f6-709f-956b-67ef010293c8 Address:127.0.0.1:11796}]" -46:11.976 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11796 [Follower]" leader= -46:11.976 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9c6759e2-76f6-709f-956b-67ef010293c8.dc1 127.0.0.1 -46:11.977 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9c6759e2-76f6-709f-956b-67ef010293c8 127.0.0.1 -2022-11-07T22:46:11.977Z [INFO] agent.router: Initializing LAN area manager -46:11.977 [INFO] TestAgent.server: Adding LAN server: server="Node-9c6759e2-76f6-709f-956b-67ef010293c8 (Addr: tcp/127.0.0.1:11796) (DC: dc1)" -46:11.977 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9c6759e2-76f6-709f-956b-67ef010293c8.dc1 area=wan -46:11.977 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11735 network=udp -46:11.977 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11735 network=tcp -46:11.977 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11750 network=tcp -46:11.978 [INFO] TestAgent: started state syncer -46:11.978 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11797 network=tcp -46:12.034 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.034 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11796 [Candidate]" term=2 -46:12.053 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.053 [DEBUG] TestAgent.server.raft: vote granted: from=9c6759e2-76f6-709f-956b-67ef010293c8 term=2 tally=1 -46:12.053 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.053 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11796 [Leader]" -46:12.053 [INFO] TestAgent.server: cluster leadership acquired -46:12.053 [INFO] TestAgent.server: New leader elected: payload=Node-9c6759e2-76f6-709f-956b-67ef010293c8 -46:12.065 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11796 -46:12.086 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.112 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.112 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.112 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.112 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.112 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.112 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c6759e2-76f6-709f-956b-67ef010293c8 -46:12.112 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9c6759e2-76f6-709f-956b-67ef010293c8 -46:12.121 [INFO] TestAgent: Requesting shutdown -46:12.121 [INFO] TestAgent.server: shutting down server -46:12.121 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.121 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.121 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.121 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.121 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.121 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.121 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.128 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:12.130 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.130 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:46:12.134Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.134Z [INFO] agent.router.manager: shutting down -46:12.134 [INFO] TestAgent: consul server down -46:12.134 [INFO] TestAgent: shutdown complete -46:12.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11735 network=tcp -46:12.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11735 network=udp -46:12.134 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11750 network=tcp -46:12.634 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.634 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_EmptyAltDomain (0.72s) -=== CONT TestRecursorAddr ---- PASS: TestRecursorAddr (0.00s) -=== CONT TestCatalogServiceNodes -2022-11-07T22:46:11.743Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.753Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.779 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7c02e2d9-74a7-026c-8233-715fd425a2dd Address:127.0.0.1:11768}]" -46:11.779 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11768 [Follower]" leader= -46:11.780 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7c02e2d9-74a7-026c-8233-715fd425a2dd.dc1 127.0.0.1 -46:11.781 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7c02e2d9-74a7-026c-8233-715fd425a2dd 127.0.0.1 -2022-11-07T22:46:11.781Z [INFO] agent.router: Initializing LAN area manager -46:11.781 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11742 network=udp -46:11.781 [INFO] TestAgent.server: Adding LAN server: server="Node-7c02e2d9-74a7-026c-8233-715fd425a2dd (Addr: tcp/127.0.0.1:11768) (DC: dc1)" -46:11.781 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7c02e2d9-74a7-026c-8233-715fd425a2dd.dc1 area=wan -46:11.781 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11742 network=tcp -46:11.782 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11729 network=tcp -46:11.782 [INFO] TestAgent: started state syncer -46:11.782 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11769 network=tcp -46:11.816 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.816 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11768 [Candidate]" term=2 -46:11.829 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.829 [DEBUG] TestAgent.server.raft: vote granted: from=7c02e2d9-74a7-026c-8233-715fd425a2dd term=2 tally=1 -46:11.829 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.829 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11768 [Leader]" -46:11.829 [INFO] TestAgent.server: cluster leadership acquired -46:11.829 [INFO] TestAgent.server: New leader elected: payload=Node-7c02e2d9-74a7-026c-8233-715fd425a2dd -46:11.837 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11768 -46:11.854 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.882 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.882 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.882 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.882 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.882 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.882 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7c02e2d9-74a7-026c-8233-715fd425a2dd -46:11.882 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7c02e2d9-74a7-026c-8233-715fd425a2dd -46:11.896 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.127 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=ANY class=IN latency=92.66µs client=127.0.0.1:36304 client_network=udp -46:12.127 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=77.554µs client=127.0.0.1:46509 client_network=udp -46:12.127 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=94.127µs client=127.0.0.1:33680 client_network=udp -46:12.128 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=85.267µs client=127.0.0.1:47695 client_network=udp -46:12.128 [INFO] TestAgent: Requesting shutdown -46:12.128 [INFO] TestAgent.server: shutting down server -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.128 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.128 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.128 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.131 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.136Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.136Z [INFO] agent.router.manager: shutting down -46:12.137 [INFO] TestAgent: consul server down -46:12.137 [INFO] TestAgent: shutdown complete -46:12.137 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11742 network=tcp -46:12.137 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11742 network=udp -46:12.137 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11729 network=tcp -46:12.637 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.637 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup (0.90s) +2021-10-05T16:31:23.297Z [WARN] agent: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:23.297Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.308Z [WARN] agent.auto_config: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:23.308Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.373 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7a84bb7b-1c45-61b5-bc31-45bcdbb6a8aa Address:127.0.0.1:30257}]" +31:23.374 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test node.dc1 127.0.0.1 +31:23.375 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test node 127.0.0.1 +2021-10-05T16:31:23.375Z [INFO] agent.router: Initializing LAN area manager +31:23.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30252 network=udp +31:23.375 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30257 [Follower]" leader= +31:23.376 [INFO] TestAgent.server: Adding LAN server: server="test node (Addr: tcp/127.0.0.1:30257) (DC: dc1)" +31:23.376 [INFO] TestAgent.server: Handled event for server in area: event=member-join server="test node.dc1" area=wan +31:23.376 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30252 network=tcp +31:23.376 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30253 network=tcp +31:23.377 [INFO] TestAgent: started state syncer +31:23.377 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30258 network=tcp +31:23.432 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.432 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30257 [Candidate]" term=2 +31:23.456 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.456 [DEBUG] TestAgent.server.raft: vote granted: from=7a84bb7b-1c45-61b5-bc31-45bcdbb6a8aa term=2 tally=1 +31:23.456 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.456 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30257 [Leader]" +31:23.456 [INFO] TestAgent.server: cluster leadership acquired +31:23.456 [INFO] TestAgent.server: New leader elected: payload="test node" +31:23.464 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30257 +31:23.479 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.542 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.542 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.542 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.542 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.542 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.542 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node="test node" +31:23.542 [INFO] TestAgent.server: member joined, marking health alive: member="test node" +31:23.563 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.648 [DEBUG] TestAgent.dns: request served from client: name=alias.service.consul. type=SRV class=IN latency=333.086µs client=127.0.0.1:49091 client_network=udp +31:23.652 [DEBUG] TestAgent.dns: request served from client: name=alias.service.CoNsUl. type=SRV class=IN latency=294.589µs client=127.0.0.1:40751 client_network=udp +31:23.654 [INFO] TestAgent: Requesting shutdown +31:23.654 [INFO] TestAgent.server: shutting down server +31:23.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.654 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.654 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.655 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.655 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.655 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:23.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.655 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.655 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.661 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.672Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.672Z [INFO] agent.router.manager: shutting down +31:23.672 [INFO] TestAgent: consul server down +31:23.672 [INFO] TestAgent: shutdown complete +31:23.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30252 network=tcp +31:23.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30252 network=udp +31:23.672 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30253 network=tcp +31:24.173 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.173 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ExternalServiceToConsulCNAMELookup (0.89s) === CONT TestDNS_ServiceLookup -2022-11-07T22:46:11.905Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.914Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.945 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8adc6e89-0417-0802-3f83-7e506387aaa0 Address:127.0.0.1:11733}]" -46:11.945 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11733 [Follower]" leader= -46:11.945 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8adc6e89-0417-0802-3f83-7e506387aaa0.dc1 127.0.0.1 -46:11.946 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8adc6e89-0417-0802-3f83-7e506387aaa0 127.0.0.1 -2022-11-07T22:46:11.946Z [INFO] agent.router: Initializing LAN area manager -46:11.946 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8adc6e89-0417-0802-3f83-7e506387aaa0.dc1 area=wan -46:11.946 [INFO] TestAgent.server: Adding LAN server: server="Node-8adc6e89-0417-0802-3f83-7e506387aaa0 (Addr: tcp/127.0.0.1:11733) (DC: dc1)" -46:11.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11707 network=udp -46:11.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11707 network=tcp -46:11.946 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11708 network=tcp -46:11.946 [INFO] TestAgent: started state syncer -46:11.947 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11734 network=tcp -46:12.001 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.001 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11733 [Candidate]" term=2 -46:12.011 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.011 [DEBUG] TestAgent.server.raft: vote granted: from=8adc6e89-0417-0802-3f83-7e506387aaa0 term=2 tally=1 -46:12.011 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.011 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11733 [Leader]" -46:12.012 [INFO] TestAgent.server: cluster leadership acquired -46:12.012 [INFO] TestAgent.server: New leader elected: payload=Node-8adc6e89-0417-0802-3f83-7e506387aaa0 -46:12.021 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11733 -46:12.043 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.076 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.076 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.076 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.076 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.076 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.076 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8adc6e89-0417-0802-3f83-7e506387aaa0 -46:12.076 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8adc6e89-0417-0802-3f83-7e506387aaa0 -46:12.092 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.127 [DEBUG] TestAgent.dns: request served from client: name=api.ingress.consul. type=A class=IN latency=130.592µs client=127.0.0.1:41006 client_network=udp -46:12.127 [DEBUG] TestAgent.dns: request served from client: name=api.ingress.dc1.consul. type=A class=IN latency=109.873µs client=127.0.0.1:58895 client_network=udp -46:12.128 [DEBUG] TestAgent.dns: request served from client: name=db.ingress.consul. type=A class=IN latency=110.606µs client=127.0.0.1:44970 client_network=udp -46:12.128 [DEBUG] TestAgent.dns: request served from client: name=db.ingress.dc1.consul. type=A class=IN latency=117.394µs client=127.0.0.1:33443 client_network=udp -46:12.128 [INFO] TestAgent: Requesting shutdown -46:12.128 [INFO] TestAgent.server: shutting down server -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.128 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.128 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.134 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.140Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.140Z [INFO] agent.router.manager: shutting down -46:12.140 [INFO] TestAgent: consul server down -46:12.140 [INFO] TestAgent: shutdown complete -46:12.140 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11707 network=tcp -46:12.140 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11707 network=udp -46:12.140 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11708 network=tcp -46:12.641 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.641 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_IngressServiceLookup (0.75s) +2021-10-05T16:31:23.269Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.285Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.352 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:79aaf8ea-4c65-a43a-0dfb-a1ce715cb7a0 Address:127.0.0.1:30250}]" +31:23.352 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30250 [Follower]" leader= +31:23.353 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: server1.dc1 127.0.0.1 +31:23.353 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: server1 127.0.0.1 +2021-10-05T16:31:23.353Z [INFO] agent.router: Initializing LAN area manager +31:23.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30245 network=udp +31:23.354 [INFO] TestAgent.server: Adding LAN server: server="server1 (Addr: tcp/127.0.0.1:30250) (DC: dc1)" +31:23.354 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=server1.dc1 area=wan +31:23.354 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30245 network=tcp +31:23.354 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30246 network=tcp +31:23.354 [INFO] TestAgent: started state syncer +31:23.354 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30251 network=tcp +31:23.392 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.392 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30250 [Candidate]" term=2 +31:23.435 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.435 [DEBUG] TestAgent.server.raft: vote granted: from=79aaf8ea-4c65-a43a-0dfb-a1ce715cb7a0 term=2 tally=1 +31:23.435 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.435 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30250 [Leader]" +31:23.435 [INFO] TestAgent.server: cluster leadership acquired +31:23.435 [INFO] TestAgent.server: New leader elected: payload=server1 +31:23.451 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30250 +31:23.469 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.504 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.504 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.504 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.504 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.504 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.505 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=server1 +31:23.505 [INFO] TestAgent.server: member joined, marking health alive: member=server1 +31:23.544 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.600 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.616 [INFO] TestAgent: Synced node info +31:23.616 [DEBUG] TestAgent: Node info in sync +31:23.701 [DEBUG] TestAgent.dns: request served from client: name=something.node.consul. type=NS class=IN latency=129.875µs client=127.0.0.1:39006 client_network=udp +31:23.701 [INFO] TestAgent: Requesting shutdown +31:23.701 [INFO] TestAgent.server: shutting down server +31:23.701 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.701 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.701 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.702 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.702 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.702 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.702 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.702 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.702 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.702 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.707 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.716Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.716Z [INFO] agent.router.manager: shutting down +31:23.716 [INFO] TestAgent: consul server down +31:23.716 [INFO] TestAgent: shutdown complete +31:23.716 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30245 network=tcp +31:23.716 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30245 network=udp +31:23.716 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30246 network=tcp +31:24.217 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.217 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NSRecords (0.96s) +=== CONT TestDNS_ServiceLookupMultiAddrNoCNAME +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:22.781Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:22.781Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.834Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:22.834Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.928 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7ba53657-7940-454c-e904-03a54fb19b86 Address:127.0.0.1:30222}]" +31:22.929 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30222 [Follower]" leader= +31:22.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 127.0.0.1 +31:22.930 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7ba53657-7940-454c-e904-03a54fb19b86 127.0.0.1 +2021-10-05T16:31:22.930Z [INFO] agent.router: Initializing LAN area manager +31:22.930 [INFO] TestAgent.server: Adding LAN server: server="Node-7ba53657-7940-454c-e904-03a54fb19b86 (Addr: tcp/127.0.0.1:30222) (DC: dc2)" +31:22.930 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 area=wan +31:22.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30231 network=tcp +31:22.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30231 network=udp +31:22.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30232 network=tcp +31:22.932 [INFO] TestAgent: started state syncer +31:22.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30223 network=tcp +31:22.981 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:22.996 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.996 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30222 [Candidate]" term=2 +31:23.023 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.023 [DEBUG] TestAgent.server.raft: vote granted: from=7ba53657-7940-454c-e904-03a54fb19b86 term=2 tally=1 +31:23.023 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.023 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30222 [Leader]" +31:23.024 [INFO] TestAgent.server: cluster leadership acquired +31:23.024 [INFO] TestAgent.server: New leader elected: payload=Node-7ba53657-7940-454c-e904-03a54fb19b86 +31:23.039 [INFO] TestAgent.server: initializing acls +31:23.045 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:23.064 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:23.064 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:23.064 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:23.064 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7ba53657-7940-454c-e904-03a54fb19b86 +31:23.064 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 +31:23.064 [INFO] TestAgent.server: Updating LAN server: server="Node-7ba53657-7940-454c-e904-03a54fb19b86 (Addr: tcp/127.0.0.1:30222) (DC: dc2)" +31:23.065 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 area=wan +31:23.099 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.115 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:30203] +31:23.115 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:30203 +31:23.116 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 127.0.0.1 +31:23.117 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:23.117 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 area=wan +31:23.164 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.164 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.164 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.164 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.164 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.164 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7ba53657-7940-454c-e904-03a54fb19b86 +31:23.164 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7ba53657-7940-454c-e904-03a54fb19b86 +31:23.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7ba53657-7940-454c-e904-03a54fb19b86 +31:23.193 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=224.021µs client=127.0.0.1:43650 client_network=udp +31:23.193 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=117.297µs client=127.0.0.1:35011 client_network=udp +31:23.193 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=109.035µs client=127.0.0.1:46594 client_network=udp +31:23.193 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=92.006µs client=127.0.0.1:57019 client_network=udp +31:23.208 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.222 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=293.956µs client=127.0.0.1:40325 client_network=udp +31:23.222 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=158.203µs client=127.0.0.1:53033 client_network=udp +31:23.222 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=103.707µs client=127.0.0.1:39122 client_network=udp +31:23.222 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=98.116µs client=127.0.0.1:57059 client_network=udp +31:23.240 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=266.581µs client=127.0.0.1:55938 client_network=udp +31:23.241 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=239.159µs client=127.0.0.1:46299 client_network=udp +31:23.241 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=143.261µs client=127.0.0.1:37431 client_network=udp +31:23.242 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=131.068µs client=127.0.0.1:35898 client_network=udp +31:23.252 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.256 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=302.801µs client=127.0.0.1:41713 client_network=udp +31:23.257 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=162.531µs client=127.0.0.1:48536 client_network=udp +31:23.257 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=146.714µs client=127.0.0.1:37943 client_network=udp +31:23.257 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=126.878µs client=127.0.0.1:33948 client_network=udp +31:23.262 [INFO] TestAgent: Synced node info +31:23.338 [INFO] TestAgent: Requesting shutdown +31:23.338 [INFO] TestAgent.server: shutting down server +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:23.338 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:23.338 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.338 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:23.338 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:23.345 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.354Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.354Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.354Z [INFO] agent.router.manager: shutting down +31:23.354 [INFO] TestAgent: consul server down +31:23.354 [INFO] TestAgent: shutdown complete +31:23.354 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30231 network=tcp +31:23.354 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30231 network=udp +31:23.354 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30232 network=tcp +31:23.854 [INFO] TestAgent: Waiting for endpoints to shut down +31:23.854 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:22.454Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:22.454Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.483Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:22.483Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.508 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c885042-4c7c-2d34-c11f-3ea0f3500f76 Address:127.0.0.1:30204}]" +31:22.508 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30204 [Follower]" leader= +31:22.509 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 127.0.0.1 +31:22.509 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 127.0.0.1 +2021-10-05T16:31:22.509Z [INFO] agent.router: Initializing LAN area manager +31:22.509 [INFO] TestAgent.server: Adding LAN server: server="Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 (Addr: tcp/127.0.0.1:30204) (DC: dc1)" +31:22.509 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30199 network=udp +31:22.510 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 area=wan +31:22.510 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30199 network=tcp +31:22.510 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30200 network=tcp +31:22.510 [INFO] TestAgent: started state syncer +31:22.510 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30191 network=tcp +31:22.560 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:22.576 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.576 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30204 [Candidate]" term=2 +31:22.605 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.605 [DEBUG] TestAgent.server.raft: vote granted: from=6c885042-4c7c-2d34-c11f-3ea0f3500f76 term=2 tally=1 +31:22.605 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.605 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30204 [Leader]" +31:22.608 [INFO] TestAgent.server: cluster leadership acquired +31:22.608 [INFO] TestAgent.server: New leader elected: payload=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.628 [INFO] TestAgent.server: initializing acls +31:22.644 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:22.660 [INFO] TestAgent.server: initializing acls +31:22.699 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:22.699 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:22.699 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:22.699 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:22.699 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.699 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 +31:22.699 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 area=wan +31:22.699 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:22.699 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.699 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 +31:22.700 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76.dc1 area=wan +31:22.700 [INFO] TestAgent.server: Updating LAN server: server="Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 (Addr: tcp/127.0.0.1:30204) (DC: dc1)" +31:22.700 [INFO] TestAgent.server: Updating LAN server: server="Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 (Addr: tcp/127.0.0.1:30204) (DC: dc1)" +31:22.724 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.814 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.814 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.814 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.814 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.814 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.814 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.814 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.847 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.847 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c885042-4c7c-2d34-c11f-3ea0f3500f76 +31:22.896 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.896 [INFO] TestAgent: Synced node info +31:22.896 [DEBUG] TestAgent: Node info in sync +31:23.116 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:59572 +31:23.116 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 127.0.0.1 +31:23.116 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7ba53657-7940-454c-e904-03a54fb19b86.dc2 area=wan +31:23.145 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=984.116µs client=127.0.0.1:44051 client_network=udp +31:23.146 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=382.497µs client=127.0.0.1:40943 client_network=udp +31:23.146 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=279.808µs client=127.0.0.1:59985 client_network=udp +31:23.147 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=282.586µs client=127.0.0.1:56743 client_network=udp +31:23.272 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=8.139011ms client=127.0.0.1:33647 client_network=udp +31:23.273 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=604.713µs client=127.0.0.1:45732 client_network=udp +31:23.273 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=338.819µs client=127.0.0.1:47438 client_network=udp +31:23.274 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=374.338µs client=127.0.0.1:54075 client_network=udp +31:23.311 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=543.105µs client=127.0.0.1:51082 client_network=udp +31:23.311 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=296.697µs client=127.0.0.1:39115 client_network=udp +31:23.316 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=4.482479ms client=127.0.0.1:60675 client_network=udp +31:23.318 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=1.696601ms client=127.0.0.1:51581 client_network=udp +31:23.331 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=SRV class=IN latency=534.028µs client=127.0.0.1:42669 client_network=udp +31:23.332 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=SRV class=IN latency=435.667µs client=127.0.0.1:40690 client_network=udp +31:23.337 [DEBUG] TestAgent.dns: request served from client: name=db.service.dc2.consul. type=A class=IN latency=873.333µs client=127.0.0.1:40769 client_network=udp +31:23.337 [DEBUG] TestAgent.dns: request served from client: name=11a3f8fc-e904-5ae4-ff20-98020f6b975b.query.dc2.consul. type=A class=IN latency=466.411µs client=127.0.0.1:38280 client_network=udp +31:23.854 [INFO] TestAgent: Requesting shutdown +31:23.854 [INFO] TestAgent.server: shutting down server +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:23.854 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:23.854 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.854 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.854 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.855 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.855 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.855 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:23.855 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:23.860 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.872Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.872Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.872Z [INFO] agent.router.manager: shutting down +31:23.872 [INFO] TestAgent: consul server down +31:23.872 [INFO] TestAgent: shutdown complete +31:23.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30199 network=tcp +31:23.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30199 network=udp +31:23.872 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30200 network=tcp +31:24.375 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.375 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_WanTranslation (1.95s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc1 (0.01s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc2 (0.05s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc2 (0.03s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc2 (0.02s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/service-wan-from-dc2 (0.02s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/node-addr-from-dc1 (0.02s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/node-wan-from-dc1 (0.05s) + --- PASS: TestDNS_ServiceLookup_WanTranslation/service-addr-from-dc1 (0.02s) +=== CONT TestDNS_ServiceLookupPreferNoCNAME +2021-10-05T16:31:23.560Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.630Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.684 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae Address:127.0.0.1:30271}]" +31:23.685 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae.dc1 127.0.0.1 +31:23.685 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae 127.0.0.1 +2021-10-05T16:31:23.685Z [INFO] agent.router: Initializing LAN area manager +31:23.686 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30266 network=udp +31:23.686 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30271 [Follower]" leader= +31:23.686 [INFO] TestAgent.server: Adding LAN server: server="Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae (Addr: tcp/127.0.0.1:30271) (DC: dc1)" +31:23.686 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae.dc1 area=wan +31:23.686 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30266 network=tcp +31:23.686 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30267 network=tcp +31:23.686 [INFO] TestAgent: started state syncer +31:23.687 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30272 network=tcp +31:23.751 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.752 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30271 [Candidate]" term=2 +31:23.768 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.768 [DEBUG] TestAgent.server.raft: vote granted: from=ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae term=2 tally=1 +31:23.768 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.768 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30271 [Leader]" +31:23.771 [INFO] TestAgent.server: cluster leadership acquired +31:23.771 [INFO] TestAgent.server: New leader elected: payload=Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae +31:23.777 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30271 +31:23.828 [INFO] TestAgent: Synced node info +31:23.828 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.866 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.866 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.866 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.866 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.866 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.866 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae +31:23.866 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ed8c5c6c-ab05-6c56-ff3c-27c8d4768fae +31:23.894 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.924 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=246.473µs client=127.0.0.1:60824 client_network=udp +31:23.928 [INFO] TestAgent: Requesting shutdown +31:23.928 [INFO] TestAgent.server: shutting down server +31:23.928 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.928 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.928 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.928 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.948Z [INFO] agent.router.manager: shutting down +31:23.948 [INFO] TestAgent: consul server down +31:23.948 [INFO] TestAgent: shutdown complete +31:23.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30266 network=tcp +31:23.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30266 network=udp +31:23.948 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30267 network=tcp +31:24.448 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.448 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ExternalServiceLookup (0.91s) +=== CONT TestDNS_ServiceLookupNoMultiCNAME +2021-10-05T16:31:23.567Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.623Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.678 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fe2ad231-b565-cf48-43ca-a747496301e5 Address:127.0.0.1:30278}]" +31:23.678 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fe2ad231-b565-cf48-43ca-a747496301e5.dc1 127.0.0.1 +31:23.679 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fe2ad231-b565-cf48-43ca-a747496301e5 127.0.0.1 +2021-10-05T16:31:23.679Z [INFO] agent.router: Initializing LAN area manager +31:23.679 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30273 network=udp +31:23.679 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30278 [Follower]" leader= +31:23.679 [INFO] TestAgent.server: Adding LAN server: server="Node-fe2ad231-b565-cf48-43ca-a747496301e5 (Addr: tcp/127.0.0.1:30278) (DC: dc1)" +31:23.679 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fe2ad231-b565-cf48-43ca-a747496301e5.dc1 area=wan +31:23.679 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30273 network=tcp +31:23.680 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30274 network=tcp +31:23.680 [INFO] TestAgent: started state syncer +31:23.680 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30279 network=tcp +31:23.719 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.719 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30278 [Candidate]" term=2 +31:23.735 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.735 [DEBUG] TestAgent.server.raft: vote granted: from=fe2ad231-b565-cf48-43ca-a747496301e5 term=2 tally=1 +31:23.735 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.735 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30278 [Leader]" +31:23.735 [INFO] TestAgent.server: cluster leadership acquired +31:23.735 [INFO] TestAgent.server: New leader elected: payload=Node-fe2ad231-b565-cf48-43ca-a747496301e5 +31:23.743 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30278 +31:23.767 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.803 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.803 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.803 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.803 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.803 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.803 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fe2ad231-b565-cf48-43ca-a747496301e5 +31:23.803 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fe2ad231-b565-cf48-43ca-a747496301e5 +31:23.845 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.941 [DEBUG] TestAgent.dns: request served from client: name=api.ingress.consul. type=A class=IN latency=184.306µs client=127.0.0.1:41344 client_network=udp +31:23.942 [DEBUG] TestAgent.dns: request served from client: name=api.ingress.dc1.consul. type=A class=IN latency=98.43µs client=127.0.0.1:54722 client_network=udp +31:23.942 [DEBUG] TestAgent.dns: request served from client: name=db.ingress.consul. type=A class=IN latency=100.028µs client=127.0.0.1:44764 client_network=udp +31:23.942 [DEBUG] TestAgent.dns: request served from client: name=db.ingress.dc1.consul. type=A class=IN latency=100.189µs client=127.0.0.1:49379 client_network=udp +31:23.942 [INFO] TestAgent: Requesting shutdown +31:23.943 [INFO] TestAgent.server: shutting down server +31:23.943 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.943 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.943 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.943 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.943 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.943 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.943 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.943 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:23.943 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.943 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.943 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.947 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.961Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.961Z [INFO] agent.router.manager: shutting down +31:23.961 [INFO] TestAgent: consul server down +31:23.961 [INFO] TestAgent: shutdown complete +31:23.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30273 network=tcp +31:23.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30273 network=udp +31:23.961 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30274 network=tcp +31:24.464 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.464 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_IngressServiceLookup (0.91s) --- PASS: TestDNS_IngressServiceLookup/api.ingress.consul. (0.00s) --- PASS: TestDNS_IngressServiceLookup/api.ingress.dc1.consul. (0.00s) --- PASS: TestDNS_IngressServiceLookup/db.ingress.consul. (0.00s) --- PASS: TestDNS_IngressServiceLookup/db.ingress.dc1.consul. (0.00s) -=== CONT TestDNS_ServiceLookupMultiAddrNoCNAME -2022-11-07T22:46:11.703Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.712Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.735 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf3575f5-c6a1-c028-9690-819423d8cb6c Address:127.0.0.1:11740}]" -46:11.735 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11740 [Follower]" leader= -46:11.735 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf3575f5-c6a1-c028-9690-819423d8cb6c.dc1 127.0.0.1 -46:11.736 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf3575f5-c6a1-c028-9690-819423d8cb6c 127.0.0.1 -2022-11-07T22:46:11.736Z [INFO] agent.router: Initializing LAN area manager -46:11.736 [INFO] TestAgent.server: Adding LAN server: server="Node-cf3575f5-c6a1-c028-9690-819423d8cb6c (Addr: tcp/127.0.0.1:11740) (DC: dc1)" -46:11.736 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf3575f5-c6a1-c028-9690-819423d8cb6c.dc1 area=wan -46:11.736 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11756 network=udp -46:11.736 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11756 network=tcp -46:11.737 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11743 network=tcp -46:11.737 [INFO] TestAgent: started state syncer -46:11.737 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11741 network=tcp -46:11.798 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.798 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11740 [Candidate]" term=2 -46:11.811 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.811 [DEBUG] TestAgent.server.raft: vote granted: from=cf3575f5-c6a1-c028-9690-819423d8cb6c term=2 tally=1 -46:11.811 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.811 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11740 [Leader]" -46:11.811 [INFO] TestAgent.server: cluster leadership acquired -46:11.812 [INFO] TestAgent.server: New leader elected: payload=Node-cf3575f5-c6a1-c028-9690-819423d8cb6c -46:11.818 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11740 -46:11.829 [INFO] TestAgent: Synced node info -46:11.847 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.864 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.864 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.864 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.864 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.864 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.864 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf3575f5-c6a1-c028-9690-819423d8cb6c -46:11.864 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cf3575f5-c6a1-c028-9690-819423d8cb6c -46:11.878 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.168 [DEBUG] TestAgent.dns: request served from client: name=fOO.node.dc1.consul. type=ANY class=IN latency=64.888µs client=127.0.0.1:58045 client_network=udp -46:12.168 [INFO] TestAgent: Requesting shutdown -46:12.168 [INFO] TestAgent.server: shutting down server -46:12.168 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.168 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.168 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.168 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.168 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.168 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.172 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.176Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.176Z [INFO] agent.router.manager: shutting down -46:12.176 [INFO] TestAgent: consul server down -46:12.176 [INFO] TestAgent: shutdown complete -46:12.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11756 network=tcp -46:12.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11756 network=udp -46:12.176 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11743 network=tcp -46:12.677 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.677 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_CaseInsensitiveNodeLookup (0.99s) +=== CONT TestDNS_ServiceReverseLookupNodeAddress +2021-10-05T16:31:23.538Z [WARN] agent: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:23.538Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.549Z [WARN] agent.auto_config: Node name "test node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:23.549Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.630 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:634f03de-420b-804a-dd91-e71fada436c1 Address:127.0.0.1:30264}]" +31:23.631 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test node.dc1 127.0.0.1 +31:23.633 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30264 [Follower]" leader= +31:23.637 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test node 127.0.0.1 +2021-10-05T16:31:23.637Z [INFO] agent.router: Initializing LAN area manager +31:23.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30259 network=udp +31:23.637 [INFO] TestAgent.server: Adding LAN server: server="test node (Addr: tcp/127.0.0.1:30264) (DC: dc1)" +31:23.637 [INFO] TestAgent.server: Handled event for server in area: event=member-join server="test node.dc1" area=wan +31:23.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30259 network=tcp +31:23.638 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30260 network=tcp +31:23.638 [INFO] TestAgent: started state syncer +31:23.639 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30265 network=tcp +31:23.694 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:23.694 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30264 [Candidate]" term=2 +31:23.707 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:23.707 [DEBUG] TestAgent.server.raft: vote granted: from=634f03de-420b-804a-dd91-e71fada436c1 term=2 tally=1 +31:23.707 [INFO] TestAgent.server.raft: election won: tally=1 +31:23.707 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30264 [Leader]" +31:23.708 [INFO] TestAgent.server: cluster leadership acquired +31:23.708 [INFO] TestAgent.server: New leader elected: payload="test node" +31:23.717 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30264 +31:23.733 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:23.769 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:23.769 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:23.769 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:23.769 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:23.769 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:23.770 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node="test node" +31:23.770 [INFO] TestAgent.server: member joined, marking health alive: member="test node" +31:23.794 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.859 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.863 [INFO] TestAgent: Synced node info +31:23.863 [DEBUG] TestAgent: Node info in sync +31:23.984 [ERROR] TestAgent.dns: Infinite recursion detected for name, won't perform any CNAME resolution.: name=web.service.consul. +31:23.984 [DEBUG] TestAgent.dns: request served from client: name=web.service.consul. type=A class=IN latency=352.495µs client=127.0.0.1:45956 client_network=udp +31:23.984 [INFO] TestAgent: Requesting shutdown +31:23.984 [INFO] TestAgent.server: shutting down server +31:23.984 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:23.984 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:23.984 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:23.984 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:23.984 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:23.984 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:23.984 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:23.984 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:23.984 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:23.984 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:23.991 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:23.999Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:23.999Z [INFO] agent.router.manager: shutting down +31:23.999 [INFO] TestAgent: consul server down +31:23.999 [INFO] TestAgent: shutdown complete +31:23.999 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30259 network=tcp +31:24.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30259 network=udp +31:24.000 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30260 network=tcp +31:24.500 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.500 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_InifiniteRecursion (0.98s) +=== CONT TestDNS_SOA_Settings +2021-10-05T16:31:20.779Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.795Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.850 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ffd3ecb-f9f7-865e-d980-2a05f7088b79 Address:127.0.0.1:30120}]" +31:20.851 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79.dc1 127.0.0.1 +31:20.851 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79 127.0.0.1 +2021-10-05T16:31:20.851Z [INFO] agent.router: Initializing LAN area manager +31:20.852 [DEBUG] TestAgent.dns: recursor enabled +31:20.852 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30115 network=udp +31:20.852 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30120 [Follower]" leader= +31:20.852 [DEBUG] TestAgent.dns: recursor enabled +31:20.852 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79.dc1 area=wan +31:20.852 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30115 network=tcp +31:20.852 [INFO] TestAgent.server: Adding LAN server: server="Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79 (Addr: tcp/127.0.0.1:30120) (DC: dc1)" +31:20.853 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30116 network=tcp +31:20.853 [INFO] TestAgent: started state syncer +31:20.853 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30121 network=tcp +31:20.920 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.920 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30120 [Candidate]" term=2 +31:20.942 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.942 [DEBUG] TestAgent.server.raft: vote granted: from=1ffd3ecb-f9f7-865e-d980-2a05f7088b79 term=2 tally=1 +31:20.942 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.942 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30120 [Leader]" +31:20.942 [INFO] TestAgent.server: cluster leadership acquired +31:20.942 [INFO] TestAgent.server: New leader elected: payload=Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79 +31:20.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30120 +31:21.000 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:21.043 [INFO] TestAgent: Synced node info +31:21.043 [DEBUG] TestAgent: Node info in sync +31:21.077 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:21.077 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:21.077 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:21.077 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:21.077 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:21.077 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79 +31:21.077 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1ffd3ecb-f9f7-865e-d980-2a05f7088b79 +31:21.091 [INFO] TestAgent.server: federation state anti-entropy synced +31:23.885 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:23.885 [DEBUG] TestAgent: Node info in sync +31:24.048 [ERROR] TestAgent.dns: recurse failed: error="read udp 127.0.0.1:32805->127.0.0.1:38000: i/o timeout" +31:24.048 [ERROR] TestAgent.dns: all resolvers failed for question from client: question="{apple.com. 255 1}" client=127.0.0.1:34403 client_network=udp +31:24.048 [DEBUG] TestAgent.dns: request served from client: question="{apple.com. 255 1}" network=udp latency=3.001684962s client=127.0.0.1:34403 client_network=udp +31:24.051 [INFO] TestAgent: Requesting shutdown +31:24.051 [INFO] TestAgent.server: shutting down server +31:24.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.051 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.051 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.051 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.052 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.052 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.052 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.052 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.052 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.060 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.064Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.064Z [INFO] agent.router.manager: shutting down +31:24.064 [INFO] TestAgent: consul server down +31:24.064 [INFO] TestAgent: shutdown complete +31:24.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30115 network=tcp +31:24.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30115 network=udp +31:24.064 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30116 network=tcp +31:24.568 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.568 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_RecursorTimeout (3.81s) +=== CONT TestDNS_ServiceReverseLookup_CustomDomain +[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:23.868Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:23.882Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:23.956 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:72e1a2b0-d46f-12af-77d2-a34754c416ca Address:127.0.0.1:30285}]" +31:23.957 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-72e1a2b0-d46f-12af-77d2-a34754c416ca.dc1 127.0.0.1 +31:23.957 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-72e1a2b0-d46f-12af-77d2-a34754c416ca 127.0.0.1 +2021-10-05T16:31:23.957Z [INFO] agent.router: Initializing LAN area manager +31:23.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30280 network=udp +31:23.957 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30285 [Follower]" leader= +31:23.958 [INFO] TestAgent.server: Adding LAN server: server="Node-72e1a2b0-d46f-12af-77d2-a34754c416ca (Addr: tcp/127.0.0.1:30285) (DC: dc1)" +31:23.958 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-72e1a2b0-d46f-12af-77d2-a34754c416ca.dc1 area=wan +31:23.958 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30280 network=tcp +31:23.958 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30281 network=tcp +31:23.958 [INFO] TestAgent: started state syncer +31:23.959 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30286 network=tcp +31:24.020 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.020 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30285 [Candidate]" term=2 +31:24.056 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.056 [DEBUG] TestAgent.server.raft: vote granted: from=72e1a2b0-d46f-12af-77d2-a34754c416ca term=2 tally=1 +31:24.056 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.056 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30285 [Leader]" +31:24.056 [INFO] TestAgent.server: cluster leadership acquired +31:24.057 [INFO] TestAgent.server: New leader elected: payload=Node-72e1a2b0-d46f-12af-77d2-a34754c416ca +31:24.089 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30285 +31:24.103 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.150 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.150 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.150 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.150 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.150 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.150 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72e1a2b0-d46f-12af-77d2-a34754c416ca +31:24.150 [INFO] TestAgent.server: member joined, marking health alive: member=Node-72e1a2b0-d46f-12af-77d2-a34754c416ca +31:24.173 [INFO] TestAgent: Synced node info +31:24.197 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.197 [DEBUG] TestAgent.dns: request served from client: name=db.connect.consul. type=SRV class=IN latency=161.483µs client=127.0.0.1:37537 client_network=udp +31:24.198 [INFO] TestAgent: Requesting shutdown +31:24.198 [INFO] TestAgent.server: shutting down server +31:24.198 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.198 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.198 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.198 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.198 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.198 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.198 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.198 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.198 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.198 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.212 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.220Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.220Z [INFO] agent.router.manager: shutting down +31:24.220 [INFO] TestAgent: consul server down +31:24.220 [INFO] TestAgent: shutdown complete +31:24.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30280 network=tcp +31:24.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30280 network=udp +31:24.220 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30281 network=tcp +31:24.721 [INFO] TestAgent: Waiting for endpoints to shut down +31:24.721 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ConnectServiceLookup (0.87s) +=== CONT TestDNS_ServiceReverseLookup_IPV6 +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:24.071Z [WARN] agent: Node name "my.test-node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:24.071Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.087Z [WARN] agent.auto_config: Node name "my.test-node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:24.087Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.121 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8d5fce15-117b-f24d-c2ed-07a1e5da618a Address:127.0.0.1:30292}]" +31:24.122 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: my.test-node.dc1 127.0.0.1 +31:24.122 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: my.test-node 127.0.0.1 +2021-10-05T16:31:24.122Z [INFO] agent.router: Initializing LAN area manager +31:24.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30287 network=udp +31:24.122 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30292 [Follower]" leader= +31:24.123 [INFO] TestAgent.server: Adding LAN server: server="my.test-node (Addr: tcp/127.0.0.1:30292) (DC: dc1)" +31:24.123 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=my.test-node.dc1 area=wan +31:24.123 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30287 network=tcp +31:24.124 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30288 network=tcp +31:24.124 [INFO] TestAgent: started state syncer +31:24.124 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30293 network=tcp +31:24.184 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.184 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30292 [Candidate]" term=2 +31:24.226 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.226 [DEBUG] TestAgent.server.raft: vote granted: from=8d5fce15-117b-f24d-c2ed-07a1e5da618a term=2 tally=1 +31:24.226 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.226 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30292 [Leader]" +31:24.228 [INFO] TestAgent.server: cluster leadership acquired +31:24.228 [INFO] TestAgent.server: New leader elected: payload=my.test-node +31:24.244 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30292 +31:24.285 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.317 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.318 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.318 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.318 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.318 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.318 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=my.test-node +31:24.318 [INFO] TestAgent.server: member joined, marking health alive: member=my.test-node +31:24.351 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.388 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:24.404 [INFO] TestAgent: Synced node info +31:24.515 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=137.71µs client=127.0.0.1:44101 client_network=udp +31:24.515 [INFO] TestAgent: Requesting shutdown +31:24.515 [INFO] TestAgent.server: shutting down server +31:24.515 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.515 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.515 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.515 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.515 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.515 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.515 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.515 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.515 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.515 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.521 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.548Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.548Z [INFO] agent.router.manager: shutting down +31:24.548 [INFO] TestAgent: consul server down +31:24.548 [INFO] TestAgent: shutdown complete +31:24.548 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30287 network=tcp +31:24.548 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30287 network=udp +31:24.548 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30288 network=tcp +31:25.049 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.049 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookupWithInternalServiceAddress (1.00s) +=== CONT TestDNS_ServiceReverseLookup +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:24.191Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.206Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.259 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cca9f60a-06e8-91a0-fd47-3889b3459c1a Address:127.0.0.1:30299}]" +31:24.260 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a.dc1 127.0.0.1 +31:24.260 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30299 [Follower]" leader= +31:24.260 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a 127.0.0.1 +2021-10-05T16:31:24.260Z [INFO] agent.router: Initializing LAN area manager +31:24.261 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30294 network=udp +31:24.261 [INFO] TestAgent.server: Adding LAN server: server="Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a (Addr: tcp/127.0.0.1:30299) (DC: dc1)" +31:24.261 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a.dc1 area=wan +31:24.261 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30294 network=tcp +31:24.262 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30295 network=tcp +31:24.262 [INFO] TestAgent: started state syncer +31:24.262 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30300 network=tcp +31:24.300 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.300 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30299 [Candidate]" term=2 +31:24.340 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.340 [DEBUG] TestAgent.server.raft: vote granted: from=cca9f60a-06e8-91a0-fd47-3889b3459c1a term=2 tally=1 +31:24.340 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.340 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30299 [Leader]" +31:24.340 [INFO] TestAgent.server: cluster leadership acquired +31:24.340 [INFO] TestAgent.server: New leader elected: payload=Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a +31:24.364 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30299 +31:24.405 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.452 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.452 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.452 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.452 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.452 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.452 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a +31:24.452 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cca9f60a-06e8-91a0-fd47-3889b3459c1a +31:24.484 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.528 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=220.235µs client=127.0.0.1:46711 client_network=udp +31:24.532 [DEBUG] TestAgent.dns: request served from client: name=7288ef3a-eeaa-290f-cd5c-240e8299c5e2.query.consul. type=SRV class=IN latency=164.873µs client=127.0.0.1:60690 client_network=udp +31:24.532 [DEBUG] TestAgent.dns: request served from client: name=nodb.service.consul. type=SRV class=IN latency=86.126µs client=127.0.0.1:35438 client_network=udp +31:24.533 [DEBUG] TestAgent.dns: request served from client: name=nope.query.consul. type=SRV class=IN latency=53.992µs client=127.0.0.1:44481 client_network=udp +31:24.533 [INFO] TestAgent: Requesting shutdown +31:24.533 [INFO] TestAgent.server: shutting down server +31:24.533 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.533 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.533 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.533 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.533 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.544 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.544 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.544 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:24.544 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.544 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.544 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.568 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.578Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.578Z [INFO] agent.router.manager: shutting down +31:24.578 [INFO] TestAgent: consul server down +31:24.578 [INFO] TestAgent: shutdown complete +31:24.578 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30294 network=tcp +31:24.578 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30294 network=udp +31:24.578 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30295 network=tcp +31:25.078 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.078 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup (0.91s) +=== CONT TestDNS_ReverseLookup_IPV6 +2021-10-05T16:31:24.234Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.247Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.284 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88304ef4-3ddb-8816-d036-d4094ba9e323 Address:127.0.0.1:30306}]" +31:24.285 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88304ef4-3ddb-8816-d036-d4094ba9e323.dc1 127.0.0.1 +31:24.285 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88304ef4-3ddb-8816-d036-d4094ba9e323 127.0.0.1 +2021-10-05T16:31:24.285Z [INFO] agent.router: Initializing LAN area manager +31:24.286 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30301 network=udp +31:24.286 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30306 [Follower]" leader= +31:24.286 [INFO] TestAgent.server: Adding LAN server: server="Node-88304ef4-3ddb-8816-d036-d4094ba9e323 (Addr: tcp/127.0.0.1:30306) (DC: dc1)" +31:24.286 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88304ef4-3ddb-8816-d036-d4094ba9e323.dc1 area=wan +31:24.286 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30301 network=tcp +31:24.287 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30302 network=tcp +31:24.287 [INFO] TestAgent: started state syncer +31:24.287 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30307 network=tcp +31:24.332 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.332 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30306 [Candidate]" term=2 +31:24.382 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.382 [DEBUG] TestAgent.server.raft: vote granted: from=88304ef4-3ddb-8816-d036-d4094ba9e323 term=2 tally=1 +31:24.382 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.382 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30306 [Leader]" +31:24.382 [INFO] TestAgent.server: cluster leadership acquired +31:24.383 [INFO] TestAgent.server: New leader elected: payload=Node-88304ef4-3ddb-8816-d036-d4094ba9e323 +31:24.409 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30306 +31:24.427 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88304ef4-3ddb-8816-d036-d4094ba9e323 +31:24.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88304ef4-3ddb-8816-d036-d4094ba9e323 +31:24.531 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.552 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:24.573 [INFO] TestAgent: Synced node info +31:24.658 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=336.458µs client=127.0.0.1:36584 client_network=udp +31:24.658 [INFO] TestAgent: Requesting shutdown +31:24.658 [INFO] TestAgent.server: shutting down server +31:24.658 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.658 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.658 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.658 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.659 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.659 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.659 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.659 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.659 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.659 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.668 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.684Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.684Z [INFO] agent.router.manager: shutting down +31:24.684 [INFO] TestAgent: consul server down +31:24.684 [INFO] TestAgent: shutdown complete +31:24.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30301 network=tcp +31:24.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30301 network=udp +31:24.685 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30302 network=tcp +31:25.185 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.185 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookupMultiAddrNoCNAME (0.97s) +=== CONT TestDNS_ReverseLookup_CustomDomain +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:24.395Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.428Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.508 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06eec532-dee6-9916-81fa-346de96b2184 Address:127.0.0.1:30313}]" +31:24.509 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-06eec532-dee6-9916-81fa-346de96b2184.dc1 127.0.0.1 +31:24.509 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-06eec532-dee6-9916-81fa-346de96b2184 127.0.0.1 +2021-10-05T16:31:24.509Z [INFO] agent.router: Initializing LAN area manager +31:24.510 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30308 network=udp +31:24.510 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30313 [Follower]" leader= +31:24.510 [INFO] TestAgent.server: Adding LAN server: server="Node-06eec532-dee6-9916-81fa-346de96b2184 (Addr: tcp/127.0.0.1:30313) (DC: dc1)" +31:24.510 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-06eec532-dee6-9916-81fa-346de96b2184.dc1 area=wan +31:24.510 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30308 network=tcp +31:24.511 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30309 network=tcp +31:24.511 [INFO] TestAgent: started state syncer +31:24.511 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30314 network=tcp +31:24.578 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.578 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30313 [Candidate]" term=2 +31:24.636 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.636 [DEBUG] TestAgent.server.raft: vote granted: from=06eec532-dee6-9916-81fa-346de96b2184 term=2 tally=1 +31:24.636 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.636 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30313 [Leader]" +31:24.636 [INFO] TestAgent.server: cluster leadership acquired +31:24.636 [INFO] TestAgent.server: New leader elected: payload=Node-06eec532-dee6-9916-81fa-346de96b2184 +31:24.668 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30313 +31:24.691 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.739 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.739 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.739 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.739 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.739 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.739 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06eec532-dee6-9916-81fa-346de96b2184 +31:24.739 [INFO] TestAgent.server: member joined, marking health alive: member=Node-06eec532-dee6-9916-81fa-346de96b2184 +31:24.777 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.805 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=219.871µs client=127.0.0.1:35537 client_network=udp +31:24.805 [INFO] TestAgent: Requesting shutdown +31:24.805 [INFO] TestAgent.server: shutting down server +31:24.805 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.805 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.805 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.805 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.805 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.805 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.805 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.805 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:24.805 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.805 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.805 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.814 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.827Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.827Z [INFO] agent.router.manager: shutting down +31:24.828 [INFO] TestAgent: consul server down +31:24.828 [INFO] TestAgent: shutdown complete +31:24.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30308 network=tcp +31:24.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30308 network=udp +31:24.828 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30309 network=tcp +31:25.328 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.328 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookupPreferNoCNAME (0.95s) +=== CONT TestDNS_ReverseLookup +2021-10-05T16:31:24.467Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.592 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6679e07a-f8d9-2af5-444b-f61e90fd397c Address:127.0.0.1:30320}]" +31:24.592 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30320 [Follower]" leader= +31:24.593 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6679e07a-f8d9-2af5-444b-f61e90fd397c.dc1 127.0.0.1 +31:24.594 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6679e07a-f8d9-2af5-444b-f61e90fd397c 127.0.0.1 +2021-10-05T16:31:24.594Z [INFO] agent.router: Initializing LAN area manager +31:24.594 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30315 network=udp +31:24.595 [INFO] TestAgent.server: Adding LAN server: server="Node-6679e07a-f8d9-2af5-444b-f61e90fd397c (Addr: tcp/127.0.0.1:30320) (DC: dc1)" +31:24.595 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6679e07a-f8d9-2af5-444b-f61e90fd397c.dc1 area=wan +31:24.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30315 network=tcp +31:24.596 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30316 network=tcp +31:24.596 [INFO] TestAgent: started state syncer +31:24.597 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30321 network=tcp +31:24.643 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.643 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30320 [Candidate]" term=2 +31:24.688 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.688 [DEBUG] TestAgent.server.raft: vote granted: from=6679e07a-f8d9-2af5-444b-f61e90fd397c term=2 tally=1 +31:24.688 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.688 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30320 [Leader]" +31:24.688 [INFO] TestAgent.server: cluster leadership acquired +31:24.689 [INFO] TestAgent.server: New leader elected: payload=Node-6679e07a-f8d9-2af5-444b-f61e90fd397c +31:24.699 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30320 +31:24.733 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.776 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.776 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.776 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.776 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.776 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.776 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6679e07a-f8d9-2af5-444b-f61e90fd397c +31:24.776 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6679e07a-f8d9-2af5-444b-f61e90fd397c +31:24.809 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.834 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=207.731µs client=127.0.0.1:38448 client_network=udp +31:24.834 [INFO] TestAgent: Requesting shutdown +31:24.834 [INFO] TestAgent.server: shutting down server +31:24.834 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.834 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.834 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.834 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.834 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.834 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.834 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.834 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:24.834 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.834 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.834 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.841 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.846Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.846Z [INFO] agent.router.manager: shutting down +31:24.846 [INFO] TestAgent: consul server down +31:24.846 [INFO] TestAgent: shutdown complete +31:24.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30315 network=tcp +31:24.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30315 network=udp +31:24.846 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30316 network=tcp +31:25.349 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.349 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookupNoMultiCNAME (0.90s) +=== CONT TestDNS_EDNS0_ECS +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:24.493Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.547Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.640 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c32bd972-bf11-2175-7d2a-937f6b5dd235 Address:127.0.0.1:30327}]" +31:24.640 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30327 [Follower]" leader= +31:24.640 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c32bd972-bf11-2175-7d2a-937f6b5dd235.dc1 127.0.0.1 +31:24.641 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c32bd972-bf11-2175-7d2a-937f6b5dd235 127.0.0.1 +2021-10-05T16:31:24.641Z [INFO] agent.router: Initializing LAN area manager +31:24.641 [INFO] TestAgent.server: Adding LAN server: server="Node-c32bd972-bf11-2175-7d2a-937f6b5dd235 (Addr: tcp/127.0.0.1:30327) (DC: dc1)" +31:24.641 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c32bd972-bf11-2175-7d2a-937f6b5dd235.dc1 area=wan +31:24.641 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30322 network=udp +31:24.641 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30322 network=tcp +31:24.642 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30323 network=tcp +31:24.642 [INFO] TestAgent: started state syncer +31:24.642 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30328 network=tcp +31:24.688 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.688 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30327 [Candidate]" term=2 +31:24.736 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.736 [DEBUG] TestAgent.server.raft: vote granted: from=c32bd972-bf11-2175-7d2a-937f6b5dd235 term=2 tally=1 +31:24.736 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.736 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30327 [Leader]" +31:24.736 [INFO] TestAgent.server: cluster leadership acquired +31:24.736 [INFO] TestAgent.server: New leader elected: payload=Node-c32bd972-bf11-2175-7d2a-937f6b5dd235 +31:24.771 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30327 +31:24.813 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.861 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.861 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.861 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.861 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.861 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.861 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c32bd972-bf11-2175-7d2a-937f6b5dd235 +31:24.861 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c32bd972-bf11-2175-7d2a-937f6b5dd235 +31:24.872 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.932 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:24.938 [INFO] TestAgent: Synced node info +31:25.065 [DEBUG] TestAgent.dns: request served from client: question="{1.0.0.127.in-addr.arpa. 255 1}" latency=101.141µs client=127.0.0.1:36214 client_network=udp +31:25.065 [INFO] TestAgent: Requesting shutdown +31:25.065 [INFO] TestAgent.server: shutting down server +31:25.065 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.065 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.065 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.065 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.065 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.065 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.065 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.065 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.069 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.075Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.075Z [INFO] agent.router.manager: shutting down +31:25.075 [INFO] TestAgent: consul server down +31:25.075 [INFO] TestAgent: shutdown complete +31:25.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30322 network=tcp +31:25.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30322 network=udp +31:25.075 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30323 network=tcp +31:25.576 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.576 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceReverseLookupNodeAddress (1.11s) +=== CONT TestDNS_EDNS0 +2021-10-05T16:31:24.736Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.749Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.851 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd2ca64a-6028-d835-2e74-1324c49add1d Address:127.0.0.1:30348}]" +31:24.852 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd2ca64a-6028-d835-2e74-1324c49add1d.dc1 127.0.0.1 +31:24.852 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd2ca64a-6028-d835-2e74-1324c49add1d 127.0.0.1 +2021-10-05T16:31:24.852Z [INFO] agent.router: Initializing LAN area manager +31:24.853 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30343 network=udp +31:24.853 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30348 [Follower]" leader= +31:24.853 [INFO] TestAgent.server: Adding LAN server: server="Node-cd2ca64a-6028-d835-2e74-1324c49add1d (Addr: tcp/127.0.0.1:30348) (DC: dc1)" +31:24.853 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd2ca64a-6028-d835-2e74-1324c49add1d.dc1 area=wan +31:24.853 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30343 network=tcp +31:24.854 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30344 network=tcp +31:24.854 [INFO] TestAgent: started state syncer +31:24.854 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30349 network=tcp +31:24.920 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.920 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30348 [Candidate]" term=2 +31:24.944 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.944 [DEBUG] TestAgent.server.raft: vote granted: from=cd2ca64a-6028-d835-2e74-1324c49add1d term=2 tally=1 +31:24.944 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.944 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30348 [Leader]" +31:24.944 [INFO] TestAgent.server: cluster leadership acquired +31:24.944 [INFO] TestAgent.server: New leader elected: payload=Node-cd2ca64a-6028-d835-2e74-1324c49add1d +31:24.954 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30348 +31:24.972 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.020 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.020 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.020 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.020 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.020 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.020 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd2ca64a-6028-d835-2e74-1324c49add1d +31:25.020 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd2ca64a-6028-d835-2e74-1324c49add1d +31:25.036 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:25.045 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.062 [INFO] TestAgent: Synced node info +31:25.062 [DEBUG] TestAgent: Node info in sync +31:25.063 [DEBUG] TestAgent.dns: request served from client: question="{9.2.3.8.2.4.0.0.0.0.f.f.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa. 255 1}" latency=94.074µs client=127.0.0.1:44972 client_network=udp +31:25.063 [INFO] TestAgent: Requesting shutdown +31:25.063 [INFO] TestAgent.server: shutting down server +31:25.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.063 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.063 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.063 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.063 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.069 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.074Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.074Z [INFO] agent.router.manager: shutting down +31:25.074 [INFO] TestAgent: consul server down +31:25.074 [INFO] TestAgent: shutdown complete +31:25.074 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30343 network=tcp +31:25.074 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30343 network=udp +31:25.074 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30344 network=tcp +31:25.576 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.576 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:24.623Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.655Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.713 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d8607efc-bb1a-ba65-66e5-fa557d7f528c Address:127.0.0.1:30341}]" +31:24.714 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c.dc1 127.0.0.1 +31:24.715 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c 127.0.0.1 +2021-10-05T16:31:24.715Z [INFO] agent.router: Initializing LAN area manager +31:24.715 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30341 [Follower]" leader= +31:24.715 [INFO] TestAgent.server: Adding LAN server: server="Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c (Addr: tcp/127.0.0.1:30341) (DC: dc1)" +31:24.715 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c.dc1 area=wan +31:24.715 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30336 network=tcp +31:24.715 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30336 network=udp +31:24.716 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30337 network=tcp +31:24.716 [INFO] TestAgent: started state syncer +31:24.716 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30342 network=tcp +31:24.760 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.760 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30341 [Candidate]" term=2 +31:24.812 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.812 [DEBUG] TestAgent.server.raft: vote granted: from=d8607efc-bb1a-ba65-66e5-fa557d7f528c term=2 tally=1 +31:24.812 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30341 [Leader]" +31:24.813 [INFO] TestAgent.server: cluster leadership acquired +31:24.813 [INFO] TestAgent.server: New leader elected: payload=Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c +31:24.840 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30341 +31:24.874 [INFO] TestAgent: Synced node info +31:24.912 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.958 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.958 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.958 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.958 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.958 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.958 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c +31:24.958 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d8607efc-bb1a-ba65-66e5-fa557d7f528c +31:24.976 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.063 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=112.264µs client=127.0.0.1:35765 client_network=udp +31:25.064 [INFO] TestAgent: Requesting shutdown +31:25.064 [INFO] TestAgent.server: shutting down server +31:25.064 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.064 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.064 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.064 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.064 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.064 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.064 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.069 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.075Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.075Z [INFO] agent.router.manager: shutting down +31:25.075 [INFO] TestAgent: consul server down +31:25.075 [INFO] TestAgent: shutdown complete +31:25.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30336 network=tcp +31:25.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30336 network=udp +31:25.075 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30337 network=tcp +31:25.576 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.576 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceReverseLookup_CustomDomain (1.01s) +=== CONT TestDNS_NodeLookup_CNAME +--- PASS: TestDNS_ServiceReverseLookup_IPV6 (0.86s) +=== CONT TestDNSCycleRecursorCheck +=== RUN TestDNS_EDNS0_ECS/global +=== RUN TestDNS_EDNS0_ECS/query +=== RUN TestDNS_EDNS0_ECS/query-subnet +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:25.071Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.085Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.156 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f96b404-471d-22da-7803-d8633c803edd Address:127.0.0.1:30355}]" +31:25.157 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4f96b404-471d-22da-7803-d8633c803edd.dc1 127.0.0.1 +31:25.158 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4f96b404-471d-22da-7803-d8633c803edd 127.0.0.1 +2021-10-05T16:31:25.158Z [INFO] agent.router: Initializing LAN area manager +31:25.158 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30350 network=udp +31:25.158 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30355 [Follower]" leader= +31:25.158 [INFO] TestAgent.server: Adding LAN server: server="Node-4f96b404-471d-22da-7803-d8633c803edd (Addr: tcp/127.0.0.1:30355) (DC: dc1)" +31:25.158 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4f96b404-471d-22da-7803-d8633c803edd.dc1 area=wan +31:25.159 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30350 network=tcp +31:25.159 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30351 network=tcp +31:25.159 [INFO] TestAgent: started state syncer +31:25.159 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30356 network=tcp +31:25.224 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.224 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30355 [Candidate]" term=2 +31:25.248 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.248 [DEBUG] TestAgent.server.raft: vote granted: from=4f96b404-471d-22da-7803-d8633c803edd term=2 tally=1 +31:25.248 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.248 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30355 [Leader]" +31:25.248 [INFO] TestAgent.server: cluster leadership acquired +31:25.248 [INFO] TestAgent.server: New leader elected: payload=Node-4f96b404-471d-22da-7803-d8633c803edd +31:25.258 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30355 +31:25.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.355 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.355 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.355 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.355 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.355 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.355 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4f96b404-471d-22da-7803-d8633c803edd +31:25.355 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4f96b404-471d-22da-7803-d8633c803edd +31:25.389 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.416 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=251.84µs client=127.0.0.1:59963 client_network=udp +31:25.416 [INFO] TestAgent: Requesting shutdown +31:25.416 [INFO] TestAgent.server: shutting down server +31:25.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.417 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.417 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:25.417 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.417 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.417 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.432 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.448Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.448Z [INFO] agent.router.manager: shutting down +31:25.449 [INFO] TestAgent: consul server down +31:25.449 [INFO] TestAgent: shutdown complete +31:25.449 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30350 network=tcp +31:25.449 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30350 network=udp +31:25.449 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30351 network=tcp +31:25.949 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.949 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceReverseLookup (0.90s) +=== CONT TestDNS_NodeLookup_AAAA +2021-10-05T16:31:25.090Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.107Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.163 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a Address:127.0.0.1:30362}]" +31:25.164 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30362 [Follower]" leader= +31:25.164 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a.dc1 127.0.0.1 +31:25.165 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a 127.0.0.1 +2021-10-05T16:31:25.165Z [INFO] agent.router: Initializing LAN area manager +31:25.165 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30357 network=udp +31:25.165 [INFO] TestAgent.server: Adding LAN server: server="Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a (Addr: tcp/127.0.0.1:30362) (DC: dc1)" +31:25.165 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a.dc1 area=wan +31:25.165 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30357 network=tcp +31:25.166 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30358 network=tcp +31:25.166 [INFO] TestAgent: started state syncer +31:25.166 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30363 network=tcp +31:25.225 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.225 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30362 [Candidate]" term=2 +31:25.245 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.245 [DEBUG] TestAgent.server.raft: vote granted: from=6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a term=2 tally=1 +31:25.245 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.245 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30362 [Leader]" +31:25.248 [INFO] TestAgent.server: cluster leadership acquired +31:25.248 [INFO] TestAgent.server: New leader elected: payload=Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a +31:25.259 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30362 +31:25.277 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.309 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.309 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.309 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.309 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.309 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.309 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a +31:25.309 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6a993fcc-fd4e-2c58-2ab4-4fdd4fd7121a +31:25.330 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.344 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:25.356 [INFO] TestAgent: Synced node info +31:25.356 [DEBUG] TestAgent: Node info in sync +31:25.516 [DEBUG] TestAgent.dns: request served from client: question="{2.4.2.4.2.4.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. 255 1}" latency=197.746µs client=127.0.0.1:39747 client_network=udp +31:25.520 [INFO] TestAgent: Requesting shutdown +31:25.520 [INFO] TestAgent.server: shutting down server +31:25.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.520 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.520 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.520 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.520 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.520 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.520 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.532 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.544Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.544Z [INFO] agent.router.manager: shutting down +31:25.544 [INFO] TestAgent: consul server down +31:25.544 [INFO] TestAgent: shutdown complete +31:25.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30357 network=tcp +31:25.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30357 network=udp +31:25.544 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30358 network=tcp +31:26.044 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.044 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ReverseLookup_IPV6 (0.97s) +=== CONT TestDNS_NodeLookup_PeriodName +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:25.211Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.229Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.296 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fcb4bb21-bd2a-e2b6-3367-bc4e376e413c Address:127.0.0.1:30369}]" +31:25.296 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30369 [Follower]" leader= +31:25.297 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c.dc1 127.0.0.1 +31:25.297 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c 127.0.0.1 +2021-10-05T16:31:25.297Z [INFO] agent.router: Initializing LAN area manager +31:25.297 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30364 network=udp +31:25.298 [INFO] TestAgent.server: Adding LAN server: server="Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c (Addr: tcp/127.0.0.1:30369) (DC: dc1)" +31:25.298 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c.dc1 area=wan +31:25.298 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30364 network=tcp +31:25.298 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30365 network=tcp +31:25.298 [INFO] TestAgent: started state syncer +31:25.298 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30370 network=tcp +31:25.353 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.353 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30369 [Candidate]" term=2 +31:25.391 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.391 [DEBUG] TestAgent.server.raft: vote granted: from=fcb4bb21-bd2a-e2b6-3367-bc4e376e413c term=2 tally=1 +31:25.391 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.391 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30369 [Leader]" +31:25.416 [INFO] TestAgent.server: cluster leadership acquired +31:25.417 [INFO] TestAgent.server: New leader elected: payload=Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c +31:25.448 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30369 +31:25.493 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.584 [INFO] TestAgent: Synced node info +31:25.584 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.584 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.584 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.584 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.584 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.584 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c +31:25.584 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fcb4bb21-bd2a-e2b6-3367-bc4e376e413c +31:25.636 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.640 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=120.415µs client=127.0.0.1:43904 client_network=udp +31:25.641 [INFO] TestAgent: Requesting shutdown +31:25.641 [INFO] TestAgent.server: shutting down server +31:25.641 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.641 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.641 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.641 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.641 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.641 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.652 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.660Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.660Z [INFO] agent.router.manager: shutting down +31:25.660 [INFO] TestAgent: consul server down +31:25.660 [INFO] TestAgent: shutdown complete +31:25.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30364 network=tcp +31:25.660 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30364 network=udp +31:25.660 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30365 network=tcp +31:26.163 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.163 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ReverseLookup_CustomDomain (0.98s) +=== CONT TestDNS_CaseInsensitiveNodeLookup +2021-10-05T16:31:25.349Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.373Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.483 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:34627bae-ad65-e2cc-0236-c961c5c78197 Address:127.0.0.1:30376}]" +31:25.484 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-34627bae-ad65-e2cc-0236-c961c5c78197.dc1 127.0.0.1 +31:25.484 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30376 [Follower]" leader= +31:25.484 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-34627bae-ad65-e2cc-0236-c961c5c78197 127.0.0.1 +2021-10-05T16:31:25.484Z [INFO] agent.router: Initializing LAN area manager +31:25.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30371 network=udp +31:25.485 [INFO] TestAgent.server: Adding LAN server: server="Node-34627bae-ad65-e2cc-0236-c961c5c78197 (Addr: tcp/127.0.0.1:30376) (DC: dc1)" +31:25.485 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-34627bae-ad65-e2cc-0236-c961c5c78197.dc1 area=wan +31:25.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30371 network=tcp +31:25.485 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30372 network=tcp +31:25.485 [INFO] TestAgent: started state syncer +31:25.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30377 network=tcp +31:25.546 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.546 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30376 [Candidate]" term=2 +31:25.568 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.568 [DEBUG] TestAgent.server.raft: vote granted: from=34627bae-ad65-e2cc-0236-c961c5c78197 term=2 tally=1 +31:25.568 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.568 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30376 [Leader]" +31:25.568 [INFO] TestAgent.server: cluster leadership acquired +31:25.569 [INFO] TestAgent.server: New leader elected: payload=Node-34627bae-ad65-e2cc-0236-c961c5c78197 +31:25.594 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30376 +31:25.642 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.680 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.680 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.680 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.680 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.680 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.680 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-34627bae-ad65-e2cc-0236-c961c5c78197 +31:25.680 [INFO] TestAgent.server: member joined, marking health alive: member=Node-34627bae-ad65-e2cc-0236-c961c5c78197 +31:25.708 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.734 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=79.94µs client=127.0.0.1:58228 client_network=udp +31:25.734 [INFO] TestAgent: Requesting shutdown +31:25.734 [INFO] TestAgent.server: shutting down server +31:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.734 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.734 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.734 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.760Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.760Z [INFO] agent.router.manager: shutting down +31:25.760 [INFO] TestAgent: consul server down +31:25.760 [INFO] TestAgent: shutdown complete +31:25.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30371 network=tcp +31:25.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30371 network=udp +31:25.760 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30372 network=tcp +31:26.264 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.264 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ReverseLookup (0.94s) +=== CONT TestDNS_NodeLookup +2021-10-05T16:31:25.375Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.390Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.470 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd Address:127.0.0.1:30383}]" +31:25.471 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd.dc1 127.0.0.1 +31:25.471 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd 127.0.0.1 +2021-10-05T16:31:25.471Z [INFO] agent.router: Initializing LAN area manager +31:25.472 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30378 network=udp +31:25.472 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30383 [Follower]" leader= +31:25.472 [INFO] TestAgent.server: Adding LAN server: server="Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd (Addr: tcp/127.0.0.1:30383) (DC: dc1)" +31:25.472 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd.dc1 area=wan +31:25.473 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30378 network=tcp +31:25.474 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30379 network=tcp +31:25.474 [INFO] TestAgent: started state syncer +31:25.474 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30384 network=tcp +31:25.522 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.522 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30383 [Candidate]" term=2 +31:25.545 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.545 [DEBUG] TestAgent.server.raft: vote granted: from=717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd term=2 tally=1 +31:25.545 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.545 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30383 [Leader]" +31:25.545 [INFO] TestAgent.server: cluster leadership acquired +31:25.545 [INFO] TestAgent.server: New leader elected: payload=Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd +31:25.563 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30383 +31:25.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.668 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.668 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.668 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.668 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.668 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.668 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd +31:25.668 [INFO] TestAgent.server: member joined, marking health alive: member=Node-717d09ba-89d3-fcb7-cd32-fe0fcca4d1cd +31:25.692 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.716 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:25.744 [INFO] TestAgent: Synced node info +31:25.744 [DEBUG] TestAgent: Node info in sync +31:25.748 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=201.064µs client=127.0.0.1:40709 client_network=udp +31:25.752 [DEBUG] TestAgent.dns: request served from client: name=test.query.consul. type=A class=IN latency=180.585µs client=127.0.0.1:42903 client_network=udp +31:25.753 [DEBUG] TestAgent.dns: request served from client: name=test.query.consul. type=A class=IN latency=89.75µs client=127.0.0.1:35333 client_network=udp +31:25.753 [INFO] TestAgent: Requesting shutdown +31:25.753 [INFO] TestAgent.server: shutting down server +31:25.753 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.753 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.753 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.753 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.753 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.753 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.753 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.753 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.753 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.753 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.759 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.770Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.770Z [INFO] agent.router.manager: shutting down +31:25.770 [INFO] TestAgent: consul server down +31:25.770 [INFO] TestAgent: shutdown complete +31:25.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30378 network=tcp +31:25.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30378 network=udp +31:25.770 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30379 network=tcp +31:26.272 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.272 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_EDNS0_ECS (0.92s) + --- PASS: TestDNS_EDNS0_ECS/global (0.01s) + --- PASS: TestDNS_EDNS0_ECS/query (0.00s) + --- PASS: TestDNS_EDNS0_ECS/query-subnet (0.00s) +=== CONT TestDNS_EmptyAltDomain +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:25.621Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.651Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.741 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d Address:127.0.0.1:30404}]" +31:25.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d.dc1 127.0.0.1 +31:25.742 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d 127.0.0.1 +2021-10-05T16:31:25.742Z [INFO] agent.router: Initializing LAN area manager +31:25.742 [DEBUG] TestAgent.dns: recursor enabled +31:25.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30399 network=udp +31:25.742 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30404 [Follower]" leader= +31:25.743 [INFO] TestAgent.server: Adding LAN server: server="Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d (Addr: tcp/127.0.0.1:30404) (DC: dc1)" +31:25.743 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d.dc1 area=wan +31:25.743 [DEBUG] TestAgent.dns: recursor enabled +31:25.743 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30399 network=tcp +31:25.745 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30400 network=tcp +31:25.745 [INFO] TestAgent: started state syncer +31:25.745 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30405 network=tcp +31:25.808 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.808 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30404 [Candidate]" term=2 +31:25.826 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.826 [DEBUG] TestAgent.server.raft: vote granted: from=8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d term=2 tally=1 +31:25.826 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.826 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30404 [Leader]" +31:25.826 [INFO] TestAgent.server: cluster leadership acquired +31:25.826 [INFO] TestAgent.server: New leader elected: payload=Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d +31:25.835 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30404 +31:25.866 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.925 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.925 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.925 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.925 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.925 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.925 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d +31:25.925 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8a93f5ba-15a1-dcc3-a1b3-a56d18fe9f6d +31:25.949 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.968 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=199.373µs +31:25.968 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=ANY class=IN latency=421.311µs client=127.0.0.1:58167 client_network=udp +31:25.968 [INFO] TestAgent: Requesting shutdown +31:25.968 [INFO] TestAgent.server: shutting down server +31:25.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.968 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.968 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.968 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.968 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.968 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.968 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.969 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:25.969 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.969 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.969 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.984 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.992Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.992Z [INFO] agent.router.manager: shutting down +31:25.992 [INFO] TestAgent: consul server down +31:25.992 [INFO] TestAgent: shutdown complete +31:25.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30399 network=tcp +31:25.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30399 network=udp +31:25.992 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30400 network=tcp +31:26.493 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.493 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_CNAME (0.91s) +=== CONT TestDNS_Over_TCP +2021-10-05T16:31:25.595Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.612Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.672 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1b577f89-087f-36f0-bc31-f012ecfb407c Address:127.0.0.1:30397}]" +31:25.672 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30397 [Follower]" leader= +31:25.673 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1b577f89-087f-36f0-bc31-f012ecfb407c.dc1 127.0.0.1 +31:25.674 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1b577f89-087f-36f0-bc31-f012ecfb407c 127.0.0.1 +2021-10-05T16:31:25.674Z [INFO] agent.router: Initializing LAN area manager +31:25.674 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30392 network=udp +31:25.674 [INFO] TestAgent.server: Adding LAN server: server="Node-1b577f89-087f-36f0-bc31-f012ecfb407c (Addr: tcp/127.0.0.1:30397) (DC: dc1)" +31:25.674 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1b577f89-087f-36f0-bc31-f012ecfb407c.dc1 area=wan +31:25.674 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30392 network=tcp +31:25.675 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30393 network=tcp +31:25.675 [INFO] TestAgent: started state syncer +31:25.675 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30398 network=tcp +31:25.730 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.730 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30397 [Candidate]" term=2 +31:25.758 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.758 [DEBUG] TestAgent.server.raft: vote granted: from=1b577f89-087f-36f0-bc31-f012ecfb407c term=2 tally=1 +31:25.758 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.758 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30397 [Leader]" +31:25.760 [INFO] TestAgent.server: cluster leadership acquired +31:25.760 [INFO] TestAgent.server: New leader elected: payload=Node-1b577f89-087f-36f0-bc31-f012ecfb407c +31:25.770 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30397 +31:25.791 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.844 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.844 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.844 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.844 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.844 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.844 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1b577f89-087f-36f0-bc31-f012ecfb407c +31:25.844 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1b577f89-087f-36f0-bc31-f012ecfb407c +31:25.884 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.937 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:25.952 [INFO] TestAgent: Synced node info +31:26.004 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=152.114µs client=127.0.0.1:60407 client_network=udp +31:26.008 [INFO] TestAgent: Requesting shutdown +31:26.008 [INFO] TestAgent.server: shutting down server +31:26.008 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.008 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.008 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.008 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.008 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.008 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.008 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.008 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.008 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.008 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.024Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.024Z [INFO] agent.router.manager: shutting down +31:26.024 [INFO] TestAgent: consul server down +31:26.024 [INFO] TestAgent: shutdown complete +31:26.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30392 network=tcp +31:26.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30392 network=udp +31:26.024 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30393 network=tcp +31:26.524 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.524 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_EDNS0 (0.95s) +=== CONT TestRecursorAddr +--- PASS: TestRecursorAddr (0.00s) === CONT TestDiscoveryChainRead -2022-11-07T22:46:12.043Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.054Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.094 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88817ca3-8659-cb43-006e-6cba1ed91ae6 Address:127.0.0.1:11775}]" -46:12.094 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11775 [Follower]" leader= -46:12.094 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88817ca3-8659-cb43-006e-6cba1ed91ae6.dc1 127.0.0.1 -46:12.095 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88817ca3-8659-cb43-006e-6cba1ed91ae6 127.0.0.1 -2022-11-07T22:46:12.095Z [INFO] agent.router: Initializing LAN area manager -46:12.095 [INFO] TestAgent.server: Adding LAN server: server="Node-88817ca3-8659-cb43-006e-6cba1ed91ae6 (Addr: tcp/127.0.0.1:11775) (DC: dc1)" -46:12.095 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88817ca3-8659-cb43-006e-6cba1ed91ae6.dc1 area=wan -46:12.095 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11812 network=udp -46:12.095 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11812 network=tcp -46:12.096 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11799 network=tcp -46:12.096 [INFO] TestAgent: started state syncer -46:12.096 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11776 network=tcp -46:12.156 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.156 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11775 [Candidate]" term=2 -46:12.165 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.165 [DEBUG] TestAgent.server.raft: vote granted: from=88817ca3-8659-cb43-006e-6cba1ed91ae6 term=2 tally=1 -46:12.165 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.165 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11775 [Leader]" -46:12.165 [INFO] TestAgent.server: cluster leadership acquired -46:12.166 [INFO] TestAgent.server: New leader elected: payload=Node-88817ca3-8659-cb43-006e-6cba1ed91ae6 -46:12.173 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11775 -46:12.187 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.193 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:12.193 [INFO] TestAgent: Requesting shutdown -46:12.194 [INFO] TestAgent.server: shutting down server -46:12.194 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.194 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.197 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.201Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.201Z [INFO] agent.router.manager: shutting down -46:12.208 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" -46:12.208 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" -46:12.208 [INFO] TestAgent: consul server down -46:12.208 [INFO] TestAgent: shutdown complete -46:12.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11812 network=tcp -46:12.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11812 network=udp -46:12.208 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11799 network=tcp -46:12.708 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.708 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#04 -2022-11-07T22:46:10.130Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.144Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.178 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4824cc11-3525-4c22-0c76-25bd3f7903f4 Address:127.0.0.1:11544}]" -46:10.178 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11544 [Follower]" leader= -46:10.179 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4824cc11-3525-4c22-0c76-25bd3f7903f4.dc1 127.0.0.1 -46:10.179 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4824cc11-3525-4c22-0c76-25bd3f7903f4 127.0.0.1 -2022-11-07T22:46:10.179Z [INFO] agent.router: Initializing LAN area manager -46:10.180 [INFO] TestAgent.server: Adding LAN server: server="Node-4824cc11-3525-4c22-0c76-25bd3f7903f4 (Addr: tcp/127.0.0.1:11544) (DC: dc1)" -46:10.180 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4824cc11-3525-4c22-0c76-25bd3f7903f4.dc1 area=wan -46:10.180 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11553 network=udp -46:10.180 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11553 network=tcp -46:10.180 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11554 network=tcp -46:10.180 [INFO] TestAgent: started state syncer -46:10.180 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11545 network=tcp -46:10.235 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.235 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11544 [Candidate]" term=2 -46:10.247 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.247 [DEBUG] TestAgent.server.raft: vote granted: from=4824cc11-3525-4c22-0c76-25bd3f7903f4 term=2 tally=1 -46:10.247 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.247 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11544 [Leader]" -46:10.247 [INFO] TestAgent.server: cluster leadership acquired -46:10.247 [INFO] TestAgent.server: New leader elected: payload=Node-4824cc11-3525-4c22-0c76-25bd3f7903f4 -46:10.254 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11544 -46:10.269 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.297 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.297 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.297 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.297 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.297 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.297 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4824cc11-3525-4c22-0c76-25bd3f7903f4 -46:10.297 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4824cc11-3525-4c22-0c76-25bd3f7903f4 -46:10.304 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.518 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.521 [INFO] TestAgent: Synced node info -46:12.273 [INFO] TestAgent: Requesting shutdown -46:12.273 [INFO] TestAgent.server: shutting down server -46:12.273 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.273 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.273 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.273 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.273 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.273 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.273 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.273 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.273 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.273 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.278 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.284Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.284Z [INFO] agent.router.manager: shutting down -46:12.284 [INFO] TestAgent: consul server down -46:12.284 [INFO] TestAgent: shutdown complete -46:12.284 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11553 network=tcp -46:12.284 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11553 network=udp -46:12.284 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11554 network=tcp -46:12.784 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.784 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_ServerHealth_Unhealthy (2.67s) -=== CONT TestDNS_ServiceLookupPreferNoCNAME -2022-11-07T22:46:10.134Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:10.143Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:10.179 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:39d8ace8-7112-80f7-5561-ce2f808ae454 Address:127.0.0.1:11593}]" -46:10.179 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11593 [Follower]" leader= -46:10.180 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-39d8ace8-7112-80f7-5561-ce2f808ae454.dc1 127.0.0.1 -46:10.180 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-39d8ace8-7112-80f7-5561-ce2f808ae454 127.0.0.1 -2022-11-07T22:46:10.180Z [INFO] agent.router: Initializing LAN area manager -46:10.180 [INFO] TestAgent.server: Adding LAN server: server="Node-39d8ace8-7112-80f7-5561-ce2f808ae454 (Addr: tcp/127.0.0.1:11593) (DC: dc1)" -46:10.180 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-39d8ace8-7112-80f7-5561-ce2f808ae454.dc1 area=wan -46:10.181 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11546 network=udp -46:10.181 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11546 network=tcp -46:10.181 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11547 network=tcp -46:10.181 [INFO] TestAgent: started state syncer -46:10.181 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11594 network=tcp -46:10.240 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:10.240 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11593 [Candidate]" term=2 -46:10.251 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:10.251 [DEBUG] TestAgent.server.raft: vote granted: from=39d8ace8-7112-80f7-5561-ce2f808ae454 term=2 tally=1 -46:10.251 [INFO] TestAgent.server.raft: election won: tally=1 -46:10.251 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11593 [Leader]" -46:10.251 [INFO] TestAgent.server: cluster leadership acquired -46:10.251 [INFO] TestAgent.server: New leader elected: payload=Node-39d8ace8-7112-80f7-5561-ce2f808ae454 -46:10.258 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11593 -46:10.273 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.303 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.303 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.303 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.303 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.303 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.303 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-39d8ace8-7112-80f7-5561-ce2f808ae454 -46:10.303 [INFO] TestAgent.server: member joined, marking health alive: member=Node-39d8ace8-7112-80f7-5561-ce2f808ae454 -46:10.319 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.497 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.504 [INFO] TestAgent: Synced node info -46:12.280 [INFO] TestAgent: Requesting shutdown -46:12.280 [INFO] TestAgent.server: shutting down server -46:12.280 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.280 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.280 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.280 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.280 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.280 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.280 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.280 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.280 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.280 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.284 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.291Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.291Z [INFO] agent.router.manager: shutting down -46:12.291 [INFO] TestAgent: consul server down -46:12.291 [INFO] TestAgent: shutdown complete -46:12.291 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11546 network=tcp -46:12.291 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11546 network=udp -46:12.291 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11547 network=tcp -46:12.791 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.791 [INFO] TestAgent: Endpoints down ---- PASS: TestOperator_ServerHealth (2.67s) -=== CONT TestCatalogRegister_checkRegistration -2022-11-07T22:46:11.885Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.895Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd1db4e8-c07d-f654-adff-4c4a77fe4d99 Address:127.0.0.1:11747}]" -46:11.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11747 [Follower]" leader= -46:11.930 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99.dc1 127.0.0.1 -46:11.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99 127.0.0.1 -2022-11-07T22:46:11.931Z [INFO] agent.router: Initializing LAN area manager -46:11.931 [INFO] TestAgent.server: Adding LAN server: server="Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99 (Addr: tcp/127.0.0.1:11747) (DC: dc1)" -46:11.931 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99.dc1 area=wan -46:11.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11763 network=udp -46:11.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11763 network=tcp -46:11.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11771 network=tcp -46:11.932 [INFO] TestAgent: started state syncer -46:11.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11748 network=tcp -46:11.995 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.995 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11747 [Candidate]" term=2 -46:12.010 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.010 [DEBUG] TestAgent.server.raft: vote granted: from=dd1db4e8-c07d-f654-adff-4c4a77fe4d99 term=2 tally=1 -46:12.010 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.010 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11747 [Leader]" -46:12.010 [INFO] TestAgent.server: cluster leadership acquired -46:12.011 [INFO] TestAgent.server: New leader elected: payload=Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99 -46:12.020 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11747 -46:12.036 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.076 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.076 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.076 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.076 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.076 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.076 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99 -46:12.076 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd1db4e8-c07d-f654-adff-4c4a77fe4d99 -46:12.093 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.104 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.112 [INFO] TestAgent: Synced node info -46:12.112 [DEBUG] TestAgent: Node info in sync -46:12.318 [INFO] TestAgent: Requesting shutdown -46:12.318 [INFO] TestAgent.server: shutting down server -46:12.318 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.318 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.318 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.318 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.318 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.318 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.318 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.318 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.318 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.318 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.322 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.326Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.326Z [INFO] agent.router.manager: shutting down -46:12.326 [INFO] TestAgent: consul server down -46:12.326 [INFO] TestAgent: shutdown complete -46:12.326 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11763 network=tcp -46:12.326 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11763 network=udp -46:12.326 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11771 network=tcp -46:12.827 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.827 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes_Filter (0.95s) +2021-10-05T16:31:25.628Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.647Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.732 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4364acbf-60dd-adbc-f993-f591ddf6c750 Address:127.0.0.1:30411}]" +31:25.732 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4364acbf-60dd-adbc-f993-f591ddf6c750.dc1 127.0.0.1 +31:25.733 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30411 [Follower]" leader= +31:25.733 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4364acbf-60dd-adbc-f993-f591ddf6c750 127.0.0.1 +2021-10-05T16:31:25.733Z [INFO] agent.router: Initializing LAN area manager +31:25.733 [DEBUG] TestAgent.dns: recursor enabled +31:25.733 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30406 network=udp +31:25.733 [INFO] TestAgent.server: Adding LAN server: server="Node-4364acbf-60dd-adbc-f993-f591ddf6c750 (Addr: tcp/127.0.0.1:30411) (DC: dc1)" +31:25.733 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4364acbf-60dd-adbc-f993-f591ddf6c750.dc1 area=wan +31:25.734 [DEBUG] TestAgent.dns: recursor enabled +31:25.734 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30406 network=tcp +31:25.734 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30407 network=tcp +31:25.734 [INFO] TestAgent: started state syncer +31:25.734 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30412 network=tcp +31:25.798 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.798 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30411 [Candidate]" term=2 +31:25.821 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.821 [DEBUG] TestAgent.server.raft: vote granted: from=4364acbf-60dd-adbc-f993-f591ddf6c750 term=2 tally=1 +31:25.822 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.822 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30411 [Leader]" +31:25.822 [INFO] TestAgent.server: cluster leadership acquired +31:25.822 [INFO] TestAgent.server: New leader elected: payload=Node-4364acbf-60dd-adbc-f993-f591ddf6c750 +31:25.833 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30411 +31:25.862 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.925 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.925 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.925 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.925 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.925 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.925 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4364acbf-60dd-adbc-f993-f591ddf6c750 +31:25.925 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4364acbf-60dd-adbc-f993-f591ddf6c750 +31:25.948 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:25.949 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.967 [INFO] TestAgent: Synced node info +31:25.967 [DEBUG] TestAgent: Node info in sync +31:26.032 [DEBUG] TestAgent.dns: recurse failed for question: question="{google.com. 1 1}" rtt=89.162µs recursor=127.0.0.1:46173 rcode=SERVFAIL +31:26.032 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{google.com. 1 1}" rtt=84.275µs recursor=127.0.0.1:43360 +31:26.033 [DEBUG] TestAgent.dns: request served from client: question="{google.com. 1 1}" network=udp latency=356.99µs client=127.0.0.1:35198 client_network=udp +31:26.033 [INFO] TestAgent: Requesting shutdown +31:26.033 [INFO] TestAgent.server: shutting down server +31:26.033 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.033 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.033 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.033 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.033 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.033 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.033 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.033 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.033 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.033 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.043 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.048Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.048Z [INFO] agent.router.manager: shutting down +31:26.048 [INFO] TestAgent: consul server down +31:26.048 [INFO] TestAgent: shutdown complete +31:26.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30406 network=tcp +31:26.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30406 network=udp +31:26.048 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30407 network=tcp +31:26.548 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.548 [INFO] TestAgent: Endpoints down +--- PASS: TestDNSCycleRecursorCheck (0.97s) === CONT TestDenylist === RUN TestDenylist/nothing_blocked_root === RUN TestDenylist/nothing_blocked_path @@ -37480,491 +30789,9 @@ --- PASS: TestDenylist/longer_subpath (0.00s) --- PASS: TestDenylist/partial_prefix (0.00s) --- PASS: TestDenylist/no_match (0.00s) -=== CONT TestDNS_ServiceLookupNoMultiCNAME -2022-11-07T22:46:11.943Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.953Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.985 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a49a9d7e-140f-c97f-83e0-06aa54b82933 Address:127.0.0.1:11810}]" -46:11.985 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11810 [Follower]" leader= -46:11.985 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a49a9d7e-140f-c97f-83e0-06aa54b82933.dc1 127.0.0.1 -46:11.986 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a49a9d7e-140f-c97f-83e0-06aa54b82933 127.0.0.1 -2022-11-07T22:46:11.986Z [INFO] agent.router: Initializing LAN area manager -46:11.986 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a49a9d7e-140f-c97f-83e0-06aa54b82933.dc1 area=wan -46:11.986 [INFO] TestAgent.server: Adding LAN server: server="Node-a49a9d7e-140f-c97f-83e0-06aa54b82933 (Addr: tcp/127.0.0.1:11810) (DC: dc1)" -46:11.986 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11798 network=tcp -46:11.986 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11798 network=udp -46:11.986 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11785 network=tcp -46:11.986 [INFO] TestAgent: started state syncer -46:11.986 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11811 network=tcp -46:12.041 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.041 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11810 [Candidate]" term=2 -46:12.061 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.061 [DEBUG] TestAgent.server.raft: vote granted: from=a49a9d7e-140f-c97f-83e0-06aa54b82933 term=2 tally=1 -46:12.061 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.061 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11810 [Leader]" -46:12.061 [INFO] TestAgent.server: cluster leadership acquired -46:12.061 [INFO] TestAgent.server: New leader elected: payload=Node-a49a9d7e-140f-c97f-83e0-06aa54b82933 -46:12.069 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11810 -46:12.088 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.117 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.117 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.117 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.117 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.117 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.117 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a49a9d7e-140f-c97f-83e0-06aa54b82933 -46:12.117 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a49a9d7e-140f-c97f-83e0-06aa54b82933 -46:12.130 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.308 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.312 [INFO] TestAgent: Synced node info -46:12.331 [DEBUG] TestAgent.dns: request served from client: name=db.connect.consul. type=SRV class=IN latency=134.463µs client=127.0.0.1:50620 client_network=udp -46:12.331 [INFO] TestAgent: Requesting shutdown -46:12.331 [INFO] TestAgent.server: shutting down server -46:12.331 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.331 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.331 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.331 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.331 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.331 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.331 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.331 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.331 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.331 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.334 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.337Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.337Z [INFO] agent.router.manager: shutting down -46:12.337 [INFO] TestAgent: consul server down -46:12.337 [INFO] TestAgent: shutdown complete -46:12.337 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11798 network=tcp -46:12.337 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11798 network=udp -46:12.337 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11785 network=tcp -46:12.838 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.838 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ConnectServiceLookup (0.91s) -=== CONT TestCatalogServices_NodeMetaFilter -[WARN] freeport: 10 out of 73 pending ports are still in use; something probably didn't wait around for the port to be closed! -46:12.397 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-cfbf8e8b-2bd9-61c2-bfbc-3b1fd01384c9 127.0.0.1 -2022-11-07T22:46:12.397Z [INFO] agent.router: Initializing LAN area manager -46:12.397 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e: 127.0.0.1:11277 -46:12.397 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -46:12.397 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11277: connect: connection refused -46:12.397 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8: 127.0.0.1:11284 -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11284: connect: connection refused -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -46:12.397 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -46:12.397 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11851 network=udp -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -46:12.397 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -46:12.397 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11851 network=tcp -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.397 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.398Z [WARN] agent.router.manager: No servers available -46:12.398 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:12.398 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:12.398 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -2022-11-07T22:46:12.397Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.398Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -46:12.399 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11852 network=tcp -2022-11-07T22:46:12.399Z [WARN] agent.router.manager: No servers available -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:12.399 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:12.400 [INFO] TestAgent: started state syncer -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11818 network=tcp -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:12.400Z [WARN] agent.router.manager: No servers available -46:12.400 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:12.400 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy -46:12.400 [INFO] TestAgent: Requesting shutdown -46:12.400 [INFO] TestAgent.client: shutting down client -46:12.400 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:12.405 [INFO] TestAgent: consul client down -46:12.405 [INFO] TestAgent: shutdown complete -46:12.405 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11851 network=tcp -46:12.405 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11851 network=udp -46:12.405 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11852 network=tcp -46:12.905 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.905 [INFO] TestAgent: Endpoints down -46:07.830 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 127.0.0.1 -2022-11-07T22:46:07.830Z [INFO] agent.router: Initializing LAN area manager -46:07.830 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -46:07.830 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11246 network=udp -46:07.830 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11246 network=tcp -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:07.830 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.830Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -46:07.831 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11247 network=tcp -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -46:07.831 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:07.831 [INFO] TestAgent: started state syncer -46:07.831 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11287 network=tcp -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -46:07.831 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:07.831Z [WARN] agent.router.manager: No servers available -46:07.831 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:07.831 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11277] -46:07.831 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11277 -46:07.832 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e 127.0.0.1 -46:07.832 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:07.832 [DEBUG] TestAgent: systemd notify failed: error="No socket" -2022-11-07T22:46:07.832Z [WARN] agent.router.manager: No servers available -46:07.832 [INFO] TestAgent.client: adding server: server="Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e (Addr: tcp/127.0.0.1:11279) (DC: dc1)" -46:08.007 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:08.208 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:08.208 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.298 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.301 [INFO] TestAgent: Synced node info -46:08.311 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -46:08.311 [DEBUG] TestAgent: Node info in sync -46:08.311 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy -46:08.408 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:08.408 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.608 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:08.608 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.608 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:11.324 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:11.324 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:11.324 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:11.324 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" -46:11.327 [DEBUG] TestAgent: Node info in sync -46:11.332 [INFO] TestAgent: Synced service: service=web-sidecar-proxy -46:11.343 [INFO] TestAgent: Requesting shutdown -46:11.343 [INFO] TestAgent.client: shutting down client -46:11.343 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:11.353 [INFO] TestAgent: consul client down -46:11.353 [INFO] TestAgent: shutdown complete -46:11.353 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11246 network=tcp -46:11.353 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11246 network=udp -46:11.353 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11247 network=tcp -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=Intention.Match server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=DiscoveryChain.Get server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=DiscoveryChain.Get server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=ConnectCA.Roots server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=Health.ServiceNodes server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=ConnectCA.Roots server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.353 [ERROR] TestAgent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=127.0.0.1:11279 error="rpc error making call: EOF" -46:11.854 [INFO] TestAgent: Waiting for endpoints to shut down -46:11.854 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:07.556Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:07.567Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:07.605 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5edc8725-ec3f-d34f-badb-466dd8f33b1e Address:127.0.0.1:11279}]" -46:07.605 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11279 [Follower]" leader= -46:07.606 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e.dc1 127.0.0.1 -46:07.607 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e 127.0.0.1 -2022-11-07T22:46:07.607Z [INFO] agent.router: Initializing LAN area manager -46:07.607 [INFO] TestAgent.server: Adding LAN server: server="Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e (Addr: tcp/127.0.0.1:11279) (DC: dc1)" -46:07.607 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e.dc1 area=wan -46:07.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11204 network=udp -46:07.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11204 network=tcp -46:07.608 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11205 network=tcp -46:07.608 [INFO] TestAgent: started state syncer -46:07.608 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11280 network=tcp -46:07.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:07.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11279 [Candidate]" term=2 -46:07.679 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:07.679 [DEBUG] TestAgent.server.raft: vote granted: from=5edc8725-ec3f-d34f-badb-466dd8f33b1e term=2 tally=1 -46:07.679 [INFO] TestAgent.server.raft: election won: tally=1 -46:07.679 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11279 [Leader]" -46:07.679 [INFO] TestAgent.server: cluster leadership acquired -46:07.680 [INFO] TestAgent.server: New leader elected: payload=Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e -46:07.688 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11279 -46:07.708 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:07.751 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:07.751 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:07.751 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:07.751 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:07.751 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:07.751 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e -46:07.751 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5edc8725-ec3f-d34f-badb-466dd8f33b1e -46:07.771 [INFO] TestAgent.server: federation state anti-entropy synced -46:07.831 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:47308 -46:07.832 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 127.0.0.1 -46:07.832 [INFO] TestAgent.server: member joined, marking health alive: member=Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.024 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:08.027 [INFO] TestAgent: Synced node info -46:08.027 [DEBUG] TestAgent: Node info in sync -46:08.030 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.231 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.430 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:08.608 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-57c6f61e-df43-58bf-3dfb-9b30f1eb42b8 -46:09.269 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.269 [DEBUG] TestAgent: Node info in sync -46:11.854 [INFO] TestAgent: Requesting shutdown -46:11.854 [INFO] TestAgent.server: shutting down server -46:11.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:11.854 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:11.854 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:11.854 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:11.854 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:11.854 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:11.854 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:11.854 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:11.854 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:11.854 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:11.858 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:11.865Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:11.865Z [INFO] agent.router.manager: shutting down -46:11.865 [INFO] TestAgent: consul server down -46:11.865 [INFO] TestAgent: shutdown complete -46:11.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11204 network=tcp -46:11.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11204 network=udp -46:11.865 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11205 network=tcp -46:12.366 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.366 [INFO] TestAgent: Endpoints down ---- PASS: TestServiceManager_PersistService_ConfigFiles (5.36s) -=== CONT TestCoordinate_Update_ACLDeny -2022-11-07T22:46:12.121Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.131Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.157 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e0f49f73-3174-e156-edce-91a237def17e Address:127.0.0.1:11712}]" -46:12.157 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11712 [Follower]" leader= -46:12.158 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e0f49f73-3174-e156-edce-91a237def17e.dc1 127.0.0.1 -46:12.158 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e0f49f73-3174-e156-edce-91a237def17e 127.0.0.1 -2022-11-07T22:46:12.158Z [INFO] agent.router: Initializing LAN area manager -46:12.158 [INFO] TestAgent.server: Adding LAN server: server="Node-e0f49f73-3174-e156-edce-91a237def17e (Addr: tcp/127.0.0.1:11712) (DC: dc1)" -46:12.158 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e0f49f73-3174-e156-edce-91a237def17e.dc1 area=wan -46:12.158 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11777 network=tcp -46:12.158 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11777 network=udp -46:12.159 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11778 network=tcp -46:12.159 [INFO] TestAgent: started state syncer -46:12.159 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11713 network=tcp -46:12.198 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.198 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11712 [Candidate]" term=2 -46:12.212 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.212 [DEBUG] TestAgent.server.raft: vote granted: from=e0f49f73-3174-e156-edce-91a237def17e term=2 tally=1 -46:12.212 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.212 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11712 [Leader]" -46:12.212 [INFO] TestAgent.server: cluster leadership acquired -46:12.212 [INFO] TestAgent.server: New leader elected: payload=Node-e0f49f73-3174-e156-edce-91a237def17e -46:12.218 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11712 -46:12.230 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.250 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.251 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.251 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.251 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.251 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.251 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e0f49f73-3174-e156-edce-91a237def17e -46:12.251 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e0f49f73-3174-e156-edce-91a237def17e -46:12.261 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.416 [INFO] TestAgent: Requesting shutdown -46:12.416 [INFO] TestAgent.server: shutting down server -46:12.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.416 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.422 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.426Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.426Z [INFO] agent.router.manager: shutting down -46:12.426 [INFO] TestAgent: consul server down -46:12.426 [INFO] TestAgent: shutdown complete -46:12.426 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11777 network=tcp -46:12.426 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11777 network=udp -46:12.426 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11778 network=tcp -46:12.927 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.927 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Get (1.61s) - --- PASS: TestPreparedQuery_Get/#00 (0.79s) - --- PASS: TestPreparedQuery_Get/#01 (0.82s) -=== CONT TestDNS_ServiceReverseLookupNodeAddress -2022-11-07T22:46:12.136Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.151Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.182 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0a5d94f9-868f-202c-15bb-022ec9cadabe Address:127.0.0.1:11828}]" -46:12.182 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11828 [Follower]" leader= -46:12.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0a5d94f9-868f-202c-15bb-022ec9cadabe.dc1 127.0.0.1 -46:12.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0a5d94f9-868f-202c-15bb-022ec9cadabe 127.0.0.1 -2022-11-07T22:46:12.184Z [INFO] agent.router: Initializing LAN area manager -46:12.184 [INFO] TestAgent.server: Adding LAN server: server="Node-0a5d94f9-868f-202c-15bb-022ec9cadabe (Addr: tcp/127.0.0.1:11828) (DC: dc1)" -46:12.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0a5d94f9-868f-202c-15bb-022ec9cadabe.dc1 area=wan -46:12.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11714 network=udp -46:12.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11714 network=tcp -46:12.185 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11722 network=tcp -46:12.185 [INFO] TestAgent: started state syncer -46:12.185 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11829 network=tcp -46:12.240 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.240 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11828 [Candidate]" term=2 -46:12.250 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.250 [DEBUG] TestAgent.server.raft: vote granted: from=0a5d94f9-868f-202c-15bb-022ec9cadabe term=2 tally=1 -46:12.250 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.250 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11828 [Leader]" -46:12.250 [INFO] TestAgent.server: cluster leadership acquired -46:12.250 [INFO] TestAgent.server: New leader elected: payload=Node-0a5d94f9-868f-202c-15bb-022ec9cadabe -46:12.268 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11828 -46:12.283 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.304 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.305 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.305 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.305 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.305 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.305 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0a5d94f9-868f-202c-15bb-022ec9cadabe -46:12.305 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0a5d94f9-868f-202c-15bb-022ec9cadabe -46:12.313 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.393 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.399 [INFO] TestAgent: Synced node info -46:12.550 [INFO] TestAgent: Requesting shutdown -46:12.550 [INFO] TestAgent.server: shutting down server -46:12.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.550 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.550 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.550 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.550 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.550 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.550 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.556 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.560Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.560Z [INFO] agent.router.manager: shutting down -46:12.560 [INFO] TestAgent: consul server down -46:12.560 [INFO] TestAgent: shutdown complete -46:12.560 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11714 network=tcp -46:12.560 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11714 network=udp -46:12.560 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11722 network=tcp -46:13.060 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.060 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes_NodeMetaFilter (0.94s) -=== CONT TestCatalogServices +=== CONT TestCoordinate_Datacenters +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDiscoveryChainRead/GET:_error_on_no_service_name === RUN TestDiscoveryChainRead/GET:_read_default_chain === RUN TestDiscoveryChainRead/GET:_read_default_chain;_evaluate_in_dc2 @@ -37976,733 +30803,418 @@ === RUN TestDiscoveryChainRead/GET:_read_modified_chain === RUN TestDiscoveryChainRead/POST:_read_modified_chain_with_overrides_(camel_case) === RUN TestDiscoveryChainRead/POST:_read_modified_chain_with_overrides_(snake_case) -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:11.889Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.900Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.938 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1dab7489-c8d8-f061-693e-94f8b375b900 Address:127.0.0.1:11705}]" -46:11.938 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11705 [Follower]" leader= -46:11.938 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1dab7489-c8d8-f061-693e-94f8b375b900.dc1 127.0.0.1 -46:11.939 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1dab7489-c8d8-f061-693e-94f8b375b900 127.0.0.1 -2022-11-07T22:46:11.939Z [INFO] agent.router: Initializing LAN area manager -46:11.939 [INFO] TestAgent.server: Adding LAN server: server="Node-1dab7489-c8d8-f061-693e-94f8b375b900 (Addr: tcp/127.0.0.1:11705) (DC: dc1)" -46:11.939 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1dab7489-c8d8-f061-693e-94f8b375b900.dc1 area=wan -46:11.939 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11749 network=udp -46:11.939 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11749 network=tcp -46:11.940 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11764 network=tcp -46:11.940 [INFO] TestAgent: started state syncer -46:11.940 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11706 network=tcp -46:11.975 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.975 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11705 [Candidate]" term=2 -46:11.987 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.987 [DEBUG] TestAgent.server.raft: vote granted: from=1dab7489-c8d8-f061-693e-94f8b375b900 term=2 tally=1 -46:11.987 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.987 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11705 [Leader]" -46:11.987 [INFO] TestAgent.server: cluster leadership acquired -46:11.987 [INFO] TestAgent.server: New leader elected: payload=Node-1dab7489-c8d8-f061-693e-94f8b375b900 -46:11.994 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11705 -46:12.013 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.060 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.060 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.060 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.060 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.060 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.060 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1dab7489-c8d8-f061-693e-94f8b375b900 -46:12.060 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1dab7489-c8d8-f061-693e-94f8b375b900 -46:12.076 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.261 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.266 [INFO] TestAgent: Synced node info -46:12.266 [DEBUG] TestAgent: Node info in sync -46:12.284 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.284 [DEBUG] TestAgent: Node info in sync -46:12.572 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=4719032 -46:12.589 [INFO] TestAgent: Requesting shutdown -46:12.589 [INFO] TestAgent.server: shutting down server -46:12.589 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.589 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.589 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.589 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.589 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.589 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.589 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.589 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.589 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.589 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.594 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.597Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.597Z [INFO] agent.router.manager: shutting down -46:12.597 [INFO] TestAgent: consul server down -46:12.597 [INFO] TestAgent: shutdown complete -46:12.597 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11749 network=tcp -46:12.597 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11749 network=udp -46:12.597 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11764 network=tcp -46:13.097 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.097 [INFO] TestAgent: Endpoints down ---- PASS: TestTxnEndpoint_Bad_Size_Net (5.55s) - --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_default_limits (1.06s) - --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_configured_max_txn_len (0.93s) - --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_default_max_kv_value_size (1.07s) - --- PASS: TestTxnEndpoint_Bad_Size_Net/allowed (1.27s) - --- PASS: TestTxnEndpoint_Bad_Size_Net/allowed_kv_max_backward_compatible (1.22s) -=== CONT TestDNS_SOA_Settings -2022-11-07T22:46:12.237Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.251Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.295 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:99d16a19-dadd-6102-7dfc-cbb11ddc676c Address:127.0.0.1:11824}]" -46:12.295 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11824 [Follower]" leader= -46:12.295 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c.dc1 127.0.0.1 -46:12.296 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c 127.0.0.1 -2022-11-07T22:46:12.296Z [INFO] agent.router: Initializing LAN area manager -46:12.296 [INFO] TestAgent.server: Adding LAN server: server="Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c (Addr: tcp/127.0.0.1:11824) (DC: dc1)" -46:12.296 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11830 network=udp -46:12.296 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c.dc1 area=wan -46:12.296 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11830 network=tcp -46:12.296 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11831 network=tcp -46:12.296 [INFO] TestAgent: started state syncer -46:12.296 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11825 network=tcp -46:12.352 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.352 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11824 [Candidate]" term=2 -46:12.361 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.361 [DEBUG] TestAgent.server.raft: vote granted: from=99d16a19-dadd-6102-7dfc-cbb11ddc676c term=2 tally=1 -46:12.361 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.361 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11824 [Leader]" -46:12.361 [INFO] TestAgent.server: cluster leadership acquired -46:12.361 [INFO] TestAgent.server: New leader elected: payload=Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c -46:12.367 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11824 -46:12.378 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.397 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.397 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.397 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.397 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.397 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.397 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c -46:12.397 [INFO] TestAgent.server: member joined, marking health alive: member=Node-99d16a19-dadd-6102-7dfc-cbb11ddc676c -46:12.406 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.515 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.524 [INFO] TestAgent: Synced node info -46:12.524 [DEBUG] TestAgent: Node info in sync -46:12.619 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=94.698µs client=127.0.0.1:55236 client_network=tcp -46:12.619 [INFO] TestAgent: Requesting shutdown -46:12.619 [INFO] TestAgent.server: shutting down server -46:12.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.619 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.619 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.619 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.619 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.619 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.619 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.622 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.627Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.627Z [INFO] agent.router.manager: shutting down -46:12.627 [INFO] TestAgent: consul server down -46:12.627 [INFO] TestAgent: shutdown complete -46:12.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11830 network=tcp -46:12.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11830 network=udp -46:12.627 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11831 network=tcp -46:13.127 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.127 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_Over_TCP (0.90s) +2021-10-05T16:31:26.061Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.101Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.188 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:015245b4-1863-8473-d268-fd692495da0e Address:127.0.0.1:29405}]" +31:26.189 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-015245b4-1863-8473-d268-fd692495da0e.dc1 127.0.0.1 +31:26.189 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-015245b4-1863-8473-d268-fd692495da0e 127.0.0.1 +2021-10-05T16:31:26.189Z [INFO] agent.router: Initializing LAN area manager +31:26.189 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp +31:26.189 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= +31:26.190 [INFO] TestAgent.server: Adding LAN server: server="Node-015245b4-1863-8473-d268-fd692495da0e (Addr: tcp/127.0.0.1:29405) (DC: dc1)" +31:26.190 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-015245b4-1863-8473-d268-fd692495da0e.dc1 area=wan +31:26.190 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp +31:26.190 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp +31:26.190 [INFO] TestAgent: started state syncer +31:26.191 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp +31:26.244 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.244 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 +31:26.289 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.289 [DEBUG] TestAgent.server.raft: vote granted: from=015245b4-1863-8473-d268-fd692495da0e term=2 tally=1 +31:26.289 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.289 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" +31:26.292 [INFO] TestAgent.server: cluster leadership acquired +31:26.292 [INFO] TestAgent.server: New leader elected: payload=Node-015245b4-1863-8473-d268-fd692495da0e +31:26.336 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 +31:26.384 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.456 [INFO] TestAgent: Synced node info +31:26.456 [DEBUG] TestAgent: Node info in sync +31:26.477 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.477 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.477 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.477 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.477 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.477 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-015245b4-1863-8473-d268-fd692495da0e +31:26.477 [INFO] TestAgent.server: member joined, marking health alive: member=Node-015245b4-1863-8473-d268-fd692495da0e +31:26.520 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.520 [DEBUG] TestAgent.dns: request served from client: name=foo.bar.node.consul. type=ANY class=IN latency=78.847µs client=127.0.0.1:46519 client_network=udp +31:26.520 [INFO] TestAgent: Requesting shutdown +31:26.520 [INFO] TestAgent.server: shutting down server +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.520 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.520 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.528 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.533Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.533Z [INFO] agent.router.manager: shutting down +31:26.533 [INFO] TestAgent: consul server down +31:26.533 [INFO] TestAgent: shutdown complete +31:26.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp +31:26.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp +31:26.533 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp +31:27.036 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.036 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_PeriodName (0.99s) +=== CONT TestCoordinate_Update_ACLDeny +2021-10-05T16:31:25.975Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.994Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.076 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7951a49c-b685-fe8c-eb1b-59e199362228 Address:127.0.0.1:30418}]" +31:26.077 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7951a49c-b685-fe8c-eb1b-59e199362228.dc1 127.0.0.1 +31:26.077 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7951a49c-b685-fe8c-eb1b-59e199362228 127.0.0.1 +2021-10-05T16:31:26.077Z [INFO] agent.router: Initializing LAN area manager +31:26.077 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30413 network=udp +31:26.077 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30418 [Follower]" leader= +31:26.078 [INFO] TestAgent.server: Adding LAN server: server="Node-7951a49c-b685-fe8c-eb1b-59e199362228 (Addr: tcp/127.0.0.1:30418) (DC: dc1)" +31:26.078 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7951a49c-b685-fe8c-eb1b-59e199362228.dc1 area=wan +31:26.078 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30413 network=tcp +31:26.078 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30414 network=tcp +31:26.079 [INFO] TestAgent: started state syncer +31:26.079 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30419 network=tcp +31:26.143 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.143 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30418 [Candidate]" term=2 +31:26.155 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.155 [DEBUG] TestAgent.server.raft: vote granted: from=7951a49c-b685-fe8c-eb1b-59e199362228 term=2 tally=1 +31:26.155 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.155 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30418 [Leader]" +31:26.156 [INFO] TestAgent.server: cluster leadership acquired +31:26.156 [INFO] TestAgent.server: New leader elected: payload=Node-7951a49c-b685-fe8c-eb1b-59e199362228 +31:26.168 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30418 +31:26.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.312 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.312 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.312 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.312 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.312 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.312 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7951a49c-b685-fe8c-eb1b-59e199362228 +31:26.312 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7951a49c-b685-fe8c-eb1b-59e199362228 +31:26.353 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.428 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:26.438 [INFO] TestAgent: Synced node info +31:26.438 [DEBUG] TestAgent: Node info in sync +31:26.516 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=AAAA class=IN latency=124.203µs client=127.0.0.1:37440 client_network=udp +31:26.520 [INFO] TestAgent: Requesting shutdown +31:26.520 [INFO] TestAgent.server: shutting down server +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.520 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.520 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.520 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.520 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.532 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.544Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.544Z [INFO] agent.router.manager: shutting down +31:26.544 [INFO] TestAgent: consul server down +31:26.544 [INFO] TestAgent: shutdown complete +31:26.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30413 network=tcp +31:26.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30413 network=udp +31:26.544 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30414 network=tcp +31:27.044 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.044 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup_AAAA (1.09s) === CONT TestCoordinate_Update -2022-11-07T22:46:12.421Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.430Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.455 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f784623b-1ea9-97a5-8fad-c038b7dc5caa Address:127.0.0.1:11842}]" -46:12.455 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11842 [Follower]" leader= -46:12.455 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa.dc1 127.0.0.1 -46:12.456 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa 127.0.0.1 -2022-11-07T22:46:12.456Z [INFO] agent.router: Initializing LAN area manager -46:12.456 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11819 network=udp -46:12.456 [INFO] TestAgent.server: Adding LAN server: server="Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa (Addr: tcp/127.0.0.1:11842) (DC: dc1)" -46:12.457 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa.dc1 area=wan -46:12.457 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11819 network=tcp -46:12.457 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11836 network=tcp -46:12.458 [INFO] TestAgent: started state syncer -46:12.458 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11843 network=tcp -46:12.505 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.505 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11842 [Candidate]" term=2 -46:12.524 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.524 [DEBUG] TestAgent.server.raft: vote granted: from=f784623b-1ea9-97a5-8fad-c038b7dc5caa term=2 tally=1 -46:12.524 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.524 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11842 [Leader]" -46:12.525 [INFO] TestAgent.server: cluster leadership acquired -46:12.525 [INFO] TestAgent.server: New leader elected: payload=Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa -46:12.537 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11842 -46:12.553 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.575 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.575 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.575 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.575 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.575 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.575 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa -46:12.575 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f784623b-1ea9-97a5-8fad-c038b7dc5caa -46:12.583 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.640 [INFO] TestAgent: Requesting shutdown -46:12.640 [INFO] TestAgent.server: shutting down server -46:12.640 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.640 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.640 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.640 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.640 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.640 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.640 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.640 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.640 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.640 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.640 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.644 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.648Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.648Z [INFO] agent.router.manager: shutting down -46:12.648 [INFO] TestAgent: consul server down -46:12.648 [INFO] TestAgent: shutdown complete -46:12.648 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11819 network=tcp -46:12.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11819 network=udp -46:12.649 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11836 network=tcp -46:13.150 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.150 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Explain/#02 -2022-11-07T22:46:11.856Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.856Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.873Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.873Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.907 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:badda641-1bbe-98ed-a6dc-6fba1d7d97bc Address:127.0.0.1:11761}]" -46:11.907 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11761 [Follower]" leader= -46:11.907 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 127.0.0.1 -46:11.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc 127.0.0.1 -2022-11-07T22:46:11.908Z [INFO] agent.router: Initializing LAN area manager -46:11.908 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 area=wan -46:11.908 [INFO] TestAgent.server: Adding LAN server: server="Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc (Addr: tcp/127.0.0.1:11761) (DC: dc2)" -46:11.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11770 network=udp -46:11.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11770 network=tcp -46:11.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11757 network=tcp -46:11.909 [INFO] TestAgent: started state syncer -46:11.909 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11762 network=tcp -46:11.949 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.949 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11761 [Candidate]" term=2 -46:11.958 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:11.964 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.964 [DEBUG] TestAgent.server.raft: vote granted: from=badda641-1bbe-98ed-a6dc-6fba1d7d97bc term=2 tally=1 -46:11.964 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.964 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11761 [Leader]" -46:11.964 [INFO] TestAgent.server: cluster leadership acquired -46:11.964 [INFO] TestAgent.server: New leader elected: payload=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc -46:11.971 [INFO] TestAgent.server: initializing acls -46:11.976 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:11.982 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:11.982 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:11.982 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:11.982 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc -46:11.982 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 -46:11.983 [INFO] TestAgent.server: Updating LAN server: server="Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc (Addr: tcp/127.0.0.1:11761) (DC: dc2)" -46:11.983 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 area=wan -46:12.005 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.023 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.023 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.023 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.023 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.024 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.024 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc -46:12.024 [INFO] TestAgent.server: member joined, marking health alive: member=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc -46:12.033 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.043 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc -46:12.178 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.183 [INFO] TestAgent: Synced node info -46:12.183 [DEBUG] TestAgent: Node info in sync -46:12.201 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11753] -46:12.201 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11753 -46:12.202 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 127.0.0.1 -46:12.202 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:12.202 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 area=wan -46:12.207 [INFO] TestAgent: Requesting shutdown -46:12.208 [INFO] TestAgent.server: shutting down server -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.208 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.208 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:12.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.212 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.218Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.218Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.218Z [INFO] agent.router.manager: shutting down -46:12.218 [INFO] TestAgent: consul server down -46:12.218 [INFO] TestAgent: shutdown complete -46:12.218 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11770 network=tcp -46:12.218 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11770 network=udp -46:12.218 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11757 network=tcp -46:12.722 [INFO] TestAgent: Waiting for endpoints to shut down -46:12.722 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:11.687Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.687Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:11.696Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:11.696Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:11.733 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 Address:127.0.0.1:11754}]" -46:11.733 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11754 [Follower]" leader= -46:11.733 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 127.0.0.1 -46:11.734 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 127.0.0.1 -2022-11-07T22:46:11.734Z [INFO] agent.router: Initializing LAN area manager -46:11.734 [INFO] TestAgent.server: Adding LAN server: server="Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 (Addr: tcp/127.0.0.1:11754) (DC: dc1)" -46:11.734 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 area=wan -46:11.734 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11721 network=tcp -46:11.734 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11721 network=udp -46:11.735 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11736 network=tcp -46:11.735 [INFO] TestAgent: started state syncer -46:11.735 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11755 network=tcp -46:11.784 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:11.796 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:11.796 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11754 [Candidate]" term=2 -46:11.806 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:11.806 [DEBUG] TestAgent.server.raft: vote granted: from=8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 term=2 tally=1 -46:11.806 [INFO] TestAgent.server.raft: election won: tally=1 -46:11.806 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11754 [Leader]" -46:11.806 [INFO] TestAgent.server: cluster leadership acquired -46:11.806 [INFO] TestAgent.server: New leader elected: payload=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 -46:11.812 [INFO] TestAgent.server: initializing acls -46:11.815 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:11.826 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:11.826 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:11.826 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:11.826 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 -46:11.826 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 -46:11.826 [INFO] TestAgent.server: Updating LAN server: server="Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 (Addr: tcp/127.0.0.1:11754) (DC: dc1)" -46:11.826 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5.dc1 area=wan -46:11.849 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:11.873 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:11.873 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:11.873 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:11.873 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:11.873 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:11.873 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 -46:11.873 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 -46:11.882 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f6c1540-7a79-f6f0-1e52-ae054fd9eca5 -46:11.888 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.102 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.112 [INFO] TestAgent: Synced node info -46:12.112 [DEBUG] TestAgent: Node info in sync -46:12.201 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:59762 -46:12.201 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 127.0.0.1 -46:12.201 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-badda641-1bbe-98ed-a6dc-6fba1d7d97bc.dc2 area=wan -46:12.722 [INFO] TestAgent: Requesting shutdown -46:12.722 [INFO] TestAgent.server: shutting down server -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.722 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.722 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.722 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.722 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.726 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.729Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.729Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.729Z [INFO] agent.router.manager: shutting down -46:12.729 [INFO] TestAgent: consul server down -46:12.729 [INFO] TestAgent: shutdown complete -46:12.729 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11721 network=tcp -46:12.729 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11721 network=udp -46:12.729 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11736 network=tcp -46:13.231 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.231 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes_WanTranslation (1.56s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:26.316Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.363Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.412 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67f1f7ff-0659-6224-52fa-90a1bed50a54 Address:127.0.0.1:29447}]" +31:26.413 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67f1f7ff-0659-6224-52fa-90a1bed50a54.dc1 127.0.0.1 +31:26.413 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67f1f7ff-0659-6224-52fa-90a1bed50a54 127.0.0.1 +2021-10-05T16:31:26.413Z [INFO] agent.router: Initializing LAN area manager +31:26.414 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29477 network=udp +31:26.414 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29447 [Follower]" leader= +31:26.414 [INFO] TestAgent.server: Adding LAN server: server="Node-67f1f7ff-0659-6224-52fa-90a1bed50a54 (Addr: tcp/127.0.0.1:29447) (DC: dc1)" +31:26.414 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67f1f7ff-0659-6224-52fa-90a1bed50a54.dc1 area=wan +31:26.414 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29477 network=tcp +31:26.415 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29478 network=tcp +31:26.415 [INFO] TestAgent: started state syncer +31:26.415 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29448 network=tcp +31:26.472 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.472 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29447 [Candidate]" term=2 +31:26.490 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.490 [DEBUG] TestAgent.server.raft: vote granted: from=67f1f7ff-0659-6224-52fa-90a1bed50a54 term=2 tally=1 +31:26.490 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.490 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29447 [Leader]" +31:26.490 [INFO] TestAgent.server: cluster leadership acquired +31:26.491 [INFO] TestAgent.server: New leader elected: payload=Node-67f1f7ff-0659-6224-52fa-90a1bed50a54 +31:26.501 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29447 +31:26.532 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.590 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.590 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.590 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.590 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.590 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.590 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67f1f7ff-0659-6224-52fa-90a1bed50a54 +31:26.590 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67f1f7ff-0659-6224-52fa-90a1bed50a54 +31:26.624 [INFO] TestAgent: Synced node info +31:26.631 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.641 [INFO] TestAgent: Requesting shutdown +31:26.641 [INFO] TestAgent.server: shutting down server +31:26.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.641 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.641 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.641 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.641 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.641 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.641 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.651 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.656Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.656Z [INFO] agent.router.manager: shutting down +31:26.656 [INFO] TestAgent: consul server down +31:26.656 [INFO] TestAgent: shutdown complete +31:26.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29477 network=tcp +31:26.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29477 network=udp +31:26.656 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29478 network=tcp +31:27.157 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.157 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_EmptyAltDomain (0.89s) === CONT TestCoordinate_Node -=== RUN TestCoordinate_Update_ACLDeny/no_token -=== RUN TestCoordinate_Update_ACLDeny/valid_token -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:12.379Z [WARN] agent: Node name "my.test-node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:12.379Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.394Z [WARN] agent.auto_config: Node name "my.test-node" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:12.394Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.425 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:027394c2-297e-7e52-f866-01dcdc44c999 Address:127.0.0.1:11849}]" -46:12.425 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11849 [Follower]" leader= -46:12.426 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: my.test-node.dc1 127.0.0.1 -46:12.426 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: my.test-node 127.0.0.1 -2022-11-07T22:46:12.426Z [INFO] agent.router: Initializing LAN area manager -46:12.426 [INFO] TestAgent.server: Adding LAN server: server="my.test-node (Addr: tcp/127.0.0.1:11849) (DC: dc1)" -46:12.426 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=my.test-node.dc1 area=wan -46:12.426 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11826 network=udp -46:12.426 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11826 network=tcp -46:12.427 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11813 network=tcp -46:12.427 [INFO] TestAgent: started state syncer -46:12.427 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11850 network=tcp -46:12.491 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.491 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11849 [Candidate]" term=2 -46:12.500 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.500 [DEBUG] TestAgent.server.raft: vote granted: from=027394c2-297e-7e52-f866-01dcdc44c999 term=2 tally=1 -46:12.500 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.500 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11849 [Leader]" -46:12.500 [INFO] TestAgent.server: cluster leadership acquired -46:12.500 [INFO] TestAgent.server: New leader elected: payload=my.test-node -46:12.511 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11849 -46:12.537 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.558 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.558 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.558 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.558 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.558 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.558 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=my.test-node -46:12.558 [INFO] TestAgent.server: member joined, marking health alive: member=my.test-node -46:12.570 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.597 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.600 [INFO] TestAgent: Synced node info -46:12.600 [DEBUG] TestAgent: Node info in sync -46:12.824 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=162.253µs client=127.0.0.1:60446 client_network=udp -46:12.824 [INFO] TestAgent: Requesting shutdown -46:12.824 [INFO] TestAgent.server: shutting down server -46:12.824 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.824 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.824 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.824 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.824 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.824 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.824 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.824 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.824 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.824 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.833 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.841Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.841Z [INFO] agent.router.manager: shutting down -46:12.841 [INFO] TestAgent: consul server down -46:12.841 [INFO] TestAgent: shutdown complete -46:12.841 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11826 network=tcp -46:12.841 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11826 network=udp -46:12.841 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11813 network=tcp -46:13.341 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.341 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookupWithInternalServiceAddress (0.98s) -=== CONT TestCatalogNodes_DistanceSort -2022-11-07T22:46:12.721Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.734Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.770 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f Address:127.0.0.1:11905}]" -46:12.770 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11905 [Follower]" leader= -46:12.771 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f.dc1 127.0.0.1 -46:12.771 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f 127.0.0.1 -2022-11-07T22:46:12.771Z [INFO] agent.router: Initializing LAN area manager -46:12.771 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f.dc1 area=wan -46:12.771 [INFO] TestAgent.server: Adding LAN server: server="Node-f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f (Addr: tcp/127.0.0.1:11905) (DC: dc1)" -46:12.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11872 network=udp -46:12.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11872 network=tcp -46:12.772 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11873 network=tcp -46:12.772 [INFO] TestAgent: started state syncer -46:12.772 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11906 network=tcp -46:12.814 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.814 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11905 [Candidate]" term=2 -46:12.828 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.828 [DEBUG] TestAgent.server.raft: vote granted: from=f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f term=2 tally=1 -46:12.828 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.828 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11905 [Leader]" -46:12.829 [INFO] TestAgent.server: cluster leadership acquired -46:12.829 [INFO] TestAgent.server: New leader elected: payload=Node-f3bd0e90-b1fd-b0eb-08e5-4ddfa702aa7f -46:12.838 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11905 -46:12.849 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:12.850 [INFO] TestAgent: Requesting shutdown -46:12.850 [INFO] TestAgent.server: shutting down server -46:12.850 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.850 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.856 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:12.859 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:12.865Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.865Z [INFO] agent.router.manager: shutting down -46:12.876 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:12.876 [INFO] TestAgent: consul server down -46:12.876 [INFO] TestAgent: shutdown complete -46:12.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11872 network=tcp -46:12.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11872 network=udp -46:12.876 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11873 network=tcp -46:13.376 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.376 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:12.649Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.658Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.691 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c8402a72-2df4-d799-7fdd-93a1a631c7ec Address:127.0.0.1:11789}]" -46:12.691 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11789 [Follower]" leader= -46:12.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec.dc1 127.0.0.1 -46:12.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec 127.0.0.1 -2022-11-07T22:46:12.692Z [INFO] agent.router: Initializing LAN area manager -46:12.692 [INFO] TestAgent.server: Adding LAN server: server="Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec (Addr: tcp/127.0.0.1:11789) (DC: dc1)" -46:12.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec.dc1 area=wan -46:12.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11805 network=udp -46:12.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11805 network=tcp -46:12.693 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11820 network=tcp -46:12.693 [INFO] TestAgent: started state syncer -46:12.693 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11790 network=tcp -46:12.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11789 [Candidate]" term=2 -46:12.749 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.749 [DEBUG] TestAgent.server.raft: vote granted: from=c8402a72-2df4-d799-7fdd-93a1a631c7ec term=2 tally=1 -46:12.749 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.749 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11789 [Leader]" -46:12.749 [INFO] TestAgent.server: cluster leadership acquired -46:12.749 [INFO] TestAgent.server: New leader elected: payload=Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec -46:12.757 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11789 -46:12.777 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.818 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.818 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.818 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.818 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.818 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.818 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec -46:12.818 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c8402a72-2df4-d799-7fdd-93a1a631c7ec -46:12.840 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.862 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=SRV class=IN latency=156.59µs client=127.0.0.1:48295 client_network=udp -46:12.862 [DEBUG] TestAgent.dns: request served from client: name=0b74a4b4-052b-6157-394d-fe290b5b5349.query.consul. type=SRV class=IN latency=127.285µs client=127.0.0.1:47108 client_network=udp -46:12.863 [DEBUG] TestAgent.dns: request served from client: name=nodb.service.consul. type=SRV class=IN latency=96.981µs client=127.0.0.1:40274 client_network=udp -46:12.863 [DEBUG] TestAgent.dns: request served from client: name=nope.query.consul. type=SRV class=IN latency=67.05µs client=127.0.0.1:33798 client_network=udp -46:12.863 [INFO] TestAgent: Requesting shutdown -46:12.863 [INFO] TestAgent.server: shutting down server -46:12.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.863 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.863 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.863 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.863 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.863 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.863 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.863 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.863 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.863 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.863 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.869 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.876Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.876Z [INFO] agent.router.manager: shutting down -46:12.876 [INFO] TestAgent: consul server down -46:12.876 [INFO] TestAgent: shutdown complete -46:12.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11805 network=tcp -46:12.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11805 network=udp -46:12.876 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11820 network=tcp -46:13.376 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.376 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#05 -=== CONT TestDNS_ServiceReverseLookup_CustomDomain ---- PASS: TestDNS_ServiceLookup (0.74s) -2022-11-07T22:46:12.657Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.672Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.700 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a06037c4-57dc-611d-123b-00a7c5b367f3 Address:127.0.0.1:11782}]" -46:12.700 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11782 [Follower]" leader= -46:12.701 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a06037c4-57dc-611d-123b-00a7c5b367f3.dc1 127.0.0.1 -46:12.701 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a06037c4-57dc-611d-123b-00a7c5b367f3 127.0.0.1 -2022-11-07T22:46:12.701Z [INFO] agent.router: Initializing LAN area manager -46:12.702 [INFO] TestAgent.server: Adding LAN server: server="Node-a06037c4-57dc-611d-123b-00a7c5b367f3 (Addr: tcp/127.0.0.1:11782) (DC: dc1)" -46:12.702 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a06037c4-57dc-611d-123b-00a7c5b367f3.dc1 area=wan -46:12.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11791 network=udp -46:12.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11791 network=tcp -46:12.702 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11806 network=tcp -46:12.702 [INFO] TestAgent: started state syncer -46:12.702 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11783 network=tcp -46:12.745 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.745 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11782 [Candidate]" term=2 -46:12.762 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.762 [DEBUG] TestAgent.server.raft: vote granted: from=a06037c4-57dc-611d-123b-00a7c5b367f3 term=2 tally=1 -46:12.762 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.762 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11782 [Leader]" -46:12.762 [INFO] TestAgent.server: cluster leadership acquired -46:12.762 [INFO] TestAgent.server: New leader elected: payload=Node-a06037c4-57dc-611d-123b-00a7c5b367f3 -46:12.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11782 -46:12.798 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.834 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.834 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.834 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.834 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.834 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.834 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a06037c4-57dc-611d-123b-00a7c5b367f3 -46:12.834 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a06037c4-57dc-611d-123b-00a7c5b367f3 -46:12.856 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.866 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.871 [INFO] TestAgent: Synced node info -46:12.893 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=171.944µs client=127.0.0.1:33599 client_network=udp -46:12.894 [INFO] TestAgent: Requesting shutdown -46:12.894 [INFO] TestAgent.server: shutting down server -46:12.894 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.894 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.894 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.894 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.894 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.894 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.894 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.894 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.894 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.894 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.897 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.901Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.901Z [INFO] agent.router.manager: shutting down -46:12.901 [INFO] TestAgent: consul server down -46:12.901 [INFO] TestAgent: shutdown complete -46:12.901 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11791 network=tcp -46:12.901 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11791 network=udp -46:12.901 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11806 network=tcp -46:13.401 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.401 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookupMultiAddrNoCNAME (0.76s) -=== CONT TestCatalogNodes_Blocking -2022-11-07T22:46:12.645Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.654Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.687 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6cfb030c-59c8-9737-5f56-d337052a7bfb Address:127.0.0.1:11803}]" -46:12.687 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11803 [Follower]" leader= -46:12.688 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6cfb030c-59c8-9737-5f56-d337052a7bfb.dc1 127.0.0.1 -46:12.688 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6cfb030c-59c8-9737-5f56-d337052a7bfb 127.0.0.1 -2022-11-07T22:46:12.688Z [INFO] agent.router: Initializing LAN area manager -46:12.688 [INFO] TestAgent.server: Adding LAN server: server="Node-6cfb030c-59c8-9737-5f56-d337052a7bfb (Addr: tcp/127.0.0.1:11803) (DC: dc1)" -46:12.688 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6cfb030c-59c8-9737-5f56-d337052a7bfb.dc1 area=wan -46:12.688 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11844 network=udp -46:12.688 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11844 network=tcp -46:12.689 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11845 network=tcp -46:12.689 [INFO] TestAgent: started state syncer -46:12.689 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11804 network=tcp -46:12.740 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.740 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11803 [Candidate]" term=2 -46:12.756 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.756 [DEBUG] TestAgent.server.raft: vote granted: from=6cfb030c-59c8-9737-5f56-d337052a7bfb term=2 tally=1 -46:12.756 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.756 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11803 [Leader]" -46:12.756 [INFO] TestAgent.server: cluster leadership acquired -46:12.756 [INFO] TestAgent.server: New leader elected: payload=Node-6cfb030c-59c8-9737-5f56-d337052a7bfb -46:12.768 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11803 -46:12.792 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.825 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.825 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.825 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.825 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.825 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.825 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6cfb030c-59c8-9737-5f56-d337052a7bfb -46:12.825 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6cfb030c-59c8-9737-5f56-d337052a7bfb -46:12.838 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.920 [INFO] TestAgent: Requesting shutdown -46:12.920 [INFO] TestAgent.server: shutting down server -46:12.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:12.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:12.920 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:12.920 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:12.920 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.920 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:12.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:12.920 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:12.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:12.921 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:12.921 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:12.924 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:12.927Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:12.927Z [INFO] agent.router.manager: shutting down -46:12.927 [INFO] TestAgent: consul server down -46:12.927 [INFO] TestAgent: shutdown complete -46:12.927 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11844 network=tcp -46:12.927 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11844 network=udp -46:12.927 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11845 network=tcp -46:13.427 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.427 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServiceNodes (0.79s) +2021-10-05T16:31:26.191Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.245Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.322 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63653eb7-67ac-e1d7-4c56-5de0aa5f98dc Address:127.0.0.1:29412}]" +31:26.322 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= +31:26.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc.dc1 127.0.0.1 +31:26.336 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc 127.0.0.1 +2021-10-05T16:31:26.336Z [INFO] agent.router: Initializing LAN area manager +31:26.336 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp +31:26.341 [INFO] TestAgent.server: Adding LAN server: server="Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc (Addr: tcp/127.0.0.1:29412) (DC: dc1)" +31:26.341 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc.dc1 area=wan +31:26.342 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp +31:26.342 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp +31:26.342 [INFO] TestAgent: started state syncer +31:26.342 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp +31:26.382 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.382 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 +31:26.428 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.428 [DEBUG] TestAgent.server.raft: vote granted: from=63653eb7-67ac-e1d7-4c56-5de0aa5f98dc term=2 tally=1 +31:26.428 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" +31:26.429 [INFO] TestAgent.server: cluster leadership acquired +31:26.429 [INFO] TestAgent.server: New leader elected: payload=Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc +31:26.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 +31:26.468 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.497 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.497 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.497 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.497 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.497 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.497 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc +31:26.497 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63653eb7-67ac-e1d7-4c56-5de0aa5f98dc +31:26.515 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.608 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:26.615 [INFO] TestAgent: Synced node info +31:26.653 [DEBUG] TestAgent.dns: request served from client: name=fOO.node.dc1.consul. type=ANY class=IN latency=86.45µs client=127.0.0.1:58996 client_network=udp +31:26.653 [INFO] TestAgent: Requesting shutdown +31:26.653 [INFO] TestAgent.server: shutting down server +31:26.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.653 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.653 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.653 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.653 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.653 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.653 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.660 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.668Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.668Z [INFO] agent.router.manager: shutting down +31:26.668 [INFO] TestAgent: consul server down +31:26.668 [INFO] TestAgent: shutdown complete +31:26.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp +31:26.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp +31:26.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp +31:27.171 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.171 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_CaseInsensitiveNodeLookup (1.01s) === CONT TestCoordinate_Nodes -2022-11-07T22:46:12.806Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.820Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.866 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf04e2db-5541-ddff-a94a-7131ea2c341a Address:127.0.0.1:11933}]" -46:12.866 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11933 [Follower]" leader= -46:12.867 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf04e2db-5541-ddff-a94a-7131ea2c341a.dc1 127.0.0.1 -46:12.867 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf04e2db-5541-ddff-a94a-7131ea2c341a 127.0.0.1 -2022-11-07T22:46:12.867Z [INFO] agent.router: Initializing LAN area manager -46:12.867 [INFO] TestAgent.server: Adding LAN server: server="Node-cf04e2db-5541-ddff-a94a-7131ea2c341a (Addr: tcp/127.0.0.1:11933) (DC: dc1)" -46:12.867 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf04e2db-5541-ddff-a94a-7131ea2c341a.dc1 area=wan -46:12.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11893 network=udp -46:12.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11893 network=tcp -46:12.868 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11894 network=tcp -46:12.868 [INFO] TestAgent: started state syncer -46:12.868 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11934 network=tcp -46:12.930 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.930 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11933 [Candidate]" term=2 -46:12.941 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.941 [DEBUG] TestAgent.server.raft: vote granted: from=cf04e2db-5541-ddff-a94a-7131ea2c341a term=2 tally=1 -46:12.941 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.941 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11933 [Leader]" -46:12.941 [INFO] TestAgent.server: cluster leadership acquired -46:12.941 [INFO] TestAgent.server: New leader elected: payload=Node-cf04e2db-5541-ddff-a94a-7131ea2c341a -46:12.951 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11933 -46:12.967 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.994 [INFO] TestAgent: Requesting shutdown -46:12.994 [INFO] TestAgent.server: shutting down server -46:12.994 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:12.994 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:13.000 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.007Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.007Z [INFO] agent.router.manager: shutting down -46:13.010 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" -46:13.010 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" -46:13.010 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:13.010 [INFO] TestAgent: consul server down -46:13.010 [INFO] TestAgent: shutdown complete -46:13.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11893 network=tcp -46:13.010 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11893 network=udp -46:13.010 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11894 network=tcp -46:13.510 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.510 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogRegister_checkRegistration (0.72s) -=== CONT TestCoordinate_Datacenters -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:12.686Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.696Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.730 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b34100e5-8fd9-d8d7-660f-902e26e5bf28 Address:127.0.0.1:11870}]" -46:12.730 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11870 [Follower]" leader= -46:12.730 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28.dc1 127.0.0.1 -46:12.731 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28 127.0.0.1 -2022-11-07T22:46:12.731Z [INFO] agent.router: Initializing LAN area manager -46:12.731 [INFO] TestAgent.server: Adding LAN server: server="Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28 (Addr: tcp/127.0.0.1:11870) (DC: dc1)" -46:12.731 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28.dc1 area=wan -46:12.731 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11784 network=udp -46:12.731 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11784 network=tcp -46:12.731 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11792 network=tcp -46:12.731 [INFO] TestAgent: started state syncer -46:12.731 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11871 network=tcp -46:12.766 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.766 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11870 [Candidate]" term=2 -46:12.778 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.778 [DEBUG] TestAgent.server.raft: vote granted: from=b34100e5-8fd9-d8d7-660f-902e26e5bf28 term=2 tally=1 -46:12.778 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.778 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11870 [Leader]" -46:12.778 [INFO] TestAgent.server: cluster leadership acquired -46:12.778 [INFO] TestAgent.server: New leader elected: payload=Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28 -46:12.789 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11870 -46:12.803 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:12.846 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:12.846 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:12.846 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:12.846 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:12.846 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:12.846 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28 -46:12.846 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b34100e5-8fd9-d8d7-660f-902e26e5bf28 -46:12.863 [INFO] TestAgent.server: federation state anti-entropy synced -46:12.986 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:12.993 [INFO] TestAgent: Synced node info -46:12.993 [DEBUG] TestAgent: Node info in sync -46:13.079 [INFO] TestAgent: Requesting shutdown -46:13.079 [INFO] TestAgent.server: shutting down server -46:13.079 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.079 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.079 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.079 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.079 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.079 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.079 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.079 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.079 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.079 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.084 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.090Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.090Z [INFO] agent.router.manager: shutting down -46:13.090 [INFO] TestAgent: consul server down -46:13.090 [INFO] TestAgent: shutdown complete -46:13.090 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11784 network=tcp -46:13.090 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11784 network=udp -46:13.090 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11792 network=tcp -46:13.590 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.590 [INFO] TestAgent: Endpoints down ---- PASS: TestDiscoveryChainRead (0.91s) +2021-10-05T16:31:26.311Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.363Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.461 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b45440f-e301-8a9f-ff67-985514004c2d Address:127.0.0.1:29489}]" +31:26.462 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29489 [Follower]" leader= +31:26.462 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b45440f-e301-8a9f-ff67-985514004c2d.dc1 127.0.0.1 +31:26.463 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b45440f-e301-8a9f-ff67-985514004c2d 127.0.0.1 +2021-10-05T16:31:26.463Z [INFO] agent.router: Initializing LAN area manager +31:26.463 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=udp +31:26.463 [INFO] TestAgent.server: Adding LAN server: server="Node-8b45440f-e301-8a9f-ff67-985514004c2d (Addr: tcp/127.0.0.1:29489) (DC: dc1)" +31:26.463 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b45440f-e301-8a9f-ff67-985514004c2d.dc1 area=wan +31:26.464 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=tcp +31:26.464 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29415 network=tcp +31:26.464 [INFO] TestAgent: started state syncer +31:26.464 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29490 network=tcp +31:26.500 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.500 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29489 [Candidate]" term=2 +31:26.524 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.524 [DEBUG] TestAgent.server.raft: vote granted: from=8b45440f-e301-8a9f-ff67-985514004c2d term=2 tally=1 +31:26.524 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.524 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29489 [Leader]" +31:26.532 [INFO] TestAgent.server: cluster leadership acquired +31:26.532 [INFO] TestAgent.server: New leader elected: payload=Node-8b45440f-e301-8a9f-ff67-985514004c2d +31:26.568 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29489 +31:26.620 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.696 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.696 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.696 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.696 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.696 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.696 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b45440f-e301-8a9f-ff67-985514004c2d +31:26.696 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b45440f-e301-8a9f-ff67-985514004c2d +31:26.728 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=ANY class=IN latency=120.594µs client=127.0.0.1:52902 client_network=udp +31:26.729 [DEBUG] TestAgent.dns: request served from client: name=foo.node.consul. type=A class=IN latency=78.313µs client=127.0.0.1:39791 client_network=udp +31:26.729 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=76.775µs client=127.0.0.1:41711 client_network=udp +31:26.729 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=117.496µs client=127.0.0.1:50785 client_network=udp +31:26.729 [INFO] TestAgent: Requesting shutdown +31:26.729 [INFO] TestAgent.server: shutting down server +31:26.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.729 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.729 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.729 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.729 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.729 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:26.729 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.729 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:26.730 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.730 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.730 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.756Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.756Z [INFO] agent.router.manager: shutting down +31:26.756 [INFO] TestAgent: consul server down +31:26.756 [INFO] TestAgent: shutdown complete +31:26.756 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=tcp +31:26.756 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=udp +31:26.756 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29415 network=tcp +31:27.262 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.262 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_NodeLookup (0.99s) +=== CONT TestCoordinate_Disabled_Response +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:26.577Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.609Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.704 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf536da1-c806-ed6c-6514-e4c701de3263 Address:127.0.0.1:29454}]" +31:26.704 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29454 [Follower]" leader= +31:26.705 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf536da1-c806-ed6c-6514-e4c701de3263.dc1 127.0.0.1 +31:26.706 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf536da1-c806-ed6c-6514-e4c701de3263 127.0.0.1 +2021-10-05T16:31:26.706Z [INFO] agent.router: Initializing LAN area manager +31:26.706 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=udp +31:26.706 [INFO] TestAgent.server: Adding LAN server: server="Node-cf536da1-c806-ed6c-6514-e4c701de3263 (Addr: tcp/127.0.0.1:29454) (DC: dc1)" +31:26.706 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf536da1-c806-ed6c-6514-e4c701de3263.dc1 area=wan +31:26.707 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=tcp +31:26.707 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29436 network=tcp +31:26.707 [INFO] TestAgent: started state syncer +31:26.707 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29455 network=tcp +31:26.764 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.764 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29454 [Candidate]" term=2 +31:26.803 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.803 [DEBUG] TestAgent.server.raft: vote granted: from=cf536da1-c806-ed6c-6514-e4c701de3263 term=2 tally=1 +31:26.803 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.803 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29454 [Leader]" +31:26.803 [INFO] TestAgent.server: cluster leadership acquired +31:26.803 [INFO] TestAgent.server: New leader elected: payload=Node-cf536da1-c806-ed6c-6514-e4c701de3263 +31:26.832 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29454 +31:26.875 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.926 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.926 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.926 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.926 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.926 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.926 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf536da1-c806-ed6c-6514-e4c701de3263 +31:26.926 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cf536da1-c806-ed6c-6514-e4c701de3263 +31:26.964 [INFO] TestAgent: Synced node info +31:26.964 [DEBUG] TestAgent: Node info in sync +31:26.976 [INFO] TestAgent: Requesting shutdown +31:26.976 [INFO] TestAgent.server: shutting down server +31:26.976 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.976 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.976 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.976 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.976 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.976 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.976 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.976 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.980 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.980 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.984 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.992Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.992Z [INFO] agent.router.manager: shutting down +31:26.992 [INFO] TestAgent: consul server down +31:26.992 [INFO] TestAgent: shutdown complete +31:26.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=tcp +31:26.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=udp +31:26.992 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29436 network=tcp +31:27.492 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.492 [INFO] TestAgent: Endpoints down +--- PASS: TestCoordinate_Datacenters (0.94s) +=== CONT TestConnectCAConfig +=== RUN TestConnectCAConfig/basic +2021-10-05T16:31:26.583Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.613Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.648 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 Address:127.0.0.1:29426}]" +31:26.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6.dc1 127.0.0.1 +31:26.650 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 127.0.0.1 +2021-10-05T16:31:26.650Z [INFO] agent.router: Initializing LAN area manager +31:26.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29470 network=udp +31:26.650 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29426 [Follower]" leader= +31:26.651 [INFO] TestAgent.server: Adding LAN server: server="Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 (Addr: tcp/127.0.0.1:29426) (DC: dc1)" +31:26.651 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6.dc1 area=wan +31:26.651 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29470 network=tcp +31:26.651 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29471 network=tcp +31:26.651 [INFO] TestAgent: started state syncer +31:26.652 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29427 network=tcp +31:26.713 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.713 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29426 [Candidate]" term=2 +31:26.748 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.748 [DEBUG] TestAgent.server.raft: vote granted: from=c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 term=2 tally=1 +31:26.748 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.748 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29426 [Leader]" +31:26.748 [INFO] TestAgent.server: cluster leadership acquired +31:26.748 [INFO] TestAgent.server: New leader elected: payload=Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 +31:26.766 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29426 +31:26.792 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 +31:26.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c2a8c3dd-1396-847c-dd0e-70f1fb8a80c6 +31:26.880 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.924 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:26.936 [INFO] TestAgent: Synced node info +31:27.025 [INFO] TestAgent: Requesting shutdown +31:27.025 [INFO] TestAgent.server: shutting down server +31:27.025 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.025 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.025 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.025 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.025 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.025 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.036 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.044Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.044Z [INFO] agent.router.manager: shutting down +31:27.044 [INFO] TestAgent: consul server down +31:27.044 [INFO] TestAgent: shutdown complete +31:27.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29470 network=tcp +31:27.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29470 network=udp +31:27.044 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29471 network=tcp +31:27.545 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.545 [INFO] TestAgent: Endpoints down +--- PASS: TestDiscoveryChainRead (1.02s) --- PASS: TestDiscoveryChainRead/GET:_error_on_no_service_name (0.00s) --- PASS: TestDiscoveryChainRead/GET:_read_default_chain (0.00s) --- PASS: TestDiscoveryChainRead/GET:_read_default_chain;_evaluate_in_dc2 (0.00s) @@ -38711,3761 +31223,14220 @@ --- PASS: TestDiscoveryChainRead/POST:_read_default_chain (0.00s) --- PASS: TestDiscoveryChainRead/POST:_read_default_chain;_evaluate_in_dc2 (0.00s) --- PASS: TestDiscoveryChainRead/POST:_read_default_chain_with_cache (0.00s) - --- PASS: TestDiscoveryChainRead/GET:_read_modified_chain (0.00s) + --- PASS: TestDiscoveryChainRead/GET:_read_modified_chain (0.03s) --- PASS: TestDiscoveryChainRead/POST:_read_modified_chain_with_overrides_(camel_case) (0.00s) --- PASS: TestDiscoveryChainRead/POST:_read_modified_chain_with_overrides_(snake_case) (0.00s) -=== CONT TestDNS_ServiceReverseLookup_IPV6 -2022-11-07T22:46:12.796Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.811Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.866 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e9b544b5-5d77-c3b9-e97c-73f804390062 Address:127.0.0.1:11891}]" -46:12.866 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11891 [Follower]" leader= -46:12.866 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e9b544b5-5d77-c3b9-e97c-73f804390062.dc1 127.0.0.1 -46:12.866 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e9b544b5-5d77-c3b9-e97c-73f804390062 127.0.0.1 -2022-11-07T22:46:12.866Z [INFO] agent.router: Initializing LAN area manager -46:12.866 [INFO] TestAgent.server: Adding LAN server: server="Node-e9b544b5-5d77-c3b9-e97c-73f804390062 (Addr: tcp/127.0.0.1:11891) (DC: dc1)" -46:12.867 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e9b544b5-5d77-c3b9-e97c-73f804390062.dc1 area=wan -46:12.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11907 network=udp -46:12.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11907 network=tcp -46:12.867 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11908 network=tcp -46:12.867 [INFO] TestAgent: started state syncer -46:12.867 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11892 network=tcp -46:12.936 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.936 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11891 [Candidate]" term=2 -46:12.947 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.947 [DEBUG] TestAgent.server.raft: vote granted: from=e9b544b5-5d77-c3b9-e97c-73f804390062 term=2 tally=1 -46:12.947 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.947 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11891 [Leader]" -46:12.947 [INFO] TestAgent.server: cluster leadership acquired -46:12.947 [INFO] TestAgent.server: New leader elected: payload=Node-e9b544b5-5d77-c3b9-e97c-73f804390062 -46:12.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11891 -46:12.978 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.009 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.009 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.009 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.009 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.009 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.009 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e9b544b5-5d77-c3b9-e97c-73f804390062 -46:13.009 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e9b544b5-5d77-c3b9-e97c-73f804390062 -46:13.023 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.085 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=158.531µs client=127.0.0.1:55082 client_network=udp -46:13.085 [INFO] TestAgent: Requesting shutdown -46:13.085 [INFO] TestAgent.server: shutting down server -46:13.085 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.085 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.085 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.085 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.085 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.085 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.085 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.085 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:13.085 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.085 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.085 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.090 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.096Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.096Z [INFO] agent.router.manager: shutting down -46:13.096 [INFO] TestAgent: consul server down -46:13.096 [INFO] TestAgent: shutdown complete -46:13.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11907 network=tcp -46:13.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11907 network=udp -46:13.096 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11908 network=tcp -46:13.596 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.596 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookupPreferNoCNAME (0.81s) -=== CONT TestCatalogNodes_WanTranslation -2022-11-07T22:46:12.939Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.948Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.995 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a196c07-c2b1-8eef-95bd-9d14e3339c43 Address:127.0.0.1:11898}]" -46:12.995 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11898 [Follower]" leader= -46:12.995 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43.dc1 127.0.0.1 -46:12.996 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43 127.0.0.1 -2022-11-07T22:46:12.996Z [INFO] agent.router: Initializing LAN area manager -46:12.996 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43.dc1 area=wan -46:12.996 [INFO] TestAgent.server: Adding LAN server: server="Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43 (Addr: tcp/127.0.0.1:11898) (DC: dc1)" -46:12.996 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11886 network=udp -46:12.996 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11886 network=tcp -46:12.996 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11887 network=tcp -46:12.996 [INFO] TestAgent: started state syncer -46:12.996 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11899 network=tcp -46:13.058 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.058 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11898 [Candidate]" term=2 -46:13.068 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.068 [DEBUG] TestAgent.server.raft: vote granted: from=9a196c07-c2b1-8eef-95bd-9d14e3339c43 term=2 tally=1 -46:13.068 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.068 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11898 [Leader]" -46:13.069 [INFO] TestAgent.server: cluster leadership acquired -46:13.069 [INFO] TestAgent.server: New leader elected: payload=Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43 -46:13.079 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11898 -46:13.095 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.120 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.120 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.120 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.120 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.120 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.120 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43 -46:13.120 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9a196c07-c2b1-8eef-95bd-9d14e3339c43 -46:13.127 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.264 [DEBUG] TestAgent.dns: request served from client: question="{1.0.0.127.in-addr.arpa. 255 1}" latency=79.141µs client=127.0.0.1:36952 client_network=udp -46:13.264 [INFO] TestAgent: Requesting shutdown -46:13.264 [INFO] TestAgent.server: shutting down server -46:13.264 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.264 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.264 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.264 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:13.264 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.264 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.264 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.271 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.279Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.279Z [INFO] agent.router.manager: shutting down -46:13.279 [INFO] TestAgent: consul server down -46:13.279 [INFO] TestAgent: shutdown complete -46:13.279 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11886 network=tcp -46:13.279 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11886 network=udp -46:13.279 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11887 network=tcp -46:13.780 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.780 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceReverseLookupNodeAddress (0.85s) -=== CONT TestDNS_ServiceReverseLookup -2022-11-07T22:46:12.847Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.856Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.895 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2d584189-7f61-022c-2182-45429f76f8c9 Address:127.0.0.1:11912}]" -46:12.895 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11912 [Follower]" leader= -46:12.895 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2d584189-7f61-022c-2182-45429f76f8c9.dc1 127.0.0.1 -46:12.896 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2d584189-7f61-022c-2182-45429f76f8c9 127.0.0.1 -2022-11-07T22:46:12.896Z [INFO] agent.router: Initializing LAN area manager -46:12.896 [INFO] TestAgent.server: Adding LAN server: server="Node-2d584189-7f61-022c-2182-45429f76f8c9 (Addr: tcp/127.0.0.1:11912) (DC: dc1)" -46:12.896 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2d584189-7f61-022c-2182-45429f76f8c9.dc1 area=wan -46:12.896 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11879 network=tcp -46:12.896 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11879 network=udp -46:12.897 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11880 network=tcp -46:12.897 [INFO] TestAgent: started state syncer -46:12.897 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11913 network=tcp -46:12.953 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.953 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11912 [Candidate]" term=2 -46:12.967 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.967 [DEBUG] TestAgent.server.raft: vote granted: from=2d584189-7f61-022c-2182-45429f76f8c9 term=2 tally=1 -46:12.967 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.967 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11912 [Leader]" -46:12.967 [INFO] TestAgent.server: cluster leadership acquired -46:12.967 [INFO] TestAgent.server: New leader elected: payload=Node-2d584189-7f61-022c-2182-45429f76f8c9 -46:12.974 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11912 -46:12.995 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.027 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.027 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.027 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.027 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.027 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.027 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2d584189-7f61-022c-2182-45429f76f8c9 -46:13.027 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2d584189-7f61-022c-2182-45429f76f8c9 -46:13.040 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.305 [INFO] TestAgent: Requesting shutdown -46:13.305 [INFO] TestAgent.server: shutting down server -46:13.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.305 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.305 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.305 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.305 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.305 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:13.305 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.305 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.312 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.317Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.317Z [INFO] agent.router.manager: shutting down -46:13.318 [INFO] TestAgent: consul server down -46:13.318 [INFO] TestAgent: shutdown complete -46:13.318 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11879 network=tcp -46:13.318 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11879 network=udp -46:13.318 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11880 network=tcp -46:13.818 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.818 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServices_NodeMetaFilter (0.98s) -=== CONT TestCoordinate_Disabled_Response -2022-11-07T22:46:12.917Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:12.917Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.927Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:12.927Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.967 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7868e903-686b-6aa3-aa50-ef8c7bfeba12 Address:127.0.0.1:11884}]" -46:12.967 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11884 [Follower]" leader= -46:12.968 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12.dc1 127.0.0.1 -46:12.968 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 127.0.0.1 -2022-11-07T22:46:12.968Z [INFO] agent.router: Initializing LAN area manager -46:12.968 [INFO] TestAgent.server: Adding LAN server: server="Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 (Addr: tcp/127.0.0.1:11884) (DC: dc1)" -46:12.968 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12.dc1 area=wan -46:12.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11914 network=udp -46:12.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11914 network=tcp -46:12.969 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11915 network=tcp -46:12.969 [INFO] TestAgent: started state syncer -46:12.969 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11885 network=tcp -46:13.019 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:13.028 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.028 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11884 [Candidate]" term=2 -46:13.036 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.036 [DEBUG] TestAgent.server.raft: vote granted: from=7868e903-686b-6aa3-aa50-ef8c7bfeba12 term=2 tally=1 -46:13.036 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.036 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11884 [Leader]" -46:13.036 [INFO] TestAgent.server: cluster leadership acquired -46:13.036 [INFO] TestAgent.server: New leader elected: payload=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.042 [INFO] TestAgent.server: initializing acls -46:13.045 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:13.045 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:13.050 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:13.061 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:13.061 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:13.061 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:13.061 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.061 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12.dc1 -46:13.061 [INFO] TestAgent.server: Updating LAN server: server="Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 (Addr: tcp/127.0.0.1:11884) (DC: dc1)" -46:13.061 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12.dc1 area=wan -46:13.073 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.094 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.094 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.094 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.094 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.094 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.095 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.095 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.106 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.106 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.309 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.310 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.310 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7868e903-686b-6aa3-aa50-ef8c7bfeba12 -46:13.310 [INFO] TestAgent: Requesting shutdown -46:13.310 [INFO] TestAgent.server: shutting down server -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.310 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.310 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.310 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.310 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.314 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:13.316 [INFO] TestAgent: Synced node info -46:13.316 [DEBUG] TestAgent: Node info in sync -2022-11-07T22:46:13.321Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.321Z [INFO] agent.router.manager: shutting down -46:13.321 [INFO] TestAgent: consul server down -46:13.321 [INFO] TestAgent: shutdown complete -46:13.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11914 network=tcp -46:13.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11914 network=udp -46:13.321 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11915 network=tcp -46:13.822 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.822 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Update_ACLDeny (0.92s) - --- PASS: TestCoordinate_Update_ACLDeny/no_token (0.00s) - --- PASS: TestCoordinate_Update_ACLDeny/valid_token (0.00s) -=== CONT TestCatalogNodes_Filter -2022-11-07T22:46:12.838Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:12.848Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:12.886 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02ebc112-e7f3-0654-f81f-10ee8d59c5f7 Address:127.0.0.1:11877}]" -46:12.886 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11877 [Follower]" leader= -46:12.886 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7.dc1 127.0.0.1 -46:12.887 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7 127.0.0.1 -2022-11-07T22:46:12.887Z [INFO] agent.router: Initializing LAN area manager -46:12.887 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7.dc1 area=wan -46:12.887 [INFO] TestAgent.server: Adding LAN server: server="Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7 (Addr: tcp/127.0.0.1:11877) (DC: dc1)" -46:12.887 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11935 network=udp -46:12.887 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11935 network=tcp -46:12.887 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11936 network=tcp -46:12.887 [INFO] TestAgent: started state syncer -46:12.887 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11878 network=tcp -46:12.945 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:12.945 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11877 [Candidate]" term=2 -46:12.962 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:12.962 [DEBUG] TestAgent.server.raft: vote granted: from=02ebc112-e7f3-0654-f81f-10ee8d59c5f7 term=2 tally=1 -46:12.962 [INFO] TestAgent.server.raft: election won: tally=1 -46:12.962 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11877 [Leader]" -46:12.962 [INFO] TestAgent.server: cluster leadership acquired -46:12.962 [INFO] TestAgent.server: New leader elected: payload=Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7 -46:12.971 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11877 -46:12.988 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.015 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.015 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.015 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.015 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.015 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.015 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7 -46:13.016 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02ebc112-e7f3-0654-f81f-10ee8d59c5f7 -46:13.031 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.096 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.101 [INFO] TestAgent: Synced node info -46:13.317 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=ANY class=IN latency=157.171µs client=127.0.0.1:48920 client_network=udp -46:13.317 [INFO] TestAgent: Requesting shutdown -46:13.317 [INFO] TestAgent.server: shutting down server -46:13.317 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.317 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.317 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.317 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.317 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.317 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.317 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.317 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.317 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.317 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.321 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.327Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.327Z [INFO] agent.router.manager: shutting down -46:13.328 [INFO] TestAgent: consul server down -46:13.328 [INFO] TestAgent: shutdown complete -46:13.328 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11935 network=tcp -46:13.328 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11935 network=udp -46:13.328 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11936 network=tcp -46:13.828 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.828 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookupNoMultiCNAME (1.00s) -=== CONT TestDNS_ReverseLookup_IPV6 -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:13.161Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.170Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.199 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6260177c-b875-3a40-245e-740bdef5af4f Address:127.0.0.1:11947}]" -46:13.199 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11947 [Follower]" leader= -46:13.199 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6260177c-b875-3a40-245e-740bdef5af4f.dc1 127.0.0.1 -46:13.200 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6260177c-b875-3a40-245e-740bdef5af4f 127.0.0.1 -2022-11-07T22:46:13.200Z [INFO] agent.router: Initializing LAN area manager -46:13.200 [INFO] TestAgent.server: Adding LAN server: server="Node-6260177c-b875-3a40-245e-740bdef5af4f (Addr: tcp/127.0.0.1:11947) (DC: dc1)" -46:13.200 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6260177c-b875-3a40-245e-740bdef5af4f.dc1 area=wan -46:13.200 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11942 network=udp -46:13.200 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11942 network=tcp -46:13.200 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11943 network=tcp -46:13.201 [INFO] TestAgent: started state syncer -46:13.201 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11948 network=tcp -46:13.262 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.262 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11947 [Candidate]" term=2 -46:13.274 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.274 [DEBUG] TestAgent.server.raft: vote granted: from=6260177c-b875-3a40-245e-740bdef5af4f term=2 tally=1 -46:13.274 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.274 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11947 [Leader]" -46:13.274 [INFO] TestAgent.server: cluster leadership acquired -46:13.274 [INFO] TestAgent.server: New leader elected: payload=Node-6260177c-b875-3a40-245e-740bdef5af4f -46:13.283 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11947 -46:13.300 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.326 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.326 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.326 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.326 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.326 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.326 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6260177c-b875-3a40-245e-740bdef5af4f -46:13.326 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6260177c-b875-3a40-245e-740bdef5af4f -46:13.337 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.345 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.350 [INFO] TestAgent: Synced node info -46:13.364 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:13.364 [INFO] TestAgent: Requesting shutdown -46:13.364 [INFO] TestAgent.server: shutting down server -46:13.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.364 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.364 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.364 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.364 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.364 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.364 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.376Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.376Z [INFO] agent.router.manager: shutting down -46:13.376 [INFO] TestAgent: consul server down -46:13.376 [INFO] TestAgent: shutdown complete -46:13.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11942 network=tcp -46:13.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11942 network=udp -46:13.376 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11943 network=tcp -46:13.877 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.877 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Explain (2.44s) - --- PASS: TestPreparedQuery_Explain/#00 (0.98s) - --- PASS: TestPreparedQuery_Explain/#01 (0.74s) - --- PASS: TestPreparedQuery_Explain/#02 (0.73s) -=== CONT TestConnectCAConfig -=== RUN TestConnectCAConfig/basic -2022-11-07T22:46:13.076Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.091Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.121 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:47040040-1354-5489-5637-3b4bb1f5d53a Address:127.0.0.1:11919}]" -46:13.121 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11919 [Follower]" leader= -46:13.121 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-47040040-1354-5489-5637-3b4bb1f5d53a.dc1 127.0.0.1 -46:13.122 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-47040040-1354-5489-5637-3b4bb1f5d53a 127.0.0.1 -2022-11-07T22:46:13.122Z [INFO] agent.router: Initializing LAN area manager -46:13.122 [INFO] TestAgent.server: Adding LAN server: server="Node-47040040-1354-5489-5637-3b4bb1f5d53a (Addr: tcp/127.0.0.1:11919) (DC: dc1)" -46:13.122 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-47040040-1354-5489-5637-3b4bb1f5d53a.dc1 area=wan -46:13.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11900 network=udp -46:13.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11900 network=tcp -46:13.123 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11901 network=tcp -46:13.123 [INFO] TestAgent: started state syncer -46:13.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11920 network=tcp -46:13.186 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.186 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11919 [Candidate]" term=2 -46:13.197 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.197 [DEBUG] TestAgent.server.raft: vote granted: from=47040040-1354-5489-5637-3b4bb1f5d53a term=2 tally=1 -46:13.197 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.197 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11919 [Leader]" -46:13.197 [INFO] TestAgent.server: cluster leadership acquired -46:13.197 [INFO] TestAgent.server: New leader elected: payload=Node-47040040-1354-5489-5637-3b4bb1f5d53a -46:13.203 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11919 -46:13.235 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.265 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.265 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.265 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.265 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.265 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.265 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-47040040-1354-5489-5637-3b4bb1f5d53a -46:13.265 [INFO] TestAgent.server: member joined, marking health alive: member=Node-47040040-1354-5489-5637-3b4bb1f5d53a -46:13.282 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.284 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.291 [INFO] TestAgent: Synced node info -46:13.291 [DEBUG] TestAgent: Node info in sync -46:13.387 [INFO] TestAgent: Requesting shutdown -46:13.387 [INFO] TestAgent.server: shutting down server -46:13.387 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.387 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.387 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.387 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.387 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.387 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.387 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.387 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.387 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.387 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.393 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.401Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.401Z [INFO] agent.router.manager: shutting down -46:13.401 [INFO] TestAgent: consul server down -46:13.401 [INFO] TestAgent: shutdown complete -46:13.401 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11900 network=tcp -46:13.401 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11900 network=udp -46:13.401 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11901 network=tcp -46:13.901 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.901 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogServices (0.84s) -=== CONT TestCatalogNodes_MetaFilter +=== CONT TestConnectCARoots_list [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:13.144Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.154Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.177 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c93cca59-a649-1bf8-2232-a207dfa33ebd Address:127.0.0.1:11940}]" -46:13.177 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11940 [Follower]" leader= -46:13.178 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c93cca59-a649-1bf8-2232-a207dfa33ebd.dc1 127.0.0.1 -46:13.179 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c93cca59-a649-1bf8-2232-a207dfa33ebd 127.0.0.1 -2022-11-07T22:46:13.179Z [INFO] agent.router: Initializing LAN area manager -46:13.179 [INFO] TestAgent.server: Adding LAN server: server="Node-c93cca59-a649-1bf8-2232-a207dfa33ebd (Addr: tcp/127.0.0.1:11940) (DC: dc1)" -46:13.179 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c93cca59-a649-1bf8-2232-a207dfa33ebd.dc1 area=wan -46:13.179 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11970 network=udp -46:13.179 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11970 network=tcp -46:13.180 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11971 network=tcp -46:13.180 [INFO] TestAgent: started state syncer -46:13.180 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11941 network=tcp -46:13.231 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.231 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11940 [Candidate]" term=2 -46:13.251 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.251 [DEBUG] TestAgent.server.raft: vote granted: from=c93cca59-a649-1bf8-2232-a207dfa33ebd term=2 tally=1 -46:13.251 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.251 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11940 [Leader]" -46:13.251 [INFO] TestAgent.server: cluster leadership acquired -46:13.252 [INFO] TestAgent.server: New leader elected: payload=Node-c93cca59-a649-1bf8-2232-a207dfa33ebd -46:13.259 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11940 -46:13.278 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.306 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.306 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.306 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.306 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.306 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.306 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c93cca59-a649-1bf8-2232-a207dfa33ebd -46:13.306 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c93cca59-a649-1bf8-2232-a207dfa33ebd -46:13.320 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.525 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.529 [INFO] TestAgent: Synced node info -46:13.627 [INFO] TestAgent: Requesting shutdown -46:13.627 [INFO] TestAgent.server: shutting down server -46:13.627 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.627 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.627 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.627 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.627 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.627 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.627 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.627 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.627 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.627 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.632 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.639Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.639Z [INFO] agent.router.manager: shutting down -46:13.640 [INFO] TestAgent: consul server down -46:13.640 [INFO] TestAgent: shutdown complete -46:13.640 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11970 network=tcp -46:13.640 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11970 network=udp -46:13.640 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11971 network=tcp -46:14.141 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.141 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Update (1.01s) -=== CONT TestCatalogNodes +=== RUN TestCoordinate_Update_ACLDeny/no_token +=== RUN TestCoordinate_Update_ACLDeny/valid_token +2021-10-05T16:31:26.517Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.577Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.672 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 Address:127.0.0.1:29468}]" +31:26.672 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29468 [Follower]" leader= +31:26.673 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0.dc1 127.0.0.1 +31:26.673 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 127.0.0.1 +2021-10-05T16:31:26.673Z [INFO] agent.router: Initializing LAN area manager +31:26.673 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29484 network=udp +31:26.674 [INFO] TestAgent.server: Adding LAN server: server="Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 (Addr: tcp/127.0.0.1:29468) (DC: dc1)" +31:26.674 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0.dc1 area=wan +31:26.674 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29484 network=tcp +31:26.675 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29485 network=tcp +31:26.675 [INFO] TestAgent: started state syncer +31:26.675 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29469 network=tcp +31:26.733 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.733 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29468 [Candidate]" term=2 +31:26.765 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.765 [DEBUG] TestAgent.server.raft: vote granted: from=fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 term=2 tally=1 +31:26.765 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.765 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29468 [Leader]" +31:26.765 [INFO] TestAgent.server: cluster leadership acquired +31:26.766 [INFO] TestAgent.server: New leader elected: payload=Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 +31:26.781 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29468 +31:26.793 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.825 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.825 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.825 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.825 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.825 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.825 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 +31:26.825 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fbacdd0f-ca3b-0c5c-9144-141bc9cc77b0 +31:26.852 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.876 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:26.888 [INFO] TestAgent: Synced node info +31:26.888 [DEBUG] TestAgent: Node info in sync +31:27.116 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=163.732µs client=127.0.0.1:38910 client_network=tcp +31:27.116 [INFO] TestAgent: Requesting shutdown +31:27.116 [INFO] TestAgent.server: shutting down server +31:27.116 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.116 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.116 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.116 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.116 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.116 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.128 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.134Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.134Z [INFO] agent.router.manager: shutting down +31:27.134 [INFO] TestAgent: consul server down +31:27.134 [INFO] TestAgent: shutdown complete +31:27.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29484 network=tcp +31:27.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29484 network=udp +31:27.134 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29485 network=tcp +31:27.635 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.635 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_Over_TCP (1.14s) +=== CONT TestConnectCARoots_empty === RUN TestCoordinate_Disabled_Response/0 === RUN TestCoordinate_Disabled_Response/1 === RUN TestCoordinate_Disabled_Response/2 === RUN TestCoordinate_Disabled_Response/3 -2022-11-07T22:46:13.521Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.530Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.568 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c Address:127.0.0.1:11989}]" -46:13.569 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11989 [Follower]" leader= -46:13.569 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c.dc1 127.0.0.1 -46:13.569 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c 127.0.0.1 -2022-11-07T22:46:13.569Z [INFO] agent.router: Initializing LAN area manager -46:13.570 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c.dc1 area=wan -46:13.570 [INFO] TestAgent.server: Adding LAN server: server="Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c (Addr: tcp/127.0.0.1:11989) (DC: dc1)" -46:13.570 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11977 network=udp -46:13.570 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11977 network=tcp -46:13.570 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11978 network=tcp -46:13.570 [INFO] TestAgent: started state syncer -46:13.570 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11990 network=tcp -46:13.615 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.615 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11989 [Candidate]" term=2 -46:13.625 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.625 [DEBUG] TestAgent.server.raft: vote granted: from=2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c term=2 tally=1 -46:13.625 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.625 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11989 [Leader]" -46:13.625 [INFO] TestAgent.server: cluster leadership acquired -46:13.625 [INFO] TestAgent.server: New leader elected: payload=Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c -46:13.635 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11989 -46:13.649 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.671 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.671 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.671 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.671 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.671 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.671 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c -46:13.671 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b5e3e60-cd9e-97f5-6c5c-a18ef012f95c -46:13.681 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.690 [INFO] TestAgent: Requesting shutdown -46:13.690 [INFO] TestAgent.server: shutting down server -46:13.690 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.690 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.690 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.690 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.690 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.690 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:13.690 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.690 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.694 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.697Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.697Z [INFO] agent.router.manager: shutting down -46:13.697 [INFO] TestAgent: consul server down -46:13.697 [INFO] TestAgent: shutdown complete -46:13.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11977 network=tcp -46:13.697 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11977 network=udp -46:13.697 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11978 network=tcp -46:14.198 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.198 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Datacenters (0.69s) -=== CONT TestCatalogDatacenters -2022-11-07T22:46:13.393Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.408Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.444 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd8a5c56-8496-8647-eb01-46e69ab1a76c Address:127.0.0.1:11954}]" -46:13.444 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11954 [Follower]" leader= -46:13.445 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c.dc1 127.0.0.1 -46:13.445 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c 127.0.0.1 -2022-11-07T22:46:13.445Z [INFO] agent.router: Initializing LAN area manager -46:13.445 [INFO] TestAgent.server: Adding LAN server: server="Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c (Addr: tcp/127.0.0.1:11954) (DC: dc1)" -46:13.445 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11928 network=udp -46:13.445 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c.dc1 area=wan -46:13.445 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11928 network=tcp -46:13.446 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11929 network=tcp -46:13.446 [INFO] TestAgent: started state syncer -46:13.446 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11955 network=tcp -46:13.488 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.488 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11954 [Candidate]" term=2 -46:13.505 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.505 [DEBUG] TestAgent.server.raft: vote granted: from=cd8a5c56-8496-8647-eb01-46e69ab1a76c term=2 tally=1 -46:13.505 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.505 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11954 [Leader]" -46:13.506 [INFO] TestAgent.server: cluster leadership acquired -46:13.506 [INFO] TestAgent.server: New leader elected: payload=Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c -46:13.518 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11954 -46:13.518 [INFO] TestAgent: Synced node info -46:13.545 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.569 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.569 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.569 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.569 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.569 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.569 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c -46:13.569 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd8a5c56-8496-8647-eb01-46e69ab1a76c -46:13.588 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.699 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.699 [DEBUG] TestAgent: Node info in sync -46:13.699 [DEBUG] TestAgent: Node info in sync -46:13.707 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=122.681µs client=127.0.0.1:41935 client_network=udp -46:13.707 [INFO] TestAgent: Requesting shutdown -46:13.707 [INFO] TestAgent.server: shutting down server -46:13.707 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.707 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.707 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.707 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.707 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.707 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.707 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.707 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.713 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.720Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.720Z [INFO] agent.router.manager: shutting down -46:13.720 [INFO] TestAgent: consul server down -46:13.720 [INFO] TestAgent: shutdown complete -46:13.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11928 network=tcp -46:13.720 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11928 network=udp -46:13.720 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11929 network=tcp -46:14.220 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.220 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceReverseLookup_CustomDomain (0.84s) -=== CONT TestDNS_ReverseLookup_CustomDomain -2022-11-07T22:46:13.244Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.253Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.297 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f5df11fb-d182-5327-b85e-b5c36137f4da Address:127.0.0.1:11856}]" -46:13.297 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11856 [Follower]" leader= -46:13.297 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f5df11fb-d182-5327-b85e-b5c36137f4da.dc1 127.0.0.1 -46:13.298 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f5df11fb-d182-5327-b85e-b5c36137f4da 127.0.0.1 -2022-11-07T22:46:13.298Z [INFO] agent.router: Initializing LAN area manager -46:13.298 [INFO] TestAgent.server: Adding LAN server: server="Node-f5df11fb-d182-5327-b85e-b5c36137f4da (Addr: tcp/127.0.0.1:11856) (DC: dc1)" -46:13.298 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f5df11fb-d182-5327-b85e-b5c36137f4da.dc1 area=wan -46:13.298 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11949 network=udp -46:13.298 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11949 network=tcp -46:13.298 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11950 network=tcp -46:13.298 [INFO] TestAgent: started state syncer -46:13.298 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11857 network=tcp -46:13.344 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.344 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11856 [Candidate]" term=2 -46:13.353 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.353 [DEBUG] TestAgent.server.raft: vote granted: from=f5df11fb-d182-5327-b85e-b5c36137f4da term=2 tally=1 -46:13.353 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.353 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11856 [Leader]" -46:13.353 [INFO] TestAgent.server: cluster leadership acquired -46:13.353 [INFO] TestAgent.server: New leader elected: payload=Node-f5df11fb-d182-5327-b85e-b5c36137f4da -46:13.360 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11856 -46:13.375 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.399 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.399 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.399 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.399 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.399 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.399 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f5df11fb-d182-5327-b85e-b5c36137f4da -46:13.399 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f5df11fb-d182-5327-b85e-b5c36137f4da -46:13.410 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.559 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.565 [INFO] TestAgent: Synced node info -46:13.565 [DEBUG] TestAgent: Node info in sync -46:13.763 [INFO] TestAgent: Requesting shutdown -46:13.763 [INFO] TestAgent.server: shutting down server -46:13.763 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.763 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.763 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.763 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.763 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.763 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.763 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.763 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.763 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.763 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.769 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.774Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.774Z [INFO] agent.router.manager: shutting down -46:13.774 [INFO] TestAgent: consul server down -46:13.774 [INFO] TestAgent: shutdown complete -46:13.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11949 network=tcp -46:13.775 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11949 network=udp -46:13.775 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11950 network=tcp -46:14.275 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.275 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Node (1.04s) -=== CONT TestConnectCARoots_list -2022-11-07T22:46:13.393Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.408Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.440 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:95fd082c-c9b1-6b29-f572-4262f17f988f Address:127.0.0.1:11926}]" -46:13.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11926 [Follower]" leader= -46:13.441 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-95fd082c-c9b1-6b29-f572-4262f17f988f.dc1 127.0.0.1 -46:13.441 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-95fd082c-c9b1-6b29-f572-4262f17f988f 127.0.0.1 -2022-11-07T22:46:13.442Z [INFO] agent.router: Initializing LAN area manager -46:13.442 [INFO] TestAgent.server: Adding LAN server: server="Node-95fd082c-c9b1-6b29-f572-4262f17f988f (Addr: tcp/127.0.0.1:11926) (DC: dc1)" -46:13.442 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-95fd082c-c9b1-6b29-f572-4262f17f988f.dc1 area=wan -46:13.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11963 network=tcp -46:13.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11963 network=udp -46:13.442 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11964 network=tcp -46:13.442 [INFO] TestAgent: started state syncer -46:13.442 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11927 network=tcp -46:13.508 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.508 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11926 [Candidate]" term=2 -46:13.521 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.521 [DEBUG] TestAgent.server.raft: vote granted: from=95fd082c-c9b1-6b29-f572-4262f17f988f term=2 tally=1 -46:13.521 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.521 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11926 [Leader]" -46:13.521 [INFO] TestAgent.server: cluster leadership acquired -46:13.521 [INFO] TestAgent.server: New leader elected: payload=Node-95fd082c-c9b1-6b29-f572-4262f17f988f -46:13.529 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11926 -46:13.549 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.598 [INFO] TestAgent: Synced node info -46:13.598 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.598 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.598 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.598 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.598 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.598 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-95fd082c-c9b1-6b29-f572-4262f17f988f -46:13.598 [INFO] TestAgent.server: member joined, marking health alive: member=Node-95fd082c-c9b1-6b29-f572-4262f17f988f -46:13.601 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.810 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:13.810 [INFO] TestAgent: Requesting shutdown -46:13.811 [INFO] TestAgent.server: shutting down server -46:13.811 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.811 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.811 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.811 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.811 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.811 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.811 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.811 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.811 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.811 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.818 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.822Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.822Z [INFO] agent.router.manager: shutting down -46:13.822 [INFO] TestAgent: consul server down -46:13.822 [INFO] TestAgent: shutdown complete -46:13.822 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11963 network=tcp -46:13.822 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11963 network=udp -46:13.822 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11964 network=tcp -46:14.323 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.323 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#06 -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:13.790Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.800Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.826 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ab5df04-af3f-c84e-a8fa-aafd67676e32 Address:127.0.0.1:12017}]" -46:13.826 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12017 [Follower]" leader= -46:13.826 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32.dc1 127.0.0.1 -46:13.827 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32 127.0.0.1 -2022-11-07T22:46:13.827Z [INFO] agent.router: Initializing LAN area manager -46:13.827 [INFO] TestAgent.server: Adding LAN server: server="Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32 (Addr: tcp/127.0.0.1:12017) (DC: dc1)" -46:13.827 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32.dc1 area=wan -46:13.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11984 network=tcp -46:13.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11984 network=udp -46:13.827 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11985 network=tcp -46:13.828 [INFO] TestAgent: started state syncer -46:13.828 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12018 network=tcp -46:13.871 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.871 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12017 [Candidate]" term=2 -46:13.894 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.894 [DEBUG] TestAgent.server.raft: vote granted: from=3ab5df04-af3f-c84e-a8fa-aafd67676e32 term=2 tally=1 -46:13.894 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.894 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12017 [Leader]" -46:13.895 [INFO] TestAgent.server: cluster leadership acquired -46:13.895 [INFO] TestAgent.server: New leader elected: payload=Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32 -46:13.901 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12017 -46:13.920 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.951 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.951 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.951 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.951 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.951 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.951 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32 -46:13.951 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ab5df04-af3f-c84e-a8fa-aafd67676e32 -46:13.969 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.973 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.982 [INFO] TestAgent: Synced node info -46:13.990 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=99.117µs client=127.0.0.1:52932 client_network=udp -46:13.990 [INFO] TestAgent: Requesting shutdown -46:13.990 [INFO] TestAgent.server: shutting down server -46:13.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.990 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.990 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.990 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.990 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.990 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.990 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.990 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.990 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.000 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.006Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.006Z [INFO] agent.router.manager: shutting down -46:14.006 [INFO] TestAgent: consul server down -46:14.006 [INFO] TestAgent: shutdown complete -46:14.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11984 network=tcp -46:14.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11984 network=udp -46:14.006 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11985 network=tcp -46:14.507 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.507 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceReverseLookup (0.73s) -=== CONT TestDNS_ReverseLookup -2022-11-07T22:46:13.600Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.608Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.642 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:da804bfa-b0b4-562b-f109-3c455028210b Address:127.0.0.1:11996}]" -46:13.642 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11996 [Follower]" leader= -46:13.642 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-da804bfa-b0b4-562b-f109-3c455028210b.dc1 127.0.0.1 -46:13.643 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-da804bfa-b0b4-562b-f109-3c455028210b 127.0.0.1 -2022-11-07T22:46:13.643Z [INFO] agent.router: Initializing LAN area manager -46:13.643 [INFO] TestAgent.server: Adding LAN server: server="Node-da804bfa-b0b4-562b-f109-3c455028210b (Addr: tcp/127.0.0.1:11996) (DC: dc1)" -46:13.643 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-da804bfa-b0b4-562b-f109-3c455028210b.dc1 area=wan -46:13.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11991 network=udp -46:13.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11991 network=tcp -46:13.643 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11992 network=tcp -46:13.643 [INFO] TestAgent: started state syncer -46:13.643 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11997 network=tcp -46:13.697 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.697 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11996 [Candidate]" term=2 -46:13.707 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.707 [DEBUG] TestAgent.server.raft: vote granted: from=da804bfa-b0b4-562b-f109-3c455028210b term=2 tally=1 -46:13.707 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.707 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11996 [Leader]" -46:13.707 [INFO] TestAgent.server: cluster leadership acquired -46:13.707 [INFO] TestAgent.server: New leader elected: payload=Node-da804bfa-b0b4-562b-f109-3c455028210b -46:13.717 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11996 -46:13.732 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.758 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.758 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.758 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.758 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.758 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.758 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-da804bfa-b0b4-562b-f109-3c455028210b -46:13.758 [INFO] TestAgent.server: member joined, marking health alive: member=Node-da804bfa-b0b4-562b-f109-3c455028210b -46:13.772 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.878 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.885 [INFO] TestAgent: Synced node info -46:14.030 [DEBUG] TestAgent.dns: request served from client: question="{9.2.3.8.2.4.0.0.0.0.f.f.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa. 255 1}" latency=128.58µs client=127.0.0.1:49792 client_network=udp -46:14.030 [INFO] TestAgent: Requesting shutdown -46:14.030 [INFO] TestAgent.server: shutting down server -46:14.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.030 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.030 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.030 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.030 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.030 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.030 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.030 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.030 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.030 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.037 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.045Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.045Z [INFO] agent.router.manager: shutting down -46:14.045 [INFO] TestAgent: consul server down -46:14.045 [INFO] TestAgent: shutdown complete -46:14.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11991 network=tcp -46:14.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11991 network=udp -46:14.045 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11992 network=tcp -46:14.545 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.545 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceReverseLookup_IPV6 (0.95s) -=== CONT TestCatalogDeregister -2022-11-07T22:46:13.440Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.449Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.484 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:130de69b-0d24-86ba-e7a1-a5b8d59411b0 Address:127.0.0.1:11975}]" -46:13.485 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11975 [Follower]" leader= -46:13.485 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0.dc1 127.0.0.1 -46:13.485 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0 127.0.0.1 -2022-11-07T22:46:13.485Z [INFO] agent.router: Initializing LAN area manager -46:13.486 [INFO] TestAgent.server: Adding LAN server: server="Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0 (Addr: tcp/127.0.0.1:11975) (DC: dc1)" -46:13.486 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0.dc1 area=wan -46:13.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12012 network=udp -46:13.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12012 network=tcp -46:13.486 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12013 network=tcp -46:13.486 [INFO] TestAgent: started state syncer -46:13.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11976 network=tcp -46:13.553 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.553 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11975 [Candidate]" term=2 -46:13.572 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.572 [DEBUG] TestAgent.server.raft: vote granted: from=130de69b-0d24-86ba-e7a1-a5b8d59411b0 term=2 tally=1 -46:13.572 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.572 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11975 [Leader]" -46:13.572 [INFO] TestAgent.server: cluster leadership acquired -46:13.572 [INFO] TestAgent.server: New leader elected: payload=Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0 -46:13.581 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11975 -46:13.597 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.624 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.624 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.624 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.624 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.624 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.624 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0 -46:13.624 [INFO] TestAgent.server: member joined, marking health alive: member=Node-130de69b-0d24-86ba-e7a1-a5b8d59411b0 -46:13.643 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.769 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.773 [INFO] TestAgent: Synced node info -46:13.773 [DEBUG] TestAgent: Node info in sync -46:13.780 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.780 [DEBUG] TestAgent: Node info in sync -46:14.044 [INFO] TestAgent: Requesting shutdown -46:14.044 [INFO] TestAgent.server: shutting down server -46:14.044 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.044 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.044 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.044 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.044 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.044 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.044 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.044 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.044 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.044 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.053 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.060Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.060Z [INFO] agent.router.manager: shutting down -46:14.060 [INFO] TestAgent: consul server down -46:14.060 [INFO] TestAgent: shutdown complete -46:14.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12012 network=tcp -46:14.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12012 network=udp -46:14.060 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12013 network=tcp -46:14.561 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.561 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Nodes (1.13s) -=== CONT TestConnectCARoots_empty -2022-11-07T22:46:13.352Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.362Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.394 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e5a6ba1d-31f8-3eba-bfea-a633c6e1508e Address:127.0.0.1:11961}]" -46:13.394 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11961 [Follower]" leader= -46:13.395 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e.dc1 127.0.0.1 -46:13.395 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e 127.0.0.1 -2022-11-07T22:46:13.395Z [INFO] agent.router: Initializing LAN area manager -46:13.395 [INFO] TestAgent.server: Adding LAN server: server="Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e (Addr: tcp/127.0.0.1:11961) (DC: dc1)" -46:13.396 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e.dc1 area=wan -46:13.396 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11858 network=udp -46:13.396 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11858 network=tcp -46:13.396 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11859 network=tcp -46:13.396 [INFO] TestAgent: started state syncer -46:13.396 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11962 network=tcp -46:13.430 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.430 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11961 [Candidate]" term=2 -46:13.443 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.443 [DEBUG] TestAgent.server.raft: vote granted: from=e5a6ba1d-31f8-3eba-bfea-a633c6e1508e term=2 tally=1 -46:13.443 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.443 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11961 [Leader]" -46:13.443 [INFO] TestAgent.server: cluster leadership acquired -46:13.443 [INFO] TestAgent.server: New leader elected: payload=Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e -46:13.450 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11961 -46:13.465 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.491 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.491 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.491 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.491 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.491 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.491 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e -46:13.491 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e5a6ba1d-31f8-3eba-bfea-a633c6e1508e -46:13.507 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.772 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.777 [INFO] TestAgent: Synced node info -46:13.777 [DEBUG] TestAgent: Node info in sync -46:14.071 [INFO] TestAgent: Requesting shutdown -46:14.071 [INFO] TestAgent.server: shutting down server -46:14.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.071 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.071 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.071 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.071 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.071 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.071 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.075 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.079Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.079Z [INFO] agent.router.manager: shutting down -46:14.079 [INFO] TestAgent: consul server down -46:14.079 [INFO] TestAgent: shutdown complete -46:14.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11858 network=tcp -46:14.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11858 network=udp -46:14.079 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11859 network=tcp -46:14.580 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.580 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes_DistanceSort (1.24s) -=== CONT TestDNS_EDNS0_ECS -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:13.834Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.848Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.897 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9eee2b34-6f1f-414e-5124-7034aba58999 Address:127.0.0.1:11116}]" -46:13.897 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11116 [Follower]" leader= -46:13.898 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9eee2b34-6f1f-414e-5124-7034aba58999.dc1 127.0.0.1 -46:13.898 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9eee2b34-6f1f-414e-5124-7034aba58999 127.0.0.1 -2022-11-07T22:46:13.898Z [INFO] agent.router: Initializing LAN area manager -46:13.898 [INFO] TestAgent.server: Adding LAN server: server="Node-9eee2b34-6f1f-414e-5124-7034aba58999 (Addr: tcp/127.0.0.1:11116) (DC: dc1)" -46:13.898 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9eee2b34-6f1f-414e-5124-7034aba58999.dc1 area=wan -46:13.899 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12033 network=udp -46:13.899 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12033 network=tcp -46:13.899 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12034 network=tcp -46:13.899 [INFO] TestAgent: started state syncer -46:13.899 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11117 network=tcp -46:13.957 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.957 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11116 [Candidate]" term=2 -46:13.975 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.975 [DEBUG] TestAgent.server.raft: vote granted: from=9eee2b34-6f1f-414e-5124-7034aba58999 term=2 tally=1 -46:13.975 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.975 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11116 [Leader]" -46:13.975 [INFO] TestAgent.server: cluster leadership acquired -46:13.975 [INFO] TestAgent.server: New leader elected: payload=Node-9eee2b34-6f1f-414e-5124-7034aba58999 -46:13.992 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11116 -46:14.008 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.043 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.043 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.043 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.043 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.043 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.043 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9eee2b34-6f1f-414e-5124-7034aba58999 -46:14.043 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9eee2b34-6f1f-414e-5124-7034aba58999 -46:14.052 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.135 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.139 [INFO] TestAgent: Synced node info -46:14.139 [DEBUG] TestAgent: Node info in sync -46:14.154 [INFO] TestAgent: Requesting shutdown -46:14.154 [INFO] TestAgent.server: shutting down server -46:14.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.154 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.154 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.154 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.154 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.154 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.154 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.160 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.165Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.165Z [INFO] agent.router.manager: shutting down -46:14.165 [INFO] TestAgent: consul server down -46:14.165 [INFO] TestAgent: shutdown complete -46:14.165 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12033 network=tcp -46:14.165 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12033 network=udp -46:14.165 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12034 network=tcp -46:14.666 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.666 [INFO] TestAgent: Endpoints down ---- PASS: TestCoordinate_Disabled_Response (0.85s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:27.281Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.295Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.356 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:27a3a10f-608c-bd86-ca3f-46e1e9d61764 Address:127.0.0.1:29517}]" +31:27.356 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29517 [Follower]" leader= +31:27.357 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764.dc1 127.0.0.1 +31:27.357 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764 127.0.0.1 +2021-10-05T16:31:27.357Z [INFO] agent.router: Initializing LAN area manager +31:27.357 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29463 network=udp +31:27.357 [INFO] TestAgent.server: Adding LAN server: server="Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764 (Addr: tcp/127.0.0.1:29517) (DC: dc1)" +31:27.357 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764.dc1 area=wan +31:27.357 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29463 network=tcp +31:27.358 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29464 network=tcp +31:27.358 [INFO] TestAgent: started state syncer +31:27.358 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29518 network=tcp +31:27.410 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.410 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29517 [Candidate]" term=2 +31:27.435 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.435 [DEBUG] TestAgent.server.raft: vote granted: from=27a3a10f-608c-bd86-ca3f-46e1e9d61764 term=2 tally=1 +31:27.435 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.435 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29517 [Leader]" +31:27.435 [INFO] TestAgent.server: cluster leadership acquired +31:27.435 [INFO] TestAgent.server: New leader elected: payload=Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764 +31:27.461 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29517 +31:27.492 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.537 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.537 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.537 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.537 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.537 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.537 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764 +31:27.537 [INFO] TestAgent.server: member joined, marking health alive: member=Node-27a3a10f-608c-bd86-ca3f-46e1e9d61764 +31:27.560 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=221.223µs client=127.0.0.1:47852 client_network=udp +31:27.560 [INFO] TestAgent: Requesting shutdown +31:27.561 [INFO] TestAgent.server: shutting down server +31:27.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.561 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.561 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.561 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.561 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:27.561 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.561 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.561 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.568 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.572 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.580Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.580Z [INFO] agent.router.manager: shutting down +31:27.580 [INFO] TestAgent: consul server down +31:27.580 [INFO] TestAgent: shutdown complete +31:27.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29463 network=tcp +31:27.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29463 network=udp +31:27.580 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29464 network=tcp +31:28.080 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.080 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:26.402Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:26.431Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:26.488 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ec32489-0c8e-f61b-d32d-937db639e379 Address:127.0.0.1:29475}]" +31:26.488 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29475 [Follower]" leader= +31:26.488 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0ec32489-0c8e-f61b-d32d-937db639e379.dc1 127.0.0.1 +31:26.489 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0ec32489-0c8e-f61b-d32d-937db639e379 127.0.0.1 +2021-10-05T16:31:26.489Z [INFO] agent.router: Initializing LAN area manager +31:26.489 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=udp +31:26.489 [INFO] TestAgent.server: Adding LAN server: server="Node-0ec32489-0c8e-f61b-d32d-937db639e379 (Addr: tcp/127.0.0.1:29475) (DC: dc1)" +31:26.489 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0ec32489-0c8e-f61b-d32d-937db639e379.dc1 area=wan +31:26.490 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=tcp +31:26.490 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29450 network=tcp +31:26.490 [INFO] TestAgent: started state syncer +31:26.490 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29476 network=tcp +31:26.528 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:26.528 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29475 [Candidate]" term=2 +31:26.576 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:26.576 [DEBUG] TestAgent.server.raft: vote granted: from=0ec32489-0c8e-f61b-d32d-937db639e379 term=2 tally=1 +31:26.576 [INFO] TestAgent.server.raft: election won: tally=1 +31:26.576 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29475 [Leader]" +31:26.580 [INFO] TestAgent.server: cluster leadership acquired +31:26.580 [INFO] TestAgent.server: New leader elected: payload=Node-0ec32489-0c8e-f61b-d32d-937db639e379 +31:26.608 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29475 +31:26.652 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:26.679 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:26.679 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:26.679 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:26.679 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:26.679 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:26.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0ec32489-0c8e-f61b-d32d-937db639e379 +31:26.679 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0ec32489-0c8e-f61b-d32d-937db639e379 +31:26.701 [INFO] TestAgent.server: federation state anti-entropy synced +31:26.706 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=130.572µs client=127.0.0.1:60278 client_network=udp +31:26.706 [INFO] TestAgent: Requesting shutdown +31:26.706 [INFO] TestAgent.server: shutting down server +31:26.706 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:26.706 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:26.706 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:26.706 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:26.706 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:26.707 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:26.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:26.707 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:26.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:26.707 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:26.707 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:26.711 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:26.714Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:26.714Z [INFO] agent.router.manager: shutting down +31:26.714 [INFO] TestAgent: consul server down +31:26.714 [INFO] TestAgent: shutdown complete +31:26.714 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=tcp +31:26.714 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=udp +31:26.714 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29450 network=tcp +31:27.215 [INFO] TestAgent: Waiting for endpoints to shut down +31:27.215 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:25.497Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:25.532Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:25.636 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ac83cc04-04ed-ebeb-bd3c-d62c0f94326d Address:127.0.0.1:30390}]" +31:25.636 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30390 [Follower]" leader= +31:25.637 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d.dc1 127.0.0.1 +31:25.637 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d 127.0.0.1 +2021-10-05T16:31:25.637Z [INFO] agent.router: Initializing LAN area manager +31:25.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30385 network=udp +31:25.638 [INFO] TestAgent.server: Adding LAN server: server="Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d (Addr: tcp/127.0.0.1:30390) (DC: dc1)" +31:25.638 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d.dc1 area=wan +31:25.638 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30385 network=tcp +31:25.638 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30386 network=tcp +31:25.638 [INFO] TestAgent: started state syncer +31:25.638 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30391 network=tcp +31:25.691 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:25.691 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30390 [Candidate]" term=2 +31:25.706 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:25.706 [DEBUG] TestAgent.server.raft: vote granted: from=ac83cc04-04ed-ebeb-bd3c-d62c0f94326d term=2 tally=1 +31:25.706 [INFO] TestAgent.server.raft: election won: tally=1 +31:25.706 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30390 [Leader]" +31:25.706 [INFO] TestAgent.server: cluster leadership acquired +31:25.706 [INFO] TestAgent.server: New leader elected: payload=Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d +31:25.720 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30390 +31:25.755 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:25.801 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:25.801 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:25.801 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:25.801 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:25.801 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:25.801 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d +31:25.801 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ac83cc04-04ed-ebeb-bd3c-d62c0f94326d +31:25.825 [INFO] TestAgent.server: federation state anti-entropy synced +31:25.842 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=322.601µs client=127.0.0.1:34974 client_network=udp +31:25.842 [INFO] TestAgent: Requesting shutdown +31:25.842 [INFO] TestAgent.server: shutting down server +31:25.842 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:25.842 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:25.842 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:25.842 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:25.842 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:25.842 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:25.842 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:25.842 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:25.842 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:25.842 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:25.842 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:25.847 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:25.859Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:25.859Z [INFO] agent.router.manager: shutting down +31:25.859 [INFO] TestAgent: consul server down +31:25.859 [INFO] TestAgent: shutdown complete +31:25.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30385 network=tcp +31:25.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30385 network=udp +31:25.859 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30386 network=tcp +31:26.360 [INFO] TestAgent: Waiting for endpoints to shut down +31:26.360 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:24.574Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:24.608Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:24.645 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b40a57a6-40ce-ccb7-ef2a-78951f7f8488 Address:127.0.0.1:30334}]" +31:24.646 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488.dc1 127.0.0.1 +31:24.647 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488 127.0.0.1 +2021-10-05T16:31:24.647Z [INFO] agent.router: Initializing LAN area manager +31:24.647 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30329 network=udp +31:24.647 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30334 [Follower]" leader= +31:24.647 [INFO] TestAgent.server: Adding LAN server: server="Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488 (Addr: tcp/127.0.0.1:30334) (DC: dc1)" +31:24.647 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488.dc1 area=wan +31:24.647 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30329 network=tcp +31:24.648 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30330 network=tcp +31:24.648 [INFO] TestAgent: started state syncer +31:24.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30335 network=tcp +31:24.705 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:24.705 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30334 [Candidate]" term=2 +31:24.724 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:24.724 [DEBUG] TestAgent.server.raft: vote granted: from=b40a57a6-40ce-ccb7-ef2a-78951f7f8488 term=2 tally=1 +31:24.724 [INFO] TestAgent.server.raft: election won: tally=1 +31:24.724 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30334 [Leader]" +31:24.726 [INFO] TestAgent.server: cluster leadership acquired +31:24.727 [INFO] TestAgent.server: New leader elected: payload=Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488 +31:24.740 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30334 +31:24.770 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:24.824 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:24.824 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:24.824 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:24.824 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:24.824 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:24.824 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488 +31:24.824 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b40a57a6-40ce-ccb7-ef2a-78951f7f8488 +31:24.859 [INFO] TestAgent.server: federation state anti-entropy synced +31:24.937 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=222.989µs client=127.0.0.1:58458 client_network=udp +31:24.938 [INFO] TestAgent: Requesting shutdown +31:24.938 [INFO] TestAgent.server: shutting down server +31:24.938 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:24.938 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:24.938 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:24.938 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:24.938 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:24.938 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:24.938 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:24.938 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:24.938 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:24.938 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:24.938 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:24.945 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:24.951Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:24.951Z [INFO] agent.router.manager: shutting down +31:24.951 [INFO] TestAgent: consul server down +31:24.951 [INFO] TestAgent: shutdown complete +31:24.951 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30329 network=tcp +31:24.951 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30329 network=udp +31:24.951 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30330 network=tcp +31:25.452 [INFO] TestAgent: Waiting for endpoints to shut down +31:25.452 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_SOA_Settings (3.58s) +=== CONT TestConfig_Apply_ProxyDefaultsExpose +2021-10-05T16:31:27.067Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:27.067Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.093Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:27.093Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.184 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:91e2d166-42ae-13fb-48ff-af942bc382f2 Address:127.0.0.1:29440}]" +31:27.186 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-91e2d166-42ae-13fb-48ff-af942bc382f2.dc1 127.0.0.1 +31:27.187 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-91e2d166-42ae-13fb-48ff-af942bc382f2 127.0.0.1 +2021-10-05T16:31:27.187Z [INFO] agent.router: Initializing LAN area manager +31:27.196 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29440 [Follower]" leader= +31:27.200 [INFO] TestAgent.server: Adding LAN server: server="Node-91e2d166-42ae-13fb-48ff-af942bc382f2 (Addr: tcp/127.0.0.1:29440) (DC: dc1)" +31:27.202 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-91e2d166-42ae-13fb-48ff-af942bc382f2.dc1 area=wan +31:27.204 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29456 network=udp +31:27.204 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29456 network=tcp +31:27.211 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29457 network=tcp +31:27.215 [INFO] TestAgent: started state syncer +31:27.215 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29441 network=tcp +31:27.238 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.238 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29440 [Candidate]" term=2 +31:27.262 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:27.267 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.267 [DEBUG] TestAgent.server.raft: vote granted: from=91e2d166-42ae-13fb-48ff-af942bc382f2 term=2 tally=1 +31:27.267 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.267 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29440 [Leader]" +31:27.268 [INFO] TestAgent.server: cluster leadership acquired +31:27.268 [INFO] TestAgent.server: New leader elected: payload=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.297 [INFO] TestAgent.server: initializing acls +31:27.303 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:27.303 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:27.319 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:27.343 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:27.343 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:27.343 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:27.343 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.344 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-91e2d166-42ae-13fb-48ff-af942bc382f2.dc1 +31:27.344 [INFO] TestAgent.server: Updating LAN server: server="Node-91e2d166-42ae-13fb-48ff-af942bc382f2 (Addr: tcp/127.0.0.1:29440) (DC: dc1)" +31:27.344 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-91e2d166-42ae-13fb-48ff-af942bc382f2.dc1 area=wan +31:27.363 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.394 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.394 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.394 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.394 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.394 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.395 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.395 [INFO] TestAgent.server: member joined, marking health alive: member=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.403 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.410 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.475 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:27.491 [INFO] TestAgent: Synced node info +31:27.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-91e2d166-42ae-13fb-48ff-af942bc382f2 +31:27.591 [INFO] TestAgent: Requesting shutdown +31:27.591 [INFO] TestAgent.server: shutting down server +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.591 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.591 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.591 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.591 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.598 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.606Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.606Z [INFO] agent.router.manager: shutting down +31:27.606 [INFO] TestAgent: consul server down +31:27.606 [INFO] TestAgent: shutdown complete +31:27.606 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29456 network=tcp +31:27.606 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29456 network=udp +31:27.606 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29457 network=tcp +31:28.107 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.107 [INFO] TestAgent: Endpoints down +--- PASS: TestCoordinate_Update_ACLDeny (1.07s) + --- PASS: TestCoordinate_Update_ACLDeny/no_token (0.00s) + --- PASS: TestCoordinate_Update_ACLDeny/valid_token (0.00s) +=== CONT TestConfig_Apply_Decoding +2021-10-05T16:31:27.071Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.104Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.188 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83a1fed0-1628-a358-b0f3-e751db2099c0 Address:127.0.0.1:29433}]" +31:27.189 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83a1fed0-1628-a358-b0f3-e751db2099c0.dc1 127.0.0.1 +31:27.190 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83a1fed0-1628-a358-b0f3-e751db2099c0 127.0.0.1 +2021-10-05T16:31:27.190Z [INFO] agent.router: Initializing LAN area manager +31:27.190 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=udp +31:27.190 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29433 [Follower]" leader= +31:27.203 [INFO] TestAgent.server: Adding LAN server: server="Node-83a1fed0-1628-a358-b0f3-e751db2099c0 (Addr: tcp/127.0.0.1:29433) (DC: dc1)" +31:27.203 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83a1fed0-1628-a358-b0f3-e751db2099c0.dc1 area=wan +31:27.204 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=tcp +31:27.209 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29443 network=tcp +31:27.209 [INFO] TestAgent: started state syncer +31:27.209 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29434 network=tcp +31:27.240 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.241 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29433 [Candidate]" term=2 +31:27.272 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.272 [DEBUG] TestAgent.server.raft: vote granted: from=83a1fed0-1628-a358-b0f3-e751db2099c0 term=2 tally=1 +31:27.272 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.272 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29433 [Leader]" +31:27.273 [INFO] TestAgent.server: cluster leadership acquired +31:27.273 [INFO] TestAgent.server: New leader elected: payload=Node-83a1fed0-1628-a358-b0f3-e751db2099c0 +31:27.298 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29433 +31:27.324 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.377 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.377 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.377 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.377 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.377 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.377 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83a1fed0-1628-a358-b0f3-e751db2099c0 +31:27.377 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83a1fed0-1628-a358-b0f3-e751db2099c0 +31:27.399 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.552 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:27.572 [INFO] TestAgent: Synced node info +31:27.572 [DEBUG] TestAgent: Node info in sync +31:27.736 [INFO] TestAgent: Requesting shutdown +31:27.736 [INFO] TestAgent.server: shutting down server +31:27.736 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.736 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.736 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.736 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.736 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.736 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.736 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.736 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.736 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.736 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.756 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.760Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.760Z [INFO] agent.router.manager: shutting down +31:27.760 [INFO] TestAgent: consul server down +31:27.760 [INFO] TestAgent: shutdown complete +31:27.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=tcp +31:27.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=udp +31:27.760 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29443 network=tcp +31:28.270 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.270 [INFO] TestAgent: Endpoints down +--- PASS: TestCoordinate_Update (1.23s) +=== CONT TestConfig_Apply_CAS +2021-10-05T16:31:27.276Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.293Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.354 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 Address:127.0.0.1:29503}]" +31:27.354 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29503 [Follower]" leader= +31:27.354 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42.dc1 127.0.0.1 +31:27.355 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 127.0.0.1 +2021-10-05T16:31:27.355Z [INFO] agent.router: Initializing LAN area manager +31:27.355 [INFO] TestAgent.server: Adding LAN server: server="Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 (Addr: tcp/127.0.0.1:29503) (DC: dc1)" +31:27.355 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42.dc1 area=wan +31:27.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29505 network=udp +31:27.356 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29505 network=tcp +31:27.357 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29506 network=tcp +31:27.357 [INFO] TestAgent: started state syncer +31:27.357 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29504 network=tcp +31:27.390 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.390 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29503 [Candidate]" term=2 +31:27.404 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.404 [DEBUG] TestAgent.server.raft: vote granted: from=45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 term=2 tally=1 +31:27.404 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.404 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29503 [Leader]" +31:27.404 [INFO] TestAgent.server: cluster leadership acquired +31:27.404 [INFO] TestAgent.server: New leader elected: payload=Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 +31:27.425 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29503 +31:27.463 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.488 [INFO] TestAgent: Synced node info +31:27.512 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.512 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.512 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.512 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.512 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.513 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 +31:27.513 [INFO] TestAgent.server: member joined, marking health alive: member=Node-45a5ddab-a5b8-d9ae-a512-cd3bf73ecc42 +31:27.534 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.769 [INFO] TestAgent: Requesting shutdown +31:27.770 [INFO] TestAgent.server: shutting down server +31:27.770 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.770 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.770 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.770 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.770 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.770 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.770 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.770 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.770 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.770 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.779 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.780Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.780Z [INFO] agent.router.manager: shutting down +31:27.786 [INFO] TestAgent: consul server down +31:27.786 [INFO] TestAgent: shutdown complete +31:27.786 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29505 network=tcp +31:27.786 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29505 network=udp +31:27.786 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29506 network=tcp +31:28.287 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.287 [INFO] TestAgent: Endpoints down +--- PASS: TestCoordinate_Disabled_Response (1.03s) --- PASS: TestCoordinate_Disabled_Response/0 (0.00s) --- PASS: TestCoordinate_Disabled_Response/1 (0.00s) --- PASS: TestCoordinate_Disabled_Response/2 (0.00s) --- PASS: TestCoordinate_Disabled_Response/3 (0.00s) -=== CONT TestAgent_RegisterCheck_BadStatus -2022-11-07T22:46:13.837Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.848Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.891 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:184bda9b-443f-0085-b9ca-a59b3b97987e Address:127.0.0.1:11109}]" -46:13.891 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11109 [Follower]" leader= -46:13.891 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-184bda9b-443f-0085-b9ca-a59b3b97987e.dc1 127.0.0.1 -46:13.892 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-184bda9b-443f-0085-b9ca-a59b3b97987e 127.0.0.1 -2022-11-07T22:46:13.892Z [INFO] agent.router: Initializing LAN area manager -46:13.892 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-184bda9b-443f-0085-b9ca-a59b3b97987e.dc1 area=wan -46:13.892 [INFO] TestAgent.server: Adding LAN server: server="Node-184bda9b-443f-0085-b9ca-a59b3b97987e (Addr: tcp/127.0.0.1:11109) (DC: dc1)" -46:13.892 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11034 network=tcp -46:13.892 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11034 network=udp -46:13.892 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11035 network=tcp -46:13.892 [INFO] TestAgent: started state syncer -46:13.892 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11110 network=tcp -46:13.953 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.953 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11109 [Candidate]" term=2 -46:13.976 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.976 [DEBUG] TestAgent.server.raft: vote granted: from=184bda9b-443f-0085-b9ca-a59b3b97987e term=2 tally=1 -46:13.976 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.976 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11109 [Leader]" -46:13.976 [INFO] TestAgent.server: cluster leadership acquired -46:13.977 [INFO] TestAgent.server: New leader elected: payload=Node-184bda9b-443f-0085-b9ca-a59b3b97987e -46:13.988 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11109 -46:14.011 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-184bda9b-443f-0085-b9ca-a59b3b97987e -46:14.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-184bda9b-443f-0085-b9ca-a59b3b97987e -46:14.056 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.180 [DEBUG] TestAgent.dns: request served from client: question="{2.4.2.4.2.4.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. 255 1}" latency=86.855µs client=127.0.0.1:53395 client_network=udp -46:14.180 [INFO] TestAgent: Requesting shutdown -46:14.180 [INFO] TestAgent.server: shutting down server -46:14.180 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.180 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.180 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.180 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.180 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.180 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.180 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.180 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.180 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.180 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.180 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.184 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.191Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.191Z [INFO] agent.router.manager: shutting down -46:14.191 [INFO] TestAgent: consul server down -46:14.191 [INFO] TestAgent: shutdown complete -46:14.191 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11034 network=tcp -46:14.191 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11034 network=udp -46:14.191 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11035 network=tcp -46:14.691 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.691 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ReverseLookup_IPV6 (0.86s) -=== CONT TestConfig_Apply_ProxyDefaultsExpose -2022-11-07T22:46:13.893Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.908Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.942 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55e24c73-3e40-40bf-2657-f4466c2264ec Address:127.0.0.1:11130}]" -46:13.942 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11130 [Follower]" leader= -46:13.943 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55e24c73-3e40-40bf-2657-f4466c2264ec.dc1 127.0.0.1 -46:13.943 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55e24c73-3e40-40bf-2657-f4466c2264ec 127.0.0.1 -2022-11-07T22:46:13.943Z [INFO] agent.router: Initializing LAN area manager -46:13.943 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55e24c73-3e40-40bf-2657-f4466c2264ec.dc1 area=wan -46:13.943 [INFO] TestAgent.server: Adding LAN server: server="Node-55e24c73-3e40-40bf-2657-f4466c2264ec (Addr: tcp/127.0.0.1:11130) (DC: dc1)" -46:13.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11027 network=udp -46:13.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11027 network=tcp -46:13.944 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11028 network=tcp -46:13.944 [INFO] TestAgent: started state syncer -46:13.944 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11131 network=tcp -46:14.005 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.005 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11130 [Candidate]" term=2 -46:14.017 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.018 [DEBUG] TestAgent.server.raft: vote granted: from=55e24c73-3e40-40bf-2657-f4466c2264ec term=2 tally=1 -46:14.018 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.018 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11130 [Leader]" -46:14.018 [INFO] TestAgent.server: cluster leadership acquired -46:14.018 [INFO] TestAgent.server: New leader elected: payload=Node-55e24c73-3e40-40bf-2657-f4466c2264ec -46:14.031 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11130 -46:14.057 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.078 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.078 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.078 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.078 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.078 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.078 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55e24c73-3e40-40bf-2657-f4466c2264ec -46:14.078 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55e24c73-3e40-40bf-2657-f4466c2264ec -46:14.090 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.214 [INFO] TestAgent.server.connect: CA provider config updated -46:14.215 [INFO] TestAgent: Requesting shutdown -46:14.215 [INFO] TestAgent.server: shutting down server -46:14.215 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.215 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.215 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.215 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.215 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.215 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.215 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.215 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.215 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.219 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.225Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.226Z [INFO] agent.router.manager: shutting down -46:14.226 [INFO] TestAgent: consul server down -46:14.226 [INFO] TestAgent: shutdown complete -46:14.226 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11027 network=tcp -46:14.226 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11027 network=udp -46:14.226 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11028 network=tcp -46:14.726 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.726 [INFO] TestAgent: Endpoints down +=== CONT TestConfig_Apply_ProxyDefaultsMeshGateway +[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:27.513Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.528Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.588 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d60964ec-3e1d-e291-6570-575215f52198 Address:127.0.0.1:29482}]" +31:27.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d60964ec-3e1d-e291-6570-575215f52198.dc1 127.0.0.1 +31:27.589 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d60964ec-3e1d-e291-6570-575215f52198 127.0.0.1 +2021-10-05T16:31:27.589Z [INFO] agent.router: Initializing LAN area manager +31:27.589 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29491 network=udp +31:27.589 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29482 [Follower]" leader= +31:27.590 [INFO] TestAgent.server: Adding LAN server: server="Node-d60964ec-3e1d-e291-6570-575215f52198 (Addr: tcp/127.0.0.1:29482) (DC: dc1)" +31:27.590 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d60964ec-3e1d-e291-6570-575215f52198.dc1 area=wan +31:27.590 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29491 network=tcp +31:27.591 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29492 network=tcp +31:27.591 [INFO] TestAgent: started state syncer +31:27.591 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29483 network=tcp +31:27.652 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.652 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29482 [Candidate]" term=2 +31:27.705 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.705 [DEBUG] TestAgent.server.raft: vote granted: from=d60964ec-3e1d-e291-6570-575215f52198 term=2 tally=1 +31:27.705 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.705 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29482 [Leader]" +31:27.705 [INFO] TestAgent.server: cluster leadership acquired +31:27.705 [INFO] TestAgent.server: New leader elected: payload=Node-d60964ec-3e1d-e291-6570-575215f52198 +31:27.736 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29482 +31:27.797 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.846 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.846 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.846 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.846 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.846 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.846 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d60964ec-3e1d-e291-6570-575215f52198 +31:27.846 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d60964ec-3e1d-e291-6570-575215f52198 +31:27.865 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.904 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:27.928 [INFO] TestAgent: Synced node info +31:27.928 [INFO] TestAgent.server.connect: CA provider config updated +31:27.928 [INFO] TestAgent: Requesting shutdown +31:27.928 [INFO] TestAgent.server: shutting down server +31:27.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.928 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.928 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.928 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.928 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.928 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.936 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.939Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.939Z [INFO] agent.router.manager: shutting down +31:27.939 [INFO] TestAgent: consul server down +31:27.939 [INFO] TestAgent: shutdown complete +31:27.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29491 network=tcp +31:27.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29491 network=udp +31:27.939 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29492 network=tcp +31:28.440 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.440 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:27.228Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.303Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.358 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fd38bd65-2c08-8d80-e67b-a5d60860e663 Address:127.0.0.1:29461}]" +31:27.359 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fd38bd65-2c08-8d80-e67b-a5d60860e663.dc1 127.0.0.1 +31:27.359 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fd38bd65-2c08-8d80-e67b-a5d60860e663 127.0.0.1 +2021-10-05T16:31:27.359Z [INFO] agent.router: Initializing LAN area manager +31:27.360 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=udp +31:27.360 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29461 [Follower]" leader= +31:27.360 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fd38bd65-2c08-8d80-e67b-a5d60860e663.dc1 area=wan +31:27.360 [INFO] TestAgent.server: Adding LAN server: server="Node-fd38bd65-2c08-8d80-e67b-a5d60860e663 (Addr: tcp/127.0.0.1:29461) (DC: dc1)" +31:27.360 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=tcp +31:27.361 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29429 network=tcp +31:27.361 [INFO] TestAgent: started state syncer +31:27.361 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29462 network=tcp +31:27.430 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.430 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29461 [Candidate]" term=2 +31:27.468 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.468 [DEBUG] TestAgent.server.raft: vote granted: from=fd38bd65-2c08-8d80-e67b-a5d60860e663 term=2 tally=1 +31:27.468 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.468 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29461 [Leader]" +31:27.468 [INFO] TestAgent.server: cluster leadership acquired +31:27.468 [INFO] TestAgent.server: New leader elected: payload=Node-fd38bd65-2c08-8d80-e67b-a5d60860e663 +31:27.488 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29461 +31:27.509 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.553 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.553 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.553 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.553 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.553 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.553 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fd38bd65-2c08-8d80-e67b-a5d60860e663 +31:27.553 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fd38bd65-2c08-8d80-e67b-a5d60860e663 +31:27.584 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.765 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:27.784 [INFO] TestAgent: Synced node info +31:27.784 [DEBUG] TestAgent: Node info in sync +31:27.932 [INFO] TestAgent: Requesting shutdown +31:27.932 [INFO] TestAgent.server: shutting down server +31:27.932 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.932 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.932 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.932 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.932 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.932 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.939 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.940Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.940Z [INFO] agent.router.manager: shutting down +31:27.940 [INFO] TestAgent: consul server down +31:27.940 [INFO] TestAgent: shutdown complete +31:27.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=tcp +31:27.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=udp +31:27.940 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29429 network=tcp +31:28.440 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.440 [INFO] TestAgent: Endpoints down === RUN TestConnectCAConfig/basic_with_IntermediateCertTTL -2022-11-07T22:46:13.837Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.851Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.898 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf932bc5-df8c-4534-3419-e7ccbfbdde3d Address:127.0.0.1:11032}]" -46:13.898 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11032 [Follower]" leader= -46:13.899 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d.dc1 127.0.0.1 -46:13.899 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d 127.0.0.1 -2022-11-07T22:46:13.899Z [INFO] agent.router: Initializing LAN area manager -46:13.899 [INFO] TestAgent.server: Adding LAN server: server="Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d (Addr: tcp/127.0.0.1:11032) (DC: dc1)" -46:13.899 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d.dc1 area=wan -46:13.899 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11118 network=udp -46:13.899 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11118 network=tcp -46:13.900 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11119 network=tcp -46:13.900 [INFO] TestAgent: started state syncer -46:13.900 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11033 network=tcp -46:13.942 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.942 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11032 [Candidate]" term=2 -46:13.960 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.960 [DEBUG] TestAgent.server.raft: vote granted: from=bf932bc5-df8c-4534-3419-e7ccbfbdde3d term=2 tally=1 -46:13.960 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.960 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11032 [Leader]" -46:13.961 [INFO] TestAgent.server: cluster leadership acquired -46:13.961 [INFO] TestAgent.server: New leader elected: payload=Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d -46:13.973 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11032 -46:14.000 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.030 [INFO] TestAgent: Synced node info -46:14.043 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.043 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.043 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.043 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.043 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.043 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d -46:14.043 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bf932bc5-df8c-4534-3419-e7ccbfbdde3d -46:14.060 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.109 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.109 [DEBUG] TestAgent: Node info in sync -46:14.109 [DEBUG] TestAgent: Node info in sync -46:14.246 [INFO] TestAgent: Requesting shutdown -46:14.246 [INFO] TestAgent.server: shutting down server -46:14.246 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.246 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.246 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.246 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.246 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.246 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.246 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.246 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.246 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.246 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.255 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.263Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.263Z [INFO] agent.router.manager: shutting down -46:14.263 [INFO] TestAgent: consul server down -46:14.263 [INFO] TestAgent: shutdown complete -46:14.263 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11118 network=tcp -46:14.263 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11118 network=udp -46:14.263 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11119 network=tcp -46:14.763 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.763 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes_Filter (0.94s) -=== CONT TestDNS_EDNS0 -=== RUN TestDNS_EDNS0_ECS/global -=== RUN TestDNS_EDNS0_ECS/query -=== RUN TestDNS_EDNS0_ECS/query-subnet -2022-11-07T22:46:13.912Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.921Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.969 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 Address:127.0.0.1:11123}]" -46:13.969 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11123 [Follower]" leader= -46:13.970 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9.dc1 127.0.0.1 -46:13.970 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 127.0.0.1 -2022-11-07T22:46:13.970Z [INFO] agent.router: Initializing LAN area manager -46:13.971 [INFO] TestAgent.server: Adding LAN server: server="Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 (Addr: tcp/127.0.0.1:11123) (DC: dc1)" -46:13.971 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9.dc1 area=wan -46:13.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11132 network=tcp -46:13.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11132 network=udp -46:13.971 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11133 network=tcp -46:13.971 [INFO] TestAgent: started state syncer -46:13.971 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11124 network=tcp -46:14.015 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.015 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11123 [Candidate]" term=2 -46:14.030 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.030 [DEBUG] TestAgent.server.raft: vote granted: from=8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 term=2 tally=1 -46:14.030 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.030 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11123 [Leader]" -46:14.030 [INFO] TestAgent.server: cluster leadership acquired -46:14.030 [INFO] TestAgent.server: New leader elected: payload=Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 -46:14.042 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11123 -46:14.060 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.088 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.088 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.088 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.088 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.088 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.088 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 -46:14.088 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8db2b7fe-eea9-ffd6-4029-a1e4afd6e3d9 -46:14.097 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.248 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.252 [INFO] TestAgent: Synced node info -46:14.296 [INFO] TestAgent: Requesting shutdown -46:14.296 [INFO] TestAgent.server: shutting down server -46:14.296 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.296 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.296 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.296 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.296 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.296 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.296 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.296 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.296 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.296 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.302 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.309Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.309Z [INFO] agent.router.manager: shutting down -46:14.309 [INFO] TestAgent: consul server down -46:14.309 [INFO] TestAgent: shutdown complete -46:14.309 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11132 network=tcp -46:14.309 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11132 network=udp -46:14.309 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11133 network=tcp -46:14.810 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.810 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes_MetaFilter (0.91s) -=== CONT TestCatalogRegister_Service_InvalidAddress +--- PASS: TestCoordinate_Nodes (1.27s) +=== CONT TestAgent_RegisterCheck +2021-10-05T16:31:27.563Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.581Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.621 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:006a8702-f059-b424-d29b-5e0d8ad6e1a4 Address:127.0.0.1:29538}]" +31:27.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4.dc1 127.0.0.1 +31:27.622 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4 127.0.0.1 +2021-10-05T16:31:27.622Z [INFO] agent.router: Initializing LAN area manager +31:27.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29498 network=udp +31:27.623 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29538 [Follower]" leader= +31:27.623 [INFO] TestAgent.server: Adding LAN server: server="Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4 (Addr: tcp/127.0.0.1:29538) (DC: dc1)" +31:27.623 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4.dc1 area=wan +31:27.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29498 network=tcp +31:27.624 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29499 network=tcp +31:27.624 [INFO] TestAgent: started state syncer +31:27.624 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29539 network=tcp +31:27.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29538 [Candidate]" term=2 +31:27.725 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.725 [DEBUG] TestAgent.server.raft: vote granted: from=006a8702-f059-b424-d29b-5e0d8ad6e1a4 term=2 tally=1 +31:27.725 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.725 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29538 [Leader]" +31:27.725 [INFO] TestAgent.server: cluster leadership acquired +31:27.725 [INFO] TestAgent.server: New leader elected: payload=Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4 +31:27.745 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29538 +31:27.799 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.851 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.851 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.851 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.851 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.851 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.851 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4 +31:27.851 [INFO] TestAgent.server: member joined, marking health alive: member=Node-006a8702-f059-b424-d29b-5e0d8ad6e1a4 +31:27.865 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.891 [DEBUG] connect.ca.consul: consul CA provider configured: id=27:7c:55:cf:7a:2f:48:78:c2:11:86:09:77:47:54:e1:7d:e9:6d:21:08:86:23:c9:9f:dd:f8:20:88:11:79:9a is_primary=true +31:27.950 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul +31:27.950 [INFO] TestAgent: Requesting shutdown +31:27.950 [INFO] TestAgent.server: shutting down server +31:27.950 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.950 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.950 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.950 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.950 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.950 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.950 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:27.950 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:27.950 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.950 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.950 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.960 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:27.961Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:27.961Z [INFO] agent.router.manager: shutting down +31:27.961 [INFO] TestAgent: consul server down +31:27.961 [INFO] TestAgent: shutdown complete +31:27.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29498 network=tcp +31:27.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29498 network=udp +31:27.961 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29499 network=tcp +31:28.461 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.461 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCARoots_list (0.92s) +=== CONT TestRemoteExecWrites_ACLDeny +=== RUN TestConfig_Apply_Decoding/No_Kind +=== RUN TestConfig_Apply_Decoding/Kind_Not_String +=== RUN TestConfig_Apply_Decoding/Lowercase_kind +2021-10-05T16:31:27.227Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.266Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d9149e0b-776e-c077-c289-b78949b5e1ae Address:127.0.0.1:29419}]" +31:27.321 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29419 [Follower]" leader= +31:27.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d9149e0b-776e-c077-c289-b78949b5e1ae.dc1 127.0.0.1 +31:27.321 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d9149e0b-776e-c077-c289-b78949b5e1ae 127.0.0.1 +2021-10-05T16:31:27.322Z [INFO] agent.router: Initializing LAN area manager +31:27.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=udp +31:27.322 [INFO] TestAgent.server: Adding LAN server: server="Node-d9149e0b-776e-c077-c289-b78949b5e1ae (Addr: tcp/127.0.0.1:29419) (DC: dc1)" +31:27.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d9149e0b-776e-c077-c289-b78949b5e1ae.dc1 area=wan +31:27.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=tcp +31:27.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29422 network=tcp +31:27.323 [INFO] TestAgent: started state syncer +31:27.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29420 network=tcp +31:27.366 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.366 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29419 [Candidate]" term=2 +31:27.400 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.400 [DEBUG] TestAgent.server.raft: vote granted: from=d9149e0b-776e-c077-c289-b78949b5e1ae term=2 tally=1 +31:27.400 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.400 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29419 [Leader]" +31:27.401 [INFO] TestAgent.server: cluster leadership acquired +31:27.401 [INFO] TestAgent.server: New leader elected: payload=Node-d9149e0b-776e-c077-c289-b78949b5e1ae +31:27.412 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29419 +31:27.440 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:27.492 [INFO] TestAgent: Synced node info +31:27.492 [DEBUG] TestAgent: Node info in sync +31:27.492 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:27.492 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.492 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.492 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:27.492 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:27.492 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d9149e0b-776e-c077-c289-b78949b5e1ae +31:27.492 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d9149e0b-776e-c077-c289-b78949b5e1ae +31:27.518 [INFO] TestAgent.server: federation state anti-entropy synced +31:27.992 [INFO] TestAgent: Requesting shutdown +31:27.992 [INFO] TestAgent.server: shutting down server +31:27.992 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:27.992 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:27.992 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:27.992 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:27.992 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:27.992 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:27.992 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:27.992 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:27.992 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:27.992 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.000 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.004Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.004Z [INFO] agent.router.manager: shutting down +31:28.004 [INFO] TestAgent: consul server down +31:28.004 [INFO] TestAgent: shutdown complete +31:28.004 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=tcp +31:28.004 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=udp +31:28.004 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29422 network=tcp +31:28.504 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.504 [INFO] TestAgent: Endpoints down +--- PASS: TestCoordinate_Node (1.35s) +=== CONT TestSessionGet +=== RUN TestSessionGet/#00 +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:27.661Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:27.732Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:27.833 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76a5b9aa-73b0-67e4-1725-a63d719acc88 Address:127.0.0.1:29496}]" +31:27.834 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76a5b9aa-73b0-67e4-1725-a63d719acc88.dc1 127.0.0.1 +31:27.835 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76a5b9aa-73b0-67e4-1725-a63d719acc88 127.0.0.1 +2021-10-05T16:31:27.835Z [INFO] agent.router: Initializing LAN area manager +31:27.835 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29540 network=udp +31:27.835 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29496 [Follower]" leader= +31:27.835 [INFO] TestAgent.server: Adding LAN server: server="Node-76a5b9aa-73b0-67e4-1725-a63d719acc88 (Addr: tcp/127.0.0.1:29496) (DC: dc1)" +31:27.836 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76a5b9aa-73b0-67e4-1725-a63d719acc88.dc1 area=wan +31:27.836 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29540 network=tcp +31:27.836 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29541 network=tcp +31:27.836 [INFO] TestAgent: started state syncer +31:27.837 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29497 network=tcp +31:27.880 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:27.880 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29496 [Candidate]" term=2 +31:27.904 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:27.904 [DEBUG] TestAgent.server.raft: vote granted: from=76a5b9aa-73b0-67e4-1725-a63d719acc88 term=2 tally=1 +31:27.904 [INFO] TestAgent.server.raft: election won: tally=1 +31:27.904 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29496 [Leader]" +31:27.904 [INFO] TestAgent.server: cluster leadership acquired +31:27.904 [INFO] TestAgent.server: New leader elected: payload=Node-76a5b9aa-73b0-67e4-1725-a63d719acc88 +31:27.924 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29496 +31:27.928 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:27.928 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:27.928 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76a5b9aa-73b0-67e4-1725-a63d719acc88 +31:27.928 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76a5b9aa-73b0-67e4-1725-a63d719acc88 +31:27.949 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.112 [INFO] TestAgent: Requesting shutdown +31:28.112 [INFO] TestAgent.server: shutting down server +31:28.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.112 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.112 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.112 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:28.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.120 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.120Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.120Z [INFO] agent.router.manager: shutting down +31:28.120 [INFO] TestAgent: consul server down +31:28.120 [INFO] TestAgent: shutdown complete +31:28.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29540 network=tcp +31:28.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29540 network=udp +31:28.120 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29541 network=tcp +31:28.621 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.621 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCARoots_empty (1.00s) +=== CONT TestSessionCustomTTL +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:28.152Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.187Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.222 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:19d8aa04-86f3-0633-d79a-b52b4ff78ed7 Address:127.0.0.1:29531}]" +31:28.223 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7.dc1 127.0.0.1 +31:28.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7 127.0.0.1 +2021-10-05T16:31:28.224Z [INFO] agent.router: Initializing LAN area manager +31:28.224 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29512 network=udp +31:28.224 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29531 [Follower]" leader= +31:28.224 [INFO] TestAgent.server: Adding LAN server: server="Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7 (Addr: tcp/127.0.0.1:29531) (DC: dc1)" +31:28.224 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7.dc1 area=wan +31:28.224 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29512 network=tcp +31:28.225 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29513 network=tcp +31:28.225 [INFO] TestAgent: started state syncer +31:28.225 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29532 network=tcp +31:28.268 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.268 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29531 [Candidate]" term=2 +31:28.300 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.300 [DEBUG] TestAgent.server.raft: vote granted: from=19d8aa04-86f3-0633-d79a-b52b4ff78ed7 term=2 tally=1 +31:28.300 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.300 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29531 [Leader]" +31:28.300 [INFO] TestAgent.server: cluster leadership acquired +31:28.300 [INFO] TestAgent.server: New leader elected: payload=Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7 +31:28.327 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29531 +31:28.349 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.381 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.381 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.381 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.381 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.381 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.381 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7 +31:28.381 [INFO] TestAgent.server: member joined, marking health alive: member=Node-19d8aa04-86f3-0633-d79a-b52b4ff78ed7 +31:28.424 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.453 [INFO] TestAgent: Requesting shutdown +31:28.453 [INFO] TestAgent.server: shutting down server +31:28.453 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.453 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.453 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.453 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.454 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.454 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.454 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.454 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:28.454 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.454 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:28.454 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:28.457 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.457Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.457Z [INFO] agent.router.manager: shutting down +31:28.457 [INFO] TestAgent: consul server down +31:28.457 [INFO] TestAgent: shutdown complete +31:28.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29512 network=tcp +31:28.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29512 network=udp +31:28.457 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29513 network=tcp +31:28.960 [INFO] TestAgent: Waiting for endpoints to shut down +31:28.960 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_ProxyDefaultsExpose (0.88s) +=== CONT TestSessionDestroy +2021-10-05T16:31:28.170Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.184Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.223 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab054159-2ad5-d838-7c04-0b4ac75cb4bd Address:127.0.0.1:29524}]" +31:28.224 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd.dc1 127.0.0.1 +31:28.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd 127.0.0.1 +2021-10-05T16:31:28.224Z [INFO] agent.router: Initializing LAN area manager +31:28.225 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29533 network=udp +31:28.225 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29524 [Follower]" leader= +31:28.225 [INFO] TestAgent.server: Adding LAN server: server="Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd (Addr: tcp/127.0.0.1:29524) (DC: dc1)" +31:28.225 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd.dc1 area=wan +31:28.225 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29533 network=tcp +31:28.226 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29534 network=tcp +31:28.226 [INFO] TestAgent: started state syncer +31:28.226 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29525 network=tcp +31:28.283 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.283 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29524 [Candidate]" term=2 +31:28.316 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.316 [DEBUG] TestAgent.server.raft: vote granted: from=ab054159-2ad5-d838-7c04-0b4ac75cb4bd term=2 tally=1 +31:28.316 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.316 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29524 [Leader]" +31:28.320 [INFO] TestAgent.server: cluster leadership acquired +31:28.320 [INFO] TestAgent.server: New leader elected: payload=Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd +31:28.333 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29524 +31:28.371 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.451 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.451 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.451 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.451 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.451 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.451 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd +31:28.451 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab054159-2ad5-d838-7c04-0b4ac75cb4bd +31:28.492 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.504 [INFO] TestAgent: Requesting shutdown +31:28.505 [INFO] TestAgent.server: shutting down server +31:28.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.505 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.505 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.505 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.505 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.505 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:28.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.505 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:28.505 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:28.516 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.516Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.516Z [INFO] agent.router.manager: shutting down +31:28.516 [INFO] TestAgent: consul server down +31:28.516 [INFO] TestAgent: shutdown complete +31:28.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29533 network=tcp +31:28.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29533 network=udp +31:28.517 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29534 network=tcp +31:29.019 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.019 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_Decoding (0.91s) + --- PASS: TestConfig_Apply_Decoding/No_Kind (0.00s) + --- PASS: TestConfig_Apply_Decoding/Kind_Not_String (0.00s) + --- PASS: TestConfig_Apply_Decoding/Lowercase_kind (0.02s) +=== CONT TestSessionCreate_NoCheck +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:28.294Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.327Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.421 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0e704649-e321-4198-2713-15902349192a Address:127.0.0.1:29545}]" +31:28.422 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0e704649-e321-4198-2713-15902349192a.dc1 127.0.0.1 +31:28.422 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0e704649-e321-4198-2713-15902349192a 127.0.0.1 +2021-10-05T16:31:28.422Z [INFO] agent.router: Initializing LAN area manager +31:28.422 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29526 network=udp +31:28.422 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29545 [Follower]" leader= +31:28.423 [INFO] TestAgent.server: Adding LAN server: server="Node-0e704649-e321-4198-2713-15902349192a (Addr: tcp/127.0.0.1:29545) (DC: dc1)" +31:28.423 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0e704649-e321-4198-2713-15902349192a.dc1 area=wan +31:28.423 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29526 network=tcp +31:28.423 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29527 network=tcp +31:28.423 [INFO] TestAgent: started state syncer +31:28.424 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29546 network=tcp +31:28.467 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.467 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29545 [Candidate]" term=2 +31:28.520 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.520 [DEBUG] TestAgent.server.raft: vote granted: from=0e704649-e321-4198-2713-15902349192a term=2 tally=1 +31:28.520 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.520 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29545 [Leader]" +31:28.524 [INFO] TestAgent.server: cluster leadership acquired +31:28.524 [INFO] TestAgent.server: New leader elected: payload=Node-0e704649-e321-4198-2713-15902349192a +31:28.552 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29545 +31:28.652 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.792 [INFO] TestAgent: Synced node info +31:28.792 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.792 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.792 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.792 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.792 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.793 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0e704649-e321-4198-2713-15902349192a +31:28.793 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0e704649-e321-4198-2713-15902349192a +31:28.822 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.858 [INFO] TestAgent: Requesting shutdown +31:28.858 [INFO] TestAgent.server: shutting down server +31:28.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.858 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.858 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.858 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.858 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.858 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:28.858 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:28.872 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.876Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.876Z [INFO] agent.router.manager: shutting down +31:28.876 [INFO] TestAgent: consul server down +31:28.876 [INFO] TestAgent: shutdown complete +31:28.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29526 network=tcp +31:28.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29526 network=udp +31:28.876 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29527 network=tcp +31:29.377 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.377 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_CAS (1.10s) +=== CONT TestSessionCreate_DefaultCheck +2021-10-05T16:31:28.303Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.319Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.417 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:910fc166-34b7-82fb-e43f-f8c58de8a5c8 Address:127.0.0.1:29552}]" +31:28.417 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8.dc1 127.0.0.1 +31:28.418 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8 127.0.0.1 +2021-10-05T16:31:28.418Z [INFO] agent.router: Initializing LAN area manager +31:28.418 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29547 network=udp +31:28.418 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29552 [Follower]" leader= +31:28.419 [INFO] TestAgent.server: Adding LAN server: server="Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8 (Addr: tcp/127.0.0.1:29552) (DC: dc1)" +31:28.419 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8.dc1 area=wan +31:28.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29547 network=tcp +31:28.419 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29548 network=tcp +31:28.420 [INFO] TestAgent: started state syncer +31:28.420 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29553 network=tcp +31:28.468 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.472 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29552 [Candidate]" term=2 +31:28.516 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.516 [DEBUG] TestAgent.server.raft: vote granted: from=910fc166-34b7-82fb-e43f-f8c58de8a5c8 term=2 tally=1 +31:28.516 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.516 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29552 [Leader]" +31:28.517 [INFO] TestAgent.server: cluster leadership acquired +31:28.517 [INFO] TestAgent.server: New leader elected: payload=Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8 +31:28.544 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29552 +31:28.649 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.769 [INFO] TestAgent: Synced node info +31:28.796 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.796 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.796 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.796 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.796 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.796 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8 +31:28.796 [INFO] TestAgent.server: member joined, marking health alive: member=Node-910fc166-34b7-82fb-e43f-f8c58de8a5c8 +31:28.853 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.866 [INFO] TestAgent: Requesting shutdown +31:28.866 [INFO] TestAgent.server: shutting down server +31:28.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.866 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.866 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.866 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.866 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.866 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:28.866 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:28.877 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:28.881Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:28.881Z [INFO] agent.router.manager: shutting down +31:28.881 [INFO] TestAgent: consul server down +31:28.881 [INFO] TestAgent: shutdown complete +31:28.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29547 network=tcp +31:28.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29547 network=udp +31:28.881 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29548 network=tcp +31:29.382 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.382 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_ProxyDefaultsMeshGateway (1.09s) +=== CONT TestSessionCreate_Delete +=== RUN TestSessionCreate_NoCheck/no_check_fields_should_yield_default_serfHealth +=== RUN TestSessionCreate_NoCheck/overwrite_nodechecks_to_associate_with_no_checks +=== RUN TestSessionCreate_NoCheck/overwrite_checks_to_associate_with_no_checks +2021-10-05T16:31:28.472Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.498Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.676 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1536c9e2-319f-522b-59ce-3f363989a2b3 Address:127.0.0.1:29566}]" +31:28.677 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1536c9e2-319f-522b-59ce-3f363989a2b3.dc1 127.0.0.1 +31:28.678 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1536c9e2-319f-522b-59ce-3f363989a2b3 127.0.0.1 +31:28.680 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29566 [Follower]" leader= +2021-10-05T16:31:28.684Z [INFO] agent.router: Initializing LAN area manager +31:28.684 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29554 network=udp +31:28.684 [INFO] TestAgent.server: Adding LAN server: server="Node-1536c9e2-319f-522b-59ce-3f363989a2b3 (Addr: tcp/127.0.0.1:29566) (DC: dc1)" +31:28.684 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1536c9e2-319f-522b-59ce-3f363989a2b3.dc1 area=wan +31:28.684 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29554 network=tcp +31:28.685 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29555 network=tcp +31:28.685 [INFO] TestAgent: started state syncer +31:28.685 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29567 network=tcp +31:28.748 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.748 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29566 [Candidate]" term=2 +31:28.797 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.797 [DEBUG] TestAgent.server.raft: vote granted: from=1536c9e2-319f-522b-59ce-3f363989a2b3 term=2 tally=1 +31:28.797 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.797 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29566 [Leader]" +31:28.799 [INFO] TestAgent.server: cluster leadership acquired +31:28.799 [INFO] TestAgent.server: New leader elected: payload=Node-1536c9e2-319f-522b-59ce-3f363989a2b3 +31:28.819 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29566 +31:28.855 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.937 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.937 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.937 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.937 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.937 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.937 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1536c9e2-319f-522b-59ce-3f363989a2b3 +31:28.937 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1536c9e2-319f-522b-59ce-3f363989a2b3 +31:28.973 [INFO] TestAgent: Synced node info +31:28.973 [DEBUG] TestAgent: Node info in sync +31:28.989 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.994 [INFO] TestAgent.server.connect: CA provider config updated +31:28.994 [INFO] TestAgent: Requesting shutdown +31:28.994 [INFO] TestAgent.server: shutting down server +31:28.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.994 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.994 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.994 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.994 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.994 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.994 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.994 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:28.994 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.005 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.006Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.006Z [INFO] agent.router.manager: shutting down +31:29.006 [INFO] TestAgent: consul server down +31:29.006 [INFO] TestAgent: shutdown complete +31:29.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29554 network=tcp +31:29.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29554 network=udp +31:29.006 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29555 network=tcp +31:29.507 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.507 [INFO] TestAgent: Endpoints down +=== RUN TestConnectCAConfig/force_without_cross_sign_CamelCase +2021-10-05T16:31:28.462Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.485Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.639 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:59e400d4-d321-576c-0976-b5d65a71af36 Address:127.0.0.1:29559}]" +31:28.640 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29559 [Follower]" leader= +31:28.640 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-59e400d4-d321-576c-0976-b5d65a71af36.dc1 127.0.0.1 +31:28.647 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-59e400d4-d321-576c-0976-b5d65a71af36 127.0.0.1 +2021-10-05T16:31:28.647Z [INFO] agent.router: Initializing LAN area manager +31:28.647 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29568 network=udp +31:28.648 [INFO] TestAgent.server: Adding LAN server: server="Node-59e400d4-d321-576c-0976-b5d65a71af36 (Addr: tcp/127.0.0.1:29559) (DC: dc1)" +31:28.648 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-59e400d4-d321-576c-0976-b5d65a71af36.dc1 area=wan +31:28.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29568 network=tcp +31:28.648 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29569 network=tcp +31:28.648 [INFO] TestAgent: started state syncer +31:28.648 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29560 network=tcp +31:28.692 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.692 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29559 [Candidate]" term=2 +31:28.765 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.765 [DEBUG] TestAgent.server.raft: vote granted: from=59e400d4-d321-576c-0976-b5d65a71af36 term=2 tally=1 +31:28.765 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.765 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29559 [Leader]" +31:28.765 [INFO] TestAgent.server: cluster leadership acquired +31:28.766 [INFO] TestAgent.server: New leader elected: payload=Node-59e400d4-d321-576c-0976-b5d65a71af36 +31:28.800 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29559 +31:28.854 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:28.940 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:28.940 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:28.940 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:28.940 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:28.940 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:28.940 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-59e400d4-d321-576c-0976-b5d65a71af36 +31:28.940 [INFO] TestAgent.server: member joined, marking health alive: member=Node-59e400d4-d321-576c-0976-b5d65a71af36 +31:28.944 [INFO] TestAgent: Synced node info +31:28.944 [DEBUG] TestAgent: Node info in sync +31:28.977 [INFO] TestAgent.server: federation state anti-entropy synced +31:28.993 [DEBUG] TestAgent: Node info in sync +31:28.999 [INFO] TestAgent: Synced check: check=test +31:28.999 [INFO] TestAgent: Requesting shutdown +31:28.999 [INFO] TestAgent.server: shutting down server +31:28.999 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:28.999 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:28.999 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:28.999 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:28.999 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:28.999 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:28.999 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:28.999 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:28.999 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:28.999 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.009 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.010Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.010Z [INFO] agent.router.manager: shutting down +31:29.011 [INFO] TestAgent: consul server down +31:29.011 [INFO] TestAgent: shutdown complete +31:29.011 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29568 network=tcp +31:29.011 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29568 network=udp +31:29.011 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29569 network=tcp +31:29.512 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.512 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheck (1.07s) +=== CONT TestSessionCreate_NodeChecks +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:28.542Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.592Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.752 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9d913b32-22ca-8236-3b9a-1ac88caa90d8 Address:127.0.0.1:29573}]" +31:28.753 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8.dc1 127.0.0.1 +31:28.754 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8 127.0.0.1 +2021-10-05T16:31:28.754Z [INFO] agent.router: Initializing LAN area manager +31:28.754 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29589 network=udp +31:28.754 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29573 [Follower]" leader= +31:28.755 [INFO] TestAgent.server: Adding LAN server: server="Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8 (Addr: tcp/127.0.0.1:29573) (DC: dc1)" +31:28.755 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8.dc1 area=wan +31:28.755 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29589 network=tcp +31:28.757 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29590 network=tcp +31:28.757 [INFO] TestAgent: started state syncer +31:28.757 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29574 network=tcp +31:28.815 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.815 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29573 [Candidate]" term=2 +31:28.864 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.864 [DEBUG] TestAgent.server.raft: vote granted: from=9d913b32-22ca-8236-3b9a-1ac88caa90d8 term=2 tally=1 +31:28.864 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.864 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29573 [Leader]" +31:28.864 [INFO] TestAgent.server: cluster leadership acquired +31:28.864 [INFO] TestAgent.server: New leader elected: payload=Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8 +31:28.908 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29573 +31:28.965 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.025 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.025 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.025 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.025 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.025 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.025 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8 +31:29.025 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9d913b32-22ca-8236-3b9a-1ac88caa90d8 +31:29.061 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.080 [INFO] TestAgent: Requesting shutdown +31:29.080 [INFO] TestAgent.server: shutting down server +31:29.080 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.080 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.080 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.080 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.080 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.080 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.080 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.080 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:29.080 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.080 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.080 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.087 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.089Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.089Z [INFO] agent.router.manager: shutting down +31:29.089 [INFO] TestAgent: consul server down +31:29.089 [INFO] TestAgent: shutdown complete +31:29.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29589 network=tcp +31:29.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29589 network=udp +31:29.089 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29590 network=tcp +31:29.590 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.590 [INFO] TestAgent: Endpoints down +=== RUN TestSessionGet/#01 +2021-10-05T16:31:28.479Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:28.479Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.509Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:28.509Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.695 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f9213833-408b-286a-d5d0-3fc7ad25b2ba Address:127.0.0.1:29587}]" +31:28.695 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29587 [Follower]" leader= +31:28.710 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 127.0.0.1 +31:28.710 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba 127.0.0.1 +2021-10-05T16:31:28.710Z [INFO] agent.router: Initializing LAN area manager +31:28.710 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29561 network=udp +31:28.711 [INFO] TestAgent.server: Adding LAN server: server="Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba (Addr: tcp/127.0.0.1:29587) (DC: dc1)" +31:28.711 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 area=wan +31:28.711 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29561 network=tcp +31:28.711 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29562 network=tcp +31:28.711 [INFO] TestAgent: started state syncer +31:28.712 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29588 network=tcp +31:28.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29587 [Candidate]" term=2 +31:28.764 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:28.792 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.792 [DEBUG] TestAgent.server.raft: vote granted: from=f9213833-408b-286a-d5d0-3fc7ad25b2ba term=2 tally=1 +31:28.792 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.792 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29587 [Leader]" +31:28.792 [INFO] TestAgent.server: cluster leadership acquired +31:28.793 [INFO] TestAgent.server: New leader elected: payload=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:28.802 [INFO] TestAgent.server: initializing acls +31:28.815 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:28.815 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:28.832 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:28.864 [INFO] TestAgent.server: initializing acls +31:28.864 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:28.892 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:28.892 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:28.892 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:28.892 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:28.892 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:28.892 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 +31:28.892 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 area=wan +31:28.892 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:28.892 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:28.893 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 +31:28.894 [INFO] TestAgent.server: Updating LAN server: server="Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba (Addr: tcp/127.0.0.1:29587) (DC: dc1)" +31:28.894 [INFO] TestAgent.server: Updating LAN server: server="Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba (Addr: tcp/127.0.0.1:29587) (DC: dc1)" +31:28.894 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba.dc1 area=wan +31:28.916 [WARN] TestAgent: Node info update blocked by ACLs: node=f9213833-408b-286a-d5d0-3fc7ad25b2ba accessorID=00000000-0000-0000-0000-000000000002 +31:28.920 [WARN] TestAgent: Node info update blocked by ACLs: node=f9213833-408b-286a-d5d0-3fc7ad25b2ba accessorID=00000000-0000-0000-0000-000000000002 +31:28.920 [DEBUG] TestAgent: Node info in sync +31:28.944 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.020 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.020 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.020 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.020 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.020 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.020 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.020 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.037 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.037 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.053 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.142 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.142 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f9213833-408b-286a-d5d0-3fc7ad25b2ba +31:29.145 [ERROR] TestAgent: failed to ack remote exec job: error="Permission denied" +31:29.146 [ERROR] TestAgent: failed to write output for remote exec job: error="Permission denied" +31:29.146 [ERROR] TestAgent: failed to write output for remote exec job: error="Permission denied" +31:29.164 [INFO] TestAgent: Requesting shutdown +31:29.164 [INFO] TestAgent.server: shutting down server +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.164 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.164 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.164 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.164 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.176 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.180Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.180Z [INFO] agent.router.manager: shutting down +31:29.180 [INFO] TestAgent: consul server down +31:29.180 [INFO] TestAgent: shutdown complete +31:29.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29561 network=tcp +31:29.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29561 network=udp +31:29.180 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29562 network=tcp +31:29.682 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.682 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecWrites_ACLDeny (1.22s) +=== CONT TestSessionCreate +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:28.986Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.026Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.107 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ca75bd2-08b4-876c-babe-cd4efab115eb Address:127.0.0.1:29510}]" +31:29.108 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29510 [Follower]" leader= +31:29.108 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1ca75bd2-08b4-876c-babe-cd4efab115eb.dc1 127.0.0.1 +31:29.109 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1ca75bd2-08b4-876c-babe-cd4efab115eb 127.0.0.1 +2021-10-05T16:31:29.109Z [INFO] agent.router: Initializing LAN area manager +31:29.109 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29582 network=udp +31:29.109 [INFO] TestAgent.server: Adding LAN server: server="Node-1ca75bd2-08b4-876c-babe-cd4efab115eb (Addr: tcp/127.0.0.1:29510) (DC: dc1)" +31:29.110 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1ca75bd2-08b4-876c-babe-cd4efab115eb.dc1 area=wan +31:29.110 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29582 network=tcp +31:29.110 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29583 network=tcp +31:29.110 [INFO] TestAgent: started state syncer +31:29.110 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29511 network=tcp +31:29.169 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.169 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29510 [Candidate]" term=2 +31:29.193 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.193 [DEBUG] TestAgent.server.raft: vote granted: from=1ca75bd2-08b4-876c-babe-cd4efab115eb term=2 tally=1 +31:29.193 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.193 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29510 [Leader]" +31:29.193 [INFO] TestAgent.server: cluster leadership acquired +31:29.193 [INFO] TestAgent.server: New leader elected: payload=Node-1ca75bd2-08b4-876c-babe-cd4efab115eb +31:29.210 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29510 +31:29.236 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.276 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.276 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.276 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.276 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.276 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.276 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ca75bd2-08b4-876c-babe-cd4efab115eb +31:29.276 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1ca75bd2-08b4-876c-babe-cd4efab115eb +31:29.332 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.332 [INFO] TestAgent: Synced node info +31:29.332 [DEBUG] TestAgent: Node info in sync +31:29.355 [INFO] TestAgent: Requesting shutdown +31:29.355 [INFO] TestAgent.server: shutting down server +31:29.355 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.355 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.355 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.355 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.355 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.355 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.355 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.355 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.355 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.355 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.359 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.359Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.359Z [INFO] agent.router.manager: shutting down +31:29.359 [INFO] TestAgent: consul server down +31:29.359 [INFO] TestAgent: shutdown complete +31:29.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29582 network=tcp +31:29.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29582 network=udp +31:29.359 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29583 network=tcp +31:29.859 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.859 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionDestroy (0.90s) +=== CONT TestServiceManager_PersistService_ConfigFiles +2021-10-05T16:31:29.043Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.059Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.093 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:504a2983-7b88-c7da-87bc-8da3d52c0bb2 Address:127.0.0.1:29608}]" +31:29.094 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2.dc1 127.0.0.1 +31:29.095 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2 127.0.0.1 +2021-10-05T16:31:29.095Z [INFO] agent.router: Initializing LAN area manager +31:29.095 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29519 network=udp +31:29.095 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29608 [Follower]" leader= +31:29.095 [INFO] TestAgent.server: Adding LAN server: server="Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2 (Addr: tcp/127.0.0.1:29608) (DC: dc1)" +31:29.095 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2.dc1 area=wan +31:29.095 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29519 network=tcp +31:29.096 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29520 network=tcp +31:29.096 [INFO] TestAgent: started state syncer +31:29.096 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29609 network=tcp +31:29.142 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.142 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29608 [Candidate]" term=2 +31:29.153 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.153 [DEBUG] TestAgent.server.raft: vote granted: from=504a2983-7b88-c7da-87bc-8da3d52c0bb2 term=2 tally=1 +31:29.153 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.153 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29608 [Leader]" +31:29.153 [INFO] TestAgent.server: cluster leadership acquired +31:29.153 [INFO] TestAgent.server: New leader elected: payload=Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2 +31:29.164 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29608 +31:29.187 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.234 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.234 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.234 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.234 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.234 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.234 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2 +31:29.234 [INFO] TestAgent.server: member joined, marking health alive: member=Node-504a2983-7b88-c7da-87bc-8da3d52c0bb2 +31:29.258 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.463 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:29.471 [INFO] TestAgent: Synced node info +31:29.471 [DEBUG] TestAgent: Node info in sync +31:29.478 [INFO] TestAgent: Requesting shutdown +31:29.479 [INFO] TestAgent.server: shutting down server +31:29.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.479 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.479 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.479 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.479 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.479 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.479 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.486 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.488Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.488Z [INFO] agent.router.manager: shutting down +31:29.488 [INFO] TestAgent: consul server down +31:29.488 [INFO] TestAgent: shutdown complete +31:29.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29519 network=tcp +31:29.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29519 network=udp +31:29.488 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29520 network=tcp +31:29.988 [INFO] TestAgent: Waiting for endpoints to shut down +31:29.988 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCreate_NoCheck (0.97s) + --- PASS: TestSessionCreate_NoCheck/no_check_fields_should_yield_default_serfHealth (0.01s) + --- PASS: TestSessionCreate_NoCheck/overwrite_nodechecks_to_associate_with_no_checks (0.01s) + --- PASS: TestSessionCreate_NoCheck/overwrite_checks_to_associate_with_no_checks (0.01s) +=== CONT TestHandleRemoteExecFailed +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:29.399Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.422Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.454 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:951b5bb9-2ecb-adf2-01ed-481e83f9766e Address:127.0.0.1:29615}]" +31:29.455 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e.dc1 127.0.0.1 +31:29.455 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e 127.0.0.1 +2021-10-05T16:31:29.455Z [INFO] agent.router: Initializing LAN area manager +31:29.456 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29610 network=udp +31:29.456 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29615 [Follower]" leader= +31:29.456 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e.dc1 area=wan +31:29.456 [INFO] TestAgent.server: Adding LAN server: server="Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e (Addr: tcp/127.0.0.1:29615) (DC: dc1)" +31:29.456 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29610 network=tcp +31:29.457 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29611 network=tcp +31:29.457 [INFO] TestAgent: started state syncer +31:29.457 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29616 network=tcp +31:29.502 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.502 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29615 [Candidate]" term=2 +31:29.532 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.532 [DEBUG] TestAgent.server.raft: vote granted: from=951b5bb9-2ecb-adf2-01ed-481e83f9766e term=2 tally=1 +31:29.532 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.532 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29615 [Leader]" +31:29.532 [INFO] TestAgent.server: cluster leadership acquired +31:29.532 [INFO] TestAgent.server: New leader elected: payload=Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e +31:29.562 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29615 +31:29.604 [INFO] TestAgent: Synced node info +31:29.604 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.628 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.628 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.628 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.628 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.628 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.628 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e +31:29.628 [INFO] TestAgent.server: member joined, marking health alive: member=Node-951b5bb9-2ecb-adf2-01ed-481e83f9766e +31:29.668 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.688 [INFO] TestAgent: Requesting shutdown +31:29.688 [INFO] TestAgent.server: shutting down server +31:29.688 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.688 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.688 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.689 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.689 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.689 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.689 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.689 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.689 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.697 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.706Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.706Z [INFO] agent.router.manager: shutting down +31:29.706 [INFO] TestAgent: consul server down +31:29.706 [INFO] TestAgent: shutdown complete +31:29.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29610 network=tcp +31:29.706 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29610 network=udp +31:29.706 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29611 network=tcp +31:30.207 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.207 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCreate_DefaultCheck (0.83s) +=== CONT TestHandleRemoteExec +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:29.533Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.589 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:24cb4837-a49b-4262-331a-6e5a3a4ca269 Address:127.0.0.1:29629}]" +31:29.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-24cb4837-a49b-4262-331a-6e5a3a4ca269.dc1 127.0.0.1 +31:29.590 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-24cb4837-a49b-4262-331a-6e5a3a4ca269 127.0.0.1 +2021-10-05T16:31:29.590Z [INFO] agent.router: Initializing LAN area manager +31:29.592 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29629 [Follower]" leader= +31:29.592 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-24cb4837-a49b-4262-331a-6e5a3a4ca269.dc1 area=wan +31:29.592 [INFO] TestAgent.server: Adding LAN server: server="Node-24cb4837-a49b-4262-331a-6e5a3a4ca269 (Addr: tcp/127.0.0.1:29629) (DC: dc1)" +31:29.592 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29624 network=tcp +31:29.592 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29624 network=udp +31:29.593 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29625 network=tcp +31:29.593 [INFO] TestAgent: started state syncer +31:29.593 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29630 network=tcp +31:29.644 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.644 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29629 [Candidate]" term=2 +31:29.684 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.684 [DEBUG] TestAgent.server.raft: vote granted: from=24cb4837-a49b-4262-331a-6e5a3a4ca269 term=2 tally=1 +31:29.684 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.684 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29629 [Leader]" +31:29.688 [INFO] TestAgent.server: cluster leadership acquired +31:29.688 [INFO] TestAgent.server: New leader elected: payload=Node-24cb4837-a49b-4262-331a-6e5a3a4ca269 +31:29.712 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29629 +31:29.749 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.795 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.795 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.795 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.795 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.795 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.795 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24cb4837-a49b-4262-331a-6e5a3a4ca269 +31:29.795 [INFO] TestAgent.server: member joined, marking health alive: member=Node-24cb4837-a49b-4262-331a-6e5a3a4ca269 +31:29.819 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.918 [INFO] TestAgent: Requesting shutdown +31:29.918 [INFO] TestAgent.server: shutting down server +31:29.918 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.918 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.918 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.918 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.918 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.918 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.918 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.918 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:29.918 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.918 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.918 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.926 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.941Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.941Z [INFO] agent.router.manager: shutting down +31:29.941 [INFO] TestAgent: consul server down +31:29.941 [INFO] TestAgent: shutdown complete +31:29.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29624 network=tcp +31:29.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29624 network=udp +31:29.941 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29625 network=tcp +31:30.444 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.444 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCreate_NodeChecks (0.93s) +=== CONT TestUiIndex +2021-10-05T16:31:29.405Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.440Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.532 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04ab51e7-3354-77e8-b49d-6a4fa120ff51 Address:127.0.0.1:29601}]" +31:29.532 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29601 [Follower]" leader= +31:29.532 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51.dc1 127.0.0.1 +31:29.533 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51 127.0.0.1 +2021-10-05T16:31:29.533Z [INFO] agent.router: Initializing LAN area manager +31:29.533 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29617 network=udp +31:29.533 [INFO] TestAgent.server: Adding LAN server: server="Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51 (Addr: tcp/127.0.0.1:29601) (DC: dc1)" +31:29.533 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51.dc1 area=wan +31:29.534 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29617 network=tcp +31:29.534 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29618 network=tcp +31:29.534 [INFO] TestAgent: started state syncer +31:29.534 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29602 network=tcp +31:29.585 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.585 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29601 [Candidate]" term=2 +31:29.605 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.606 [DEBUG] TestAgent.server.raft: vote granted: from=04ab51e7-3354-77e8-b49d-6a4fa120ff51 term=2 tally=1 +31:29.606 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.606 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29601 [Leader]" +31:29.606 [INFO] TestAgent.server: cluster leadership acquired +31:29.606 [INFO] TestAgent.server: New leader elected: payload=Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51 +31:29.615 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29601 +31:29.622 [INFO] TestAgent: Synced node info +31:29.676 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.723 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.723 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.723 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.723 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.723 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.723 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51 +31:29.723 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04ab51e7-3354-77e8-b49d-6a4fa120ff51 +31:29.763 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.960 [INFO] TestAgent: Requesting shutdown +31:29.960 [INFO] TestAgent.server: shutting down server +31:29.960 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.960 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.960 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.960 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.960 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.960 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.960 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.960 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.960 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.960 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.970 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:29.974Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:29.974Z [INFO] agent.router.manager: shutting down +31:29.975 [INFO] TestAgent: consul server down +31:29.975 [INFO] TestAgent: shutdown complete +31:29.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29617 network=tcp +31:29.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29617 network=udp +31:29.975 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29618 network=tcp +31:30.475 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.475 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCreate_Delete (1.09s) +=== CONT TestFireReceiveEvent +2021-10-05T16:31:28.662Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:28.706Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:28.831 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3030616-3392-fad3-afb6-4a4f9b854ec5 Address:127.0.0.1:29580}]" +31:28.831 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3030616-3392-fad3-afb6-4a4f9b854ec5.dc1 127.0.0.1 +31:28.832 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29580 [Follower]" leader= +31:28.832 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3030616-3392-fad3-afb6-4a4f9b854ec5 127.0.0.1 +2021-10-05T16:31:28.832Z [INFO] agent.router: Initializing LAN area manager +31:28.833 [INFO] TestAgent.server: Adding LAN server: server="Node-e3030616-3392-fad3-afb6-4a4f9b854ec5 (Addr: tcp/127.0.0.1:29580) (DC: dc1)" +31:28.833 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3030616-3392-fad3-afb6-4a4f9b854ec5.dc1 area=wan +31:28.833 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29575 network=udp +31:28.833 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29575 network=tcp +31:28.834 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29576 network=tcp +31:28.834 [INFO] TestAgent: started state syncer +31:28.834 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29581 network=tcp +31:28.878 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:28.878 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29580 [Candidate]" term=2 +31:28.892 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:28.892 [DEBUG] TestAgent.server.raft: vote granted: from=e3030616-3392-fad3-afb6-4a4f9b854ec5 term=2 tally=1 +31:28.892 [INFO] TestAgent.server.raft: election won: tally=1 +31:28.892 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29580 [Leader]" +31:28.894 [INFO] TestAgent.server: cluster leadership acquired +31:28.895 [INFO] TestAgent.server: New leader elected: payload=Node-e3030616-3392-fad3-afb6-4a4f9b854ec5 +31:28.936 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29580 +31:28.991 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.059 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.059 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.059 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.059 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.059 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.059 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3030616-3392-fad3-afb6-4a4f9b854ec5 +31:29.059 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3030616-3392-fad3-afb6-4a4f9b854ec5 +31:29.060 [INFO] TestAgent: Synced node info +31:29.092 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.756 [DEBUG] TestAgent.server: Session TTL expired: session=ecf9e049-5567-8800-4350-45fa1c02ed42 +31:29.818 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:29.818 [DEBUG] TestAgent: Node info in sync +31:29.818 [DEBUG] TestAgent: Node info in sync +31:29.986 [INFO] TestAgent: Requesting shutdown +31:29.986 [INFO] TestAgent.server: shutting down server +31:29.986 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.986 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.986 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.986 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.986 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.986 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.986 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.986 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.986 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:29.986 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.993 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.000Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.000Z [INFO] agent.router.manager: shutting down +31:30.000 [INFO] TestAgent: consul server down +31:30.000 [INFO] TestAgent: shutdown complete +31:30.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29575 network=tcp +31:30.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29575 network=udp +31:30.000 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29576 network=tcp +31:30.500 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.500 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCustomTTL (1.86s) +=== CONT TestIngestUserEvent +2021-10-05T16:31:29.647Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.691Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.760 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:295d6698-6c70-5971-c0fb-31867b981ecd Address:127.0.0.1:29636}]" +31:29.761 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-295d6698-6c70-5971-c0fb-31867b981ecd.dc1 127.0.0.1 +31:29.761 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-295d6698-6c70-5971-c0fb-31867b981ecd 127.0.0.1 +2021-10-05T16:31:29.761Z [INFO] agent.router: Initializing LAN area manager +31:29.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29638 network=udp +31:29.761 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29636 [Follower]" leader= +31:29.762 [INFO] TestAgent.server: Adding LAN server: server="Node-295d6698-6c70-5971-c0fb-31867b981ecd (Addr: tcp/127.0.0.1:29636) (DC: dc1)" +31:29.762 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-295d6698-6c70-5971-c0fb-31867b981ecd.dc1 area=wan +31:29.762 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29638 network=tcp +31:29.762 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29639 network=tcp +31:29.762 [INFO] TestAgent: started state syncer +31:29.762 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29637 network=tcp +31:29.815 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.815 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29636 [Candidate]" term=2 +31:29.845 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.845 [DEBUG] TestAgent.server.raft: vote granted: from=295d6698-6c70-5971-c0fb-31867b981ecd term=2 tally=1 +31:29.845 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.845 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29636 [Leader]" +31:29.845 [INFO] TestAgent.server: cluster leadership acquired +31:29.845 [INFO] TestAgent.server: New leader elected: payload=Node-295d6698-6c70-5971-c0fb-31867b981ecd +31:29.854 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29636 +31:29.873 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.948 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.948 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.948 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.948 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.948 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.948 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-295d6698-6c70-5971-c0fb-31867b981ecd +31:29.948 [INFO] TestAgent.server: member joined, marking health alive: member=Node-295d6698-6c70-5971-c0fb-31867b981ecd +31:29.962 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.972 [INFO] TestAgent: Synced node info +31:29.994 [INFO] TestAgent: Requesting shutdown +31:29.994 [INFO] TestAgent.server: shutting down server +31:29.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:29.994 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:29.994 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:29.994 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:29.994 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:29.994 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:29.994 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:29.994 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:29.994 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:29.994 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.007 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.020Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.020Z [INFO] agent.router.manager: shutting down +31:30.020 [INFO] TestAgent: consul server down +31:30.020 [INFO] TestAgent: shutdown complete +31:30.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29638 network=tcp +31:30.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29638 network=udp +31:30.020 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29639 network=tcp +31:30.528 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.528 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionGet (2.02s) + --- PASS: TestSessionGet/#00 (1.08s) + --- PASS: TestSessionGet/#01 (0.94s) +=== CONT TestShouldProcessUserEvent +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:29.547Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.615Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.745 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:59451b31-7502-1022-9693-4b7a3f52eaba Address:127.0.0.1:29622}]" +31:29.745 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29622 [Follower]" leader= +31:29.746 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-59451b31-7502-1022-9693-4b7a3f52eaba.dc1 127.0.0.1 +31:29.746 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-59451b31-7502-1022-9693-4b7a3f52eaba 127.0.0.1 +2021-10-05T16:31:29.746Z [INFO] agent.router: Initializing LAN area manager +31:29.746 [INFO] TestAgent.server: Adding LAN server: server="Node-59451b31-7502-1022-9693-4b7a3f52eaba (Addr: tcp/127.0.0.1:29622) (DC: dc1)" +31:29.746 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-59451b31-7502-1022-9693-4b7a3f52eaba.dc1 area=wan +31:29.746 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=udp +31:29.746 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=tcp +31:29.747 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29604 network=tcp +31:29.747 [INFO] TestAgent: started state syncer +31:29.747 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29623 network=tcp +31:29.788 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.788 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29622 [Candidate]" term=2 +31:29.804 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.804 [DEBUG] TestAgent.server.raft: vote granted: from=59451b31-7502-1022-9693-4b7a3f52eaba term=2 tally=1 +31:29.804 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.804 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29622 [Leader]" +31:29.805 [INFO] TestAgent.server: cluster leadership acquired +31:29.805 [INFO] TestAgent.server: New leader elected: payload=Node-59451b31-7502-1022-9693-4b7a3f52eaba +31:29.818 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29622 +31:29.844 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.870 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.870 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.870 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.870 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.870 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.870 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-59451b31-7502-1022-9693-4b7a3f52eaba +31:29.870 [INFO] TestAgent.server: member joined, marking health alive: member=Node-59451b31-7502-1022-9693-4b7a3f52eaba +31:29.901 [INFO] TestAgent.server: federation state anti-entropy synced +31:29.981 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:29.988 [INFO] TestAgent: Synced node info +31:30.088 [INFO] TestAgent.server.connect: CA provider config updated +31:30.088 [INFO] TestAgent: Requesting shutdown +31:30.088 [INFO] TestAgent.server: shutting down server +31:30.088 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.088 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.088 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.088 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.088 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.088 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.088 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.088 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.088 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.088 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.095 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.105Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.105Z [INFO] agent.router.manager: shutting down +31:30.105 [INFO] TestAgent: consul server down +31:30.105 [INFO] TestAgent: shutdown complete +31:30.105 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=tcp +31:30.105 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=udp +31:30.105 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29604 network=tcp +31:30.605 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.605 [INFO] TestAgent: Endpoints down +=== RUN TestConnectCAConfig/force_without_cross_sign_snake_case +2021-10-05T16:31:29.730Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.765Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.813 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:858301dc-c2ba-b456-0901-0acf6e74d32d Address:127.0.0.1:29657}]" +31:29.814 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-858301dc-c2ba-b456-0901-0acf6e74d32d.dc1 127.0.0.1 +31:29.814 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-858301dc-c2ba-b456-0901-0acf6e74d32d 127.0.0.1 +2021-10-05T16:31:29.814Z [INFO] agent.router: Initializing LAN area manager +31:29.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=udp +31:29.815 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29657 [Follower]" leader= +31:29.815 [INFO] TestAgent.server: Adding LAN server: server="Node-858301dc-c2ba-b456-0901-0acf6e74d32d (Addr: tcp/127.0.0.1:29657) (DC: dc1)" +31:29.815 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-858301dc-c2ba-b456-0901-0acf6e74d32d.dc1 area=wan +31:29.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=tcp +31:29.816 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29646 network=tcp +31:29.816 [INFO] TestAgent: started state syncer +31:29.816 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29658 network=tcp +31:29.880 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:29.880 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29657 [Candidate]" term=2 +31:29.911 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:29.911 [DEBUG] TestAgent.server.raft: vote granted: from=858301dc-c2ba-b456-0901-0acf6e74d32d term=2 tally=1 +31:29.911 [INFO] TestAgent.server.raft: election won: tally=1 +31:29.911 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29657 [Leader]" +31:29.911 [INFO] TestAgent.server: cluster leadership acquired +31:29.911 [INFO] TestAgent.server: New leader elected: payload=Node-858301dc-c2ba-b456-0901-0acf6e74d32d +31:29.927 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29657 +31:29.946 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:29.993 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:29.993 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:29.993 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:29.993 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:29.993 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:29.993 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-858301dc-c2ba-b456-0901-0acf6e74d32d +31:29.993 [INFO] TestAgent.server: member joined, marking health alive: member=Node-858301dc-c2ba-b456-0901-0acf6e74d32d +31:30.035 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.165 [INFO] TestAgent: Requesting shutdown +31:30.165 [INFO] TestAgent.server: shutting down server +31:30.165 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.165 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.165 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.165 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.165 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.166 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.166 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.166 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:30.166 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.166 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.166 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.175 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.183Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.183Z [INFO] agent.router.manager: shutting down +31:30.183 [INFO] TestAgent: consul server down +31:30.183 [INFO] TestAgent: shutdown complete +31:30.183 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=tcp +31:30.183 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=udp +31:30.183 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29646 network=tcp +31:30.683 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.683 [INFO] TestAgent: Endpoints down +--- PASS: TestSessionCreate (1.00s) +=== CONT TestValidateUserEventParams +--- PASS: TestValidateUserEventParams (0.00s) +=== CONT TestUIGatewayServiceNodes_Ingress +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:30.008Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.021Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.073 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7470fa14-7c35-3dad-db17-46c109f65cee Address:127.0.0.1:29678}]" +31:30.074 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7470fa14-7c35-3dad-db17-46c109f65cee.dc1 127.0.0.1 +31:30.074 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7470fa14-7c35-3dad-db17-46c109f65cee 127.0.0.1 +2021-10-05T16:31:30.074Z [INFO] agent.router: Initializing LAN area manager +31:30.075 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=udp +31:30.075 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29678 [Follower]" leader= +31:30.075 [INFO] TestAgent.server: Adding LAN server: server="Node-7470fa14-7c35-3dad-db17-46c109f65cee (Addr: tcp/127.0.0.1:29678) (DC: dc1)" +31:30.075 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7470fa14-7c35-3dad-db17-46c109f65cee.dc1 area=wan +31:30.075 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=tcp +31:30.076 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29653 network=tcp +31:30.076 [INFO] TestAgent: started state syncer +31:30.076 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29679 network=tcp +31:30.118 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.118 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29678 [Candidate]" term=2 +31:30.138 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.138 [DEBUG] TestAgent.server.raft: vote granted: from=7470fa14-7c35-3dad-db17-46c109f65cee term=2 tally=1 +31:30.138 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29678 [Leader]" +31:30.139 [INFO] TestAgent.server: cluster leadership acquired +31:30.139 [INFO] TestAgent.server: New leader elected: payload=Node-7470fa14-7c35-3dad-db17-46c109f65cee +31:30.165 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29678 +31:30.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.248 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.248 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.248 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.248 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.248 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7470fa14-7c35-3dad-db17-46c109f65cee +31:30.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7470fa14-7c35-3dad-db17-46c109f65cee +31:30.271 [INFO] TestAgent: Synced node info +31:30.271 [DEBUG] TestAgent: Node info in sync +31:30.278 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.299 [DEBUG] TestAgent: received remote exec event: id=c72a2f13-d3f7-26a8-3385-a0d2585a0734 +31:30.304 [INFO] TestAgent: remote exec script: script="echo failing;exit 2" +31:30.349 [INFO] TestAgent: Requesting shutdown +31:30.350 [INFO] TestAgent.server: shutting down server +31:30.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.350 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.350 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.350 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.350 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.350 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.350 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.353 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.357Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.357Z [INFO] agent.router.manager: shutting down +31:30.357 [INFO] TestAgent: consul server down +31:30.357 [INFO] TestAgent: shutdown complete +31:30.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=tcp +31:30.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=udp +31:30.357 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29653 network=tcp +31:30.858 [INFO] TestAgent: Waiting for endpoints to shut down +31:30.858 [INFO] TestAgent: Endpoints down +--- PASS: TestHandleRemoteExecFailed (0.87s) +=== CONT TestUIGatewayServiceNodes_Terminating +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:30.491Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.505Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.580 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d Address:127.0.0.1:29594}]" +31:30.580 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29594 [Follower]" leader= +31:30.581 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d.dc1 127.0.0.1 +31:30.581 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d 127.0.0.1 +2021-10-05T16:31:30.581Z [INFO] agent.router: Initializing LAN area manager +31:30.582 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=udp +31:30.582 [INFO] TestAgent.server: Adding LAN server: server="Node-7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d (Addr: tcp/127.0.0.1:29594) (DC: dc1)" +31:30.582 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d.dc1 area=wan +31:30.582 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=tcp +31:30.582 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29632 network=tcp +31:30.582 [INFO] TestAgent: started state syncer +31:30.583 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29595 network=tcp +31:30.624 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.624 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29594 [Candidate]" term=2 +31:30.646 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.646 [DEBUG] TestAgent.server.raft: vote granted: from=7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d term=2 tally=1 +31:30.646 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.646 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29594 [Leader]" +31:30.646 [INFO] TestAgent.server: cluster leadership acquired +31:30.647 [INFO] TestAgent.server: New leader elected: payload=Node-7726e4ef-6c2d-10e5-b91c-85c8be7b9e0d +31:30.661 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29594 +31:30.669 [DEBUG] TestAgent.server: User event: event=deploy +31:30.669 [DEBUG] TestAgent.server: User event: event=deploy +31:30.669 [DEBUG] TestAgent: new event: event_name=deploy event_id=2b09e82c-3e09-c11e-6f61-f68dad26e66b +31:30.688 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.696 [INFO] TestAgent: Requesting shutdown +31:30.698 [INFO] TestAgent.server: shutting down server +31:30.698 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.700 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:30.716 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.726Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.726Z [INFO] agent.router.manager: shutting down +31:30.736 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +31:30.736 [INFO] TestAgent: consul server down +31:30.736 [INFO] TestAgent: shutdown complete +31:30.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=tcp +31:30.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=udp +31:30.736 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29632 network=tcp +31:31.237 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.237 [INFO] TestAgent: Endpoints down +--- PASS: TestFireReceiveEvent (0.76s) +=== CONT TestUiServices +2021-10-05T16:31:30.247Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.269Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.314 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a1b5141-a22f-1206-545e-49332761d0d1 Address:127.0.0.1:29664}]" +31:30.314 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29664 [Follower]" leader= +31:30.315 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9a1b5141-a22f-1206-545e-49332761d0d1.dc1 127.0.0.1 +31:30.316 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9a1b5141-a22f-1206-545e-49332761d0d1 127.0.0.1 +2021-10-05T16:31:30.316Z [INFO] agent.router: Initializing LAN area manager +31:30.317 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=udp +31:30.317 [INFO] TestAgent.server: Adding LAN server: server="Node-9a1b5141-a22f-1206-545e-49332761d0d1 (Addr: tcp/127.0.0.1:29664) (DC: dc1)" +31:30.317 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9a1b5141-a22f-1206-545e-49332761d0d1.dc1 area=wan +31:30.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=tcp +31:30.319 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29681 network=tcp +31:30.319 [INFO] TestAgent: started state syncer +31:30.319 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29665 network=tcp +31:30.361 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.361 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29664 [Candidate]" term=2 +31:30.376 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.376 [DEBUG] TestAgent.server.raft: vote granted: from=9a1b5141-a22f-1206-545e-49332761d0d1 term=2 tally=1 +31:30.376 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.376 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29664 [Leader]" +31:30.377 [INFO] TestAgent.server: cluster leadership acquired +31:30.377 [INFO] TestAgent.server: New leader elected: payload=Node-9a1b5141-a22f-1206-545e-49332761d0d1 +31:30.388 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29664 +31:30.416 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.459 [INFO] TestAgent: Synced node info +31:30.482 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.482 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.483 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.483 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.483 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.483 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9a1b5141-a22f-1206-545e-49332761d0d1 +31:30.483 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9a1b5141-a22f-1206-545e-49332761d0d1 +31:30.504 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.676 [DEBUG] TestAgent: received remote exec event: id=249a0b56-1b99-0b2c-8136-88bb0937af4b +31:30.690 [INFO] TestAgent: remote exec script: script=uptime +31:30.749 [INFO] TestAgent: Requesting shutdown +31:30.749 [INFO] TestAgent.server: shutting down server +31:30.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.749 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.749 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.749 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.750 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.750 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.750 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.750 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.750 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.756 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.767Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.768Z [INFO] agent.router.manager: shutting down +31:30.768 [INFO] TestAgent: consul server down +31:30.768 [INFO] TestAgent: shutdown complete +31:30.768 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=tcp +31:30.768 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=udp +31:30.768 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29681 network=tcp +31:31.269 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.269 [INFO] TestAgent: Endpoints down +--- PASS: TestHandleRemoteExec (1.06s) +=== CONT TestUiNodeInfo +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:30.468Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.489Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.570 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 Address:127.0.0.1:29643}]" +31:30.571 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9.dc1 127.0.0.1 +31:30.572 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 127.0.0.1 +2021-10-05T16:31:30.572Z [INFO] agent.router: Initializing LAN area manager +31:30.572 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29643 [Follower]" leader= +31:30.572 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=udp +31:30.572 [INFO] TestAgent.server: Adding LAN server: server="Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 (Addr: tcp/127.0.0.1:29643) (DC: dc1)" +31:30.572 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9.dc1 area=wan +31:30.572 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=tcp +31:30.573 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29688 network=tcp +31:30.573 [INFO] TestAgent: started state syncer +31:30.573 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29644 network=tcp +31:30.636 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.636 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29643 [Candidate]" term=2 +31:30.668 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.668 [DEBUG] TestAgent.server.raft: vote granted: from=1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 term=2 tally=1 +31:30.668 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.668 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29643 [Leader]" +31:30.668 [INFO] TestAgent.server: cluster leadership acquired +31:30.668 [INFO] TestAgent.server: New leader elected: payload=Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 +31:30.676 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29643 +31:30.735 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.782 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.782 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.782 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.782 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.782 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.782 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 +31:30.782 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e421480-0abe-f5eb-7a5b-4df3f4d68ad9 +31:30.800 [INFO] TestAgent: Requesting shutdown +31:30.800 [INFO] TestAgent.server: shutting down server +31:30.800 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.800 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.800 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.800 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.800 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.800 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:30.800 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.800 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.800 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.805 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.805 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.814 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.824Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.824Z [INFO] agent.router.manager: shutting down +31:30.824 [INFO] TestAgent: consul server down +31:30.824 [INFO] TestAgent: shutdown complete +31:30.824 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=tcp +31:30.824 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=udp +31:30.824 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29688 network=tcp +31:31.327 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.327 [INFO] TestAgent: Endpoints down +--- PASS: TestUiIndex (0.89s) +=== CONT TestUiNodes_Filter +2021-10-05T16:31:30.574Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.590Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.660 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 Address:127.0.0.1:29699}]" +31:30.660 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29699 [Follower]" leader= +31:30.660 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4.dc1 127.0.0.1 +31:30.661 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 127.0.0.1 +2021-10-05T16:31:30.661Z [INFO] agent.router: Initializing LAN area manager +31:30.661 [INFO] TestAgent.server: Adding LAN server: server="Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 (Addr: tcp/127.0.0.1:29699) (DC: dc1)" +31:30.661 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4.dc1 area=wan +31:30.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=udp +31:30.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=tcp +31:30.663 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29695 network=tcp +31:30.663 [INFO] TestAgent: started state syncer +31:30.663 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29700 network=tcp +31:30.730 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.730 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29699 [Candidate]" term=2 +31:30.756 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.756 [DEBUG] TestAgent.server.raft: vote granted: from=0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 term=2 tally=1 +31:30.756 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.756 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29699 [Leader]" +31:30.756 [INFO] TestAgent.server: cluster leadership acquired +31:30.756 [INFO] TestAgent.server: New leader elected: payload=Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 +31:30.777 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29699 +31:30.800 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.837 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.837 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.837 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.837 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.837 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.837 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 +31:30.837 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0af92b7e-fbe6-0364-e7c2-8235d8d90ff4 +31:30.850 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.871 [INFO] TestAgent: Requesting shutdown +31:30.871 [INFO] TestAgent.server: shutting down server +31:30.871 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.871 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.871 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.871 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.871 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.871 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.871 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.871 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:30.871 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.871 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.871 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.883 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.892Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.892Z [INFO] agent.router.manager: shutting down +31:30.892 [INFO] TestAgent: consul server down +31:30.892 [INFO] TestAgent: shutdown complete +31:30.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=tcp +31:30.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=udp +31:30.892 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29695 network=tcp +31:31.395 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.395 [INFO] TestAgent: Endpoints down +--- PASS: TestShouldProcessUserEvent (0.87s) +=== CONT TestUiNodes +2021-10-05T16:31:30.622Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.637Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.680 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d Address:127.0.0.1:29727}]" +31:30.680 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29727 [Follower]" leader= +31:30.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d.dc1 127.0.0.1 +31:30.682 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d 127.0.0.1 +2021-10-05T16:31:30.682Z [INFO] agent.router: Initializing LAN area manager +31:30.682 [INFO] TestAgent.server: Adding LAN server: server="Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d (Addr: tcp/127.0.0.1:29727) (DC: dc1)" +31:30.682 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d.dc1 area=wan +31:30.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=udp +31:30.682 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=tcp +31:30.683 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29702 network=tcp +31:30.683 [INFO] TestAgent: started state syncer +31:30.683 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29728 network=tcp +31:30.724 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.724 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29727 [Candidate]" term=2 +31:30.749 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.749 [DEBUG] TestAgent.server.raft: vote granted: from=b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d term=2 tally=1 +31:30.749 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.749 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29727 [Leader]" +31:30.750 [INFO] TestAgent.server: cluster leadership acquired +31:30.750 [INFO] TestAgent.server: New leader elected: payload=Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d +31:30.764 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29727 +31:30.780 [INFO] TestAgent: Synced node info +31:30.814 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.872 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.872 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.872 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.872 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.872 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d +31:30.872 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0e64e9f-c0fb-f4e5-e4a6-c1a054e2c48d +31:30.912 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.923 [INFO] TestAgent.server.connect: CA provider config updated +31:30.923 [INFO] TestAgent: Requesting shutdown +31:30.923 [INFO] TestAgent.server: shutting down server +31:30.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.923 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.923 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.923 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.923 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.923 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.923 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.929 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:30.936Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:30.936Z [INFO] agent.router.manager: shutting down +31:30.936 [INFO] TestAgent: consul server down +31:30.936 [INFO] TestAgent: shutdown complete +31:30.936 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=tcp +31:30.936 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=udp +31:30.936 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29702 network=tcp +31:31.437 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.437 [INFO] TestAgent: Endpoints down +=== RUN TestConnectCAConfig/setting_state_fails +2021-10-05T16:31:30.552Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.584Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.648 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:28a1cbd3-adb2-a0d3-d105-279502aab738 Address:127.0.0.1:29692}]" +31:30.648 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29692 [Follower]" leader= +31:30.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-28a1cbd3-adb2-a0d3-d105-279502aab738.dc1 127.0.0.1 +31:30.650 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-28a1cbd3-adb2-a0d3-d105-279502aab738 127.0.0.1 +2021-10-05T16:31:30.650Z [INFO] agent.router: Initializing LAN area manager +31:30.650 [INFO] TestAgent.server: Adding LAN server: server="Node-28a1cbd3-adb2-a0d3-d105-279502aab738 (Addr: tcp/127.0.0.1:29692) (DC: dc1)" +31:30.650 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-28a1cbd3-adb2-a0d3-d105-279502aab738.dc1 area=wan +31:30.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=udp +31:30.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=tcp +31:30.651 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29597 network=tcp +31:30.651 [INFO] TestAgent: started state syncer +31:30.651 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29693 network=tcp +31:30.684 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.684 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29692 [Candidate]" term=2 +31:30.713 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.713 [DEBUG] TestAgent.server.raft: vote granted: from=28a1cbd3-adb2-a0d3-d105-279502aab738 term=2 tally=1 +31:30.713 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.713 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29692 [Leader]" +31:30.716 [INFO] TestAgent.server: cluster leadership acquired +31:30.716 [INFO] TestAgent.server: New leader elected: payload=Node-28a1cbd3-adb2-a0d3-d105-279502aab738 +31:30.723 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29692 +31:30.766 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.819 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.819 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.819 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.819 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.819 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.819 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28a1cbd3-adb2-a0d3-d105-279502aab738 +31:30.819 [INFO] TestAgent.server: member joined, marking health alive: member=Node-28a1cbd3-adb2-a0d3-d105-279502aab738 +31:30.833 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.896 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:30.912 [INFO] TestAgent: Synced node info +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.970 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.971 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.972 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.973 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [DEBUG] TestAgent: new event: event_name=test event_id= +31:30.974 [INFO] TestAgent: Requesting shutdown +31:30.975 [INFO] TestAgent.server: shutting down server +31:30.975 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:30.975 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:30.975 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:30.975 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:30.975 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:30.975 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:30.975 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:30.975 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:30.975 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:30.975 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:30.983 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.000Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.000Z [INFO] agent.router.manager: shutting down +31:31.000 [INFO] TestAgent: consul server down +31:31.000 [INFO] TestAgent: shutdown complete +31:31.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=tcp +31:31.000 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=udp +31:31.000 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29597 network=tcp +31:31.500 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.500 [INFO] TestAgent: Endpoints down +--- PASS: TestIngestUserEvent (1.00s) +=== CONT TestTxnEndpoint_Bad_JSON +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestUiServices/No_Filter +=== PAUSE TestUiServices/No_Filter +=== RUN TestUiServices/Filtered +2021-10-05T16:31:30.723Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.751Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:30.820 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6e1de79c-81c4-fbbf-6876-af8f08ef6916 Address:127.0.0.1:29713}]" +31:30.821 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29713 [Follower]" leader= +31:30.821 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916.dc1 127.0.0.1 +31:30.822 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916 127.0.0.1 +2021-10-05T16:31:30.822Z [INFO] agent.router: Initializing LAN area manager +31:30.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=udp +31:30.823 [INFO] TestAgent.server: Adding LAN server: server="Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916 (Addr: tcp/127.0.0.1:29713) (DC: dc1)" +31:30.823 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916.dc1 area=wan +31:30.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=tcp +31:30.824 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29737 network=tcp +31:30.824 [INFO] TestAgent: started state syncer +31:30.824 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29714 network=tcp +31:30.864 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.864 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29713 [Candidate]" term=2 +31:30.893 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.893 [DEBUG] TestAgent.server.raft: vote granted: from=6e1de79c-81c4-fbbf-6876-af8f08ef6916 term=2 tally=1 +31:30.893 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.893 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29713 [Leader]" +31:30.896 [INFO] TestAgent.server: cluster leadership acquired +31:30.896 [INFO] TestAgent.server: New leader elected: payload=Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916 +31:30.914 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29713 +31:30.935 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.967 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.967 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.967 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.967 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.967 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.967 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916 +31:30.967 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6e1de79c-81c4-fbbf-6876-af8f08ef6916 +31:31.000 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.017 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:31.032 [INFO] TestAgent: Synced node info +31:31.160 [INFO] TestAgent: Requesting shutdown +31:31.160 [INFO] TestAgent.server: shutting down server +31:31.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.160 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.160 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.160 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.160 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.160 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.184 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.188Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.188Z [INFO] agent.router.manager: shutting down +31:31.188 [INFO] TestAgent: consul server down +31:31.188 [INFO] TestAgent: shutdown complete +31:31.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=tcp +31:31.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=udp +31:31.188 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29737 network=tcp +31:31.688 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.688 [INFO] TestAgent: Endpoints down +--- PASS: TestUIGatewayServiceNodes_Ingress (1.00s) +=== CONT TestTxnEndpoint_UpdateCheck +2021-10-05T16:31:30.898Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:30.918Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.004 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 Address:127.0.0.1:29720}]" +31:31.005 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238.dc1 127.0.0.1 +31:31.006 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 127.0.0.1 +2021-10-05T16:31:31.006Z [INFO] agent.router: Initializing LAN area manager +31:31.006 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=udp +31:31.006 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29720 [Follower]" leader= +31:31.007 [INFO] TestAgent.server: Adding LAN server: server="Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 (Addr: tcp/127.0.0.1:29720) (DC: dc1)" +31:31.007 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238.dc1 area=wan +31:31.007 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=tcp +31:31.008 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29716 network=tcp +31:31.008 [INFO] TestAgent: started state syncer +31:31.008 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29721 network=tcp +31:31.073 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.073 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29720 [Candidate]" term=2 +31:31.082 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.082 [DEBUG] TestAgent.server.raft: vote granted: from=22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 term=2 tally=1 +31:31.082 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.082 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29720 [Leader]" +31:31.084 [INFO] TestAgent.server: cluster leadership acquired +31:31.084 [INFO] TestAgent.server: New leader elected: payload=Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 +31:31.094 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29720 +31:31.110 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.220 [INFO] TestAgent: Synced node info +31:31.220 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.220 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.220 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.220 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.220 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.220 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 +31:31.220 [INFO] TestAgent.server: member joined, marking health alive: member=Node-22c4e7c2-e089-4cb6-fb1a-b8eef4a16238 +31:31.247 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.259 [INFO] TestAgent: Requesting shutdown +31:31.259 [INFO] TestAgent.server: shutting down server +31:31.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.259 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.259 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.259 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.259 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.259 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.259 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.276 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.279Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.279Z [INFO] agent.router.manager: shutting down +31:31.280 [INFO] TestAgent: consul server down +31:31.280 [INFO] TestAgent: shutdown complete +31:31.280 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=tcp +31:31.280 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=udp +31:31.280 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29716 network=tcp +31:31.780 [INFO] TestAgent: Waiting for endpoints to shut down +31:31.780 [INFO] TestAgent: Endpoints down +--- PASS: TestUIGatewayServiceNodes_Terminating (0.93s) +=== CONT TestTxnEndpoint_KV_Actions +=== RUN TestTxnEndpoint_KV_Actions/#00 +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== CONT TestTxnEndpoint_Bad_Size_Ops +2021-10-05T16:31:31.313Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.379Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.461 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a3f10913-abab-26f9-4b9d-6ee3bfd90130 Address:127.0.0.1:29755}]" +31:31.462 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130.dc1 127.0.0.1 +31:31.463 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130 127.0.0.1 +2021-10-05T16:31:31.463Z [INFO] agent.router: Initializing LAN area manager +31:31.463 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=udp +31:31.463 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29755 [Follower]" leader= +31:31.463 [INFO] TestAgent.server: Adding LAN server: server="Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130 (Addr: tcp/127.0.0.1:29755) (DC: dc1)" +31:31.464 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130.dc1 area=wan +31:31.464 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=tcp +31:31.465 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29744 network=tcp +31:31.465 [INFO] TestAgent: started state syncer +31:31.465 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29756 network=tcp +31:31.518 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.518 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29755 [Candidate]" term=2 +31:31.562 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.562 [DEBUG] TestAgent.server.raft: vote granted: from=a3f10913-abab-26f9-4b9d-6ee3bfd90130 term=2 tally=1 +31:31.562 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.562 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29755 [Leader]" +31:31.564 [INFO] TestAgent.server: cluster leadership acquired +31:31.564 [INFO] TestAgent.server: New leader elected: payload=Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130 +31:31.592 [INFO] TestAgent: Synced node info +31:31.592 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29755 +31:31.637 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.683 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.683 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.683 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.683 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.683 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.684 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130 +31:31.684 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a3f10913-abab-26f9-4b9d-6ee3bfd90130 +31:31.724 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.737 [INFO] TestAgent: Requesting shutdown +31:31.737 [INFO] TestAgent.server: shutting down server +31:31.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.737 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.737 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.737 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.737 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.737 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.737 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.749 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.750Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.750Z [INFO] agent.router.manager: shutting down +31:31.750 [INFO] TestAgent: consul server down +31:31.750 [INFO] TestAgent: shutdown complete +31:31.750 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=tcp +31:31.750 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=udp +31:31.750 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29744 network=tcp +31:32.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.252 [INFO] TestAgent: Endpoints down +--- PASS: TestUiNodeInfo (0.98s) +=== CONT TestTxnEndpoint_Bad_Size_Net +=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_default_limits +2021-10-05T16:31:31.521Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.539Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.587 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08262107-cbbd-7877-428d-5857bbca6995 Address:127.0.0.1:29748}]" +31:31.587 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-08262107-cbbd-7877-428d-5857bbca6995.dc1 127.0.0.1 +31:31.588 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29748 [Follower]" leader= +31:31.588 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08262107-cbbd-7877-428d-5857bbca6995 127.0.0.1 +2021-10-05T16:31:31.588Z [INFO] agent.router: Initializing LAN area manager +31:31.588 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-08262107-cbbd-7877-428d-5857bbca6995.dc1 area=wan +31:31.588 [INFO] TestAgent.server: Adding LAN server: server="Node-08262107-cbbd-7877-428d-5857bbca6995 (Addr: tcp/127.0.0.1:29748) (DC: dc1)" +31:31.589 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=udp +31:31.589 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=tcp +31:31.589 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29730 network=tcp +31:31.589 [INFO] TestAgent: started state syncer +31:31.589 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29749 network=tcp +31:31.644 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.644 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29748 [Candidate]" term=2 +31:31.676 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.676 [DEBUG] TestAgent.server.raft: vote granted: from=08262107-cbbd-7877-428d-5857bbca6995 term=2 tally=1 +31:31.676 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.676 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29748 [Leader]" +31:31.676 [INFO] TestAgent.server: cluster leadership acquired +31:31.676 [INFO] TestAgent.server: New leader elected: payload=Node-08262107-cbbd-7877-428d-5857bbca6995 +31:31.686 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29748 +31:31.707 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.763 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.763 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.763 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.763 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.763 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.763 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08262107-cbbd-7877-428d-5857bbca6995 +31:31.763 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08262107-cbbd-7877-428d-5857bbca6995 +31:31.772 [INFO] TestAgent: Requesting shutdown +31:31.772 [INFO] TestAgent.server: shutting down server +31:31.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.772 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.772 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.772 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.772 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.772 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:31.772 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.772 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.774 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.784 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.789 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.791Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.791Z [INFO] agent.router.manager: shutting down +31:31.791 [INFO] TestAgent: consul server down +31:31.791 [INFO] TestAgent: shutdown complete +31:31.791 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=tcp +31:31.791 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=udp +31:31.791 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29730 network=tcp +31:32.291 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.291 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_Bad_JSON (0.79s) +=== CONT TestTxnEndpoint_Bad_Size_Item +=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_default_limits +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:31.383Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.427Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.524 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ffdf23bc-4305-cf95-1e3a-1e67800fa56a Address:127.0.0.1:29706}]" +31:31.524 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a.dc1 127.0.0.1 +31:31.525 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a 127.0.0.1 +2021-10-05T16:31:31.525Z [INFO] agent.router: Initializing LAN area manager +31:31.525 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=udp +31:31.525 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29706 [Follower]" leader= +31:31.526 [INFO] TestAgent.server: Adding LAN server: server="Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a (Addr: tcp/127.0.0.1:29706) (DC: dc1)" +31:31.526 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a.dc1 area=wan +31:31.526 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=tcp +31:31.526 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29758 network=tcp +31:31.526 [INFO] TestAgent: started state syncer +31:31.526 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29707 network=tcp +31:31.567 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.567 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29706 [Candidate]" term=2 +31:31.593 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.593 [DEBUG] TestAgent.server.raft: vote granted: from=ffdf23bc-4305-cf95-1e3a-1e67800fa56a term=2 tally=1 +31:31.593 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.593 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29706 [Leader]" +31:31.593 [INFO] TestAgent.server: cluster leadership acquired +31:31.593 [INFO] TestAgent.server: New leader elected: payload=Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a +31:31.605 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29706 +31:31.652 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.734 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.734 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.734 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.734 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.734 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.734 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a +31:31.734 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ffdf23bc-4305-cf95-1e3a-1e67800fa56a +31:31.771 [INFO] TestAgent: Synced node info +31:31.777 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.821 [INFO] TestAgent: Requesting shutdown +31:31.821 [INFO] TestAgent.server: shutting down server +31:31.821 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.821 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.821 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.821 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.821 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.821 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.821 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.821 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.821 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.821 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.827 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.828Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.828Z [INFO] agent.router.manager: shutting down +31:31.828 [INFO] TestAgent: consul server down +31:31.828 [INFO] TestAgent: shutdown complete +31:31.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=tcp +31:31.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=udp +31:31.828 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29758 network=tcp +31:32.329 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.329 [INFO] TestAgent: Endpoints down +--- PASS: TestUiNodes_Filter (0.99s) +=== CONT TestAgentConnectAuthorize_badBody +2021-10-05T16:31:31.457Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.480Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.568 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:93ef7d96-366d-e3a8-be83-a3a4b53e8698 Address:127.0.0.1:29741}]" +31:31.568 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698.dc1 127.0.0.1 +31:31.569 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29741 [Follower]" leader= +31:31.569 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698 127.0.0.1 +2021-10-05T16:31:31.569Z [INFO] agent.router: Initializing LAN area manager +31:31.569 [INFO] TestAgent.server: Adding LAN server: server="Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698 (Addr: tcp/127.0.0.1:29741) (DC: dc1)" +31:31.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=udp +31:31.569 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698.dc1 area=wan +31:31.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=tcp +31:31.570 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29674 network=tcp +31:31.570 [INFO] TestAgent: started state syncer +31:31.570 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29742 network=tcp +31:31.606 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.607 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29741 [Candidate]" term=2 +31:31.620 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.620 [DEBUG] TestAgent.server.raft: vote granted: from=93ef7d96-366d-e3a8-be83-a3a4b53e8698 term=2 tally=1 +31:31.620 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.620 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29741 [Leader]" +31:31.620 [INFO] TestAgent.server: cluster leadership acquired +31:31.620 [INFO] TestAgent.server: New leader elected: payload=Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698 +31:31.632 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29741 +31:31.678 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.713 [INFO] TestAgent: Synced node info +31:31.713 [DEBUG] TestAgent: Node info in sync +31:31.763 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.763 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.763 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.763 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.763 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.764 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698 +31:31.764 [INFO] TestAgent.server: member joined, marking health alive: member=Node-93ef7d96-366d-e3a8-be83-a3a4b53e8698 +31:31.795 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.860 [INFO] TestAgent: Requesting shutdown +31:31.860 [INFO] TestAgent.server: shutting down server +31:31.860 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.860 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.860 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.860 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.860 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.860 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.860 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.860 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.860 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.860 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.866 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.868Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.868Z [INFO] agent.router.manager: shutting down +31:31.868 [INFO] TestAgent: consul server down +31:31.868 [INFO] TestAgent: shutdown complete +31:31.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=tcp +31:31.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=udp +31:31.868 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29674 network=tcp +31:32.368 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.368 [INFO] TestAgent: Endpoints down +=== RUN TestConnectCAConfig/updating_config_with_same_state +2021-10-05T16:31:31.432Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.466Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.560 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2332e38c-8ec8-3d04-aed1-c3d049fbdf21 Address:127.0.0.1:29671}]" +31:31.560 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29671 [Follower]" leader= +31:31.561 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21.dc1 127.0.0.1 +31:31.563 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21 127.0.0.1 +2021-10-05T16:31:31.563Z [INFO] agent.router: Initializing LAN area manager +31:31.563 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=udp +31:31.563 [INFO] TestAgent.server: Adding LAN server: server="Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21 (Addr: tcp/127.0.0.1:29671) (DC: dc1)" +31:31.563 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21.dc1 area=wan +31:31.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=tcp +31:31.565 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29709 network=tcp +31:31.566 [INFO] TestAgent: started state syncer +31:31.566 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29672 network=tcp +31:31.628 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.628 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29671 [Candidate]" term=2 +31:31.671 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.671 [DEBUG] TestAgent.server.raft: vote granted: from=2332e38c-8ec8-3d04-aed1-c3d049fbdf21 term=2 tally=1 +31:31.671 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.671 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29671 [Leader]" +31:31.671 [INFO] TestAgent.server: cluster leadership acquired +31:31.671 [INFO] TestAgent.server: New leader elected: payload=Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21 +31:31.688 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29671 +31:31.711 [INFO] TestAgent: Synced node info +31:31.740 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.771 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.771 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.771 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.771 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.771 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.771 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21 +31:31.771 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2332e38c-8ec8-3d04-aed1-c3d049fbdf21 +31:31.797 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.972 [INFO] TestAgent: Requesting shutdown +31:31.972 [INFO] TestAgent.server: shutting down server +31:31.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.972 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.972 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.972 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.972 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.972 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.972 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.983 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.984Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.984Z [INFO] agent.router.manager: shutting down +31:31.984 [INFO] TestAgent: consul server down +31:31.984 [INFO] TestAgent: shutdown complete +31:31.984 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=tcp +31:31.984 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=udp +31:31.984 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29709 network=tcp +31:32.484 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.484 [INFO] TestAgent: Endpoints down +--- PASS: TestUiNodes (1.09s) +=== CONT TestConfig_Apply_TerminatingGateway +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:31.804Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.817Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.900 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fae6279e-d2fc-99ce-9d42-b88392ced73c Address:127.0.0.1:29783}]" +31:31.900 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29783 [Follower]" leader= +31:31.901 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fae6279e-d2fc-99ce-9d42-b88392ced73c.dc1 127.0.0.1 +31:31.901 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fae6279e-d2fc-99ce-9d42-b88392ced73c 127.0.0.1 +2021-10-05T16:31:31.901Z [INFO] agent.router: Initializing LAN area manager +31:31.902 [INFO] TestAgent.server: Adding LAN server: server="Node-fae6279e-d2fc-99ce-9d42-b88392ced73c (Addr: tcp/127.0.0.1:29783) (DC: dc1)" +31:31.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=udp +31:31.902 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fae6279e-d2fc-99ce-9d42-b88392ced73c.dc1 area=wan +31:31.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=tcp +31:31.902 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29765 network=tcp +31:31.902 [INFO] TestAgent: started state syncer +31:31.902 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29784 network=tcp +31:31.972 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.972 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29783 [Candidate]" term=2 +31:32.005 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.005 [DEBUG] TestAgent.server.raft: vote granted: from=fae6279e-d2fc-99ce-9d42-b88392ced73c term=2 tally=1 +31:32.005 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.005 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29783 [Leader]" +31:32.006 [INFO] TestAgent.server: cluster leadership acquired +31:32.006 [INFO] TestAgent.server: New leader elected: payload=Node-fae6279e-d2fc-99ce-9d42-b88392ced73c +31:32.036 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29783 +31:32.054 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.087 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.087 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.087 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.087 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.087 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.087 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fae6279e-d2fc-99ce-9d42-b88392ced73c +31:32.087 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fae6279e-d2fc-99ce-9d42-b88392ced73c +31:32.111 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.139 [INFO] TestAgent: Requesting shutdown +31:32.140 [INFO] TestAgent.server: shutting down server +31:32.140 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.140 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.140 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.140 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.140 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.140 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.140 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:32.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.140 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.140 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.147 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.168Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.169Z [INFO] agent.router.manager: shutting down +31:32.169 [INFO] TestAgent: consul server down +31:32.169 [INFO] TestAgent: shutdown complete +31:32.169 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=tcp +31:32.169 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=udp +31:32.169 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29765 network=tcp +31:32.669 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.669 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_KV_Actions/#01 +2021-10-05T16:31:31.706Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.728Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.789 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a7fdb0c3-f73b-db5c-c600-caa04626f65d Address:127.0.0.1:29762}]" +31:31.790 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d.dc1 127.0.0.1 +31:31.790 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d 127.0.0.1 +2021-10-05T16:31:31.790Z [INFO] agent.router: Initializing LAN area manager +31:31.791 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29762 [Follower]" leader= +31:31.792 [INFO] TestAgent.server: Adding LAN server: server="Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d (Addr: tcp/127.0.0.1:29762) (DC: dc1)" +31:31.792 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d.dc1 area=wan +31:31.792 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=tcp +31:31.792 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=udp +31:31.793 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29751 network=tcp +31:31.793 [INFO] TestAgent: started state syncer +31:31.793 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29763 network=tcp +31:31.852 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.852 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29762 [Candidate]" term=2 +31:31.869 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.869 [DEBUG] TestAgent.server.raft: vote granted: from=a7fdb0c3-f73b-db5c-c600-caa04626f65d term=2 tally=1 +31:31.869 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.869 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29762 [Leader]" +31:31.869 [INFO] TestAgent.server: cluster leadership acquired +31:31.869 [INFO] TestAgent.server: New leader elected: payload=Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d +31:31.878 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29762 +31:31.895 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.923 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.923 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.923 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.923 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.923 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.923 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d +31:31.923 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a7fdb0c3-f73b-db5c-c600-caa04626f65d +31:31.939 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.076 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:32.081 [INFO] TestAgent: Synced node info +31:32.081 [DEBUG] TestAgent: Node info in sync +31:32.169 [INFO] TestAgent: Requesting shutdown +31:32.169 [INFO] TestAgent.server: shutting down server +31:32.169 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.169 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.169 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.169 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.170 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.170 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.170 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.170 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.180 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.180Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.180Z [INFO] agent.router.manager: shutting down +31:32.180 [INFO] TestAgent: consul server down +31:32.180 [INFO] TestAgent: shutdown complete +31:32.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=tcp +31:32.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=udp +31:32.180 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29751 network=tcp +31:32.680 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.680 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_UpdateCheck (0.99s) +=== CONT TestConfig_Apply +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:32.254Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.271Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.353 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bcd50b92-8150-9194-9fda-2577dbbd2ba7 Address:127.0.0.1:29790}]" +31:32.354 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7.dc1 127.0.0.1 +31:32.354 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7 127.0.0.1 +2021-10-05T16:31:32.354Z [INFO] agent.router: Initializing LAN area manager +31:32.354 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=udp +31:32.354 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29790 [Follower]" leader= +31:32.355 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7.dc1 area=wan +31:32.355 [INFO] TestAgent.server: Adding LAN server: server="Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7 (Addr: tcp/127.0.0.1:29790) (DC: dc1)" +31:32.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=tcp +31:32.356 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29786 network=tcp +31:32.356 [INFO] TestAgent: started state syncer +31:32.356 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29791 network=tcp +31:32.393 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.393 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29790 [Candidate]" term=2 +31:32.419 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.419 [DEBUG] TestAgent.server.raft: vote granted: from=bcd50b92-8150-9194-9fda-2577dbbd2ba7 term=2 tally=1 +31:32.419 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.419 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29790 [Leader]" +31:32.420 [INFO] TestAgent.server: cluster leadership acquired +31:32.420 [INFO] TestAgent.server: New leader elected: payload=Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7 +31:32.430 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29790 +31:32.457 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.491 [INFO] TestAgent: Synced node info +31:32.528 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.528 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.528 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.528 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.528 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.528 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7 +31:32.528 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bcd50b92-8150-9194-9fda-2577dbbd2ba7 +31:32.604 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.604 [INFO] TestAgent: Requesting shutdown +31:32.604 [INFO] TestAgent.server: shutting down server +31:32.604 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.604 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.604 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.605 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.605 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.605 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.618 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.632Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.632Z [INFO] agent.router.manager: shutting down +31:32.632 [INFO] TestAgent: consul server down +31:32.632 [INFO] TestAgent: shutdown complete +31:32.632 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=tcp +31:32.632 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=udp +31:32.632 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29786 network=tcp +31:33.133 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.133 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_Bad_Size_Ops (0.94s) +=== CONT TestConfig_Delete +2021-10-05T16:31:32.307Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.321Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.377 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:929efaa5-76a0-8ec8-dbaf-cc92983231c3 Address:127.0.0.1:29839}]" +31:32.378 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3.dc1 127.0.0.1 +31:32.379 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3 127.0.0.1 +2021-10-05T16:31:32.379Z [INFO] agent.router: Initializing LAN area manager +31:32.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=udp +31:32.379 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29839 [Follower]" leader= +31:32.380 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3.dc1 area=wan +31:32.380 [INFO] TestAgent.server: Adding LAN server: server="Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3 (Addr: tcp/127.0.0.1:29839) (DC: dc1)" +31:32.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=tcp +31:32.381 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29779 network=tcp +31:32.381 [INFO] TestAgent: started state syncer +31:32.381 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29840 network=tcp +31:32.435 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.435 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29839 [Candidate]" term=2 +31:32.457 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.457 [DEBUG] TestAgent.server.raft: vote granted: from=929efaa5-76a0-8ec8-dbaf-cc92983231c3 term=2 tally=1 +31:32.457 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.457 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29839 [Leader]" +31:32.460 [INFO] TestAgent.server: cluster leadership acquired +31:32.460 [INFO] TestAgent.server: New leader elected: payload=Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3 +31:32.486 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29839 +31:32.518 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.608 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.608 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.608 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.608 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.608 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.609 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3 +31:32.609 [INFO] TestAgent.server: member joined, marking health alive: member=Node-929efaa5-76a0-8ec8-dbaf-cc92983231c3 +31:32.640 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.683 [INFO] TestAgent: Requesting shutdown +31:32.683 [INFO] TestAgent.server: shutting down server +31:32.683 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.683 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.683 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.683 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.683 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.683 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.683 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.683 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:32.683 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.683 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.683 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.695 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.703Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.703Z [INFO] agent.router.manager: shutting down +31:32.703 [INFO] TestAgent: consul server down +31:32.703 [INFO] TestAgent: shutdown complete +31:32.703 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=tcp +31:32.703 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=udp +31:32.704 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29779 network=tcp +31:33.204 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.204 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_configured_max_txn_len +2021-10-05T16:31:32.341Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.352Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.436 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7517cedb-4028-489e-d5cd-15dcbc434503 Address:127.0.0.1:29825}]" +31:32.436 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7517cedb-4028-489e-d5cd-15dcbc434503.dc1 127.0.0.1 +31:32.437 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7517cedb-4028-489e-d5cd-15dcbc434503 127.0.0.1 +2021-10-05T16:31:32.437Z [INFO] agent.router: Initializing LAN area manager +31:32.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=udp +31:32.437 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29825 [Follower]" leader= +31:32.437 [INFO] TestAgent.server: Adding LAN server: server="Node-7517cedb-4028-489e-d5cd-15dcbc434503 (Addr: tcp/127.0.0.1:29825) (DC: dc1)" +31:32.437 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7517cedb-4028-489e-d5cd-15dcbc434503.dc1 area=wan +31:32.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=tcp +31:32.438 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29821 network=tcp +31:32.438 [INFO] TestAgent: started state syncer +31:32.438 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29826 network=tcp +31:32.488 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.488 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29825 [Candidate]" term=2 +31:32.522 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.522 [DEBUG] TestAgent.server.raft: vote granted: from=7517cedb-4028-489e-d5cd-15dcbc434503 term=2 tally=1 +31:32.522 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.522 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29825 [Leader]" +31:32.529 [INFO] TestAgent.server: cluster leadership acquired +31:32.529 [INFO] TestAgent.server: New leader elected: payload=Node-7517cedb-4028-489e-d5cd-15dcbc434503 +31:32.554 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29825 +31:32.638 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.672 [INFO] TestAgent: Synced node info +31:32.672 [DEBUG] TestAgent: Node info in sync +31:32.689 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.689 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.689 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.689 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.689 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.689 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7517cedb-4028-489e-d5cd-15dcbc434503 +31:32.689 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7517cedb-4028-489e-d5cd-15dcbc434503 +31:32.695 [DEBUG] TestAgent: Node info in sync +31:32.707 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.741 [INFO] TestAgent: Requesting shutdown +31:32.741 [INFO] TestAgent.server: shutting down server +31:32.741 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.741 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.741 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.741 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.741 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.741 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.741 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.741 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.751 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.762Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.762Z [INFO] agent.router.manager: shutting down +31:32.762 [INFO] TestAgent: consul server down +31:32.762 [INFO] TestAgent: shutdown complete +31:32.762 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=tcp +31:32.762 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=udp +31:32.762 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29821 network=tcp +31:33.263 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.263 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_badBody (0.93s) +=== CONT TestConfig_Get +2021-10-05T16:31:32.381Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.394Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.432 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7c879a0-352c-91a8-622b-4fdfbf14cbaa Address:127.0.0.1:29818}]" +31:32.432 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29818 [Follower]" leader= +31:32.433 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa.dc1 127.0.0.1 +31:32.433 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa 127.0.0.1 +2021-10-05T16:31:32.433Z [INFO] agent.router: Initializing LAN area manager +31:32.433 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=udp +31:32.434 [INFO] TestAgent.server: Adding LAN server: server="Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa (Addr: tcp/127.0.0.1:29818) (DC: dc1)" +31:32.434 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa.dc1 area=wan +31:32.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=tcp +31:32.434 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29814 network=tcp +31:32.434 [INFO] TestAgent: started state syncer +31:32.434 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29819 network=tcp +31:32.476 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.476 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29818 [Candidate]" term=2 +31:32.528 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.528 [DEBUG] TestAgent.server.raft: vote granted: from=c7c879a0-352c-91a8-622b-4fdfbf14cbaa term=2 tally=1 +31:32.528 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.528 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29818 [Leader]" +31:32.532 [INFO] TestAgent.server: cluster leadership acquired +31:32.532 [INFO] TestAgent.server: New leader elected: payload=Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa +31:32.559 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29818 +31:32.627 [INFO] TestAgent: Synced node info +31:32.666 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.702 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.702 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.702 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.702 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.702 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.702 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa +31:32.702 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7c879a0-352c-91a8-622b-4fdfbf14cbaa +31:32.720 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.743 [INFO] TestAgent.server.connect: CA provider config updated +31:32.743 [INFO] TestAgent: Requesting shutdown +31:32.743 [INFO] TestAgent.server: shutting down server +31:32.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.743 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.743 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.743 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.743 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.743 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.744 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.744 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.757 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.769Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.769Z [INFO] agent.router.manager: shutting down +31:32.770 [INFO] TestAgent: consul server down +31:32.770 [INFO] TestAgent: shutdown complete +31:32.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=tcp +31:32.770 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=udp +31:32.770 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29814 network=tcp +31:33.272 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.272 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCAConfig (5.78s) + --- PASS: TestConnectCAConfig/basic (0.95s) + --- PASS: TestConnectCAConfig/basic_with_IntermediateCertTTL (1.07s) + --- PASS: TestConnectCAConfig/force_without_cross_sign_CamelCase (1.10s) + --- PASS: TestConnectCAConfig/force_without_cross_sign_snake_case (0.84s) + --- PASS: TestConnectCAConfig/setting_state_fails (0.93s) + --- PASS: TestConnectCAConfig/updating_config_with_same_state (0.90s) +=== CONT TestCatalog_GatewayServices_Ingress +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:32.542Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.606Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.700 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6f212a8f-5533-12d4-e989-e481ead2b056 Address:127.0.0.1:29797}]" +31:32.700 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29797 [Follower]" leader= +31:32.701 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6f212a8f-5533-12d4-e989-e481ead2b056.dc1 127.0.0.1 +31:32.701 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6f212a8f-5533-12d4-e989-e481ead2b056 127.0.0.1 +2021-10-05T16:31:32.701Z [INFO] agent.router: Initializing LAN area manager +31:32.702 [INFO] TestAgent.server: Adding LAN server: server="Node-6f212a8f-5533-12d4-e989-e481ead2b056 (Addr: tcp/127.0.0.1:29797) (DC: dc1)" +31:32.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=udp +31:32.702 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6f212a8f-5533-12d4-e989-e481ead2b056.dc1 area=wan +31:32.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=tcp +31:32.702 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29828 network=tcp +31:32.702 [INFO] TestAgent: started state syncer +31:32.703 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29798 network=tcp +31:32.745 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.745 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29797 [Candidate]" term=2 +31:32.769 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.769 [DEBUG] TestAgent.server.raft: vote granted: from=6f212a8f-5533-12d4-e989-e481ead2b056 term=2 tally=1 +31:32.769 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.769 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29797 [Leader]" +31:32.770 [INFO] TestAgent.server: cluster leadership acquired +31:32.770 [INFO] TestAgent.server: New leader elected: payload=Node-6f212a8f-5533-12d4-e989-e481ead2b056 +31:32.786 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29797 +31:32.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6f212a8f-5533-12d4-e989-e481ead2b056 +31:32.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6f212a8f-5533-12d4-e989-e481ead2b056 +31:32.901 [INFO] TestAgent: Requesting shutdown +31:32.902 [INFO] TestAgent.server: shutting down server +31:32.902 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.902 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.902 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.902 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.902 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.902 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.902 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:32.902 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.902 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.902 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.916 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.925Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.925Z [INFO] agent.router.manager: shutting down +31:32.925 [INFO] TestAgent: consul server down +31:32.925 [INFO] TestAgent: shutdown complete +31:32.925 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=tcp +31:32.925 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=udp +31:32.925 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29828 network=tcp +31:33.427 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.427 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply_TerminatingGateway (0.94s) +=== CONT TestCatalog_GatewayServices_Terminating +2021-10-05T16:31:32.281Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.302Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.384 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2412d7f2-cb2d-ab52-4bec-d28745791105 Address:127.0.0.1:29769}]" +31:32.384 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29769 [Follower]" leader= +31:32.384 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2412d7f2-cb2d-ab52-4bec-d28745791105.dc1 127.0.0.1 +31:32.385 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2412d7f2-cb2d-ab52-4bec-d28745791105 127.0.0.1 +2021-10-05T16:31:32.385Z [INFO] agent.router: Initializing LAN area manager +31:32.385 [INFO] TestAgent.server: Adding LAN server: server="Node-2412d7f2-cb2d-ab52-4bec-d28745791105 (Addr: tcp/127.0.0.1:29769) (DC: dc1)" +31:32.385 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=udp +31:32.385 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2412d7f2-cb2d-ab52-4bec-d28745791105.dc1 area=wan +31:32.385 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=tcp +31:32.386 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29835 network=tcp +31:32.386 [INFO] TestAgent: started state syncer +31:32.387 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29770 network=tcp +31:32.429 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.429 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29769 [Candidate]" term=2 +31:32.447 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.447 [DEBUG] TestAgent.server.raft: vote granted: from=2412d7f2-cb2d-ab52-4bec-d28745791105 term=2 tally=1 +31:32.448 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.448 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29769 [Leader]" +31:32.448 [INFO] TestAgent.server: cluster leadership acquired +31:32.448 [INFO] TestAgent.server: New leader elected: payload=Node-2412d7f2-cb2d-ab52-4bec-d28745791105 +31:32.484 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29769 +31:32.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.626 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.626 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.626 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.626 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.626 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.626 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2412d7f2-cb2d-ab52-4bec-d28745791105 +31:32.626 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2412d7f2-cb2d-ab52-4bec-d28745791105 +31:32.661 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.679 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:32.692 [INFO] TestAgent: Synced node info +31:32.692 [DEBUG] TestAgent: Node info in sync +31:33.034 [INFO] TestAgent: Requesting shutdown +31:33.034 [INFO] TestAgent.server: shutting down server +31:33.034 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.034 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.034 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.034 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.034 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.034 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.056Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.056Z [INFO] agent.router.manager: shutting down +31:33.056 [INFO] TestAgent: consul server down +31:33.056 [INFO] TestAgent: shutdown complete +31:33.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=tcp +31:33.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=udp +31:33.056 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29835 network=tcp +31:33.559 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.559 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_configured_max_txn_len +2021-10-05T16:31:32.687Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.725Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.792 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 Address:127.0.0.1:29832}]" +31:32.792 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29832 [Follower]" leader= +31:32.793 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492.dc1 127.0.0.1 +31:32.793 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 127.0.0.1 +2021-10-05T16:31:32.793Z [INFO] agent.router: Initializing LAN area manager +31:32.793 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=udp +31:32.793 [INFO] TestAgent.server: Adding LAN server: server="Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 (Addr: tcp/127.0.0.1:29832) (DC: dc1)" +31:32.793 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492.dc1 area=wan +31:32.794 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=tcp +31:32.794 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29800 network=tcp +31:32.794 [INFO] TestAgent: started state syncer +31:32.794 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29833 network=tcp +31:32.844 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.844 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29832 [Candidate]" term=2 +31:32.872 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.872 [DEBUG] TestAgent.server.raft: vote granted: from=c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 term=2 tally=1 +31:32.872 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.872 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29832 [Leader]" +31:32.873 [INFO] TestAgent.server: cluster leadership acquired +31:32.873 [INFO] TestAgent.server: New leader elected: payload=Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 +31:32.908 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29832 +31:32.955 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.994 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.994 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.994 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.994 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.994 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.994 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 +31:32.994 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c2b54acd-16cb-d111-ab1c-b0a2a3ccd492 +31:33.013 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.051 [INFO] TestAgent: Requesting shutdown +31:33.051 [INFO] TestAgent.server: shutting down server +31:33.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.051 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.051 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.051 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.051 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.051 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:33.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.051 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.051 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.059 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.064Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.064Z [INFO] agent.router.manager: shutting down +31:33.064 [INFO] TestAgent: consul server down +31:33.064 [INFO] TestAgent: shutdown complete +31:33.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=tcp +31:33.064 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=udp +31:33.064 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29800 network=tcp +31:33.568 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.568 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_KV_Actions (1.78s) + --- PASS: TestTxnEndpoint_KV_Actions/#00 (0.88s) + --- PASS: TestTxnEndpoint_KV_Actions/#01 (0.90s) +=== CONT TestCatalogNodeServices_WanTranslation +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestConfig_Get/get_a_single_service_entry +=== RUN TestConfig_Get/list_both_service_entries +=== RUN TestConfig_Get/get_global_proxy_config +=== RUN TestConfig_Get/error_on_no_arguments +2021-10-05T16:31:32.709Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.752Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.856 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4beaadf6-7268-14e4-880f-29e905271b64 Address:127.0.0.1:29804}]" +31:32.857 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4beaadf6-7268-14e4-880f-29e905271b64.dc1 127.0.0.1 +31:32.857 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4beaadf6-7268-14e4-880f-29e905271b64 127.0.0.1 +2021-10-05T16:31:32.857Z [INFO] agent.router: Initializing LAN area manager +31:32.857 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=udp +31:32.857 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29804 [Follower]" leader= +31:32.858 [INFO] TestAgent.server: Adding LAN server: server="Node-4beaadf6-7268-14e4-880f-29e905271b64 (Addr: tcp/127.0.0.1:29804) (DC: dc1)" +31:32.858 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4beaadf6-7268-14e4-880f-29e905271b64.dc1 area=wan +31:32.858 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=tcp +31:32.859 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29793 network=tcp +31:32.859 [INFO] TestAgent: started state syncer +31:32.859 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29805 network=tcp +31:32.899 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.899 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29804 [Candidate]" term=2 +31:32.939 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.939 [DEBUG] TestAgent.server.raft: vote granted: from=4beaadf6-7268-14e4-880f-29e905271b64 term=2 tally=1 +31:32.939 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.939 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29804 [Leader]" +31:32.944 [INFO] TestAgent.server: cluster leadership acquired +31:32.944 [INFO] TestAgent.server: New leader elected: payload=Node-4beaadf6-7268-14e4-880f-29e905271b64 +31:32.958 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29804 +31:33.000 [INFO] TestAgent: Synced node info +31:33.000 [DEBUG] TestAgent: Node info in sync +31:33.017 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.039 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.039 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.039 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.039 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.039 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.039 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4beaadf6-7268-14e4-880f-29e905271b64 +31:33.039 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4beaadf6-7268-14e4-880f-29e905271b64 +31:33.060 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.229 [INFO] TestAgent: Requesting shutdown +31:33.229 [INFO] TestAgent.server: shutting down server +31:33.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.229 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.229 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.229 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.230 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.230 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.230 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.230 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.230 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.237 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.244Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.244Z [INFO] agent.router.manager: shutting down +31:33.244 [INFO] TestAgent: consul server down +31:33.244 [INFO] TestAgent: shutdown complete +31:33.244 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=tcp +31:33.244 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=udp +31:33.244 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29793 network=tcp +31:33.745 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.745 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Apply (1.06s) +=== CONT TestCatalogNodeServices_ConnectProxy +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:33.145Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.156Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.186 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55aa24cc-9854-d153-17ee-0fcfb7da8b3f Address:127.0.0.1:29811}]" +31:33.187 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f.dc1 127.0.0.1 +31:33.187 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f 127.0.0.1 +2021-10-05T16:31:33.187Z [INFO] agent.router: Initializing LAN area manager +31:33.188 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29811 [Follower]" leader= +31:33.188 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=udp +31:33.188 [INFO] TestAgent.server: Adding LAN server: server="Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f (Addr: tcp/127.0.0.1:29811) (DC: dc1)" +31:33.188 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=tcp +31:33.188 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f.dc1 area=wan +31:33.188 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29807 network=tcp +31:33.188 [INFO] TestAgent: started state syncer +31:33.189 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29812 network=tcp +31:33.232 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.232 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29811 [Candidate]" term=2 +31:33.253 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.253 [DEBUG] TestAgent.server.raft: vote granted: from=55aa24cc-9854-d153-17ee-0fcfb7da8b3f term=2 tally=1 +31:33.253 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.253 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29811 [Leader]" +31:33.253 [INFO] TestAgent.server: cluster leadership acquired +31:33.253 [INFO] TestAgent.server: New leader elected: payload=Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f +31:33.262 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29811 +31:33.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.325 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.325 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.325 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.325 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.325 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.325 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f +31:33.325 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55aa24cc-9854-d153-17ee-0fcfb7da8b3f +31:33.355 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.377 [INFO] TestAgent: Requesting shutdown +31:33.377 [INFO] TestAgent.server: shutting down server +31:33.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.377 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.377 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.377 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.377 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.377 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:33.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.377 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.377 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.386 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.390Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.390Z [INFO] agent.router.manager: shutting down +31:33.390 [INFO] TestAgent: consul server down +31:33.390 [INFO] TestAgent: shutdown complete +31:33.390 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=tcp +31:33.390 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=udp +31:33.390 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29807 network=tcp +31:33.893 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.893 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Delete (0.76s) +=== CONT TestCatalogNodeServices_Filter +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:33.285Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.296Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.346 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:773f834a-d7e6-3ad2-72dc-64aee180fc13 Address:127.0.0.1:29776}]" +31:33.346 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-773f834a-d7e6-3ad2-72dc-64aee180fc13.dc1 127.0.0.1 +31:33.347 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-773f834a-d7e6-3ad2-72dc-64aee180fc13 127.0.0.1 +2021-10-05T16:31:33.347Z [INFO] agent.router: Initializing LAN area manager +31:33.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29776 [Follower]" leader= +31:33.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29855 network=udp +31:33.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29855 network=tcp +31:33.348 [INFO] TestAgent.server: Adding LAN server: server="Node-773f834a-d7e6-3ad2-72dc-64aee180fc13 (Addr: tcp/127.0.0.1:29776) (DC: dc1)" +31:33.348 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-773f834a-d7e6-3ad2-72dc-64aee180fc13.dc1 area=wan +31:33.349 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29856 network=tcp +31:33.349 [INFO] TestAgent: started state syncer +31:33.349 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29777 network=tcp +31:33.389 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.389 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29776 [Candidate]" term=2 +31:33.416 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.416 [DEBUG] TestAgent.server.raft: vote granted: from=773f834a-d7e6-3ad2-72dc-64aee180fc13 term=2 tally=1 +31:33.416 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.416 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29776 [Leader]" +31:33.416 [INFO] TestAgent.server: cluster leadership acquired +31:33.416 [INFO] TestAgent.server: New leader elected: payload=Node-773f834a-d7e6-3ad2-72dc-64aee180fc13 +31:33.433 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29776 +31:33.464 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.537 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.538 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.538 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.538 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.538 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.538 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-773f834a-d7e6-3ad2-72dc-64aee180fc13 +31:33.538 [INFO] TestAgent.server: member joined, marking health alive: member=Node-773f834a-d7e6-3ad2-72dc-64aee180fc13 +31:33.557 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.588 [INFO] TestAgent: Requesting shutdown +31:33.588 [INFO] TestAgent.server: shutting down server +31:33.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.589 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.589 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.589 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.589 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.589 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.589 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:33.589 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.589 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.589 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.596 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.605Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.605Z [INFO] agent.router.manager: shutting down +31:33.605 [INFO] TestAgent: consul server down +31:33.605 [INFO] TestAgent: shutdown complete +31:33.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29855 network=tcp +31:33.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29855 network=udp +31:33.605 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29856 network=tcp +31:34.107 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.107 [INFO] TestAgent: Endpoints down +--- PASS: TestConfig_Get (0.84s) + --- PASS: TestConfig_Get/get_a_single_service_entry (0.00s) + --- PASS: TestConfig_Get/list_both_service_entries (0.00s) + --- PASS: TestConfig_Get/get_global_proxy_config (0.00s) + --- PASS: TestConfig_Get/error_on_no_arguments (0.00s) +=== CONT TestCatalogNodeServiceList +2021-10-05T16:31:33.226Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.239Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.274 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0bf544df-bfb0-6f29-17bd-eb92143c1714 Address:127.0.0.1:29853}]" +31:33.275 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0bf544df-bfb0-6f29-17bd-eb92143c1714.dc1 127.0.0.1 +31:33.276 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0bf544df-bfb0-6f29-17bd-eb92143c1714 127.0.0.1 +2021-10-05T16:31:33.276Z [INFO] agent.router: Initializing LAN area manager +31:33.276 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29853 [Follower]" leader= +31:33.276 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=udp +31:33.276 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0bf544df-bfb0-6f29-17bd-eb92143c1714.dc1 area=wan +31:33.276 [INFO] TestAgent.server: Adding LAN server: server="Node-0bf544df-bfb0-6f29-17bd-eb92143c1714 (Addr: tcp/127.0.0.1:29853) (DC: dc1)" +31:33.276 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=tcp +31:33.277 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29772 network=tcp +31:33.277 [INFO] TestAgent: started state syncer +31:33.277 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29854 network=tcp +31:33.324 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.324 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29853 [Candidate]" term=2 +31:33.358 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.358 [DEBUG] TestAgent.server.raft: vote granted: from=0bf544df-bfb0-6f29-17bd-eb92143c1714 term=2 tally=1 +31:33.358 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.358 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29853 [Leader]" +31:33.358 [INFO] TestAgent.server: cluster leadership acquired +31:33.359 [INFO] TestAgent.server: New leader elected: payload=Node-0bf544df-bfb0-6f29-17bd-eb92143c1714 +31:33.367 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29853 +31:33.388 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.427 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.427 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.427 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.427 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.427 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.428 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0bf544df-bfb0-6f29-17bd-eb92143c1714 +31:33.428 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0bf544df-bfb0-6f29-17bd-eb92143c1714 +31:33.472 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.573 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:33.583 [INFO] TestAgent: Synced node info +31:33.583 [DEBUG] TestAgent: Node info in sync +31:33.615 [INFO] TestAgent: Requesting shutdown +31:33.615 [INFO] TestAgent.server: shutting down server +31:33.615 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.615 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.615 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.615 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.615 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.615 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.615 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.615 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.615 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.615 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.633 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.640Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.640Z [INFO] agent.router.manager: shutting down +31:33.640 [INFO] TestAgent: consul server down +31:33.640 [INFO] TestAgent: shutdown complete +31:33.640 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=tcp +31:33.640 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=udp +31:33.640 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29772 network=tcp +31:34.141 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.141 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Item/exceeds_default_max_kv_value_size +31:33.656 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-3bdd4432-80ea-b5eb-ae1d-f843ab756d8c 127.0.0.1 +2021-10-05T16:31:33.656Z [INFO] agent.router: Initializing LAN area manager +31:33.656 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-0865308a-0797-fd61-f89f-7f1d17c865fe: 127.0.0.1:29648 +31:33.657 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +31:33.657 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:29648: connect: connection refused +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [INFO] TestAgent.client.serf.lan: serf: Attempting re-join to previously known node: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239: 127.0.0.1:29683 +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:29683: connect: connection refused +31:33.657 [WARN] TestAgent.client.serf.lan: serf: Failed to re-join any previously known node +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=udp +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=tcp +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +31:33.657 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29884 network=tcp +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +31:33.657 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:33.657 [INFO] TestAgent: started state syncer +31:33.657 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +2021-10-05T16:31:33.657Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29889 network=tcp +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [INFO] TestAgent: Requesting shutdown +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [INFO] TestAgent.client: shutting down client +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.658 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:33.658 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:33.658Z [WARN] agent.router.manager: No servers available +31:33.677 [INFO] TestAgent: consul client down +31:33.677 [INFO] TestAgent: shutdown complete +31:33.677 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=tcp +31:33.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=udp +31:33.678 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29884 network=tcp +31:34.178 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.178 [INFO] TestAgent: Endpoints down +31:30.375 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 127.0.0.1 +2021-10-05T16:31:30.375Z [INFO] agent.router: Initializing LAN area manager +31:30.375 [DEBUG] TestAgent: added local registration for service: service=web-sidecar-proxy +31:30.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=udp +2021-10-05T16:31:30.375Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.375Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.375Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.375Z [WARN] agent.router.manager: No servers available +31:30.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=tcp +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +31:30.376 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:30.376 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +31:30.376 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +31:30.376 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.376Z [WARN] agent.router.manager: No servers available +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +31:30.377 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=service-http-checks:web error="error filling agent cache: Internal cache failure: service 'web' not in agent state" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.377Z [WARN] agent.router.manager: No servers available +31:30.377 [ERROR] TestAgent: error handling service update: error="error watching service config: No known Consul servers" +31:30.377 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29667 network=tcp +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [INFO] TestAgent: started state syncer +31:30.380 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [ERROR] TestAgent.proxycfg: watch error: service_id=web-sidecar-proxy id=discovery-chain:redis error="error filling agent cache: No known Consul servers" +31:30.380 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29686 network=tcp +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:30.380Z [WARN] agent.router.manager: No servers available +31:30.380 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:30.381 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:29648] +31:30.381 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:29648 +31:30.384 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-0865308a-0797-fd61-f89f-7f1d17c865fe 127.0.0.1 +31:30.384 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:30.384 [DEBUG] TestAgent: systemd notify failed: error="No socket" +2021-10-05T16:31:30.384Z [WARN] agent.router.manager: No servers available +31:30.384 [INFO] TestAgent.client: adding server: server="Node-0865308a-0797-fd61-f89f-7f1d17c865fe (Addr: tcp/127.0.0.1:29650) (DC: dc1)" +31:30.581 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:30.581 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.780 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:30.780 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.981 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:30.981 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.981 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:30.981 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:32.597 [INFO] TestAgent: Requesting shutdown +31:32.604 [INFO] TestAgent.client: shutting down client +31:32.604 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:32.616 [INFO] TestAgent: consul client down +31:32.616 [INFO] TestAgent: shutdown complete +31:32.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=tcp +31:32.616 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=udp +31:32.616 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29667 network=tcp +31:32.616 [ERROR] TestAgent.client: RPC failed to server: method=ConnectCA.Roots server=127.0.0.1:29650 error="rpc error making call: EOF" +31:32.616 [ERROR] TestAgent.client: RPC failed to server: method=ConfigEntry.ResolveServiceConfig server=127.0.0.1:29650 error="rpc error making call: EOF" +31:33.120 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.120 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:29.883Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:29.899Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:29.978 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0865308a-0797-fd61-f89f-7f1d17c865fe Address:127.0.0.1:29650}]" +31:29.978 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0865308a-0797-fd61-f89f-7f1d17c865fe.dc1 127.0.0.1 +31:29.979 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0865308a-0797-fd61-f89f-7f1d17c865fe 127.0.0.1 +2021-10-05T16:31:29.979Z [INFO] agent.router: Initializing LAN area manager +31:29.979 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=udp +31:29.979 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29650 [Follower]" leader= +31:29.979 [INFO] TestAgent.server: Adding LAN server: server="Node-0865308a-0797-fd61-f89f-7f1d17c865fe (Addr: tcp/127.0.0.1:29650) (DC: dc1)" +31:29.979 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0865308a-0797-fd61-f89f-7f1d17c865fe.dc1 area=wan +31:29.980 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=tcp +31:29.980 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29660 network=tcp +31:29.980 [INFO] TestAgent: started state syncer +31:29.980 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29651 network=tcp +31:30.039 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:30.039 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29650 [Candidate]" term=2 +31:30.057 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:30.057 [DEBUG] TestAgent.server.raft: vote granted: from=0865308a-0797-fd61-f89f-7f1d17c865fe term=2 tally=1 +31:30.057 [INFO] TestAgent.server.raft: election won: tally=1 +31:30.057 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29650 [Leader]" +31:30.060 [INFO] TestAgent.server: cluster leadership acquired +31:30.060 [INFO] TestAgent.server: New leader elected: payload=Node-0865308a-0797-fd61-f89f-7f1d17c865fe +31:30.073 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29650 +31:30.106 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:30.137 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:30.137 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:30.137 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:30.137 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:30.137 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:30.137 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0865308a-0797-fd61-f89f-7f1d17c865fe +31:30.137 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0865308a-0797-fd61-f89f-7f1d17c865fe +31:30.177 [INFO] TestAgent.server: federation state anti-entropy synced +31:30.357 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:30.362 [INFO] TestAgent: Synced node info +31:30.362 [DEBUG] TestAgent: Node info in sync +31:30.382 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:55700 +31:30.382 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 127.0.0.1 +31:30.382 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.576 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.775 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.976 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:30.982 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 +31:31.490 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:31.490 [DEBUG] TestAgent: Node info in sync +31:33.120 [INFO] TestAgent: Requesting shutdown +31:33.120 [INFO] TestAgent.server: shutting down server +31:33.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.120 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.120 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.120 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.120 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.120 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.120 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.120 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.120 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.124 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.128Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.128Z [INFO] agent.router.manager: shutting down +31:33.128 [INFO] TestAgent: consul server down +31:33.128 [INFO] TestAgent: shutdown complete +31:33.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=tcp +31:33.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=udp +31:33.128 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29660 network=tcp +31:33.493 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 (timeout reached) +31:33.629 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.629 [INFO] TestAgent: Endpoints down +31:33.980 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-2b0dda7f-32d4-6db2-3361-47c9e426c239 has failed, no acks received +--- PASS: TestServiceManager_PersistService_ConfigFiles (4.32s) +=== CONT TestCatalogNodeServices [WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:14.152Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.163Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.190 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:24dc964f-1814-3bd6-e49c-f0cc8075e827 Address:127.0.0.1:11095}]" -46:14.190 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11095 [Follower]" leader= -46:14.191 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-24dc964f-1814-3bd6-e49c-f0cc8075e827.dc1 127.0.0.1 -46:14.191 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-24dc964f-1814-3bd6-e49c-f0cc8075e827 127.0.0.1 -2022-11-07T22:46:14.191Z [INFO] agent.router: Initializing LAN area manager -46:14.191 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11125 network=udp -46:14.191 [INFO] TestAgent.server: Adding LAN server: server="Node-24dc964f-1814-3bd6-e49c-f0cc8075e827 (Addr: tcp/127.0.0.1:11095) (DC: dc1)" -46:14.191 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-24dc964f-1814-3bd6-e49c-f0cc8075e827.dc1 area=wan -46:14.191 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11125 network=tcp -46:14.192 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11126 network=tcp -46:14.192 [INFO] TestAgent: started state syncer -46:14.192 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11096 network=tcp -46:14.230 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.230 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11095 [Candidate]" term=2 -46:14.245 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.245 [DEBUG] TestAgent.server.raft: vote granted: from=24dc964f-1814-3bd6-e49c-f0cc8075e827 term=2 tally=1 -46:14.245 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.245 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11095 [Leader]" -46:14.246 [INFO] TestAgent.server: cluster leadership acquired -46:14.246 [INFO] TestAgent.server: New leader elected: payload=Node-24dc964f-1814-3bd6-e49c-f0cc8075e827 -46:14.257 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11095 -46:14.281 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.317 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.317 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.317 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.317 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.317 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.317 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24dc964f-1814-3bd6-e49c-f0cc8075e827 -46:14.317 [INFO] TestAgent.server: member joined, marking health alive: member=Node-24dc964f-1814-3bd6-e49c-f0cc8075e827 -46:14.335 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.408 [INFO] TestAgent: Requesting shutdown -46:14.408 [INFO] TestAgent.server: shutting down server -46:14.408 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.408 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.408 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.408 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.408 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.408 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.408 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.408 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.408 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.408 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.408 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.412 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.415Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.415Z [INFO] agent.router.manager: shutting down -46:14.415 [INFO] TestAgent: consul server down -46:14.415 [INFO] TestAgent: shutdown complete -46:14.415 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11125 network=tcp -46:14.415 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11125 network=udp -46:14.415 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11126 network=tcp -46:14.916 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.916 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes (0.78s) -=== CONT TestDNS_NodeLookup_CNAME -2022-11-07T22:46:14.208Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.217Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.258 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8e75db91-d51b-e778-175d-cfea3d921203 Address:127.0.0.1:11102}]" -46:14.258 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11102 [Follower]" leader= -46:14.259 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8e75db91-d51b-e778-175d-cfea3d921203.dc1 127.0.0.1 -46:14.259 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8e75db91-d51b-e778-175d-cfea3d921203 127.0.0.1 -2022-11-07T22:46:14.259Z [INFO] agent.router: Initializing LAN area manager -46:14.259 [INFO] TestAgent.server: Adding LAN server: server="Node-8e75db91-d51b-e778-175d-cfea3d921203 (Addr: tcp/127.0.0.1:11102) (DC: dc1)" -46:14.259 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8e75db91-d51b-e778-175d-cfea3d921203.dc1 area=wan -46:14.259 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11097 network=udp -46:14.259 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11097 network=tcp -46:14.260 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11098 network=tcp -46:14.260 [INFO] TestAgent: started state syncer -46:14.260 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11103 network=tcp -46:14.302 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.302 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11102 [Candidate]" term=2 -46:14.312 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.312 [DEBUG] TestAgent.server.raft: vote granted: from=8e75db91-d51b-e778-175d-cfea3d921203 term=2 tally=1 -46:14.312 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.312 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11102 [Leader]" -46:14.312 [INFO] TestAgent.server: cluster leadership acquired -46:14.312 [INFO] TestAgent.server: New leader elected: payload=Node-8e75db91-d51b-e778-175d-cfea3d921203 -46:14.322 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11102 -46:14.340 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.365 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.365 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.365 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.365 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.365 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.365 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e75db91-d51b-e778-175d-cfea3d921203 -46:14.365 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8e75db91-d51b-e778-175d-cfea3d921203 -46:14.371 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.492 [INFO] TestAgent: Requesting shutdown -46:14.492 [INFO] TestAgent.server: shutting down server -46:14.492 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.492 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.492 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.492 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.492 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.492 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.492 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.492 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.492 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.492 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.492 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.496 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.500Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.500Z [INFO] agent.router.manager: shutting down -46:14.500 [INFO] TestAgent: consul server down -46:14.500 [INFO] TestAgent: shutdown complete -46:14.500 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11097 network=tcp -46:14.500 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11097 network=udp -46:14.500 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11098 network=tcp -46:15.001 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.001 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogDatacenters (0.80s) -=== CONT TestConfig_Apply_Decoding +2021-10-05T16:31:33.310Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.337Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.403 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08d18ef2-ef16-71ff-26a8-eda800df430e Address:127.0.0.1:29846}]" +31:33.404 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29846 [Follower]" leader= +31:33.404 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-08d18ef2-ef16-71ff-26a8-eda800df430e.dc1 127.0.0.1 +31:33.404 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08d18ef2-ef16-71ff-26a8-eda800df430e 127.0.0.1 +2021-10-05T16:31:33.405Z [INFO] agent.router: Initializing LAN area manager +31:33.405 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=udp +31:33.405 [INFO] TestAgent.server: Adding LAN server: server="Node-08d18ef2-ef16-71ff-26a8-eda800df430e (Addr: tcp/127.0.0.1:29846) (DC: dc1)" +31:33.405 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-08d18ef2-ef16-71ff-26a8-eda800df430e.dc1 area=wan +31:33.405 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=tcp +31:33.406 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29842 network=tcp +31:33.406 [INFO] TestAgent: started state syncer +31:33.406 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29847 network=tcp +31:33.472 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.472 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29846 [Candidate]" term=2 +31:33.532 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.532 [DEBUG] TestAgent.server.raft: vote granted: from=08d18ef2-ef16-71ff-26a8-eda800df430e term=2 tally=1 +31:33.532 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.532 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29846 [Leader]" +31:33.533 [INFO] TestAgent.server: cluster leadership acquired +31:33.533 [INFO] TestAgent.server: New leader elected: payload=Node-08d18ef2-ef16-71ff-26a8-eda800df430e +31:33.565 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29846 +31:33.599 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.644 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.644 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.644 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.644 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.644 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.645 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08d18ef2-ef16-71ff-26a8-eda800df430e +31:33.645 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08d18ef2-ef16-71ff-26a8-eda800df430e +31:33.682 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.787 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:33.799 [INFO] TestAgent: Synced node info +31:33.799 [DEBUG] TestAgent: Node info in sync +31:33.832 [INFO] TestAgent: Requesting shutdown +31:33.833 [INFO] TestAgent.server: shutting down server +31:33.833 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.833 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.833 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.833 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.833 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.833 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.833 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.833 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.833 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.833 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.848 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.855Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.855Z [INFO] agent.router.manager: shutting down +31:33.855 [INFO] TestAgent: consul server down +31:33.855 [INFO] TestAgent: shutdown complete +31:33.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=tcp +31:33.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=udp +31:33.855 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29842 network=tcp +31:34.356 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.356 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalog_GatewayServices_Ingress (1.08s) +=== CONT TestCatalogConnectServiceNodes_good +2021-10-05T16:31:33.447Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.538Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.606 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04341cef-9df5-d94f-20cc-cf32f08050df Address:127.0.0.1:29874}]" +31:33.606 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04341cef-9df5-d94f-20cc-cf32f08050df.dc1 127.0.0.1 +31:33.607 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29874 [Follower]" leader= +31:33.607 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04341cef-9df5-d94f-20cc-cf32f08050df 127.0.0.1 +2021-10-05T16:31:33.607Z [INFO] agent.router: Initializing LAN area manager +31:33.607 [INFO] TestAgent.server: Adding LAN server: server="Node-04341cef-9df5-d94f-20cc-cf32f08050df (Addr: tcp/127.0.0.1:29874) (DC: dc1)" +31:33.607 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04341cef-9df5-d94f-20cc-cf32f08050df.dc1 area=wan +31:33.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=udp +31:33.607 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=tcp +31:33.608 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29849 network=tcp +31:33.608 [INFO] TestAgent: started state syncer +31:33.608 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29875 network=tcp +31:33.674 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.675 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29874 [Candidate]" term=2 +31:33.716 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.716 [DEBUG] TestAgent.server.raft: vote granted: from=04341cef-9df5-d94f-20cc-cf32f08050df term=2 tally=1 +31:33.716 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.716 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29874 [Leader]" +31:33.716 [INFO] TestAgent.server: cluster leadership acquired +31:33.716 [INFO] TestAgent.server: New leader elected: payload=Node-04341cef-9df5-d94f-20cc-cf32f08050df +31:33.739 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29874 +31:33.800 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.800 [INFO] TestAgent: Synced node info +31:33.835 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.835 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.835 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.835 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.835 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.835 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04341cef-9df5-d94f-20cc-cf32f08050df +31:33.835 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04341cef-9df5-d94f-20cc-cf32f08050df +31:33.863 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.964 [INFO] TestAgent: Requesting shutdown +31:33.964 [INFO] TestAgent.server: shutting down server +31:33.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.964 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.964 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.964 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.964 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.965 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.965 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.965 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.975 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.988Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.988Z [INFO] agent.router.manager: shutting down +31:33.988 [INFO] TestAgent: consul server down +31:33.988 [INFO] TestAgent: shutdown complete +31:33.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=tcp +31:33.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=udp +31:33.988 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29849 network=tcp +31:34.492 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.492 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalog_GatewayServices_Terminating (1.06s) +=== CONT TestCatalogServiceNodes_ConnectProxy +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:33.765Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.785Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.852 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fe11598e-031b-a157-4f29-f40d66db152c Address:127.0.0.1:29908}]" +31:33.852 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29908 [Follower]" leader= +31:33.853 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fe11598e-031b-a157-4f29-f40d66db152c.dc1 127.0.0.1 +31:33.853 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fe11598e-031b-a157-4f29-f40d66db152c 127.0.0.1 +2021-10-05T16:31:33.853Z [INFO] agent.router: Initializing LAN area manager +31:33.853 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=udp +31:33.854 [INFO] TestAgent.server: Adding LAN server: server="Node-fe11598e-031b-a157-4f29-f40d66db152c (Addr: tcp/127.0.0.1:29908) (DC: dc1)" +31:33.854 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fe11598e-031b-a157-4f29-f40d66db152c.dc1 area=wan +31:33.854 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=tcp +31:33.854 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29877 network=tcp +31:33.854 [INFO] TestAgent: started state syncer +31:33.854 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29909 network=tcp +31:33.912 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.912 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29908 [Candidate]" term=2 +31:33.968 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.968 [DEBUG] TestAgent.server.raft: vote granted: from=fe11598e-031b-a157-4f29-f40d66db152c term=2 tally=1 +31:33.968 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.968 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29908 [Leader]" +31:33.968 [INFO] TestAgent.server: cluster leadership acquired +31:33.968 [INFO] TestAgent.server: New leader elected: payload=Node-fe11598e-031b-a157-4f29-f40d66db152c +31:34.004 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29908 +31:34.027 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.067 [INFO] TestAgent: Synced node info +31:34.085 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.085 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.085 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.085 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.085 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.085 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fe11598e-031b-a157-4f29-f40d66db152c +31:34.085 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fe11598e-031b-a157-4f29-f40d66db152c +31:34.108 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.108 [INFO] TestAgent: Requesting shutdown +31:34.108 [INFO] TestAgent.server: shutting down server +31:34.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.108 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.108 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.108 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.108 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.108 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.108 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.120 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.130Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.130Z [INFO] agent.router.manager: shutting down +31:34.130 [INFO] TestAgent: consul server down +31:34.130 [INFO] TestAgent: shutdown complete +31:34.130 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=tcp +31:34.130 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=udp +31:34.130 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29877 network=tcp +31:34.632 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.632 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodeServices_ConnectProxy (0.89s) +=== CONT TestCatalogServiceNodes_DistanceSort +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:33.572Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.599Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.683 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f9d08421-26b7-024a-9d10-581a9adfdaef Address:127.0.0.1:29860}]" +31:33.683 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29860 [Follower]" leader= +31:33.684 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f9d08421-26b7-024a-9d10-581a9adfdaef.dc1 127.0.0.1 +31:33.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f9d08421-26b7-024a-9d10-581a9adfdaef 127.0.0.1 +2021-10-05T16:31:33.692Z [INFO] agent.router: Initializing LAN area manager +31:33.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=udp +31:33.692 [INFO] TestAgent.server: Adding LAN server: server="Node-f9d08421-26b7-024a-9d10-581a9adfdaef (Addr: tcp/127.0.0.1:29860) (DC: dc1)" +31:33.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f9d08421-26b7-024a-9d10-581a9adfdaef.dc1 area=wan +31:33.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=tcp +31:33.693 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29863 network=tcp +31:33.693 [INFO] TestAgent: started state syncer +31:33.693 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29861 network=tcp +31:33.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29860 [Candidate]" term=2 +31:33.751 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.751 [DEBUG] TestAgent.server.raft: vote granted: from=f9d08421-26b7-024a-9d10-581a9adfdaef term=2 tally=1 +31:33.751 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.751 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29860 [Leader]" +31:33.751 [INFO] TestAgent.server: cluster leadership acquired +31:33.751 [INFO] TestAgent.server: New leader elected: payload=Node-f9d08421-26b7-024a-9d10-581a9adfdaef +31:33.775 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29860 +31:33.826 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.908 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.908 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.908 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.908 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.908 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.909 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9d08421-26b7-024a-9d10-581a9adfdaef +31:33.909 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f9d08421-26b7-024a-9d10-581a9adfdaef +31:33.930 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.028 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:34.044 [INFO] TestAgent: Synced node info +31:34.044 [DEBUG] TestAgent: Node info in sync +31:34.232 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:34.232 [DEBUG] TestAgent: Node info in sync +31:34.322 [INFO] TestAgent: Requesting shutdown +31:34.322 [INFO] TestAgent.server: shutting down server +31:34.322 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.322 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.322 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.322 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.322 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.322 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.322 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.322 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.328 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.332Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.332Z [INFO] agent.router.manager: shutting down +31:34.332 [INFO] TestAgent: consul server down +31:34.332 [INFO] TestAgent: shutdown complete +31:34.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=tcp +31:34.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=udp +31:34.332 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29863 network=tcp +31:34.833 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.833 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Net/exceeds_default_max_kv_value_size +2021-10-05T16:31:33.947Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.998Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.035 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6a375cd2-7f4f-5903-969d-8e43871df074 Address:127.0.0.1:29922}]" +31:34.036 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29922 [Follower]" leader= +31:34.036 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6a375cd2-7f4f-5903-969d-8e43871df074.dc1 127.0.0.1 +31:34.037 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6a375cd2-7f4f-5903-969d-8e43871df074 127.0.0.1 +2021-10-05T16:31:34.037Z [INFO] agent.router: Initializing LAN area manager +31:34.037 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29910 network=udp +31:34.037 [INFO] TestAgent.server: Adding LAN server: server="Node-6a375cd2-7f4f-5903-969d-8e43871df074 (Addr: tcp/127.0.0.1:29922) (DC: dc1)" +31:34.037 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6a375cd2-7f4f-5903-969d-8e43871df074.dc1 area=wan +31:34.037 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29910 network=tcp +31:34.038 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29897 network=tcp +31:34.038 [INFO] TestAgent: started state syncer +31:34.038 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29923 network=tcp +31:34.097 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.097 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29922 [Candidate]" term=2 +31:34.125 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.125 [DEBUG] TestAgent.server.raft: vote granted: from=6a375cd2-7f4f-5903-969d-8e43871df074 term=2 tally=1 +31:34.125 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.125 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29922 [Leader]" +31:34.126 [INFO] TestAgent.server: cluster leadership acquired +31:34.126 [INFO] TestAgent.server: New leader elected: payload=Node-6a375cd2-7f4f-5903-969d-8e43871df074 +31:34.137 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29922 +31:34.187 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.253 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.253 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.253 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.253 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.253 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.253 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a375cd2-7f4f-5903-969d-8e43871df074 +31:34.253 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6a375cd2-7f4f-5903-969d-8e43871df074 +31:34.320 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.344 [INFO] TestAgent: Requesting shutdown +31:34.344 [INFO] TestAgent.server: shutting down server +31:34.344 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.344 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.344 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.344 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.344 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:34.344 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.344 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.349 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.355Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.355Z [INFO] agent.router.manager: shutting down +31:34.355 [INFO] TestAgent: consul server down +31:34.355 [INFO] TestAgent: shutdown complete +31:34.355 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29910 network=tcp +31:34.355 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29910 network=udp +31:34.355 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29897 network=tcp +31:34.856 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.856 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodeServices_Filter (0.96s) +=== CONT TestCatalogServiceNodes_WanTranslation +2021-10-05T16:31:34.134Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.150Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9656f8c8-bdcc-6d25-7636-df8cf1823825 Address:127.0.0.1:29929}]" +31:34.212 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29929 [Follower]" leader= +31:34.213 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9656f8c8-bdcc-6d25-7636-df8cf1823825.dc1 127.0.0.1 +31:34.213 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9656f8c8-bdcc-6d25-7636-df8cf1823825 127.0.0.1 +2021-10-05T16:31:34.213Z [INFO] agent.router: Initializing LAN area manager +31:34.214 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29892 network=udp +31:34.214 [INFO] TestAgent.server: Adding LAN server: server="Node-9656f8c8-bdcc-6d25-7636-df8cf1823825 (Addr: tcp/127.0.0.1:29929) (DC: dc1)" +31:34.214 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9656f8c8-bdcc-6d25-7636-df8cf1823825.dc1 area=wan +31:34.214 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29892 network=tcp +31:34.214 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29904 network=tcp +31:34.214 [INFO] TestAgent: started state syncer +31:34.215 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29930 network=tcp +31:34.252 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.252 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29929 [Candidate]" term=2 +31:34.277 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.277 [DEBUG] TestAgent.server.raft: vote granted: from=9656f8c8-bdcc-6d25-7636-df8cf1823825 term=2 tally=1 +31:34.277 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.277 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29929 [Leader]" +31:34.280 [INFO] TestAgent.server: cluster leadership acquired +31:34.282 [INFO] TestAgent.server: New leader elected: payload=Node-9656f8c8-bdcc-6d25-7636-df8cf1823825 +31:34.321 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29929 +31:34.356 [INFO] TestAgent: Synced node info +31:34.356 [DEBUG] TestAgent: Node info in sync +31:34.385 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.417 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.417 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.417 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.417 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.417 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.417 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9656f8c8-bdcc-6d25-7636-df8cf1823825 +31:34.417 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9656f8c8-bdcc-6d25-7636-df8cf1823825 +31:34.435 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.480 [INFO] TestAgent: Requesting shutdown +31:34.480 [INFO] TestAgent.server: shutting down server +31:34.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.480 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.480 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.480 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.480 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.480 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.480 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.480 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:34.480 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.492 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.508Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.508Z [INFO] agent.router.manager: shutting down +31:34.508 [INFO] TestAgent: consul server down +31:34.508 [INFO] TestAgent: shutdown complete +31:34.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29892 network=tcp +31:34.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29892 network=udp +31:34.508 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29904 network=tcp +31:35.012 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.012 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodeServiceList (0.90s) +=== CONT TestCatalogServiceNodes_Filter +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:34.204Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.218Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.300 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07d09523-14db-7de8-3cf6-a4ee9e24a9a5 Address:127.0.0.1:29901}]" +31:34.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29901 [Follower]" leader= +31:34.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5.dc1 127.0.0.1 +31:34.322 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5 127.0.0.1 +2021-10-05T16:31:34.322Z [INFO] agent.router: Initializing LAN area manager +31:34.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29917 network=udp +31:34.322 [INFO] TestAgent.server: Adding LAN server: server="Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5 (Addr: tcp/127.0.0.1:29901) (DC: dc1)" +31:34.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5.dc1 area=wan +31:34.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29917 network=tcp +31:34.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29918 network=tcp +31:34.323 [INFO] TestAgent: started state syncer +31:34.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29902 network=tcp +31:34.389 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.389 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29901 [Candidate]" term=2 +31:34.417 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.417 [DEBUG] TestAgent.server.raft: vote granted: from=07d09523-14db-7de8-3cf6-a4ee9e24a9a5 term=2 tally=1 +31:34.417 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.417 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29901 [Leader]" +31:34.420 [INFO] TestAgent.server: cluster leadership acquired +31:34.420 [INFO] TestAgent.server: New leader elected: payload=Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5 +31:34.427 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29901 +31:34.469 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.521 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.521 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.521 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.521 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.521 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.521 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5 +31:34.521 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07d09523-14db-7de8-3cf6-a4ee9e24a9a5 +31:34.543 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.567 [INFO] TestAgent: Requesting shutdown +31:34.567 [INFO] TestAgent.server: shutting down server +31:34.567 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.567 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.567 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.567 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.567 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.567 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.567 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.567 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:34.567 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.567 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.567 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.580 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.592Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.592Z [INFO] agent.router.manager: shutting down +31:34.592 [INFO] TestAgent: consul server down +31:34.592 [INFO] TestAgent: shutdown complete +31:34.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29917 network=tcp +31:34.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29917 network=udp +31:34.592 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29918 network=tcp +31:35.092 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.092 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodeServices (0.91s) +=== CONT TestCatalogServiceNodes_NodeMetaFilter +2021-10-05T16:31:34.154Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.165Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.254 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:90b641d9-8057-44ab-34b5-9be6af6a8195 Address:127.0.0.1:29915}]" +31:34.256 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29915 [Follower]" leader= +31:34.261 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-90b641d9-8057-44ab-34b5-9be6af6a8195.dc1 127.0.0.1 +31:34.262 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-90b641d9-8057-44ab-34b5-9be6af6a8195 127.0.0.1 +2021-10-05T16:31:34.262Z [INFO] agent.router: Initializing LAN area manager +31:34.262 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29931 network=udp +31:34.262 [INFO] TestAgent.server: Adding LAN server: server="Node-90b641d9-8057-44ab-34b5-9be6af6a8195 (Addr: tcp/127.0.0.1:29915) (DC: dc1)" +31:34.262 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-90b641d9-8057-44ab-34b5-9be6af6a8195.dc1 area=wan +31:34.264 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29931 network=tcp +31:34.265 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29932 network=tcp +31:34.265 [INFO] TestAgent: started state syncer +31:34.266 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29916 network=tcp +31:34.302 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.302 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29915 [Candidate]" term=2 +31:34.320 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.320 [DEBUG] TestAgent.server.raft: vote granted: from=90b641d9-8057-44ab-34b5-9be6af6a8195 term=2 tally=1 +31:34.320 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.320 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29915 [Leader]" +31:34.322 [INFO] TestAgent.server: cluster leadership acquired +31:34.322 [INFO] TestAgent.server: New leader elected: payload=Node-90b641d9-8057-44ab-34b5-9be6af6a8195 +31:34.360 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29915 +31:34.402 [INFO] TestAgent: Synced node info +31:34.414 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.440 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.440 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.440 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.440 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.440 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.440 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90b641d9-8057-44ab-34b5-9be6af6a8195 +31:34.440 [INFO] TestAgent.server: member joined, marking health alive: member=Node-90b641d9-8057-44ab-34b5-9be6af6a8195 +31:34.460 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.677 [INFO] TestAgent: Requesting shutdown +31:34.677 [INFO] TestAgent.server: shutting down server +31:34.677 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.677 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.677 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.678 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.678 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.678 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.678 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.678 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.691 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.695Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.695Z [INFO] agent.router.manager: shutting down +31:34.696 [INFO] TestAgent: consul server down +31:34.696 [INFO] TestAgent: shutdown complete +31:34.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29931 network=tcp +31:34.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29931 network=udp +31:34.696 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29932 network=tcp +31:35.198 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.198 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Item/allowed +2021-10-05T16:31:34.392Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.409Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.469 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 Address:127.0.0.1:29936}]" +31:34.469 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29936 [Follower]" leader= +31:34.470 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6.dc1 127.0.0.1 +31:34.470 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 127.0.0.1 +2021-10-05T16:31:34.470Z [INFO] agent.router: Initializing LAN area manager +31:34.471 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29903 network=udp +31:34.471 [INFO] TestAgent.server: Adding LAN server: server="Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 (Addr: tcp/127.0.0.1:29936) (DC: dc1)" +31:34.471 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6.dc1 area=wan +31:34.471 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29903 network=tcp +31:34.472 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29911 network=tcp +31:34.472 [INFO] TestAgent: started state syncer +31:34.472 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29937 network=tcp +31:34.528 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.528 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29936 [Candidate]" term=2 +31:34.562 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.563 [DEBUG] TestAgent.server.raft: vote granted: from=f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 term=2 tally=1 +31:34.563 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.563 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29936 [Leader]" +31:34.563 [INFO] TestAgent.server: cluster leadership acquired +31:34.563 [INFO] TestAgent.server: New leader elected: payload=Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 +31:34.571 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29936 +31:34.597 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.664 [INFO] TestAgent: Synced node info +31:34.664 [DEBUG] TestAgent: Node info in sync +31:34.664 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.664 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.664 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.664 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.664 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.664 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 +31:34.665 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f5410bbc-0bc6-95c8-312a-b7e7db8ec4e6 +31:34.732 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.732 [INFO] TestAgent: Requesting shutdown +31:34.733 [INFO] TestAgent.server: shutting down server +31:34.733 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.733 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.733 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.733 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.733 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.733 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.733 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.733 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.733 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.733 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.740 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.752Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.752Z [INFO] agent.router.manager: shutting down +31:34.752 [INFO] TestAgent: consul server down +31:34.752 [INFO] TestAgent: shutdown complete +31:34.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29903 network=tcp +31:34.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29903 network=udp +31:34.752 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29911 network=tcp +31:35.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.252 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogConnectServiceNodes_good (0.90s) +=== CONT TestCatalogServiceNodes +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:33.992Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:33.992Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.011Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:34.011Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.056 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4d75d19b-f91b-db38-b2de-0810ef71dda5 Address:127.0.0.1:29881}]" +31:34.057 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 127.0.0.1 +31:34.057 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 127.0.0.1 +2021-10-05T16:31:34.057Z [INFO] agent.router: Initializing LAN area manager +31:34.058 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29924 network=udp +31:34.058 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29881 [Follower]" leader= +31:34.058 [INFO] TestAgent.server: Adding LAN server: server="Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 (Addr: tcp/127.0.0.1:29881) (DC: dc2)" +31:34.058 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 area=wan +31:34.058 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29924 network=tcp +31:34.059 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29925 network=tcp +31:34.059 [INFO] TestAgent: started state syncer +31:34.059 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29882 network=tcp +31:34.104 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.104 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29881 [Candidate]" term=2 +31:34.109 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:34.134 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.134 [DEBUG] TestAgent.server.raft: vote granted: from=4d75d19b-f91b-db38-b2de-0810ef71dda5 term=2 tally=1 +31:34.134 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.134 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29881 [Leader]" +31:34.135 [INFO] TestAgent.server: cluster leadership acquired +31:34.135 [INFO] TestAgent.server: New leader elected: payload=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 +31:34.148 [INFO] TestAgent.server: initializing acls +31:34.156 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:34.156 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:34.165 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:34.165 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:34.165 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:34.165 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 +31:34.165 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 +31:34.166 [INFO] TestAgent.server: Updating LAN server: server="Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 (Addr: tcp/127.0.0.1:29881) (DC: dc2)" +31:34.166 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 area=wan +31:34.206 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.264 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.264 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.264 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.264 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.264 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.264 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 +31:34.264 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 +31:34.276 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5 +31:34.288 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.320 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29866] +31:34.321 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29866 +31:34.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 127.0.0.1 +31:34.322 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:34.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 area=wan +31:34.337 [INFO] TestAgent: Requesting shutdown +31:34.337 [INFO] TestAgent.server: shutting down server +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.337 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.337 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.337 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.337 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.345 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.349Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.349Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.349Z [INFO] agent.router.manager: shutting down +31:34.349 [INFO] TestAgent: consul server down +31:34.349 [INFO] TestAgent: shutdown complete +31:34.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29924 network=tcp +31:34.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29924 network=udp +31:34.349 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29925 network=tcp +31:34.849 [INFO] TestAgent: Waiting for endpoints to shut down +31:34.849 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:33.607Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:33.607Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:33.637Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:33.637Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:33.712 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3e6f7546-d43c-b410-1efb-00a4db5a6927 Address:127.0.0.1:29867}]" +31:33.713 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 127.0.0.1 +31:33.713 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 127.0.0.1 +2021-10-05T16:31:33.713Z [INFO] agent.router: Initializing LAN area manager +31:33.713 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29867 [Follower]" leader= +31:33.714 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=udp +31:33.714 [INFO] TestAgent.server: Adding LAN server: server="Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 (Addr: tcp/127.0.0.1:29867) (DC: dc1)" +31:33.714 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 area=wan +31:33.714 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=tcp +31:33.715 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29870 network=tcp +31:33.715 [INFO] TestAgent: started state syncer +31:33.715 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29868 network=tcp +31:33.757 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:33.758 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29867 [Candidate]" term=2 +31:33.764 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:33.786 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:33.786 [DEBUG] TestAgent.server.raft: vote granted: from=3e6f7546-d43c-b410-1efb-00a4db5a6927 term=2 tally=1 +31:33.786 [INFO] TestAgent.server.raft: election won: tally=1 +31:33.786 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29867 [Leader]" +31:33.786 [INFO] TestAgent.server: cluster leadership acquired +31:33.786 [INFO] TestAgent.server: New leader elected: payload=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 +31:33.801 [INFO] TestAgent.server: initializing acls +31:33.813 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:33.830 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:33.830 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:33.830 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:33.830 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 +31:33.830 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 +31:33.831 [INFO] TestAgent.server: Updating LAN server: server="Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 (Addr: tcp/127.0.0.1:29867) (DC: dc1)" +31:33.831 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927.dc1 area=wan +31:33.848 [INFO] TestAgent: Synced node info +31:33.848 [DEBUG] TestAgent: Node info in sync +31:33.881 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:33.929 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:33.929 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:33.929 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:33.929 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:33.929 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:33.929 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 +31:33.929 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 +31:33.961 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e6f7546-d43c-b410-1efb-00a4db5a6927 +31:33.972 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.321 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:44286 +31:34.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 127.0.0.1 +31:34.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4d75d19b-f91b-db38-b2de-0810ef71dda5.dc2 area=wan +31:34.849 [INFO] TestAgent: Requesting shutdown +31:34.850 [INFO] TestAgent.server: shutting down server +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.850 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:34.850 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.850 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.850 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:34.857 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.871Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.871Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.871Z [INFO] agent.router.manager: shutting down +31:34.872 [INFO] TestAgent: consul server down +31:34.872 [INFO] TestAgent: shutdown complete +31:34.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=tcp +31:34.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=udp +31:34.872 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29870 network=tcp +31:35.388 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.388 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodeServices_WanTranslation (1.82s) +=== CONT TestCatalogRegister_checkRegistration +2021-10-05T16:31:34.518Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.556Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.635 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04b09b2b-4c5f-d493-72af-8a34a72af29e Address:127.0.0.1:29943}]" +31:34.636 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29943 [Follower]" leader= +31:34.636 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04b09b2b-4c5f-d493-72af-8a34a72af29e.dc1 127.0.0.1 +31:34.637 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04b09b2b-4c5f-d493-72af-8a34a72af29e 127.0.0.1 +2021-10-05T16:31:34.637Z [INFO] agent.router: Initializing LAN area manager +31:34.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29938 network=udp +31:34.637 [INFO] TestAgent.server: Adding LAN server: server="Node-04b09b2b-4c5f-d493-72af-8a34a72af29e (Addr: tcp/127.0.0.1:29943) (DC: dc1)" +31:34.637 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04b09b2b-4c5f-d493-72af-8a34a72af29e.dc1 area=wan +31:34.638 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29938 network=tcp +31:34.639 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29939 network=tcp +31:34.639 [INFO] TestAgent: started state syncer +31:34.646 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29944 network=tcp +31:34.696 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.696 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29943 [Candidate]" term=2 +31:34.748 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.748 [DEBUG] TestAgent.server.raft: vote granted: from=04b09b2b-4c5f-d493-72af-8a34a72af29e term=2 tally=1 +31:34.748 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.748 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29943 [Leader]" +31:34.748 [INFO] TestAgent.server: cluster leadership acquired +31:34.748 [INFO] TestAgent.server: New leader elected: payload=Node-04b09b2b-4c5f-d493-72af-8a34a72af29e +31:34.767 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29943 +31:34.791 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.829 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.829 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.829 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.829 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.829 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.829 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04b09b2b-4c5f-d493-72af-8a34a72af29e +31:34.829 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04b09b2b-4c5f-d493-72af-8a34a72af29e +31:34.852 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.892 [INFO] TestAgent: Requesting shutdown +31:34.893 [INFO] TestAgent.server: shutting down server +31:34.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.893 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.893 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.893 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.893 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:34.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.893 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.893 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.893 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:34.898 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.903Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.903Z [INFO] agent.router.manager: shutting down +31:34.904 [INFO] TestAgent: consul server down +31:34.904 [INFO] TestAgent: shutdown complete +31:34.904 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29938 network=tcp +31:34.904 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29938 network=udp +31:34.904 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29939 network=tcp +31:35.404 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.404 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes_ConnectProxy (0.91s) +=== CONT TestCatalogServices_NodeMetaFilter +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:34.665Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.690Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.731 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7455e268-89b0-5a7b-3dc4-3495da0cd0ca Address:127.0.0.1:29950}]" +31:34.732 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29950 [Follower]" leader= +31:34.732 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca.dc1 127.0.0.1 +31:34.732 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca 127.0.0.1 +2021-10-05T16:31:34.732Z [INFO] agent.router: Initializing LAN area manager +31:34.733 [INFO] TestAgent.server: Adding LAN server: server="Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca (Addr: tcp/127.0.0.1:29950) (DC: dc1)" +31:34.733 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29945 network=udp +31:34.733 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29945 network=tcp +31:34.733 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca.dc1 area=wan +31:34.733 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29946 network=tcp +31:34.733 [INFO] TestAgent: started state syncer +31:34.734 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29951 network=tcp +31:34.790 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.790 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29950 [Candidate]" term=2 +31:34.807 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.807 [DEBUG] TestAgent.server.raft: vote granted: from=7455e268-89b0-5a7b-3dc4-3495da0cd0ca term=2 tally=1 +31:34.807 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.807 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29950 [Leader]" +31:34.808 [INFO] TestAgent.server: cluster leadership acquired +31:34.808 [INFO] TestAgent.server: New leader elected: payload=Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca +31:34.817 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29950 +31:34.836 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.905 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.905 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.905 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.905 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.905 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.906 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca +31:34.906 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7455e268-89b0-5a7b-3dc4-3495da0cd0ca +31:34.923 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.121 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:35.153 [INFO] TestAgent: Synced node info +31:35.153 [DEBUG] TestAgent: Node info in sync +31:35.205 [INFO] TestAgent: Requesting shutdown +31:35.205 [INFO] TestAgent.server: shutting down server +31:35.205 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.205 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.205 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.205 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.205 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.205 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.205 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.205 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.205 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.205 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.214 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.220Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.220Z [INFO] agent.router.manager: shutting down +31:35.220 [INFO] TestAgent: consul server down +31:35.220 [INFO] TestAgent: shutdown complete +31:35.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29945 network=tcp +31:35.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29945 network=udp +31:35.220 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29946 network=tcp +31:35.720 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.720 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes_DistanceSort (1.09s) +=== CONT TestCatalogServices +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.108Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.122Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.184 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eea08ab8-7458-9819-488a-e1f8e320fbfd Address:127.0.0.1:29992}]" +31:35.184 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29992 [Follower]" leader= +31:35.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eea08ab8-7458-9819-488a-e1f8e320fbfd.dc1 127.0.0.1 +31:35.185 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eea08ab8-7458-9819-488a-e1f8e320fbfd 127.0.0.1 +2021-10-05T16:31:35.185Z [INFO] agent.router: Initializing LAN area manager +31:35.185 [INFO] TestAgent.server: Adding LAN server: server="Node-eea08ab8-7458-9819-488a-e1f8e320fbfd (Addr: tcp/127.0.0.1:29992) (DC: dc1)" +31:35.185 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eea08ab8-7458-9819-488a-e1f8e320fbfd.dc1 area=wan +31:35.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30001 network=udp +31:35.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30001 network=tcp +31:35.186 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29988 network=tcp +31:35.186 [INFO] TestAgent: started state syncer +31:35.186 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29993 network=tcp +31:35.224 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.224 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29992 [Candidate]" term=2 +31:35.239 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.239 [DEBUG] TestAgent.server.raft: vote granted: from=eea08ab8-7458-9819-488a-e1f8e320fbfd term=2 tally=1 +31:35.239 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.239 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29992 [Leader]" +31:35.239 [INFO] TestAgent.server: cluster leadership acquired +31:35.239 [INFO] TestAgent.server: New leader elected: payload=Node-eea08ab8-7458-9819-488a-e1f8e320fbfd +31:35.252 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29992 +31:35.320 [INFO] TestAgent: Requesting shutdown +31:35.320 [INFO] TestAgent.server: shutting down server +31:35.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.321 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:35.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.340 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.347Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.347Z [INFO] agent.router.manager: shutting down +31:35.356 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:35.356 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:35.356 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:35.356 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:35.356 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:35.356 [INFO] TestAgent: consul server down +31:35.356 [INFO] TestAgent: shutdown complete +31:35.356 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30001 network=tcp +31:35.356 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30001 network=udp +31:35.356 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29988 network=tcp +31:35.857 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.857 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes_NodeMetaFilter (0.79s) +=== CONT TestCatalogNodes_DistanceSort +2021-10-05T16:31:35.033Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.070Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.160 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 Address:127.0.0.1:29999}]" +31:35.160 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29999 [Follower]" leader= +31:35.161 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571.dc1 127.0.0.1 +31:35.161 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 127.0.0.1 +2021-10-05T16:31:35.161Z [INFO] agent.router: Initializing LAN area manager +31:35.162 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571.dc1 area=wan +31:35.162 [INFO] TestAgent.server: Adding LAN server: server="Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 (Addr: tcp/127.0.0.1:29999) (DC: dc1)" +31:35.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29973 network=udp +31:35.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29973 network=tcp +31:35.163 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29974 network=tcp +31:35.163 [INFO] TestAgent: started state syncer +31:35.164 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30000 network=tcp +31:35.215 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.215 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29999 [Candidate]" term=2 +31:35.234 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.234 [DEBUG] TestAgent.server.raft: vote granted: from=b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 term=2 tally=1 +31:35.234 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.234 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29999 [Leader]" +31:35.234 [INFO] TestAgent.server: cluster leadership acquired +31:35.235 [INFO] TestAgent.server: New leader elected: payload=Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 +31:35.246 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29999 +31:35.293 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.356 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.356 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.356 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.356 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.356 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.356 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 +31:35.356 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b38a1ccd-dca6-95bb-0d1a-3704c3ac5571 +31:35.416 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.442 [INFO] TestAgent: Requesting shutdown +31:35.442 [INFO] TestAgent.server: shutting down server +31:35.442 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.442 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.442 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.442 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.442 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.442 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.442 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.442 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:35.442 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.442 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.442 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.447 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.453Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.453Z [INFO] agent.router.manager: shutting down +31:35.453 [INFO] TestAgent: consul server down +31:35.453 [INFO] TestAgent: shutdown complete +31:35.453 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29973 network=tcp +31:35.453 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29973 network=udp +31:35.453 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29974 network=tcp +31:35.954 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.954 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes_Filter (0.94s) +=== CONT TestCatalogNodes_Blocking +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.308Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.342Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.426 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7773224f-46a5-0fd0-224c-037ee419ab59 Address:127.0.0.1:30020}]" +31:35.427 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7773224f-46a5-0fd0-224c-037ee419ab59.dc1 127.0.0.1 +31:35.427 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7773224f-46a5-0fd0-224c-037ee419ab59 127.0.0.1 +2021-10-05T16:31:35.427Z [INFO] agent.router: Initializing LAN area manager +31:35.427 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29980 network=udp +31:35.427 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30020 [Follower]" leader= +31:35.428 [INFO] TestAgent.server: Adding LAN server: server="Node-7773224f-46a5-0fd0-224c-037ee419ab59 (Addr: tcp/127.0.0.1:30020) (DC: dc1)" +31:35.428 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7773224f-46a5-0fd0-224c-037ee419ab59.dc1 area=wan +31:35.428 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29980 network=tcp +31:35.428 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29995 network=tcp +31:35.429 [INFO] TestAgent: started state syncer +31:35.429 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30021 network=tcp +31:35.471 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.471 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30020 [Candidate]" term=2 +31:35.486 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.486 [DEBUG] TestAgent.server.raft: vote granted: from=7773224f-46a5-0fd0-224c-037ee419ab59 term=2 tally=1 +31:35.486 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.486 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30020 [Leader]" +31:35.491 [INFO] TestAgent.server: cluster leadership acquired +31:35.492 [INFO] TestAgent.server: New leader elected: payload=Node-7773224f-46a5-0fd0-224c-037ee419ab59 +31:35.513 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30020 +31:35.515 [INFO] TestAgent: Synced node info +31:35.515 [DEBUG] TestAgent: Node info in sync +31:35.571 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.619 [INFO] TestAgent: Requesting shutdown +31:35.619 [INFO] TestAgent.server: shutting down server +31:35.619 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.631 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:35.640 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.640 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.640 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.640 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.640 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.640 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7773224f-46a5-0fd0-224c-037ee419ab59 +31:35.640 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7773224f-46a5-0fd0-224c-037ee419ab59 +2021-10-05T16:31:35.641Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.641Z [INFO] agent.router.manager: shutting down +31:35.664 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-7773224f-46a5-0fd0-224c-037ee419ab59 127.0.0.1 30018 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:7773224f-46a5-0fd0-224c-037ee419ab59 port:30020 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30019] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:35.664 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.664 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.664 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.664 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.664 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.664 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.664 [INFO] TestAgent: consul server down +31:35.664 [INFO] TestAgent: shutdown complete +31:35.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29980 network=tcp +31:35.664 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29980 network=udp +31:35.664 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29995 network=tcp +31:36.164 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.164 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes (0.91s) +=== CONT TestCatalogNodes_WanTranslation +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.423Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.437Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.482 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ed6153ae-8529-f842-774f-34d16cb7e9ff Address:127.0.0.1:29957}]" +31:35.482 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29957 [Follower]" leader= +31:35.484 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ed6153ae-8529-f842-774f-34d16cb7e9ff.dc1 127.0.0.1 +31:35.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ed6153ae-8529-f842-774f-34d16cb7e9ff 127.0.0.1 +2021-10-05T16:31:35.486Z [INFO] agent.router: Initializing LAN area manager +31:35.486 [INFO] TestAgent.server: Adding LAN server: server="Node-ed6153ae-8529-f842-774f-34d16cb7e9ff (Addr: tcp/127.0.0.1:29957) (DC: dc1)" +31:35.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29987 network=udp +31:35.487 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ed6153ae-8529-f842-774f-34d16cb7e9ff.dc1 area=wan +31:35.495 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29987 network=tcp +31:35.501 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30016 network=tcp +31:35.501 [INFO] TestAgent: started state syncer +31:35.501 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29958 network=tcp +31:35.553 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.553 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29957 [Candidate]" term=2 +31:35.564 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.564 [DEBUG] TestAgent.server.raft: vote granted: from=ed6153ae-8529-f842-774f-34d16cb7e9ff term=2 tally=1 +31:35.564 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.564 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29957 [Leader]" +31:35.565 [INFO] TestAgent.server: cluster leadership acquired +31:35.565 [INFO] TestAgent.server: New leader elected: payload=Node-ed6153ae-8529-f842-774f-34d16cb7e9ff +31:35.588 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29957 +31:35.628 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.700 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.700 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.700 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.700 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.700 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.700 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ed6153ae-8529-f842-774f-34d16cb7e9ff +31:35.700 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ed6153ae-8529-f842-774f-34d16cb7e9ff +31:35.748 [INFO] TestAgent: Synced node info +31:35.748 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.856 [INFO] TestAgent: Requesting shutdown +31:35.856 [INFO] TestAgent.server: shutting down server +31:35.856 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.856 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.856 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.856 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.856 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.857 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.857 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.857 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.868 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.880Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.880Z [INFO] agent.router.manager: shutting down +31:35.881 [INFO] TestAgent: consul server down +31:35.881 [INFO] TestAgent: shutdown complete +31:35.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29987 network=tcp +31:35.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29987 network=udp +31:35.881 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30016 network=tcp +31:36.381 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.381 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:35.417Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.441Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.533 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67998661-84b2-8f04-2935-cf2627cce119 Address:127.0.0.1:29985}]" +31:35.534 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29985 [Follower]" leader= +31:35.535 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67998661-84b2-8f04-2935-cf2627cce119.dc1 127.0.0.1 +31:35.537 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67998661-84b2-8f04-2935-cf2627cce119 127.0.0.1 +2021-10-05T16:31:35.537Z [INFO] agent.router: Initializing LAN area manager +31:35.537 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30008 network=udp +31:35.537 [INFO] TestAgent.server: Adding LAN server: server="Node-67998661-84b2-8f04-2935-cf2627cce119 (Addr: tcp/127.0.0.1:29985) (DC: dc1)" +31:35.537 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67998661-84b2-8f04-2935-cf2627cce119.dc1 area=wan +31:35.538 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30008 network=tcp +31:35.538 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29981 network=tcp +31:35.539 [INFO] TestAgent: started state syncer +31:35.539 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29986 network=tcp +31:35.585 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.585 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29985 [Candidate]" term=2 +31:35.594 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.595 [DEBUG] TestAgent.server.raft: vote granted: from=67998661-84b2-8f04-2935-cf2627cce119 term=2 tally=1 +31:35.595 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.595 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29985 [Leader]" +31:35.595 [INFO] TestAgent.server: cluster leadership acquired +31:35.595 [INFO] TestAgent.server: New leader elected: payload=Node-67998661-84b2-8f04-2935-cf2627cce119 +31:35.617 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29985 +31:35.660 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.728 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.728 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.728 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.728 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.728 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.728 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67998661-84b2-8f04-2935-cf2627cce119 +31:35.728 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67998661-84b2-8f04-2935-cf2627cce119 +31:35.750 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.858 [INFO] TestAgent: Requesting shutdown +31:35.858 [INFO] TestAgent.server: shutting down server +31:35.858 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.858 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.858 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.859 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.859 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:35.859 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.859 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.868 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.881Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.881Z [INFO] agent.router.manager: shutting down +31:35.881 [INFO] TestAgent: consul server down +31:35.881 [INFO] TestAgent: shutdown complete +31:35.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30008 network=tcp +31:35.881 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30008 network=udp +31:35.881 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29981 network=tcp +31:36.381 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.381 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServices_NodeMetaFilter (0.98s) +=== CONT TestCatalogNodes_Filter +--- PASS: TestCatalogRegister_checkRegistration (0.99s) +=== CONT TestCatalogNodes_MetaFilter +2021-10-05T16:31:34.853Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.874Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.927 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46e55962-7338-9397-85cf-a9a0102ed226 Address:127.0.0.1:29964}]" +31:34.927 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29964 [Follower]" leader= +31:34.928 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46e55962-7338-9397-85cf-a9a0102ed226.dc1 127.0.0.1 +31:34.929 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46e55962-7338-9397-85cf-a9a0102ed226 127.0.0.1 +2021-10-05T16:31:34.929Z [INFO] agent.router: Initializing LAN area manager +31:34.929 [INFO] TestAgent.server: Adding LAN server: server="Node-46e55962-7338-9397-85cf-a9a0102ed226 (Addr: tcp/127.0.0.1:29964) (DC: dc1)" +31:34.929 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46e55962-7338-9397-85cf-a9a0102ed226.dc1 area=wan +31:34.929 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29952 network=udp +31:34.929 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29952 network=tcp +31:34.930 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29960 network=tcp +31:34.930 [INFO] TestAgent: started state syncer +31:34.930 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29965 network=tcp +31:34.980 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.980 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29964 [Candidate]" term=2 +31:34.996 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.996 [DEBUG] TestAgent.server.raft: vote granted: from=46e55962-7338-9397-85cf-a9a0102ed226 term=2 tally=1 +31:34.996 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.996 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29964 [Leader]" +31:34.996 [INFO] TestAgent.server: cluster leadership acquired +31:34.996 [INFO] TestAgent.server: New leader elected: payload=Node-46e55962-7338-9397-85cf-a9a0102ed226 +31:35.004 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29964 +31:35.031 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.081 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.081 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.081 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.081 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.081 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.081 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46e55962-7338-9397-85cf-a9a0102ed226 +31:35.081 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46e55962-7338-9397-85cf-a9a0102ed226 +31:35.103 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.390 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:35.405 [INFO] TestAgent: Synced node info +31:35.900 [INFO] TestAgent: Requesting shutdown +31:35.900 [INFO] TestAgent.server: shutting down server +31:35.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.900 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.900 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.900 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.900 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.900 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.900 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.900 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.900 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.904 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.914Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.914Z [INFO] agent.router.manager: shutting down +31:35.914 [INFO] TestAgent: consul server down +31:35.914 [INFO] TestAgent: shutdown complete +31:35.914 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29952 network=tcp +31:35.914 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29952 network=udp +31:35.914 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29960 network=tcp +31:36.415 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.415 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Net/allowed +2021-10-05T16:31:35.210Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.221Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.294 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7e2c87e5-3239-613a-d02f-06e2c8524679 Address:127.0.0.1:29978}]" +31:35.295 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7e2c87e5-3239-613a-d02f-06e2c8524679.dc1 127.0.0.1 +31:35.296 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29978 [Follower]" leader= +31:35.296 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7e2c87e5-3239-613a-d02f-06e2c8524679 127.0.0.1 +2021-10-05T16:31:35.296Z [INFO] agent.router: Initializing LAN area manager +31:35.297 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29994 network=udp +31:35.297 [INFO] TestAgent.server: Adding LAN server: server="Node-7e2c87e5-3239-613a-d02f-06e2c8524679 (Addr: tcp/127.0.0.1:29978) (DC: dc1)" +31:35.297 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7e2c87e5-3239-613a-d02f-06e2c8524679.dc1 area=wan +31:35.297 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29994 network=tcp +31:35.300 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30002 network=tcp +31:35.300 [INFO] TestAgent: started state syncer +31:35.300 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29979 network=tcp +31:35.334 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.334 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29978 [Candidate]" term=2 +31:35.392 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.392 [DEBUG] TestAgent.server.raft: vote granted: from=7e2c87e5-3239-613a-d02f-06e2c8524679 term=2 tally=1 +31:35.392 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.392 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29978 [Leader]" +31:35.396 [INFO] TestAgent.server: cluster leadership acquired +31:35.396 [INFO] TestAgent.server: New leader elected: payload=Node-7e2c87e5-3239-613a-d02f-06e2c8524679 +31:35.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29978 +31:35.432 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.467 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.467 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.467 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.467 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.467 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.467 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7e2c87e5-3239-613a-d02f-06e2c8524679 +31:35.467 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7e2c87e5-3239-613a-d02f-06e2c8524679 +31:35.482 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.677 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:35.687 [INFO] TestAgent: Synced node info +31:35.687 [DEBUG] TestAgent: Node info in sync +31:35.930 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=1573029 +31:35.952 [INFO] TestAgent: Requesting shutdown +31:35.952 [INFO] TestAgent.server: shutting down server +31:35.952 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.952 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.952 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.952 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.952 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.952 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.952 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.952 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.952 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.952 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.957 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.963Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.963Z [INFO] agent.router.manager: shutting down +31:35.963 [INFO] TestAgent: consul server down +31:35.963 [INFO] TestAgent: shutdown complete +31:35.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29994 network=tcp +31:35.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29994 network=udp +31:35.964 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30002 network=tcp +31:36.465 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.465 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_Bad_Size_Item (4.17s) + --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_default_limits (0.91s) + --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_configured_max_txn_len (0.94s) + --- PASS: TestTxnEndpoint_Bad_Size_Item/exceeds_default_max_kv_value_size (1.06s) + --- PASS: TestTxnEndpoint_Bad_Size_Item/allowed (1.27s) +=== CONT TestCatalogNodes +2021-10-05T16:31:35.743Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.759Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.805 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd28c398-360c-2130-85f9-baf4679aff67 Address:127.0.0.1:30027}]" +31:35.805 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30027 [Follower]" leader= +31:35.807 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd28c398-360c-2130-85f9-baf4679aff67.dc1 127.0.0.1 +31:35.807 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd28c398-360c-2130-85f9-baf4679aff67 127.0.0.1 +2021-10-05T16:31:35.807Z [INFO] agent.router: Initializing LAN area manager +31:35.807 [INFO] TestAgent.server: Adding LAN server: server="Node-dd28c398-360c-2130-85f9-baf4679aff67 (Addr: tcp/127.0.0.1:30027) (DC: dc1)" +31:35.808 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd28c398-360c-2130-85f9-baf4679aff67.dc1 area=wan +31:35.808 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29959 network=tcp +31:35.808 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29959 network=udp +31:35.808 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29967 network=tcp +31:35.808 [INFO] TestAgent: started state syncer +31:35.808 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30028 network=tcp +31:35.852 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.852 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30027 [Candidate]" term=2 +31:35.880 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.880 [DEBUG] TestAgent.server.raft: vote granted: from=dd28c398-360c-2130-85f9-baf4679aff67 term=2 tally=1 +31:35.880 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.880 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30027 [Leader]" +31:35.880 [INFO] TestAgent.server: cluster leadership acquired +31:35.880 [INFO] TestAgent.server: New leader elected: payload=Node-dd28c398-360c-2130-85f9-baf4679aff67 +31:35.891 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30027 +31:35.912 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.950 [INFO] TestAgent: Synced node info +31:35.969 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.969 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.969 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.969 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.969 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.969 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd28c398-360c-2130-85f9-baf4679aff67 +31:35.969 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd28c398-360c-2130-85f9-baf4679aff67 +31:36.006 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.010 [INFO] TestAgent: Requesting shutdown +31:36.011 [INFO] TestAgent.server: shutting down server +31:36.011 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.011 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.011 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.011 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.011 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.011 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.011 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.011 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.011 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.011 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.024Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.024Z [INFO] agent.router.manager: shutting down +31:36.024 [INFO] TestAgent: consul server down +31:36.024 [INFO] TestAgent: shutdown complete +31:36.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29959 network=tcp +31:36.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29959 network=udp +31:36.024 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29967 network=tcp +31:36.524 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.524 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServices (0.80s) +=== CONT TestCatalogDatacenters +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.326Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:35.326Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.414Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:35.414Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.450 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:137ab113-4e0a-bec8-0b63-96709d67cddc Address:127.0.0.1:30006}]" +31:35.451 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 127.0.0.1 +31:35.451 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-137ab113-4e0a-bec8-0b63-96709d67cddc 127.0.0.1 +2021-10-05T16:31:35.451Z [INFO] agent.router: Initializing LAN area manager +31:35.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30006 [Follower]" leader= +31:35.452 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30022 network=udp +31:35.452 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 area=wan +31:35.452 [INFO] TestAgent.server: Adding LAN server: server="Node-137ab113-4e0a-bec8-0b63-96709d67cddc (Addr: tcp/127.0.0.1:30006) (DC: dc2)" +31:35.452 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30022 network=tcp +31:35.453 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30009 network=tcp +31:35.453 [INFO] TestAgent: started state syncer +31:35.453 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30007 network=tcp +31:35.504 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:35.515 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.515 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30006 [Candidate]" term=2 +31:35.524 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.524 [DEBUG] TestAgent.server.raft: vote granted: from=137ab113-4e0a-bec8-0b63-96709d67cddc term=2 tally=1 +31:35.524 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.524 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30006 [Leader]" +31:35.524 [INFO] TestAgent.server: cluster leadership acquired +31:35.524 [INFO] TestAgent.server: New leader elected: payload=Node-137ab113-4e0a-bec8-0b63-96709d67cddc +31:35.536 [INFO] TestAgent.server: initializing acls +31:35.540 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:35.553 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:35.553 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:35.553 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:35.553 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-137ab113-4e0a-bec8-0b63-96709d67cddc +31:35.554 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 +31:35.555 [INFO] TestAgent.server: Updating LAN server: server="Node-137ab113-4e0a-bec8-0b63-96709d67cddc (Addr: tcp/127.0.0.1:30006) (DC: dc2)" +31:35.555 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 area=wan +31:35.556 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29970] +31:35.572 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29970 +31:35.584 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 127.0.0.1 +31:35.584 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:35.585 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 area=wan +31:35.601 [INFO] TestAgent: Requesting shutdown +31:35.601 [INFO] TestAgent.server: shutting down server +31:35.601 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:35.601 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:35.601 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.601 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:35.601 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:35.601 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:35.612 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:35.616 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:31:35.625Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.625Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.625Z [INFO] agent.router.manager: shutting down +31:35.636 [INFO] TestAgent: consul server down +31:35.636 [INFO] TestAgent: shutdown complete +31:35.636 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30022 network=tcp +31:35.636 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30022 network=udp +31:35.636 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30009 network=tcp +31:35.636 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:35.636 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:36.136 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.136 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:34.894Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:34.894Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.918Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:34.918Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.970 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c620d067-0af0-c618-e96e-a20c31ee0858 Address:127.0.0.1:29971}]" +31:34.971 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 127.0.0.1 +31:34.971 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c620d067-0af0-c618-e96e-a20c31ee0858 127.0.0.1 +2021-10-05T16:31:34.971Z [INFO] agent.router: Initializing LAN area manager +31:34.972 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29971 [Follower]" leader= +31:34.972 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29966 network=udp +31:34.972 [INFO] TestAgent.server: Adding LAN server: server="Node-c620d067-0af0-c618-e96e-a20c31ee0858 (Addr: tcp/127.0.0.1:29971) (DC: dc1)" +31:34.972 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29966 network=tcp +31:34.972 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 area=wan +31:34.972 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29953 network=tcp +31:34.973 [INFO] TestAgent: started state syncer +31:34.973 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29972 network=tcp +31:35.009 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:35.009 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29971 [Candidate]" term=2 +31:35.022 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:35.040 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:35.040 [DEBUG] TestAgent.server.raft: vote granted: from=c620d067-0af0-c618-e96e-a20c31ee0858 term=2 tally=1 +31:35.040 [INFO] TestAgent.server.raft: election won: tally=1 +31:35.040 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29971 [Leader]" +31:35.041 [INFO] TestAgent.server: cluster leadership acquired +31:35.041 [INFO] TestAgent.server: New leader elected: payload=Node-c620d067-0af0-c618-e96e-a20c31ee0858 +31:35.056 [INFO] TestAgent.server: initializing acls +31:35.066 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:35.077 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:35.077 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:35.077 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:35.077 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c620d067-0af0-c618-e96e-a20c31ee0858 +31:35.077 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 +31:35.078 [INFO] TestAgent.server: Updating LAN server: server="Node-c620d067-0af0-c618-e96e-a20c31ee0858 (Addr: tcp/127.0.0.1:29971) (DC: dc1)" +31:35.078 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c620d067-0af0-c618-e96e-a20c31ee0858.dc1 area=wan +31:35.113 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:35.149 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:35.149 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:35.149 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:35.149 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:35.149 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:35.149 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c620d067-0af0-c618-e96e-a20c31ee0858 +31:35.149 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c620d067-0af0-c618-e96e-a20c31ee0858 +31:35.169 [INFO] TestAgent: Synced node info +31:35.169 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c620d067-0af0-c618-e96e-a20c31ee0858 +31:35.177 [INFO] TestAgent.server: federation state anti-entropy synced +31:35.321 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:35.334 [INFO] TestAgent: Synced node info +31:35.334 [DEBUG] TestAgent: Node info in sync +31:35.568 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:48878 +31:35.584 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 127.0.0.1 +31:35.584 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-137ab113-4e0a-bec8-0b63-96709d67cddc.dc2 area=wan +31:36.136 [INFO] TestAgent: Requesting shutdown +31:36.136 [INFO] TestAgent.server: shutting down server +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.136 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.137 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:36.137 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.141 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.154Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.154Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.154Z [INFO] agent.router.manager: shutting down +31:36.154 [INFO] TestAgent: consul server down +31:36.154 [INFO] TestAgent: shutdown complete +31:36.154 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29966 network=tcp +31:36.154 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29966 network=udp +31:36.154 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29953 network=tcp +31:36.654 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.654 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogServiceNodes_WanTranslation (1.80s) +=== CONT TestCatalogDeregister +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.919Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.931Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:35.992 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:99b8be02-856c-830e-aabf-4a8438f5a23b Address:127.0.0.1:30055}]" +31:35.993 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-99b8be02-856c-830e-aabf-4a8438f5a23b.dc1 127.0.0.1 +31:35.993 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30055 [Follower]" leader= +31:35.996 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-99b8be02-856c-830e-aabf-4a8438f5a23b 127.0.0.1 +2021-10-05T16:31:35.996Z [INFO] agent.router: Initializing LAN area manager +31:35.997 [INFO] TestAgent.server: Adding LAN server: server="Node-99b8be02-856c-830e-aabf-4a8438f5a23b (Addr: tcp/127.0.0.1:30055) (DC: dc1)" +31:35.997 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-99b8be02-856c-830e-aabf-4a8438f5a23b.dc1 area=wan +31:35.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30029 network=tcp +31:35.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30029 network=udp +31:35.998 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30051 network=tcp +31:35.998 [INFO] TestAgent: started state syncer +31:35.998 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30056 network=tcp +31:36.042 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.042 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30055 [Candidate]" term=2 +31:36.054 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.054 [DEBUG] TestAgent.server.raft: vote granted: from=99b8be02-856c-830e-aabf-4a8438f5a23b term=2 tally=1 +31:36.054 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.054 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30055 [Leader]" +31:36.058 [INFO] TestAgent.server: cluster leadership acquired +31:36.060 [INFO] TestAgent.server: New leader elected: payload=Node-99b8be02-856c-830e-aabf-4a8438f5a23b +31:36.063 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30055 +31:36.082 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.117 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.117 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.117 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.117 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.117 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.117 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-99b8be02-856c-830e-aabf-4a8438f5a23b +31:36.117 [INFO] TestAgent.server: member joined, marking health alive: member=Node-99b8be02-856c-830e-aabf-4a8438f5a23b +31:36.129 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.172 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.185 [INFO] TestAgent: Synced node info +31:36.185 [DEBUG] TestAgent: Node info in sync +31:36.512 [INFO] TestAgent: Requesting shutdown +31:36.512 [INFO] TestAgent.server: shutting down server +31:36.512 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.512 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.512 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.512 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.512 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.512 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.524 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.532Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.532Z [INFO] agent.router.manager: shutting down +31:36.532 [INFO] TestAgent: consul server down +31:36.532 [INFO] TestAgent: shutdown complete +31:36.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30029 network=tcp +31:36.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30029 network=udp +31:36.532 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30051 network=tcp +31:37.035 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.035 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes_DistanceSort (1.16s) +=== CONT TestCatalogRegister_Service_InvalidAddress +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:35.968Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:35.982Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.021 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 Address:127.0.0.1:30041}]" +31:36.021 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30041 [Follower]" leader= +31:36.022 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748.dc1 127.0.0.1 +31:36.022 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 127.0.0.1 +2021-10-05T16:31:36.022Z [INFO] agent.router: Initializing LAN area manager +31:36.023 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30057 network=udp +31:36.023 [INFO] TestAgent.server: Adding LAN server: server="Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 (Addr: tcp/127.0.0.1:30041) (DC: dc1)" +31:36.023 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30057 network=tcp +31:36.023 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748.dc1 area=wan +31:36.023 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30037 network=tcp +31:36.023 [INFO] TestAgent: started state syncer +31:36.023 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30042 network=tcp +31:36.068 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.068 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30041 [Candidate]" term=2 +31:36.120 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.120 [DEBUG] TestAgent.server.raft: vote granted: from=431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 term=2 tally=1 +31:36.120 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.120 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30041 [Leader]" +31:36.120 [INFO] TestAgent.server: cluster leadership acquired +31:36.120 [INFO] TestAgent.server: New leader elected: payload=Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 +31:36.152 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30041 +31:36.200 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.240 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.240 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.240 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.240 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.240 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.240 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 +31:36.240 [INFO] TestAgent.server: member joined, marking health alive: member=Node-431c8c1f-f169-9ff5-ea2a-0b15a3c1e748 +31:36.250 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.439 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.456 [INFO] TestAgent: Synced node info +31:36.590 [INFO] TestAgent: Requesting shutdown +31:36.590 [INFO] TestAgent.server: shutting down server +31:36.590 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.590 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.590 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.590 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.590 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.590 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.590 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.590 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.590 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.590 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.604 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.611Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.611Z [INFO] agent.router.manager: shutting down +31:36.611 [INFO] TestAgent: consul server down +31:36.611 [INFO] TestAgent: shutdown complete +31:36.611 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30057 network=tcp +31:36.611 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30057 network=udp +31:36.611 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30037 network=tcp +31:37.112 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.112 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes_Blocking (1.16s) +=== CONT TestAgent_JoinWAN_viaMeshGateway === RUN TestCatalogRegister_Service_InvalidAddress/addr_0.0.0.0 === RUN TestCatalogRegister_Service_InvalidAddress/addr_:: === RUN TestCatalogRegister_Service_InvalidAddress/addr_[::] -2022-11-07T22:46:14.231Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.241Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.294 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c80364ab-5e4e-8190-5998-1ecf12b50d0d Address:127.0.0.1:11039}]" -46:14.294 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11039 [Follower]" leader= -46:14.294 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d.dc1 127.0.0.1 -46:14.295 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d 127.0.0.1 -2022-11-07T22:46:14.295Z [INFO] agent.router: Initializing LAN area manager -46:14.295 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11104 network=udp -46:14.295 [INFO] TestAgent.server: Adding LAN server: server="Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d (Addr: tcp/127.0.0.1:11039) (DC: dc1)" -46:14.295 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d.dc1 area=wan -46:14.295 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11104 network=tcp -46:14.295 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11105 network=tcp -46:14.295 [INFO] TestAgent: started state syncer -46:14.296 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11040 network=tcp -46:14.339 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.339 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11039 [Candidate]" term=2 -46:14.353 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.353 [DEBUG] TestAgent.server.raft: vote granted: from=c80364ab-5e4e-8190-5998-1ecf12b50d0d term=2 tally=1 -46:14.353 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.353 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11039 [Leader]" -46:14.353 [INFO] TestAgent.server: cluster leadership acquired -46:14.353 [INFO] TestAgent.server: New leader elected: payload=Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d -46:14.361 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11039 -46:14.380 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.400 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.400 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.400 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.400 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.400 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.400 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d -46:14.400 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c80364ab-5e4e-8190-5998-1ecf12b50d0d -46:14.411 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.559 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=73.382µs client=127.0.0.1:59673 client_network=udp -46:14.559 [INFO] TestAgent: Requesting shutdown -46:14.559 [INFO] TestAgent.server: shutting down server -46:14.559 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.559 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.559 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.559 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.559 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.559 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.559 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.559 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.559 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.559 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.559 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.562 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.568Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.568Z [INFO] agent.router.manager: shutting down -46:14.568 [INFO] TestAgent: consul server down -46:14.568 [INFO] TestAgent: shutdown complete -46:14.568 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11104 network=tcp -46:14.568 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11104 network=udp -46:14.568 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11105 network=tcp -46:15.068 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.068 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ReverseLookup_CustomDomain (0.85s) -=== CONT TestAgent_AddCheck_Alias_setToken +2021-10-05T16:31:36.394Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.407Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.512 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b6336e9e-429d-d683-6644-92f7e41aeaeb Address:127.0.0.1:30048}]" +31:36.512 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30048 [Follower]" leader= +31:36.513 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b6336e9e-429d-d683-6644-92f7e41aeaeb.dc1 127.0.0.1 +31:36.513 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b6336e9e-429d-d683-6644-92f7e41aeaeb 127.0.0.1 +2021-10-05T16:31:36.513Z [INFO] agent.router: Initializing LAN area manager +31:36.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30064 network=udp +31:36.514 [INFO] TestAgent.server: Adding LAN server: server="Node-b6336e9e-429d-d683-6644-92f7e41aeaeb (Addr: tcp/127.0.0.1:30048) (DC: dc1)" +31:36.514 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b6336e9e-429d-d683-6644-92f7e41aeaeb.dc1 area=wan +31:36.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30064 network=tcp +31:36.514 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30044 network=tcp +31:36.514 [INFO] TestAgent: started state syncer +31:36.515 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30049 network=tcp +31:36.564 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.564 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30048 [Candidate]" term=2 +31:36.590 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.590 [DEBUG] TestAgent.server.raft: vote granted: from=b6336e9e-429d-d683-6644-92f7e41aeaeb term=2 tally=1 +31:36.590 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.590 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30048 [Leader]" +31:36.590 [INFO] TestAgent.server: cluster leadership acquired +31:36.590 [INFO] TestAgent.server: New leader elected: payload=Node-b6336e9e-429d-d683-6644-92f7e41aeaeb +31:36.610 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30048 +31:36.658 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.693 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.693 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.693 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.693 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.693 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.693 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b6336e9e-429d-d683-6644-92f7e41aeaeb +31:36.693 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b6336e9e-429d-d683-6644-92f7e41aeaeb +31:36.720 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.734 [INFO] TestAgent: Requesting shutdown +31:36.734 [INFO] TestAgent.server: shutting down server +31:36.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.734 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.734 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.734 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.734 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.734 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:36.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.734 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.734 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.740 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.744Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.744Z [INFO] agent.router.manager: shutting down +31:36.744 [INFO] TestAgent: consul server down +31:36.744 [INFO] TestAgent: shutdown complete +31:36.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30064 network=tcp +31:36.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30064 network=udp +31:36.744 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30044 network=tcp +31:37.248 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.248 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes_MetaFilter (0.87s) +=== CONT TestAgent_RerouteNewHTTPChecks +2021-10-05T16:31:36.542Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.556Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.611 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6fb11786-6934-caf2-f1ca-0e55e7d7644e Address:127.0.0.1:30111}]" +31:36.611 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30111 [Follower]" leader= +31:36.613 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e.dc1 127.0.0.1 +31:36.614 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e 127.0.0.1 +2021-10-05T16:31:36.614Z [INFO] agent.router: Initializing LAN area manager +31:36.614 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30015 network=udp +31:36.614 [INFO] TestAgent.server: Adding LAN server: server="Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e (Addr: tcp/127.0.0.1:30111) (DC: dc1)" +31:36.614 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e.dc1 area=wan +31:36.614 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30015 network=tcp +31:36.615 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30030 network=tcp +31:36.615 [INFO] TestAgent: started state syncer +31:36.615 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30112 network=tcp +31:36.679 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.679 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30111 [Candidate]" term=2 +31:36.698 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.698 [DEBUG] TestAgent.server.raft: vote granted: from=6fb11786-6934-caf2-f1ca-0e55e7d7644e term=2 tally=1 +31:36.698 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.698 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30111 [Leader]" +31:36.698 [INFO] TestAgent.server: cluster leadership acquired +31:36.699 [INFO] TestAgent.server: New leader elected: payload=Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e +31:36.709 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30111 +31:36.746 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.773 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.773 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.773 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.773 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.773 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.773 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e +31:36.773 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6fb11786-6934-caf2-f1ca-0e55e7d7644e +31:36.788 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.796 [INFO] TestAgent: Requesting shutdown +31:36.796 [INFO] TestAgent.server: shutting down server +31:36.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.796 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.796 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.796 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.796 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.796 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:36.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.796 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.796 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.804 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.817Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.817Z [INFO] agent.router.manager: shutting down +31:36.817 [INFO] TestAgent: consul server down +31:36.817 [INFO] TestAgent: shutdown complete +31:36.817 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30015 network=tcp +31:36.817 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30015 network=udp +31:36.817 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30030 network=tcp +31:37.319 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.319 [INFO] TestAgent: Endpoints down [WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:14.285Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.295Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.331 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dadabf35-193d-ecd6-2201-07441d93b058 Address:127.0.0.1:11046}]" -46:14.331 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11046 [Follower]" leader= -46:14.331 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dadabf35-193d-ecd6-2201-07441d93b058.dc1 127.0.0.1 -46:14.332 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dadabf35-193d-ecd6-2201-07441d93b058 127.0.0.1 -2022-11-07T22:46:14.332Z [INFO] agent.router: Initializing LAN area manager -46:14.332 [INFO] TestAgent.server: Adding LAN server: server="Node-dadabf35-193d-ecd6-2201-07441d93b058 (Addr: tcp/127.0.0.1:11046) (DC: dc1)" -46:14.332 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dadabf35-193d-ecd6-2201-07441d93b058.dc1 area=wan -46:14.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11041 network=udp -46:14.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11041 network=tcp -46:14.332 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11042 network=tcp -46:14.332 [INFO] TestAgent: started state syncer -46:14.332 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11047 network=tcp -46:14.367 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.367 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11046 [Candidate]" term=2 -46:14.378 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.378 [DEBUG] TestAgent.server.raft: vote granted: from=dadabf35-193d-ecd6-2201-07441d93b058 term=2 tally=1 -46:14.378 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.378 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11046 [Leader]" -46:14.378 [INFO] TestAgent.server: cluster leadership acquired -46:14.378 [INFO] TestAgent.server: New leader elected: payload=Node-dadabf35-193d-ecd6-2201-07441d93b058 -46:14.386 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11046 -46:14.401 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.423 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.423 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.423 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.423 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.423 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.423 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dadabf35-193d-ecd6-2201-07441d93b058 -46:14.423 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dadabf35-193d-ecd6-2201-07441d93b058 -46:14.434 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.683 [DEBUG] connect.ca.consul: consul CA provider configured: id=a3:30:f9:2c:b8:8b:23:9b:3e:38:97:8a:c3:90:d9:b4:19:3c:02:af:b5:14:ad:93:8b:93:ee:ae:62:ad:45:3e is_primary=true -46:14.711 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.727 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul -46:14.727 [INFO] TestAgent: Requesting shutdown -46:14.727 [INFO] TestAgent.server: shutting down server -46:14.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.727 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.727 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.727 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.727 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.727 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.727 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.727 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.727 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.732 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:14.734 [INFO] TestAgent: Synced node info -2022-11-07T22:46:14.740Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.740Z [INFO] agent.router.manager: shutting down -46:14.740 [INFO] TestAgent: consul server down -46:14.740 [INFO] TestAgent: shutdown complete -46:14.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11041 network=tcp -46:14.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11041 network=udp -46:14.740 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11042 network=tcp -46:15.241 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.241 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCARoots_list (0.97s) -=== CONT TestDNSCycleRecursorCheck -2022-11-07T22:46:14.334Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.352Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.386 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:790866d2-2bda-866f-675b-a173aeec1827 Address:127.0.0.1:11088}]" -46:14.386 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11088 [Follower]" leader= -46:14.386 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-790866d2-2bda-866f-675b-a173aeec1827.dc1 127.0.0.1 -46:14.387 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-790866d2-2bda-866f-675b-a173aeec1827 127.0.0.1 -2022-11-07T22:46:14.387Z [INFO] agent.router: Initializing LAN area manager -46:14.387 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-790866d2-2bda-866f-675b-a173aeec1827.dc1 area=wan -46:14.387 [INFO] TestAgent.server: Adding LAN server: server="Node-790866d2-2bda-866f-675b-a173aeec1827 (Addr: tcp/127.0.0.1:11088) (DC: dc1)" -46:14.387 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11048 network=tcp -46:14.387 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11048 network=udp -46:14.387 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11049 network=tcp -46:14.388 [INFO] TestAgent: started state syncer -46:14.388 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11089 network=tcp -46:14.428 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.428 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11088 [Candidate]" term=2 -46:14.439 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.439 [DEBUG] TestAgent.server.raft: vote granted: from=790866d2-2bda-866f-675b-a173aeec1827 term=2 tally=1 -46:14.439 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.439 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11088 [Leader]" -46:14.439 [INFO] TestAgent.server: cluster leadership acquired -46:14.439 [INFO] TestAgent.server: New leader elected: payload=Node-790866d2-2bda-866f-675b-a173aeec1827 -46:14.445 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11088 -46:14.457 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.481 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.481 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.481 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.481 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.481 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.481 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-790866d2-2bda-866f-675b-a173aeec1827 -46:14.481 [INFO] TestAgent.server: member joined, marking health alive: member=Node-790866d2-2bda-866f-675b-a173aeec1827 -46:14.493 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.555 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.559 [INFO] TestAgent: Synced node info -46:14.736 [WARN] TestAgent.server: endpoint injected; this should only be used for testing -46:14.736 [INFO] TestAgent: Requesting shutdown -46:14.736 [INFO] TestAgent.server: shutting down server -46:14.736 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.736 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.736 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.736 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.736 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.736 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.736 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.736 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.736 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.736 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.744 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.751Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.751Z [INFO] agent.router.manager: shutting down -46:14.751 [INFO] TestAgent: consul server down -46:14.751 [INFO] TestAgent: shutdown complete -46:14.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11048 network=tcp -46:14.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11048 network=udp -46:14.751 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11049 network=tcp -46:15.251 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.251 [INFO] TestAgent: Endpoints down -=== RUN TestPreparedQuery_Execute/#07 -2022-11-07T22:46:14.518Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.528Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.551 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88f3c407-3b12-f106-0cbc-d8b72a5143a0 Address:127.0.0.1:12003}]" -46:14.551 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12003 [Follower]" leader= -46:14.551 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0.dc1 127.0.0.1 -46:14.552 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0 127.0.0.1 -2022-11-07T22:46:14.552Z [INFO] agent.router: Initializing LAN area manager -46:14.552 [INFO] TestAgent.server: Adding LAN server: server="Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0 (Addr: tcp/127.0.0.1:12003) (DC: dc1)" -46:14.552 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0.dc1 area=wan -46:14.552 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11090 network=udp -46:14.552 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11090 network=tcp -46:14.552 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11091 network=tcp -46:14.552 [INFO] TestAgent: started state syncer -46:14.553 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12004 network=tcp -46:14.591 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.591 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12003 [Candidate]" term=2 -46:14.604 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.604 [DEBUG] TestAgent.server.raft: vote granted: from=88f3c407-3b12-f106-0cbc-d8b72a5143a0 term=2 tally=1 -46:14.604 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.604 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12003 [Leader]" -46:14.604 [INFO] TestAgent.server: cluster leadership acquired -46:14.604 [INFO] TestAgent.server: New leader elected: payload=Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0 -46:14.615 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12003 -46:14.631 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.653 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.653 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.653 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.653 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.653 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.653 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0 -46:14.653 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88f3c407-3b12-f106-0cbc-d8b72a5143a0 -46:14.671 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.705 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.712 [INFO] TestAgent: Synced node info -46:14.712 [DEBUG] TestAgent: Node info in sync -46:14.742 [DEBUG] TestAgent.dns: request served from client: question="{2.0.0.127.in-addr.arpa. 255 1}" latency=69.355µs client=127.0.0.1:58464 client_network=udp -46:14.742 [INFO] TestAgent: Requesting shutdown -46:14.742 [INFO] TestAgent.server: shutting down server -46:14.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.742 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.742 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.742 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.742 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.742 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.742 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.742 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.742 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.753Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.753Z [INFO] agent.router.manager: shutting down -46:14.753 [INFO] TestAgent: consul server down -46:14.753 [INFO] TestAgent: shutdown complete -46:14.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11090 network=tcp -46:14.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11090 network=udp -46:14.753 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11091 network=tcp -46:15.253 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.253 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ReverseLookup (0.75s) +--- PASS: TestCatalogDatacenters (0.81s) +=== CONT TestAgent_RerouteExistingHTTPChecks +2021-10-05T16:31:36.399Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.413Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.451 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 Address:127.0.0.1:30062}]" +31:36.451 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30062 [Follower]" leader= +31:36.452 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67.dc1 127.0.0.1 +31:36.454 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 127.0.0.1 +2021-10-05T16:31:36.454Z [INFO] agent.router: Initializing LAN area manager +31:36.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30036 network=udp +31:36.454 [INFO] TestAgent.server: Adding LAN server: server="Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 (Addr: tcp/127.0.0.1:30062) (DC: dc1)" +31:36.454 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67.dc1 area=wan +31:36.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30036 network=tcp +31:36.455 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30023 network=tcp +31:36.455 [INFO] TestAgent: started state syncer +31:36.455 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30063 network=tcp +31:36.519 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.519 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30062 [Candidate]" term=2 +31:36.543 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.543 [DEBUG] TestAgent.server.raft: vote granted: from=3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 term=2 tally=1 +31:36.543 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.543 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30062 [Leader]" +31:36.544 [INFO] TestAgent.server: cluster leadership acquired +31:36.544 [INFO] TestAgent.server: New leader elected: payload=Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 +31:36.563 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30062 +31:36.598 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.643 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.643 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.643 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.643 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.643 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.643 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 +31:36.643 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3e4ddb12-f2b8-9a7e-6cca-7d7f27894a67 +31:36.660 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.665 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.672 [INFO] TestAgent: Synced node info +31:36.859 [INFO] TestAgent: Requesting shutdown +31:36.859 [INFO] TestAgent.server: shutting down server +31:36.859 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.859 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.859 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.859 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.859 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.859 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.859 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.859 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.859 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.864 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.876Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.876Z [INFO] agent.router.manager: shutting down +31:36.876 [INFO] TestAgent: consul server down +31:36.876 [INFO] TestAgent: shutdown complete +31:36.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30036 network=tcp +31:36.876 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30036 network=udp +31:36.876 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30023 network=tcp +31:37.376 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.376 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes_Filter (1.00s) +=== CONT TestAgent_consulConfig_RaftTrailingLogs +2021-10-05T16:31:36.492Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.515Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.593 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7636d4cb-5a0e-2052-3fc8-20acb6214378 Address:127.0.0.1:30069}]" +31:36.594 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7636d4cb-5a0e-2052-3fc8-20acb6214378.dc1 127.0.0.1 +31:36.595 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7636d4cb-5a0e-2052-3fc8-20acb6214378 127.0.0.1 +2021-10-05T16:31:36.595Z [INFO] agent.router: Initializing LAN area manager +31:36.595 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30069 [Follower]" leader= +31:36.595 [INFO] TestAgent.server: Adding LAN server: server="Node-7636d4cb-5a0e-2052-3fc8-20acb6214378 (Addr: tcp/127.0.0.1:30069) (DC: dc1)" +31:36.596 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7636d4cb-5a0e-2052-3fc8-20acb6214378.dc1 area=wan +31:36.596 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30106 network=udp +31:36.596 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30106 network=tcp +31:36.596 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30093 network=tcp +31:36.597 [INFO] TestAgent: started state syncer +31:36.597 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30070 network=tcp +31:36.659 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.659 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30069 [Candidate]" term=2 +31:36.675 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.675 [DEBUG] TestAgent.server.raft: vote granted: from=7636d4cb-5a0e-2052-3fc8-20acb6214378 term=2 tally=1 +31:36.675 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.675 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30069 [Leader]" +31:36.675 [INFO] TestAgent.server: cluster leadership acquired +31:36.675 [INFO] TestAgent.server: New leader elected: payload=Node-7636d4cb-5a0e-2052-3fc8-20acb6214378 +31:36.682 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30069 +31:36.714 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.739 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.739 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.739 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.739 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.739 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.739 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7636d4cb-5a0e-2052-3fc8-20acb6214378 +31:36.739 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7636d4cb-5a0e-2052-3fc8-20acb6214378 +31:36.763 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.886 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.903 [INFO] TestAgent: Synced node info +31:36.903 [DEBUG] TestAgent: Node info in sync +31:36.927 [INFO] TestAgent: Requesting shutdown +31:36.927 [INFO] TestAgent.server: shutting down server +31:36.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.928 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.928 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.928 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.928 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.928 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.928 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.928 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.928 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.933 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:36.940Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.940Z [INFO] agent.router.manager: shutting down +31:36.941 [INFO] TestAgent: consul server down +31:36.941 [INFO] TestAgent: shutdown complete +31:36.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30106 network=tcp +31:36.941 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30106 network=udp +31:36.941 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30093 network=tcp +31:37.442 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.442 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes (0.98s) +=== CONT TestAgent_consulConfig_AutoEncryptAllowTLS +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:36.672Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.686Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.735 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f9dee954-19b2-aeae-d6ec-854464a3f9ff Address:127.0.0.1:30097}]" +31:36.735 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff.dc1 127.0.0.1 +31:36.736 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff 127.0.0.1 +2021-10-05T16:31:36.736Z [INFO] agent.router: Initializing LAN area manager +31:36.736 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30113 network=udp +31:36.736 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30097 [Follower]" leader= +31:36.737 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff.dc1 area=wan +31:36.737 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30113 network=tcp +31:36.737 [INFO] TestAgent.server: Adding LAN server: server="Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff (Addr: tcp/127.0.0.1:30097) (DC: dc1)" +31:36.737 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30114 network=tcp +31:36.737 [INFO] TestAgent: started state syncer +31:36.737 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30098 network=tcp +31:36.784 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.784 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30097 [Candidate]" term=2 +31:36.801 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.801 [DEBUG] TestAgent.server.raft: vote granted: from=f9dee954-19b2-aeae-d6ec-854464a3f9ff term=2 tally=1 +31:36.801 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.801 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30097 [Leader]" +31:36.801 [INFO] TestAgent.server: cluster leadership acquired +31:36.801 [INFO] TestAgent.server: New leader elected: payload=Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff +31:36.821 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30097 +31:36.856 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.930 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.930 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.930 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.930 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.930 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.930 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff +31:36.930 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f9dee954-19b2-aeae-d6ec-854464a3f9ff +31:36.951 [INFO] TestAgent.server: federation state anti-entropy synced +31:37.120 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:37.136 [INFO] TestAgent: Synced node info +31:37.168 [INFO] TestAgent: Requesting shutdown +31:37.168 [INFO] TestAgent.server: shutting down server +31:37.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.168 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.168 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.168 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.168 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.168 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.168 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.168 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.168 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.176 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.180Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.180Z [INFO] agent.router.manager: shutting down +31:37.180 [INFO] TestAgent: consul server down +31:37.180 [INFO] TestAgent: shutdown complete +31:37.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30113 network=tcp +31:37.181 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30113 network=udp +31:37.181 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30114 network=tcp +31:37.681 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.681 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogDeregister (1.03s) +=== CONT TestAgent_ReloadConfigTLSConfigFailure === CONT TestAgent_JoinWAN_viaMeshGateway -2022-11-07T22:46:14.591Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.601Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.635 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a7246d02-17f4-fa02-5a26-9bba10ecc916 Address:127.0.0.1:11081}]" -46:14.635 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11081 [Follower]" leader= -46:14.636 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a7246d02-17f4-fa02-5a26-9bba10ecc916.dc1 127.0.0.1 -46:14.636 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a7246d02-17f4-fa02-5a26-9bba10ecc916 127.0.0.1 -2022-11-07T22:46:14.636Z [INFO] agent.router: Initializing LAN area manager -46:14.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11139 network=udp -46:14.636 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a7246d02-17f4-fa02-5a26-9bba10ecc916.dc1 area=wan -46:14.636 [INFO] TestAgent.server: Adding LAN server: server="Node-a7246d02-17f4-fa02-5a26-9bba10ecc916 (Addr: tcp/127.0.0.1:11081) (DC: dc1)" -46:14.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11139 network=tcp -46:14.637 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11140 network=tcp -46:14.637 [INFO] TestAgent: started state syncer -46:14.637 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11082 network=tcp -46:14.670 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.670 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11081 [Candidate]" term=2 -46:14.687 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.687 [DEBUG] TestAgent.server.raft: vote granted: from=a7246d02-17f4-fa02-5a26-9bba10ecc916 term=2 tally=1 -46:14.687 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.687 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11081 [Leader]" -46:14.687 [INFO] TestAgent.server: cluster leadership acquired -46:14.687 [INFO] TestAgent.server: New leader elected: payload=Node-a7246d02-17f4-fa02-5a26-9bba10ecc916 -46:14.696 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11081 -46:14.717 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.750 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.750 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.750 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.750 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.750 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.750 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a7246d02-17f4-fa02-5a26-9bba10ecc916 -46:14.750 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a7246d02-17f4-fa02-5a26-9bba10ecc916 -46:14.758 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.784 [DEBUG] TestAgent.dns: request served from client: name=db.service.consul. type=A class=IN latency=126.65µs client=127.0.0.1:57665 client_network=udp -46:14.785 [DEBUG] TestAgent.dns: request served from client: name=test.query.consul. type=A class=IN latency=119.89µs client=127.0.0.1:57917 client_network=udp -46:14.785 [DEBUG] TestAgent.dns: request served from client: name=test.query.consul. type=A class=IN latency=104.003µs client=127.0.0.1:35632 client_network=udp -46:14.785 [INFO] TestAgent: Requesting shutdown -46:14.785 [INFO] TestAgent.server: shutting down server -46:14.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.785 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.785 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.785 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.785 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.785 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.785 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.785 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.785 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.785 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.790 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.794Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.794Z [INFO] agent.router.manager: shutting down -46:14.794 [INFO] TestAgent: consul server down -46:14.794 [INFO] TestAgent: shutdown complete -46:14.794 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11139 network=tcp -46:14.794 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11139 network=udp -46:14.794 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11140 network=tcp -46:15.294 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.294 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_EDNS0_ECS (0.71s) - --- PASS: TestDNS_EDNS0_ECS/global (0.00s) - --- PASS: TestDNS_EDNS0_ECS/query (0.00s) - --- PASS: TestDNS_EDNS0_ECS/query-subnet (0.00s) -=== CONT TestDNS_NodeLookup_AAAA -2022-11-07T22:46:13.826Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:13.826Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.836Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:13.836Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 Address:127.0.0.1:12031}]" -46:13.859 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12031 [Follower]" leader= -46:13.862 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 127.0.0.1 -46:13.863 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 127.0.0.1 -2022-11-07T22:46:13.863Z [INFO] agent.router: Initializing LAN area manager -46:13.864 [INFO] TestAgent.server: Adding LAN server: server="Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 (Addr: tcp/127.0.0.1:12031) (DC: dc2)" -46:13.864 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 area=wan -46:13.864 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12019 network=udp -46:13.864 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12019 network=tcp -46:13.865 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12020 network=tcp -46:13.865 [INFO] TestAgent: started state syncer -46:13.865 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12032 network=tcp -46:13.903 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.903 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12031 [Candidate]" term=2 -46:13.914 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:13.915 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.915 [DEBUG] TestAgent.server.raft: vote granted: from=bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 term=2 tally=1 -46:13.915 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.915 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12031 [Leader]" -46:13.915 [INFO] TestAgent.server: cluster leadership acquired -46:13.915 [INFO] TestAgent.server: New leader elected: payload=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 -46:13.922 [INFO] TestAgent.server: initializing acls -46:13.925 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:13.932 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:13.932 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:13.932 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:13.932 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 -46:13.932 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 -46:13.932 [INFO] TestAgent.server: Updating LAN server: server="Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 (Addr: tcp/127.0.0.1:12031) (DC: dc2)" -46:13.932 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 area=wan -46:13.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.991 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.991 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.991 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.991 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.991 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.991 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 -46:13.991 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 -46:14.009 [INFO] TestAgent: Synced node info -46:14.009 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.009 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0 -46:14.262 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11981] -46:14.262 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11981 -46:14.263 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 127.0.0.1 -46:14.263 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:14.263 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 area=wan -46:14.270 [INFO] TestAgent: Requesting shutdown -46:14.271 [INFO] TestAgent.server: shutting down server -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:14.271 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.271 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.271 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.271 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:14.285 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.295Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.295Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.295Z [INFO] agent.router.manager: shutting down -46:14.295 [INFO] TestAgent: consul server down -46:14.295 [INFO] TestAgent: shutdown complete -46:14.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12019 network=tcp -46:14.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12019 network=udp -46:14.295 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12020 network=tcp -46:14.795 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.795 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:13.608Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:13.608Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.620Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:13.620Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.655 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c1a49dcc-c655-4362-e93a-b4684a56064b Address:127.0.0.1:11982}]" -46:13.655 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11982 [Follower]" leader= -46:13.656 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 127.0.0.1 -46:13.656 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c1a49dcc-c655-4362-e93a-b4684a56064b 127.0.0.1 -2022-11-07T22:46:13.656Z [INFO] agent.router: Initializing LAN area manager -46:13.656 [INFO] TestAgent.server: Adding LAN server: server="Node-c1a49dcc-c655-4362-e93a-b4684a56064b (Addr: tcp/127.0.0.1:11982) (DC: dc1)" -46:13.656 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 area=wan -46:13.656 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11998 network=udp -46:13.656 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11998 network=tcp -46:13.657 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11999 network=tcp -46:13.657 [INFO] TestAgent: started state syncer -46:13.657 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11983 network=tcp -46:13.707 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:13.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11982 [Candidate]" term=2 -46:13.729 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.729 [DEBUG] TestAgent.server.raft: vote granted: from=c1a49dcc-c655-4362-e93a-b4684a56064b term=2 tally=1 -46:13.729 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.729 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11982 [Leader]" -46:13.730 [INFO] TestAgent.server: cluster leadership acquired -46:13.730 [INFO] TestAgent.server: New leader elected: payload=Node-c1a49dcc-c655-4362-e93a-b4684a56064b -46:13.737 [INFO] TestAgent.server: initializing acls -46:13.743 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:13.748 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:13.748 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:13.748 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:13.748 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c1a49dcc-c655-4362-e93a-b4684a56064b -46:13.748 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 -46:13.748 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c1a49dcc-c655-4362-e93a-b4684a56064b.dc1 area=wan -46:13.748 [INFO] TestAgent.server: Updating LAN server: server="Node-c1a49dcc-c655-4362-e93a-b4684a56064b (Addr: tcp/127.0.0.1:11982) (DC: dc1)" -46:13.770 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.783 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.783 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.783 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.783 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.783 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.783 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c1a49dcc-c655-4362-e93a-b4684a56064b -46:13.783 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c1a49dcc-c655-4362-e93a-b4684a56064b -46:13.789 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c1a49dcc-c655-4362-e93a-b4684a56064b -46:13.794 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.008 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.017 [INFO] TestAgent: Synced node info -46:14.262 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:37778 -46:14.263 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 127.0.0.1 -46:14.263 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc6fcb56-889d-de2f-7608-a3f6b0cfc7f0.dc2 area=wan -46:14.795 [INFO] TestAgent: Requesting shutdown -46:14.795 [INFO] TestAgent.server: shutting down server -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.795 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.795 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.795 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:14.795 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.800 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.806Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.806Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.806Z [INFO] agent.router.manager: shutting down -46:14.806 [INFO] TestAgent: consul server down -46:14.806 [INFO] TestAgent: shutdown complete -46:14.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11998 network=tcp -46:14.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11998 network=udp -46:14.806 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11999 network=tcp -46:15.307 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.307 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes_WanTranslation (1.71s) -=== CONT TestConfig_Apply_CAS -2022-11-07T22:46:14.574Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.584Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.619 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:89570799-e3c1-9a64-9b50-244f10e9ee2e Address:127.0.0.1:11137}]" -46:14.619 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11137 [Follower]" leader= -46:14.620 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-89570799-e3c1-9a64-9b50-244f10e9ee2e.dc1 127.0.0.1 -46:14.620 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-89570799-e3c1-9a64-9b50-244f10e9ee2e 127.0.0.1 -2022-11-07T22:46:14.620Z [INFO] agent.router: Initializing LAN area manager -46:14.620 [INFO] TestAgent.server: Adding LAN server: server="Node-89570799-e3c1-9a64-9b50-244f10e9ee2e (Addr: tcp/127.0.0.1:11137) (DC: dc1)" -46:14.620 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-89570799-e3c1-9a64-9b50-244f10e9ee2e.dc1 area=wan -46:14.620 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11160 network=udp -46:14.620 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11160 network=tcp -46:14.621 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11161 network=tcp -46:14.621 [INFO] TestAgent: started state syncer -46:14.621 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11138 network=tcp -46:14.666 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.666 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11137 [Candidate]" term=2 -46:14.679 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.679 [DEBUG] TestAgent.server.raft: vote granted: from=89570799-e3c1-9a64-9b50-244f10e9ee2e term=2 tally=1 -46:14.679 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.679 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11137 [Leader]" -46:14.680 [INFO] TestAgent.server: cluster leadership acquired -46:14.680 [INFO] TestAgent.server: New leader elected: payload=Node-89570799-e3c1-9a64-9b50-244f10e9ee2e -46:14.692 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11137 -46:14.698 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.698 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.698 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-89570799-e3c1-9a64-9b50-244f10e9ee2e -46:14.698 [INFO] TestAgent.server: member joined, marking health alive: member=Node-89570799-e3c1-9a64-9b50-244f10e9ee2e -46:14.715 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.802 [INFO] TestAgent: Requesting shutdown -46:14.802 [INFO] TestAgent.server: shutting down server -46:14.802 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.802 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.802 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.802 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.802 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.802 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.802 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.806 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.815Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.815Z [INFO] agent.router.manager: shutting down -46:14.815 [INFO] TestAgent: consul server down -46:14.816 [INFO] TestAgent: shutdown complete -46:14.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11160 network=tcp -46:14.816 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11160 network=udp -46:14.816 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11161 network=tcp -46:15.318 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.318 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCARoots_empty (0.76s) -=== CONT TestAgent_AddCheck_Alias -=== RUN TestConfig_Apply_Decoding/No_Kind -=== RUN TestConfig_Apply_Decoding/Kind_Not_String -=== RUN TestConfig_Apply_Decoding/Lowercase_kind -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:14.556Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.569Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.598 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3bf76f2-03d7-e0dd-c019-0a55a8b3209e Address:127.0.0.1:11144}]" -46:14.598 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11144 [Follower]" leader= -46:14.598 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e.dc1 127.0.0.1 -46:14.599 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e 127.0.0.1 -2022-11-07T22:46:14.599Z [INFO] agent.router: Initializing LAN area manager -46:14.599 [INFO] TestAgent.server: Adding LAN server: server="Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e (Addr: tcp/127.0.0.1:11144) (DC: dc1)" -46:14.599 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12005 network=udp -46:14.599 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e.dc1 area=wan -46:14.599 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12005 network=tcp -46:14.599 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12006 network=tcp -46:14.599 [INFO] TestAgent: started state syncer -46:14.599 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11145 network=tcp -46:14.641 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.641 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11144 [Candidate]" term=2 -46:14.652 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.652 [DEBUG] TestAgent.server.raft: vote granted: from=f3bf76f2-03d7-e0dd-c019-0a55a8b3209e term=2 tally=1 -46:14.652 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.652 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11144 [Leader]" -46:14.652 [INFO] TestAgent.server: cluster leadership acquired -46:14.652 [INFO] TestAgent.server: New leader elected: payload=Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e -46:14.663 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11144 -46:14.684 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.719 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.719 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.719 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.719 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.719 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.719 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e -46:14.719 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f3bf76f2-03d7-e0dd-c019-0a55a8b3209e -46:14.734 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.853 [INFO] TestAgent: Requesting shutdown -46:14.853 [INFO] TestAgent.server: shutting down server -46:14.853 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.853 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.853 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.853 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.853 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.853 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.853 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.853 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.853 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.853 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.853 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.858 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.868Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.868Z [INFO] agent.router.manager: shutting down -46:14.868 [INFO] TestAgent: consul server down -46:14.868 [INFO] TestAgent: shutdown complete -46:14.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12005 network=tcp -46:14.868 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12005 network=udp -46:14.868 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12006 network=tcp -46:15.369 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.369 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogDeregister (0.82s) -=== CONT TestAgent_RerouteNewHTTPChecks -2022-11-07T22:46:14.676Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.685Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.724 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:513c6aa2-13a4-bded-598e-2da69d620b25 Address:127.0.0.1:11151}]" -46:14.724 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11151 [Follower]" leader= -46:14.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-513c6aa2-13a4-bded-598e-2da69d620b25.dc1 127.0.0.1 -46:14.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-513c6aa2-13a4-bded-598e-2da69d620b25 127.0.0.1 -2022-11-07T22:46:14.725Z [INFO] agent.router: Initializing LAN area manager -46:14.725 [INFO] TestAgent.server: Adding LAN server: server="Node-513c6aa2-13a4-bded-598e-2da69d620b25 (Addr: tcp/127.0.0.1:11151) (DC: dc1)" -46:14.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11083 network=udp -46:14.725 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-513c6aa2-13a4-bded-598e-2da69d620b25.dc1 area=wan -46:14.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11083 network=tcp -46:14.726 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11084 network=tcp -46:14.726 [INFO] TestAgent: started state syncer -46:14.726 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11152 network=tcp -46:14.785 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.785 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11151 [Candidate]" term=2 -46:14.796 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.796 [DEBUG] TestAgent.server.raft: vote granted: from=513c6aa2-13a4-bded-598e-2da69d620b25 term=2 tally=1 -46:14.796 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.796 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11151 [Leader]" -46:14.796 [INFO] TestAgent.server: cluster leadership acquired -46:14.796 [INFO] TestAgent.server: New leader elected: payload=Node-513c6aa2-13a4-bded-598e-2da69d620b25 -46:14.803 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11151 -46:14.822 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-513c6aa2-13a4-bded-598e-2da69d620b25 -46:14.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-513c6aa2-13a4-bded-598e-2da69d620b25 -46:14.856 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.883 [INFO] TestAgent: Requesting shutdown -46:14.883 [INFO] TestAgent.server: shutting down server -46:14.883 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.883 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.883 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.883 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.883 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.883 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.883 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.883 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.883 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.883 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.883 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.888 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.893Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.893Z [INFO] agent.router.manager: shutting down -46:14.893 [INFO] TestAgent: consul server down -46:14.893 [INFO] TestAgent: shutdown complete -46:14.893 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11083 network=tcp -46:14.893 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11083 network=udp -46:14.893 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11084 network=tcp -46:15.393 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.393 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck_BadStatus (0.73s) -=== CONT TestDNS_NodeLookup_PeriodName -2022-11-07T22:46:14.702Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.711Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.751 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a784d388-a971-7663-5df7-54930a7771d2 Address:127.0.0.1:11067}]" -46:14.751 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11067 [Follower]" leader= -46:14.752 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a784d388-a971-7663-5df7-54930a7771d2.dc1 127.0.0.1 -46:14.752 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a784d388-a971-7663-5df7-54930a7771d2 127.0.0.1 -2022-11-07T22:46:14.752Z [INFO] agent.router: Initializing LAN area manager -46:14.752 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a784d388-a971-7663-5df7-54930a7771d2.dc1 area=wan -46:14.752 [INFO] TestAgent.server: Adding LAN server: server="Node-a784d388-a971-7663-5df7-54930a7771d2 (Addr: tcp/127.0.0.1:11067) (DC: dc1)" -46:14.752 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11153 network=udp -46:14.752 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11153 network=tcp -46:14.753 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11154 network=tcp -46:14.753 [INFO] TestAgent: started state syncer -46:14.753 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11068 network=tcp -46:14.808 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.808 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11067 [Candidate]" term=2 -46:14.821 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.821 [DEBUG] TestAgent.server.raft: vote granted: from=a784d388-a971-7663-5df7-54930a7771d2 term=2 tally=1 -46:14.821 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.821 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11067 [Leader]" -46:14.821 [INFO] TestAgent.server: cluster leadership acquired -46:14.821 [INFO] TestAgent.server: New leader elected: payload=Node-a784d388-a971-7663-5df7-54930a7771d2 -46:14.829 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11067 -46:14.835 [INFO] TestAgent: Synced node info -46:14.867 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.888 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.888 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.888 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.888 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.888 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.888 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a784d388-a971-7663-5df7-54930a7771d2 -46:14.888 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a784d388-a971-7663-5df7-54930a7771d2 -46:14.897 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.917 [INFO] TestAgent: Requesting shutdown -46:14.917 [INFO] TestAgent.server: shutting down server -46:14.917 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.917 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.917 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.917 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.917 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.917 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.917 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.917 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.917 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.917 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.924 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.929Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.929Z [INFO] agent.router.manager: shutting down -46:14.929 [INFO] TestAgent: consul server down -46:14.929 [INFO] TestAgent: shutdown complete -46:14.929 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11153 network=tcp -46:14.929 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11153 network=udp -46:14.929 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11154 network=tcp -46:15.429 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.429 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_ProxyDefaultsExpose (0.74s) -=== CONT TestConfig_Apply_ProxyDefaultsMeshGateway -2022-11-07T22:46:14.735Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.744Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.776 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ccd7c943-781c-c302-4634-b699e40f850c Address:127.0.0.1:11207}]" -46:14.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11207 [Follower]" leader= -46:14.777 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ccd7c943-781c-c302-4634-b699e40f850c.dc1 127.0.0.1 -46:14.778 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ccd7c943-781c-c302-4634-b699e40f850c 127.0.0.1 -2022-11-07T22:46:14.778Z [INFO] agent.router: Initializing LAN area manager -46:14.778 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ccd7c943-781c-c302-4634-b699e40f850c.dc1 area=wan -46:14.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11069 network=udp -46:14.778 [INFO] TestAgent.server: Adding LAN server: server="Node-ccd7c943-781c-c302-4634-b699e40f850c (Addr: tcp/127.0.0.1:11207) (DC: dc1)" -46:14.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11069 network=tcp -46:14.779 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11070 network=tcp -46:14.779 [INFO] TestAgent: started state syncer -46:14.779 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11208 network=tcp -46:14.838 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.838 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11207 [Candidate]" term=2 -46:14.855 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.855 [DEBUG] TestAgent.server.raft: vote granted: from=ccd7c943-781c-c302-4634-b699e40f850c term=2 tally=1 -46:14.855 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.855 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11207 [Leader]" -46:14.855 [INFO] TestAgent.server: cluster leadership acquired -46:14.855 [INFO] TestAgent.server: New leader elected: payload=Node-ccd7c943-781c-c302-4634-b699e40f850c -46:14.863 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11207 -46:14.883 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.909 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.909 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.909 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.909 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.909 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.909 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ccd7c943-781c-c302-4634-b699e40f850c -46:14.909 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ccd7c943-781c-c302-4634-b699e40f850c -46:14.921 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.945 [INFO] TestAgent.server.connect: CA provider config updated -46:14.945 [INFO] TestAgent: Requesting shutdown -46:14.945 [INFO] TestAgent.server: shutting down server -46:14.945 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.945 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.945 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.945 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.945 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.945 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.945 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.945 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.945 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.945 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.945 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.951 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.961Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.961Z [INFO] agent.router.manager: shutting down -46:14.961 [INFO] TestAgent: consul server down -46:14.961 [INFO] TestAgent: shutdown complete -46:14.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11069 network=tcp -46:14.961 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11069 network=udp -46:14.961 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11070 network=tcp -46:15.461 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.461 [INFO] TestAgent: Endpoints down -=== RUN TestConnectCAConfig/force_without_cross_sign_CamelCase -2022-11-07T22:46:14.781Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.790Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.823 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:adfaea05-e0d2-e6e8-cc09-6ca1d2000608 Address:127.0.0.1:11060}]" -46:14.823 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11060 [Follower]" leader= -46:14.824 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608.dc1 127.0.0.1 -46:14.825 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608 127.0.0.1 -2022-11-07T22:46:14.825Z [INFO] agent.router: Initializing LAN area manager -46:14.825 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608.dc1 area=wan -46:14.825 [INFO] TestAgent.server: Adding LAN server: server="Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608 (Addr: tcp/127.0.0.1:11060) (DC: dc1)" -46:14.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11209 network=udp -46:14.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11209 network=tcp -46:14.826 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11210 network=tcp -46:14.826 [INFO] TestAgent: started state syncer -46:14.826 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11061 network=tcp -46:14.874 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.874 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11060 [Candidate]" term=2 -46:14.888 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.888 [DEBUG] TestAgent.server.raft: vote granted: from=adfaea05-e0d2-e6e8-cc09-6ca1d2000608 term=2 tally=1 -46:14.888 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.888 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11060 [Leader]" -46:14.888 [INFO] TestAgent.server: cluster leadership acquired -46:14.888 [INFO] TestAgent.server: New leader elected: payload=Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608 -46:14.895 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11060 -46:14.911 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.938 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.938 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.938 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.938 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.938 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.938 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608 -46:14.938 [INFO] TestAgent.server: member joined, marking health alive: member=Node-adfaea05-e0d2-e6e8-cc09-6ca1d2000608 -46:14.957 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.962 [DEBUG] TestAgent.dns: request served from client: name=foo.node.dc1.consul. type=ANY class=IN latency=83.951µs client=127.0.0.1:34880 client_network=udp -46:14.962 [INFO] TestAgent: Requesting shutdown -46:14.962 [INFO] TestAgent.server: shutting down server -46:14.962 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.962 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.962 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.962 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.962 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.962 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.962 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.962 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.962 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.962 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.962 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.967 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.974Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.974Z [INFO] agent.router.manager: shutting down -46:14.974 [INFO] TestAgent: consul server down -46:14.974 [INFO] TestAgent: shutdown complete -46:14.974 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11209 network=tcp -46:14.974 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11209 network=udp -46:14.974 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11210 network=tcp -46:15.475 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.475 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_EDNS0 (0.71s) -=== CONT TestAgent_AddCheck_GRPC -2022-11-07T22:46:14.820Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.830Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.869 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fda3fd64-5eee-0581-f207-9f3d4412a1d9 Address:127.0.0.1:11074}]" -46:14.869 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11074 [Follower]" leader= -46:14.869 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9.dc1 127.0.0.1 -46:14.870 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9 127.0.0.1 -2022-11-07T22:46:14.870Z [INFO] agent.router: Initializing LAN area manager -46:14.870 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9.dc1 area=wan -46:14.870 [INFO] TestAgent.server: Adding LAN server: server="Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9 (Addr: tcp/127.0.0.1:11074) (DC: dc1)" -46:14.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11062 network=tcp -46:14.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11062 network=udp -46:14.871 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11063 network=tcp -46:14.871 [INFO] TestAgent: started state syncer -46:14.871 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11075 network=tcp -46:14.930 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.930 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11074 [Candidate]" term=2 -46:14.943 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.943 [DEBUG] TestAgent.server.raft: vote granted: from=fda3fd64-5eee-0581-f207-9f3d4412a1d9 term=2 tally=1 -46:14.943 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.943 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11074 [Leader]" -46:14.943 [INFO] TestAgent.server: cluster leadership acquired -46:14.943 [INFO] TestAgent.server: New leader elected: payload=Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9 -46:14.949 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11074 -46:14.965 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.987 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.987 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.987 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.987 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.987 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.987 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9 -46:14.987 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fda3fd64-5eee-0581-f207-9f3d4412a1d9 -46:14.997 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.032 [INFO] TestAgent: Requesting shutdown -46:15.032 [INFO] TestAgent.server: shutting down server -46:15.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.032 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.032 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.032 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.032 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.032 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.032 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.032 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.038 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.045Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.045Z [INFO] agent.router.manager: shutting down -46:15.045 [INFO] TestAgent: consul server down -46:15.045 [INFO] TestAgent: shutdown complete -46:15.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11062 network=tcp -46:15.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11062 network=udp -46:15.045 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11063 network=tcp -46:15.546 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.546 [INFO] TestAgent: Endpoints down + agent_test.go:4395: routing {bob.,}server.dc1.consul => 127.0.0.1:30142 + agent_test.go:4396: routing {betty.,}server.dc2.consul => 127.0.0.1:30128 + agent_test.go:4397: routing {bonnie.,}server.dc3.consul => 127.0.0.1:30129 +2021-10-05T16:31:37.052Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.067Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.124 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a552e86-6965-f97e-e77e-3e1eccadf9ee Address:127.0.0.1:30125}]" +31:37.124 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30125 [Follower]" leader= +31:37.124 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a552e86-6965-f97e-e77e-3e1eccadf9ee.dc1 127.0.0.1 +31:37.125 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a552e86-6965-f97e-e77e-3e1eccadf9ee 127.0.0.1 +2021-10-05T16:31:37.125Z [INFO] agent.router: Initializing LAN area manager +31:37.125 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30099 network=udp +31:37.126 [INFO] TestAgent.server: Adding LAN server: server="Node-1a552e86-6965-f97e-e77e-3e1eccadf9ee (Addr: tcp/127.0.0.1:30125) (DC: dc1)" +31:37.126 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a552e86-6965-f97e-e77e-3e1eccadf9ee.dc1 area=wan +31:37.126 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30099 network=tcp +31:37.126 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30107 network=tcp +31:37.126 [INFO] TestAgent: started state syncer +31:37.126 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30126 network=tcp +31:37.183 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.183 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30125 [Candidate]" term=2 +31:37.208 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.208 [DEBUG] TestAgent.server.raft: vote granted: from=1a552e86-6965-f97e-e77e-3e1eccadf9ee term=2 tally=1 +31:37.208 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.208 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30125 [Leader]" +31:37.212 [INFO] TestAgent.server: cluster leadership acquired +31:37.212 [INFO] TestAgent.server: New leader elected: payload=Node-1a552e86-6965-f97e-e77e-3e1eccadf9ee +31:37.225 [INFO] TestAgent: Requesting shutdown +31:37.225 [INFO] TestAgent.server: shutting down server +31:37.225 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.225 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:37.232 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:37.232 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30125 +2021-10-05T16:31:37.240Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.240Z [INFO] agent.router.manager: shutting down +31:37.249 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:37.249 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:37.249 [INFO] TestAgent: consul server down +31:37.249 [INFO] TestAgent: shutdown complete +31:37.249 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30099 network=tcp +31:37.249 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30099 network=udp +31:37.249 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30107 network=tcp +31:37.752 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.752 [INFO] TestAgent: Endpoints down --- PASS: TestCatalogRegister_Service_InvalidAddress (0.74s) --- PASS: TestCatalogRegister_Service_InvalidAddress/addr_0.0.0.0 (0.00s) --- PASS: TestCatalogRegister_Service_InvalidAddress/addr_:: (0.00s) --- PASS: TestCatalogRegister_Service_InvalidAddress/addr_[::] (0.00s) -=== CONT TestConfig_Apply_IngressGateway +=== CONT TestAgent_ReloadConfigIncomingRPCConfig +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:36.486Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:36.486Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.521Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:36.521Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.610 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b86e4c3e-b84d-aeab-9487-522c74062f4d Address:127.0.0.1:30013}]" +31:36.610 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30013 [Follower]" leader= +31:36.611 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 127.0.0.1 +31:36.612 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b86e4c3e-b84d-aeab-9487-522c74062f4d 127.0.0.1 +2021-10-05T16:31:36.612Z [INFO] agent.router: Initializing LAN area manager +31:36.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30071 network=udp +31:36.612 [INFO] TestAgent.server: Adding LAN server: server="Node-b86e4c3e-b84d-aeab-9487-522c74062f4d (Addr: tcp/127.0.0.1:30013) (DC: dc2)" +31:36.612 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 area=wan +31:36.613 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30071 network=tcp +31:36.613 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30072 network=tcp +31:36.613 [INFO] TestAgent: started state syncer +31:36.613 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30014 network=tcp +31:36.664 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:36.669 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.669 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30013 [Candidate]" term=2 +31:36.691 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.691 [DEBUG] TestAgent.server.raft: vote granted: from=b86e4c3e-b84d-aeab-9487-522c74062f4d term=2 tally=1 +31:36.691 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.691 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30013 [Leader]" +31:36.691 [INFO] TestAgent.server: cluster leadership acquired +31:36.691 [INFO] TestAgent.server: New leader elected: payload=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d +31:36.701 [INFO] TestAgent.server: initializing acls +31:36.709 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:36.721 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:36.721 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:36.721 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:36.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b86e4c3e-b84d-aeab-9487-522c74062f4d +31:36.721 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 +31:36.721 [INFO] TestAgent.server: Updating LAN server: server="Node-b86e4c3e-b84d-aeab-9487-522c74062f4d (Addr: tcp/127.0.0.1:30013) (DC: dc2)" +31:36.721 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 area=wan +31:36.760 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.790 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.790 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.790 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.790 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.790 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.790 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d +31:36.790 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d +31:36.800 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d +31:36.817 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.908 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:30033] +31:36.908 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:30033 +31:36.911 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 127.0.0.1 +31:36.911 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:36.912 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 area=wan +31:36.926 [INFO] TestAgent: Requesting shutdown +31:36.926 [INFO] TestAgent.server: shutting down server +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:36.926 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:36.926 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:36.926 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:36.926 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:36.928 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:36.942 [INFO] TestAgent: Synced node info +2021-10-05T16:31:36.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.948Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:36.948Z [INFO] agent.router.manager: shutting down +31:36.948 [INFO] TestAgent: consul server down +31:36.948 [INFO] TestAgent: shutdown complete +31:36.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30071 network=tcp +31:36.948 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30071 network=udp +31:36.948 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30072 network=tcp +31:37.448 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.448 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:36.189Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:36.189Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.207Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:36.207Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.248 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f4719165-1ede-f617-6840-8c1098c9744f Address:127.0.0.1:30034}]" +31:36.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30034 [Follower]" leader= +31:36.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 127.0.0.1 +31:36.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4719165-1ede-f617-6840-8c1098c9744f 127.0.0.1 +2021-10-05T16:31:36.249Z [INFO] agent.router: Initializing LAN area manager +31:36.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30043 network=udp +31:36.249 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 area=wan +31:36.249 [INFO] TestAgent.server: Adding LAN server: server="Node-f4719165-1ede-f617-6840-8c1098c9744f (Addr: tcp/127.0.0.1:30034) (DC: dc1)" +31:36.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30043 network=tcp +31:36.250 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30058 network=tcp +31:36.250 [INFO] TestAgent: started state syncer +31:36.250 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30035 network=tcp +31:36.300 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:36.314 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.314 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30034 [Candidate]" term=2 +31:36.333 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.333 [DEBUG] TestAgent.server.raft: vote granted: from=f4719165-1ede-f617-6840-8c1098c9744f term=2 tally=1 +31:36.333 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.333 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30034 [Leader]" +31:36.334 [INFO] TestAgent.server: cluster leadership acquired +31:36.334 [INFO] TestAgent.server: New leader elected: payload=Node-f4719165-1ede-f617-6840-8c1098c9744f +31:36.339 [INFO] TestAgent.server: initializing acls +31:36.343 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:36.356 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:36.356 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:36.356 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:36.356 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:36.356 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f4719165-1ede-f617-6840-8c1098c9744f +31:36.356 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 +31:36.357 [INFO] TestAgent.server: Updating LAN server: server="Node-f4719165-1ede-f617-6840-8c1098c9744f (Addr: tcp/127.0.0.1:30034) (DC: dc1)" +31:36.357 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f4719165-1ede-f617-6840-8c1098c9744f.dc1 area=wan +31:36.400 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.451 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.451 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.451 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.451 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.451 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.451 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4719165-1ede-f617-6840-8c1098c9744f +31:36.451 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4719165-1ede-f617-6840-8c1098c9744f +31:36.463 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4719165-1ede-f617-6840-8c1098c9744f +31:36.470 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.909 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:40736 +31:36.910 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 127.0.0.1 +31:36.910 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b86e4c3e-b84d-aeab-9487-522c74062f4d.dc2 area=wan +31:37.031 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:37.041 [INFO] TestAgent: Synced node info +31:37.041 [DEBUG] TestAgent: Node info in sync +31:37.448 [INFO] TestAgent: Requesting shutdown +31:37.448 [INFO] TestAgent.server: shutting down server +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.448 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.448 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.449 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.449 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.465 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.472Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.472Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.472Z [INFO] agent.router.manager: shutting down +31:37.472 [INFO] TestAgent: consul server down +31:37.472 [INFO] TestAgent: shutdown complete +31:37.472 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30043 network=tcp +31:37.472 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30043 network=udp +31:37.472 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30058 network=tcp +31:37.972 [INFO] TestAgent: Waiting for endpoints to shut down +31:37.972 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogNodes_WanTranslation (1.81s) +=== CONT TestAgent_ReloadConfigOutgoingRPCConfig +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:37.375Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.411Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.500 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d99d5d14-12b7-5654-5c72-b3aa6b0be399 Address:127.0.0.1:30149}]" +31:37.500 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30149 [Follower]" leader= +31:37.504 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399.dc1 127.0.0.1 +31:37.505 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399 127.0.0.1 +2021-10-05T16:31:37.505Z [INFO] agent.router: Initializing LAN area manager +31:37.505 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30137 network=udp +31:37.505 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399.dc1 area=wan +31:37.505 [INFO] TestAgent.server: Adding LAN server: server="Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399 (Addr: tcp/127.0.0.1:30149) (DC: dc1)" +31:37.505 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30137 network=tcp +31:37.506 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30138 network=tcp +31:37.506 [INFO] TestAgent: started state syncer +31:37.506 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30143 network=tcp +31:37.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30149 [Candidate]" term=2 +31:37.570 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.570 [DEBUG] TestAgent.server.raft: vote granted: from=d99d5d14-12b7-5654-5c72-b3aa6b0be399 term=2 tally=1 +31:37.570 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.570 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30149 [Leader]" +31:37.571 [INFO] TestAgent.server: cluster leadership acquired +31:37.571 [INFO] TestAgent.server: New leader elected: payload=Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399 +31:37.579 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30149 +31:37.599 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.644 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.644 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:37.644 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:37.644 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:37.644 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:37.644 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399 +31:37.644 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d99d5d14-12b7-5654-5c72-b3aa6b0be399 +31:37.669 [INFO] TestAgent.server: federation state anti-entropy synced +31:37.689 [WARN] TestAgent: check has interval below minimum: check=http minimum_interval=1s +31:37.689 [WARN] TestAgent: check has interval below minimum: check=grpc minimum_interval=1s +31:37.689 [INFO] TestAgent: Requesting shutdown +31:37.689 [INFO] TestAgent.server: shutting down server +31:37.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.689 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.689 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.689 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.690 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.690 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:37.690 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.690 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.690 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.712 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.719Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.719Z [INFO] agent.router.manager: shutting down +31:37.719 [INFO] TestAgent: consul server down +31:37.719 [INFO] TestAgent: shutdown complete +31:37.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30137 network=tcp +31:37.719 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30137 network=udp +31:37.719 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30138 network=tcp +31:38.220 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.220 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RerouteExistingHTTPChecks (0.89s) +=== CONT TestAgent_SecurityChecks +2021-10-05T16:31:37.294Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.368Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.436 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:13bbfd6b-cb40-0c48-768f-ee256a0c174c Address:127.0.0.1:30086}]" +31:37.436 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30086 [Follower]" leader= +31:37.437 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c.dc1 127.0.0.1 +31:37.437 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c 127.0.0.1 +2021-10-05T16:31:37.437Z [INFO] agent.router: Initializing LAN area manager +31:37.437 [INFO] TestAgent.server: Adding LAN server: server="Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c (Addr: tcp/127.0.0.1:30086) (DC: dc1)" +31:37.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30074 network=udp +31:37.437 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c.dc1 area=wan +31:37.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30074 network=tcp +31:37.438 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30075 network=tcp +31:37.438 [INFO] TestAgent: started state syncer +31:37.438 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30136 network=tcp +31:37.498 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.498 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30086 [Candidate]" term=2 +31:37.510 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.510 [DEBUG] TestAgent.server.raft: vote granted: from=13bbfd6b-cb40-0c48-768f-ee256a0c174c term=2 tally=1 +31:37.510 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.510 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30086 [Leader]" +31:37.511 [INFO] TestAgent.server: cluster leadership acquired +31:37.511 [INFO] TestAgent.server: New leader elected: payload=Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c +31:37.520 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30086 +31:37.536 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.563 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.563 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:37.563 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:37.563 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:37.563 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:37.563 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c +31:37.563 [INFO] TestAgent.server: member joined, marking health alive: member=Node-13bbfd6b-cb40-0c48-768f-ee256a0c174c +31:37.579 [INFO] TestAgent.server: federation state anti-entropy synced +31:37.785 [WARN] TestAgent: check has interval below minimum: check=http minimum_interval=1s +31:37.785 [WARN] TestAgent: check has interval below minimum: check=grpc minimum_interval=1s +31:37.785 [INFO] TestAgent: Requesting shutdown +31:37.785 [INFO] TestAgent.server: shutting down server +31:37.785 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.785 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.785 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.785 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.786 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:37.786 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.786 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.786 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.787 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.787 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.792 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.799Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.799Z [INFO] agent.router.manager: shutting down +31:37.799 [INFO] TestAgent: consul server down +31:37.799 [INFO] TestAgent: shutdown complete +31:37.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30074 network=tcp +31:37.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30074 network=udp +31:37.799 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30075 network=tcp +31:38.299 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.299 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RerouteNewHTTPChecks (1.05s) +=== CONT TestAgent_loadTokens +2021-10-05T16:31:37.434Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.472Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.549 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 Address:127.0.0.1:30135}]" +31:37.549 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30135 [Follower]" leader= +31:37.550 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10.dc1 127.0.0.1 +31:37.550 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 127.0.0.1 +2021-10-05T16:31:37.550Z [INFO] agent.router: Initializing LAN area manager +31:37.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30144 network=udp +31:37.550 [INFO] TestAgent.server: Adding LAN server: server="Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 (Addr: tcp/127.0.0.1:30135) (DC: dc1)" +31:37.551 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10.dc1 area=wan +31:37.551 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30144 network=tcp +31:37.551 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30145 network=tcp +31:37.551 [INFO] TestAgent: started state syncer +31:37.551 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30150 network=tcp +31:37.586 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.586 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30135 [Candidate]" term=2 +31:37.600 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.600 [DEBUG] TestAgent.server.raft: vote granted: from=71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 term=2 tally=1 +31:37.600 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.600 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30135 [Leader]" +31:37.601 [INFO] TestAgent.server: cluster leadership acquired +31:37.601 [INFO] TestAgent.server: New leader elected: payload=Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 +31:37.613 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30135 +31:37.648 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.712 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.712 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:37.712 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:37.712 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:37.712 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:37.712 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 +31:37.712 [INFO] TestAgent.server: member joined, marking health alive: member=Node-71fdfcc4-03fa-8296-d9e7-47b8abbf6a10 +31:37.748 [INFO] TestAgent.server: federation state anti-entropy synced +31:37.784 [INFO] TestAgent: Requesting shutdown +31:37.784 [INFO] TestAgent.server: shutting down server +31:37.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.784 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.784 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.784 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.785 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.785 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.785 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:37.785 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.785 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.785 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.796 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.804Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.804Z [INFO] agent.router.manager: shutting down +31:37.804 [INFO] TestAgent: consul server down +31:37.804 [INFO] TestAgent: shutdown complete +31:37.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30144 network=tcp +31:37.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30144 network=udp +31:37.804 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30145 network=tcp +31:38.305 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.305 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_consulConfig_RaftTrailingLogs (0.92s) +=== CONT TestAgent_reloadWatchesHTTPS +2021-10-05T16:31:37.461Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.492Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.584 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3d24ea61-3542-96e4-fa85-f24f8b9c85f7 Address:127.0.0.1:30156}]" +31:37.584 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30156 [Follower]" leader= +31:37.584 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7.dc1 127.0.0.1 +31:37.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 127.0.0.1 +2021-10-05T16:31:37.585Z [INFO] agent.router: Initializing LAN area manager +31:37.585 [INFO] TestAgent.server: Adding LAN server: server="Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 (Addr: tcp/127.0.0.1:30156) (DC: dc1)" +31:37.585 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30151 network=udp +31:37.585 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7.dc1 area=wan +31:37.585 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30151 network=tcp +31:37.586 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30152 network=tcp +31:37.586 [INFO] TestAgent: started state syncer +31:37.586 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30080 network=tcp +31:37.653 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.653 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30156 [Candidate]" term=2 +31:37.674 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.674 [DEBUG] TestAgent.server.raft: vote granted: from=3d24ea61-3542-96e4-fa85-f24f8b9c85f7 term=2 tally=1 +31:37.674 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.674 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30156 [Leader]" +31:37.674 [INFO] TestAgent.server: cluster leadership acquired +31:37.674 [INFO] TestAgent.server: New leader elected: payload=Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 +31:37.688 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30156 +31:37.748 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.789 [INFO] TestAgent: Requesting shutdown +31:37.789 [INFO] TestAgent.server: shutting down server +31:37.789 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.789 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:37.789 [INFO] TestAgent.server: shutting down Connect CA roots monitor +31:37.796 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:37.803 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.803 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:37.803 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:37.803 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:37.803 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:37.803 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 +31:37.803 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 +2021-10-05T16:31:37.803Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.803Z [INFO] agent.router.manager: shutting down +31:37.803 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:37.803 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:37.803 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:37.803 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:37.803 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:37.813 [INFO] TestAgent: consul server down +31:37.813 [INFO] TestAgent: shutdown complete +31:37.813 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30151 network=tcp +31:37.813 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30151 network=udp +31:37.813 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30152 network=tcp +31:37.813 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-3d24ea61-3542-96e4-fa85-f24f8b9c85f7 127.0.0.1 30154 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:3d24ea61-3542-96e4-fa85-f24f8b9c85f7 port:30156 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30155] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:37.813 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.813 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.813 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.813 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.813 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.813 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.813 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.813 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:38.325 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.325 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_consulConfig_AutoEncryptAllowTLS (0.88s) +=== CONT TestAgent_reloadWatches +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:36.430Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:36.445Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:36.519 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c64765a2-919b-0da1-c797-d0bb5b5991e5 Address:127.0.0.1:30104}]" +31:36.520 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30104 [Follower]" leader= +31:36.520 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c64765a2-919b-0da1-c797-d0bb5b5991e5.dc1 127.0.0.1 +31:36.521 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c64765a2-919b-0da1-c797-d0bb5b5991e5 127.0.0.1 +2021-10-05T16:31:36.522Z [INFO] agent.router: Initializing LAN area manager +31:36.522 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30050 network=udp +31:36.522 [INFO] TestAgent.server: Adding LAN server: server="Node-c64765a2-919b-0da1-c797-d0bb5b5991e5 (Addr: tcp/127.0.0.1:30104) (DC: dc1)" +31:36.522 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c64765a2-919b-0da1-c797-d0bb5b5991e5.dc1 area=wan +31:36.522 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30050 network=tcp +31:36.523 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30065 network=tcp +31:36.523 [INFO] TestAgent: started state syncer +31:36.523 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30105 network=tcp +31:36.568 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:36.568 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30104 [Candidate]" term=2 +31:36.596 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:36.596 [DEBUG] TestAgent.server.raft: vote granted: from=c64765a2-919b-0da1-c797-d0bb5b5991e5 term=2 tally=1 +31:36.596 [INFO] TestAgent.server.raft: election won: tally=1 +31:36.596 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30104 [Leader]" +31:36.596 [INFO] TestAgent.server: cluster leadership acquired +31:36.596 [INFO] TestAgent.server: New leader elected: payload=Node-c64765a2-919b-0da1-c797-d0bb5b5991e5 +31:36.608 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30104 +31:36.640 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:36.677 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:36.677 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:36.677 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:36.677 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:36.677 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:36.677 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c64765a2-919b-0da1-c797-d0bb5b5991e5 +31:36.677 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c64765a2-919b-0da1-c797-d0bb5b5991e5 +31:36.709 [INFO] TestAgent.server: federation state anti-entropy synced +31:36.802 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:36.815 [INFO] TestAgent: Synced node info +31:36.815 [DEBUG] TestAgent: Node info in sync +31:37.713 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=4719032 +31:37.848 [INFO] TestAgent: Requesting shutdown +31:37.848 [INFO] TestAgent.server: shutting down server +31:37.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:37.848 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:37.848 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:37.848 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:37.848 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:37.848 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:37.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:37.848 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:37.848 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:37.848 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:37.860 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:37.865Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:37.865Z [INFO] agent.router.manager: shutting down +31:37.865 [INFO] TestAgent: consul server down +31:37.865 [INFO] TestAgent: shutdown complete +31:37.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30050 network=tcp +31:37.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30050 network=udp +31:37.865 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30065 network=tcp +31:38.365 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.365 [INFO] TestAgent: Endpoints down +=== RUN TestTxnEndpoint_Bad_Size_Net/allowed_kv_max_backward_compatible +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:38.088Z [WARN] TestAgent_ReloadConfigIncomingRPCConfig: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.807Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.824Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.858 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c086b67-e9ec-5197-8aaa-37a94e669776 Address:127.0.0.1:30163}]" +31:37.859 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c086b67-e9ec-5197-8aaa-37a94e669776.dc1 127.0.0.1 +31:37.859 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c086b67-e9ec-5197-8aaa-37a94e669776 127.0.0.1 +2021-10-05T16:31:37.859Z [INFO] agent.router: Initializing LAN area manager +31:37.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30158 network=udp +31:37.860 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30163 [Follower]" leader= +31:37.860 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c086b67-e9ec-5197-8aaa-37a94e669776.dc1 area=wan +31:37.860 [INFO] TestAgent.server: Adding LAN server: server="Node-0c086b67-e9ec-5197-8aaa-37a94e669776 (Addr: tcp/127.0.0.1:30163) (DC: dc1)" +31:37.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30158 network=tcp +31:37.860 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30159 network=tcp +31:37.861 [INFO] TestAgent: started state syncer +31:37.861 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30164 network=tcp +31:37.911 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.911 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30163 [Candidate]" term=2 +31:37.927 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.927 [DEBUG] TestAgent.server.raft: vote granted: from=0c086b67-e9ec-5197-8aaa-37a94e669776 term=2 tally=1 +31:37.927 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.927 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30163 [Leader]" +31:37.928 [INFO] TestAgent.server: cluster leadership acquired +31:37.928 [INFO] TestAgent.server: New leader elected: payload=Node-0c086b67-e9ec-5197-8aaa-37a94e669776 +31:37.938 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30163 +31:37.961 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.998 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.998 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:37.998 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:37.998 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:37.998 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:37.998 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c086b67-e9ec-5197-8aaa-37a94e669776 +31:37.998 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c086b67-e9ec-5197-8aaa-37a94e669776 +31:38.014 [INFO] TestAgent.server: federation state anti-entropy synced +31:38.090 [INFO] TestAgent: Requesting shutdown +31:38.090 [INFO] TestAgent.server: shutting down server +31:38.090 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.090 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:38.090 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:38.094 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:38.098Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.098Z [INFO] agent.router.manager: shutting down +31:38.098 [INFO] TestAgent: consul server down +31:38.098 [INFO] TestAgent: shutdown complete +31:38.098 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30158 network=tcp +31:38.098 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30158 network=udp +31:38.098 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30159 network=tcp +31:38.600 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.600 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReloadConfigIncomingRPCConfig (0.83s) +=== CONT TestAgent_GetCoordinate +2021-10-05T16:31:38.112Z [WARN] TestAgent_ReloadConfigTLSConfigFailure: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.719Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.788Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.865 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:37f5fad2-b1e1-a077-62e2-c29a5be5607c Address:127.0.0.1:30100}]" +31:37.866 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c.dc1 127.0.0.1 +31:37.867 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c 127.0.0.1 +2021-10-05T16:31:37.867Z [INFO] agent.router: Initializing LAN area manager +31:37.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30081 network=udp +31:37.867 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30100 [Follower]" leader= +31:37.867 [INFO] TestAgent.server: Adding LAN server: server="Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c (Addr: tcp/127.0.0.1:30100) (DC: dc1)" +31:37.867 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c.dc1 area=wan +31:37.867 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30081 network=tcp +31:37.868 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30082 network=tcp +31:37.868 [INFO] TestAgent: started state syncer +31:37.868 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30157 network=tcp +31:37.908 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.908 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30100 [Candidate]" term=2 +31:37.928 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:37.928 [DEBUG] TestAgent.server.raft: vote granted: from=37f5fad2-b1e1-a077-62e2-c29a5be5607c term=2 tally=1 +31:37.928 [INFO] TestAgent.server.raft: election won: tally=1 +31:37.928 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30100 [Leader]" +31:37.928 [INFO] TestAgent.server: cluster leadership acquired +31:37.928 [INFO] TestAgent.server: New leader elected: payload=Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c +31:37.948 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30100 +31:37.948 [INFO] TestAgent: Synced node info +31:37.948 [DEBUG] TestAgent: Node info in sync +31:37.980 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.037 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.038 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.038 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.038 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.038 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.038 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c +31:38.038 [INFO] TestAgent.server: member joined, marking health alive: member=Node-37f5fad2-b1e1-a077-62e2-c29a5be5607c +31:38.049 [INFO] TestAgent.server: federation state anti-entropy synced +31:38.112 [INFO] TestAgent: Requesting shutdown +31:38.112 [INFO] TestAgent.server: shutting down server +31:38.112 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.112 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:38.116 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:38.120Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.120Z [INFO] agent.router.manager: shutting down +31:38.120 [INFO] TestAgent: consul server down +31:38.120 [INFO] TestAgent: shutdown complete +31:38.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30081 network=tcp +31:38.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30081 network=udp +31:38.120 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30082 network=tcp +31:38.622 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.622 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReloadConfigTLSConfigFailure (0.94s) +=== CONT TestAgent_purgeCheckState +=== RUN TestAgent_loadTokens/original-configuration +=== RUN TestAgent_loadTokens/updated-configuration +=== RUN TestAgent_loadTokens/persisted-tokens +=== RUN TestAgent_loadTokens/persisted-tokens-override +=== RUN TestAgent_loadTokens/partial-persisted +=== RUN TestAgent_loadTokens/persistence-error-not-json +=== RUN TestAgent_loadTokens/persistence-error-wrong-top-level +2021-10-05T16:31:38.224Z [WARN] TestAgent_ReloadConfigOutgoingRPCConfig: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.007Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.034Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.083 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 Address:127.0.0.1:30170}]" +31:38.084 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30170 [Follower]" leader= +31:38.084 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343.dc1 127.0.0.1 +31:38.085 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 127.0.0.1 +2021-10-05T16:31:38.085Z [INFO] agent.router: Initializing LAN area manager +31:38.085 [INFO] TestAgent.server: Adding LAN server: server="Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 (Addr: tcp/127.0.0.1:30170) (DC: dc1)" +31:38.085 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343.dc1 area=wan +31:38.085 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30165 network=udp +31:38.085 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30165 network=tcp +31:38.086 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30166 network=tcp +31:38.086 [INFO] TestAgent: started state syncer +31:38.086 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30171 network=tcp +31:38.151 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.151 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30170 [Candidate]" term=2 +31:38.164 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.164 [DEBUG] TestAgent.server.raft: vote granted: from=bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 term=2 tally=1 +31:38.164 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.164 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30170 [Leader]" +31:38.164 [INFO] TestAgent.server: cluster leadership acquired +31:38.164 [INFO] TestAgent.server: New leader elected: payload=Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 +31:38.173 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30170 +31:38.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.225 [INFO] TestAgent: Requesting shutdown +31:38.225 [INFO] TestAgent.server: shutting down server +31:38.225 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.225 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:38.226 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.226 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.226 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.226 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.226 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.226 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 +31:38.229 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:38.234Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.234Z [INFO] agent.router.manager: shutting down +31:38.249 [INFO] TestAgent: consul server down +31:38.249 [INFO] TestAgent: shutdown complete +31:38.249 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30165 network=tcp +31:38.249 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30165 network=udp +31:38.249 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30166 network=tcp +31:38.249 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 127.0.0.1 30168 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:bd7782ce-a2ab-5e11-4e52-4d82f8ad6343 port:30170 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30169] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:38.750 [INFO] TestAgent: Waiting for endpoints to shut down +31:38.750 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReloadConfigOutgoingRPCConfig (0.78s) +=== CONT TestAgent_loadCheckState +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +--- PASS: TestAgent_SecurityChecks (0.84s) +=== CONT TestAgent_persistCheckState +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:38.328Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.341Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.385 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:af7f633f-c516-f1ed-214a-2afaa66b93d3 Address:127.0.0.1:30177}]" +31:38.385 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-af7f633f-c516-f1ed-214a-2afaa66b93d3.dc1 127.0.0.1 +31:38.386 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-af7f633f-c516-f1ed-214a-2afaa66b93d3 127.0.0.1 +2021-10-05T16:31:38.386Z [INFO] agent.router: Initializing LAN area manager +31:38.386 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30186 network=udp +31:38.386 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30177 [Follower]" leader= +31:38.387 [INFO] TestAgent.server: Adding LAN server: server="Node-af7f633f-c516-f1ed-214a-2afaa66b93d3 (Addr: tcp/127.0.0.1:30177) (DC: dc1)" +31:38.387 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-af7f633f-c516-f1ed-214a-2afaa66b93d3.dc1 area=wan +31:38.387 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30186 network=tcp +31:38.387 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:30188 network=tcp +31:38.387 [INFO] TestAgent: started state syncer +31:38.387 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30216 network=tcp +31:38.450 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.450 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30177 [Candidate]" term=2 +31:38.463 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.463 [DEBUG] TestAgent.server.raft: vote granted: from=af7f633f-c516-f1ed-214a-2afaa66b93d3 term=2 tally=1 +31:38.463 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.463 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30177 [Leader]" +31:38.463 [INFO] TestAgent.server: cluster leadership acquired +31:38.463 [INFO] TestAgent.server: New leader elected: payload=Node-af7f633f-c516-f1ed-214a-2afaa66b93d3 +31:38.473 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30177 +31:38.491 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.543 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.543 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.543 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.543 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.543 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.543 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-af7f633f-c516-f1ed-214a-2afaa66b93d3 +31:38.543 [INFO] TestAgent.server: member joined, marking health alive: member=Node-af7f633f-c516-f1ed-214a-2afaa66b93d3 +31:38.568 [INFO] TestAgent.server: federation state anti-entropy synced +31:38.643 [INFO] TestAgent: Requesting shutdown +31:38.643 [INFO] TestAgent.server: shutting down server +31:38.643 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:38.643 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:38.643 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:38.643 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:38.643 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.643 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:38.643 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:38.643 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:38.643 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:38.643 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:38.643 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:38.656 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:38.668Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.668Z [INFO] agent.router.manager: shutting down +31:38.668 [INFO] TestAgent: consul server down +31:38.668 [INFO] TestAgent: shutdown complete +31:38.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30186 network=tcp +31:38.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30186 network=udp +31:38.668 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:30188 network=tcp +31:39.176 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.176 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_reloadWatchesHTTPS (0.87s) +=== CONT TestAgent_loadChecks_checkFails +2021-10-05T16:31:38.358Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.374Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.429 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5c2e290b-f5ef-c583-c17c-f72444da5ebb Address:127.0.0.1:30198}]" +31:38.430 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 127.0.0.1 +31:38.431 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb 127.0.0.1 +2021-10-05T16:31:38.431Z [INFO] agent.router: Initializing LAN area manager +31:38.431 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30179 network=udp +31:38.431 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30198 [Follower]" leader= +31:38.431 [INFO] TestAgent.server: Adding LAN server: server="Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb (Addr: tcp/127.0.0.1:30198) (DC: dc1)" +31:38.432 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 area=wan +31:38.432 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30179 network=tcp +31:38.432 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30180 network=tcp +31:38.432 [INFO] TestAgent: started state syncer +31:38.432 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30185 network=tcp +31:38.482 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:38.500 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.500 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30198 [Candidate]" term=2 +31:38.549 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.549 [DEBUG] TestAgent.server.raft: vote granted: from=5c2e290b-f5ef-c583-c17c-f72444da5ebb term=2 tally=1 +31:38.549 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.549 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30198 [Leader]" +31:38.550 [INFO] TestAgent.server: cluster leadership acquired +31:38.550 [INFO] TestAgent.server: New leader elected: payload=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb +31:38.556 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:38.568 [INFO] TestAgent.server: initializing acls +31:38.582 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:38.584 [INFO] TestAgent.server: initializing acls +31:38.596 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:38.596 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:38.596 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:38.608 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb +31:38.608 [INFO] TestAgent.server: Updating LAN server: server="Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb (Addr: tcp/127.0.0.1:30198) (DC: dc1)" +31:38.608 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 +31:38.608 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 area=wan +31:38.624 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:38.624 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:38.624 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb +31:38.624 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 +31:38.624 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb.dc1 area=wan +31:38.624 [INFO] TestAgent.server: Updating LAN server: server="Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb (Addr: tcp/127.0.0.1:30198) (DC: dc1)" +31:38.645 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.663 [WARN] TestAgent: "default" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "replication" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "default" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token +31:38.663 [WARN] TestAgent: "replication" token present in both the configuration and persisted token store, using the persisted token +31:38.672 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token +31:38.672 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token +31:38.708 [WARN] TestAgent: unable to load persisted tokens: error="failed to decode tokens file "/tmp/consul-test/TestAgent_loadTokens-TestAgent-agent029863868/acl-tokens.json": invalid character '\x01' looking for beginning of value" +31:38.711 [WARN] TestAgent: unable to load persisted tokens: error="failed to decode tokens file "/tmp/consul-test/TestAgent_loadTokens-TestAgent-agent029863868/acl-tokens.json": json: cannot unmarshal array into Go value of type agent.persistedTokens" +31:38.711 [INFO] TestAgent: Requesting shutdown +31:38.712 [INFO] TestAgent.server: shutting down server +31:38.712 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:38.712 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:38.712 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.712 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:38.712 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:38.728 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:38.737 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.737 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.737 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.737 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.737 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.737 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb +31:38.737 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb +2021-10-05T16:31:38.738Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.738Z [INFO] agent.router.manager: shutting down +31:38.738 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:38.738 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:38.738 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:38.738 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:38.738 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +31:38.750 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-5c2e290b-f5ef-c583-c17c-f72444da5ebb 127.0.0.1 30182 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:5c2e290b-f5ef-c583-c17c-f72444da5ebb port:30198 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30183] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:38.750 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:38.750 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:38.750 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:38.750 [INFO] TestAgent: consul server down +31:38.750 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:38.750 [INFO] TestAgent: shutdown complete +31:38.750 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:38.750 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30179 network=tcp +31:38.750 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:38.750 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:38.750 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:38.750 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30179 network=udp +31:38.750 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30180 network=tcp +31:39.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.252 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_loadTokens (0.95s) + --- PASS: TestAgent_loadTokens/original-configuration (0.00s) + --- PASS: TestAgent_loadTokens/updated-configuration (0.00s) + --- PASS: TestAgent_loadTokens/persisted-tokens (0.00s) + --- PASS: TestAgent_loadTokens/persisted-tokens-override (0.00s) + --- PASS: TestAgent_loadTokens/partial-persisted (0.01s) + --- PASS: TestAgent_loadTokens/persistence-error-not-json (0.04s) + --- PASS: TestAgent_loadTokens/persistence-error-wrong-top-level (0.00s) +=== CONT TestAgent_checkStateSnapshot +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:38.341Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.354Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.409 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4fceb84e-720d-f76b-9337-375a16001b5c Address:127.0.0.1:30209}]" +31:38.409 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30209 [Follower]" leader= +31:38.410 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4fceb84e-720d-f76b-9337-375a16001b5c.dc1 127.0.0.1 +31:38.410 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4fceb84e-720d-f76b-9337-375a16001b5c 127.0.0.1 +2021-10-05T16:31:38.410Z [INFO] agent.router: Initializing LAN area manager +31:38.411 [INFO] TestAgent.server: Adding LAN server: server="Node-4fceb84e-720d-f76b-9337-375a16001b5c (Addr: tcp/127.0.0.1:30209) (DC: dc1)" +31:38.411 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4fceb84e-720d-f76b-9337-375a16001b5c.dc1 area=wan +31:38.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30217 network=udp +31:38.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30217 network=tcp +31:38.412 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30205 network=tcp +31:38.414 [INFO] TestAgent: started state syncer +31:38.414 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30210 network=tcp +31:38.480 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.480 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30209 [Candidate]" term=2 +31:38.517 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.517 [DEBUG] TestAgent.server.raft: vote granted: from=4fceb84e-720d-f76b-9337-375a16001b5c term=2 tally=1 +31:38.517 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.517 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30209 [Leader]" +31:38.518 [INFO] TestAgent.server: cluster leadership acquired +31:38.518 [INFO] TestAgent.server: New leader elected: payload=Node-4fceb84e-720d-f76b-9337-375a16001b5c +31:38.540 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30209 +31:38.632 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.636 [INFO] TestAgent: Synced node info +31:38.636 [DEBUG] TestAgent: Node info in sync +31:38.671 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.671 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.671 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.671 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.671 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.671 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4fceb84e-720d-f76b-9337-375a16001b5c +31:38.671 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4fceb84e-720d-f76b-9337-375a16001b5c +31:38.728 [INFO] TestAgent.server: federation state anti-entropy synced +31:38.836 [INFO] TestAgent: Requesting shutdown +31:38.836 [INFO] TestAgent.server: shutting down server +31:38.836 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:38.836 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:38.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:38.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:38.836 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.836 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:38.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:38.836 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:38.836 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:38.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:38.843 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:38.853Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.853Z [INFO] agent.router.manager: shutting down +31:38.853 [INFO] TestAgent: consul server down +31:38.853 [INFO] TestAgent: shutdown complete +31:38.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30217 network=tcp +31:38.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30217 network=udp +31:38.853 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30205 network=tcp +31:39.354 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.354 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_reloadWatches (1.03s) +=== CONT TestAgent_NodeMaintenanceMode +2021-10-05T16:31:38.805Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.827Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.869 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:588787c8-7777-d7e8-e059-fed60f829407 Address:127.0.0.1:30244}]" +31:38.870 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-588787c8-7777-d7e8-e059-fed60f829407.dc1 127.0.0.1 +31:38.870 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-588787c8-7777-d7e8-e059-fed60f829407 127.0.0.1 +2021-10-05T16:31:38.870Z [INFO] agent.router: Initializing LAN area manager +31:38.870 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30239 network=udp +31:38.870 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30244 [Follower]" leader= +31:38.871 [INFO] TestAgent.server: Adding LAN server: server="Node-588787c8-7777-d7e8-e059-fed60f829407 (Addr: tcp/127.0.0.1:30244) (DC: dc1)" +31:38.871 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-588787c8-7777-d7e8-e059-fed60f829407.dc1 area=wan +31:38.871 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30239 network=tcp +31:38.872 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30240 network=tcp +31:38.872 [INFO] TestAgent: started state syncer +31:38.872 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30252 network=tcp +31:38.912 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.912 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30244 [Candidate]" term=2 +31:38.936 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.936 [DEBUG] TestAgent.server.raft: vote granted: from=588787c8-7777-d7e8-e059-fed60f829407 term=2 tally=1 +31:38.936 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.936 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30244 [Leader]" +31:38.940 [INFO] TestAgent.server: cluster leadership acquired +31:38.940 [INFO] TestAgent.server: New leader elected: payload=Node-588787c8-7777-d7e8-e059-fed60f829407 +31:38.948 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30244 +31:38.972 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.001 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.001 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.001 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.001 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.001 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.001 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-588787c8-7777-d7e8-e059-fed60f829407 +31:39.001 [INFO] TestAgent.server: member joined, marking health alive: member=Node-588787c8-7777-d7e8-e059-fed60f829407 +31:39.032 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.037 [DEBUG] TestAgent: check state expired, not restoring: check=check1 +31:39.037 [INFO] TestAgent: Requesting shutdown +31:39.037 [INFO] TestAgent.server: shutting down server +31:39.037 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.037 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.037 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.037 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.037 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.037 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.038 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:39.038 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.038 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.038 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.045 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.049Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.049Z [INFO] agent.router.manager: shutting down +31:39.050 [INFO] TestAgent: consul server down +31:39.050 [INFO] TestAgent: shutdown complete +31:39.050 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30239 network=tcp +31:39.050 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30239 network=udp +31:39.050 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30240 network=tcp +31:39.551 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.551 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_loadCheckState (0.80s) +=== CONT TestAgent_AddCheck_restoresSnapshot +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:38.640Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.655Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.786 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:30391565-950e-6ec7-b66a-fb122719d886 Address:127.0.0.1:30237}]" +31:38.786 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30237 [Follower]" leader= +31:38.792 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-30391565-950e-6ec7-b66a-fb122719d886.dc1 127.0.0.1 +31:38.793 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-30391565-950e-6ec7-b66a-fb122719d886 127.0.0.1 +2021-10-05T16:31:38.793Z [INFO] agent.router: Initializing LAN area manager +31:38.793 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30225 network=udp +31:38.793 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-30391565-950e-6ec7-b66a-fb122719d886.dc1 area=wan +31:38.793 [INFO] TestAgent.server: Adding LAN server: server="Node-30391565-950e-6ec7-b66a-fb122719d886 (Addr: tcp/127.0.0.1:30237) (DC: dc1)" +31:38.793 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30225 network=tcp +31:38.794 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30233 network=tcp +31:38.794 [INFO] TestAgent: started state syncer +31:38.794 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30238 network=tcp +31:38.848 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.848 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30237 [Candidate]" term=2 +31:38.883 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.883 [DEBUG] TestAgent.server.raft: vote granted: from=30391565-950e-6ec7-b66a-fb122719d886 term=2 tally=1 +31:38.883 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.883 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30237 [Leader]" +31:38.884 [INFO] TestAgent.server: cluster leadership acquired +31:38.884 [INFO] TestAgent.server: New leader elected: payload=Node-30391565-950e-6ec7-b66a-fb122719d886 +31:38.894 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30237 +31:38.909 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.942 [INFO] TestAgent: Synced node info +31:38.942 [DEBUG] TestAgent: Node info in sync +31:38.956 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.956 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.956 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.956 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.956 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.956 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30391565-950e-6ec7-b66a-fb122719d886 +31:38.956 [INFO] TestAgent.server: member joined, marking health alive: member=Node-30391565-950e-6ec7-b66a-fb122719d886 +31:38.967 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.136 [INFO] TestAgent: Requesting shutdown +31:39.136 [INFO] TestAgent.server: shutting down server +31:39.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.136 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.136 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.136 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.136 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.136 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.136 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.136 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.136 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.136 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.144 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.151Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.151Z [INFO] agent.router.manager: shutting down +31:39.151 [INFO] TestAgent: consul server down +31:39.151 [INFO] TestAgent: shutdown complete +31:39.151 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30225 network=tcp +31:39.151 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30225 network=udp +31:39.151 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30233 network=tcp +31:39.652 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.652 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_purgeCheckState (1.03s) +=== CONT TestAgent_Service_MaintenanceMode +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:39.083Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.121Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.215 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:deb8d63b-39f5-9bee-bfe4-55a6600cfcea Address:127.0.0.1:30258}]" +31:39.215 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30258 [Follower]" leader= +31:39.216 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea.dc1 127.0.0.1 +31:39.217 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea 127.0.0.1 +2021-10-05T16:31:39.217Z [INFO] agent.router: Initializing LAN area manager +31:39.217 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30253 network=udp +31:39.217 [INFO] TestAgent.server: Adding LAN server: server="Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea (Addr: tcp/127.0.0.1:30258) (DC: dc1)" +31:39.217 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea.dc1 area=wan +31:39.217 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30253 network=tcp +31:39.218 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30254 network=tcp +31:39.218 [INFO] TestAgent: started state syncer +31:39.218 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30245 network=tcp +31:39.281 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.281 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30258 [Candidate]" term=2 +31:39.332 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.332 [DEBUG] TestAgent.server.raft: vote granted: from=deb8d63b-39f5-9bee-bfe4-55a6600cfcea term=2 tally=1 +31:39.332 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.332 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30258 [Leader]" +31:39.332 [INFO] TestAgent.server: cluster leadership acquired +31:39.333 [INFO] TestAgent.server: New leader elected: payload=Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea +31:39.351 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30258 +31:39.369 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.430 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.430 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.430 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.430 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.430 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.430 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea +31:39.430 [INFO] TestAgent.server: member joined, marking health alive: member=Node-deb8d63b-39f5-9bee-bfe4-55a6600cfcea +31:39.454 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.464 [INFO] TestAgent: Requesting shutdown +31:39.464 [INFO] TestAgent.server: shutting down server +31:39.464 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.464 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.464 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.464 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.464 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:39.464 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.464 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.473 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.492Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.492Z [INFO] agent.router.manager: shutting down +31:39.492 [INFO] TestAgent: consul server down +31:39.492 [INFO] TestAgent: shutdown complete +31:39.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30253 network=tcp +31:39.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30253 network=udp +31:39.492 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30254 network=tcp +31:39.993 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.993 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_persistCheckState (0.93s) +=== CONT TestAgent_unloadChecks +2021-10-05T16:31:39.206Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.280Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.348 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3981e5a-6d1b-5031-40c2-9c43e49ca16a Address:127.0.0.1:30251}]" +31:39.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30251 [Follower]" leader= +31:39.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3981e5a-6d1b-5031-40c2-9c43e49ca16a.dc1 127.0.0.1 +31:39.350 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3981e5a-6d1b-5031-40c2-9c43e49ca16a 127.0.0.1 +2021-10-05T16:31:39.350Z [INFO] agent.router: Initializing LAN area manager +31:39.350 [INFO] TestAgent.server: Adding LAN server: server="Node-e3981e5a-6d1b-5031-40c2-9c43e49ca16a (Addr: tcp/127.0.0.1:30251) (DC: dc1)" +31:39.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30246 network=udp +31:39.350 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3981e5a-6d1b-5031-40c2-9c43e49ca16a.dc1 area=wan +31:39.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30246 network=tcp +31:39.351 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30247 network=tcp +31:39.351 [INFO] TestAgent: started state syncer +31:39.351 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30231 network=tcp +31:39.394 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.394 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30251 [Candidate]" term=2 +31:39.417 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.417 [DEBUG] TestAgent.server.raft: vote granted: from=e3981e5a-6d1b-5031-40c2-9c43e49ca16a term=2 tally=1 +31:39.417 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.417 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30251 [Leader]" +31:39.417 [INFO] TestAgent.server: cluster leadership acquired +31:39.417 [INFO] TestAgent.server: New leader elected: payload=Node-e3981e5a-6d1b-5031-40c2-9c43e49ca16a +31:39.439 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30251 +31:39.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.496 [WARN] TestAgent: Failed to restore check: check=service:redis error="ServiceID "nope" does not exist" +31:39.496 [DEBUG] TestAgent: restored health check from file: check=service:redis file=/tmp/consul-test/TestAgent_loadChecks_checkFails-TestAgent-agent858968292/checks/60a2ef12de014a05ecdc850d9aab46da +31:39.496 [INFO] TestAgent: Requesting shutdown +31:39.496 [INFO] TestAgent.server: shutting down server +31:39.496 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.496 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:39.508 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.516Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.516Z [INFO] agent.router.manager: shutting down +31:39.528 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +31:39.528 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:39.528 [INFO] TestAgent: consul server down +31:39.528 [INFO] TestAgent: shutdown complete +31:39.528 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30246 network=tcp +31:39.528 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30246 network=udp +31:39.528 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30247 network=tcp +31:40.032 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.032 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_loadChecks_checkFails (0.86s) +=== CONT TestAgent_loadChecks_token +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:38.387Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.401Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.447 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fcd4dbfd-9a61-ad84-1410-477f0c7cebaf Address:127.0.0.1:30230}]" +31:38.448 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf.dc1 127.0.0.1 +31:38.448 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30230 [Follower]" leader= +31:38.449 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf 127.0.0.1 +2021-10-05T16:31:38.449Z [INFO] agent.router: Initializing LAN area manager +31:38.449 [INFO] TestAgent.server: Adding LAN server: server="Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf (Addr: tcp/127.0.0.1:30230) (DC: dc1)" +31:38.449 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf.dc1 area=wan +31:38.449 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30211 network=tcp +31:38.449 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30211 network=udp +31:38.450 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30226 network=tcp +31:38.450 [INFO] TestAgent: started state syncer +31:38.450 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30192 network=tcp +31:38.502 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.503 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30230 [Candidate]" term=2 +31:38.561 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.561 [DEBUG] TestAgent.server.raft: vote granted: from=fcd4dbfd-9a61-ad84-1410-477f0c7cebaf term=2 tally=1 +31:38.561 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.561 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30230 [Leader]" +31:38.561 [INFO] TestAgent.server: cluster leadership acquired +31:38.561 [INFO] TestAgent.server: New leader elected: payload=Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf +31:38.571 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30230 +31:38.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:38.744 [INFO] TestAgent: Synced node info +31:38.744 [DEBUG] TestAgent: Node info in sync +31:38.744 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:38.744 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:38.744 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:38.744 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:38.744 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:38.744 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf +31:38.744 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fcd4dbfd-9a61-ad84-1410-477f0c7cebaf +31:38.784 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.591 [WARN] TestAgent.server.rpc: Attempting to apply large raft entry: size_in_bytes=4719032 +31:39.726 [INFO] TestAgent: Requesting shutdown +31:39.726 [INFO] TestAgent.server: shutting down server +31:39.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.726 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.726 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.726 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.726 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.726 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.726 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.735 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.751Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.751Z [INFO] agent.router.manager: shutting down +31:39.751 [INFO] TestAgent: consul server down +31:39.751 [INFO] TestAgent: shutdown complete +31:39.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30211 network=tcp +31:39.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30211 network=udp +31:39.751 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30226 network=tcp +31:40.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.252 [INFO] TestAgent: Endpoints down +--- PASS: TestTxnEndpoint_Bad_Size_Net (8.00s) + --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_default_limits (1.31s) + --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_configured_max_txn_len (1.27s) + --- PASS: TestTxnEndpoint_Bad_Size_Net/exceeds_default_max_kv_value_size (1.58s) + --- PASS: TestTxnEndpoint_Bad_Size_Net/allowed (1.95s) + --- PASS: TestTxnEndpoint_Bad_Size_Net/allowed_kv_max_backward_compatible (1.89s) +=== CONT TestAgent_DeregisterPersistedSidecarAfterRestart +2021-10-05T16:31:39.285Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.315Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.440 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a9d62bb-3753-1a2f-d380-cc6bf92adc01 Address:127.0.0.1:30223}]" +31:39.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30223 [Follower]" leader= +31:39.441 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01.dc1 127.0.0.1 +31:39.441 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01 127.0.0.1 +2021-10-05T16:31:39.441Z [INFO] agent.router: Initializing LAN area manager +31:39.441 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30232 network=udp +31:39.442 [INFO] TestAgent.server: Adding LAN server: server="Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01 (Addr: tcp/127.0.0.1:30223) (DC: dc1)" +31:39.442 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01.dc1 area=wan +31:39.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30232 network=tcp +31:39.442 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30219 network=tcp +31:39.442 [INFO] TestAgent: started state syncer +31:39.442 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30199 network=tcp +31:39.503 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.503 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30223 [Candidate]" term=2 +31:39.544 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.544 [DEBUG] TestAgent.server.raft: vote granted: from=1a9d62bb-3753-1a2f-d380-cc6bf92adc01 term=2 tally=1 +31:39.544 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.544 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30223 [Leader]" +31:39.544 [INFO] TestAgent.server: cluster leadership acquired +31:39.545 [INFO] TestAgent.server: New leader elected: payload=Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01 +31:39.566 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30223 +31:39.635 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.732 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.732 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.732 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.732 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.732 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.732 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01 +31:39.732 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a9d62bb-3753-1a2f-d380-cc6bf92adc01 +31:39.743 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.750 [DEBUG] TestAgent: removed check: check=service:redis +31:39.751 [DEBUG] TestAgent: restored health check from file: check=service:redis file=/tmp/consul-test/TestAgent_checkStateSnapshot-TestAgent-agent280678131/checks/60a2ef12de014a05ecdc850d9aab46da +31:39.751 [INFO] TestAgent: Requesting shutdown +31:39.751 [INFO] TestAgent.server: shutting down server +31:39.751 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.751 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.751 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.751 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.751 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.751 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:39.751 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.751 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.751 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.763 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.763 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.764 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.774Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.774Z [INFO] agent.router.manager: shutting down +31:39.774 [INFO] TestAgent: consul server down +31:39.774 [INFO] TestAgent: shutdown complete +31:39.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30232 network=tcp +31:39.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30232 network=udp +31:39.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30219 network=tcp +31:40.276 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.276 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_checkStateSnapshot (1.02s) +=== CONT TestAgent_PurgeCheckOnDuplicate +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:39.374Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.389Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.472 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:934420e6-2d15-a077-ce44-d92467819f70 Address:127.0.0.1:30272}]" +31:39.473 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-934420e6-2d15-a077-ce44-d92467819f70.dc1 127.0.0.1 +31:39.473 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30272 [Follower]" leader= +31:39.484 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-934420e6-2d15-a077-ce44-d92467819f70 127.0.0.1 +2021-10-05T16:31:39.485Z [INFO] agent.router: Initializing LAN area manager +31:39.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30267 network=udp +31:39.485 [INFO] TestAgent.server: Adding LAN server: server="Node-934420e6-2d15-a077-ce44-d92467819f70 (Addr: tcp/127.0.0.1:30272) (DC: dc1)" +31:39.485 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-934420e6-2d15-a077-ce44-d92467819f70.dc1 area=wan +31:39.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30267 network=tcp +31:39.486 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30268 network=tcp +31:39.486 [INFO] TestAgent: started state syncer +31:39.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30273 network=tcp +31:39.540 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.540 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30272 [Candidate]" term=2 +31:39.588 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.588 [DEBUG] TestAgent.server.raft: vote granted: from=934420e6-2d15-a077-ce44-d92467819f70 term=2 tally=1 +31:39.588 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.588 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30272 [Leader]" +31:39.588 [INFO] TestAgent.server: cluster leadership acquired +31:39.588 [INFO] TestAgent.server: New leader elected: payload=Node-934420e6-2d15-a077-ce44-d92467819f70 +31:39.640 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30272 +31:39.718 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.764 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.764 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.764 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.764 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.764 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.764 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-934420e6-2d15-a077-ce44-d92467819f70 +31:39.764 [INFO] TestAgent.server: member joined, marking health alive: member=Node-934420e6-2d15-a077-ce44-d92467819f70 +31:39.780 [INFO] TestAgent: Node entered maintenance mode +31:39.785 [INFO] TestAgent: Synced node info +31:39.817 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.817 [INFO] TestAgent: Synced check: check=_node_maintenance +31:39.817 [DEBUG] TestAgent: Node info in sync +31:39.817 [DEBUG] TestAgent: Check in sync: check=_node_maintenance +31:39.817 [DEBUG] TestAgent: removed check: check=_node_maintenance +31:39.817 [INFO] TestAgent: Node left maintenance mode +31:39.827 [INFO] TestAgent: Node entered maintenance mode +31:39.827 [INFO] TestAgent: Requesting shutdown +31:39.828 [INFO] TestAgent.server: shutting down server +31:39.828 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.828 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.828 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.828 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.828 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.828 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.828 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.828 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.828 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.828 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.839 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.845Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.845Z [INFO] agent.router.manager: shutting down +31:39.845 [INFO] TestAgent: consul server down +31:39.845 [INFO] TestAgent: shutdown complete +31:39.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30267 network=tcp +31:39.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30267 network=udp +31:39.846 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30268 network=tcp +31:40.349 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.349 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_NodeMaintenanceMode (1.00s) +=== CONT TestAgent_PurgeCheck +2021-10-05T16:31:39.431Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.466Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.548 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9ed3be2a-39ed-6592-aee4-a99f67e6eba0 Address:127.0.0.1:30279}]" +31:39.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0.dc1 127.0.0.1 +31:39.549 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0 127.0.0.1 +2021-10-05T16:31:39.549Z [INFO] agent.router: Initializing LAN area manager +31:39.549 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30274 network=udp +31:39.549 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30279 [Follower]" leader= +31:39.550 [INFO] TestAgent.server: Adding LAN server: server="Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0 (Addr: tcp/127.0.0.1:30279) (DC: dc1)" +31:39.550 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0.dc1 area=wan +31:39.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30274 network=tcp +31:39.550 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30275 network=tcp +31:39.551 [INFO] TestAgent: started state syncer +31:39.551 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30259 network=tcp +31:39.616 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.616 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30279 [Candidate]" term=2 +31:39.663 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.663 [DEBUG] TestAgent.server.raft: vote granted: from=9ed3be2a-39ed-6592-aee4-a99f67e6eba0 term=2 tally=1 +31:39.663 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.663 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30279 [Leader]" +31:39.664 [INFO] TestAgent.server: cluster leadership acquired +31:39.664 [INFO] TestAgent.server: New leader elected: payload=Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0 +31:39.720 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30279 +31:39.755 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.804 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.804 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.804 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.804 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.804 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.804 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0 +31:39.804 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9ed3be2a-39ed-6592-aee4-a99f67e6eba0 +31:39.843 [INFO] TestAgent.server: federation state anti-entropy synced +31:39.864 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:39.869 [INFO] TestAgent: Synced node info +31:39.869 [DEBUG] TestAgent: Node info in sync +31:39.882 [INFO] TestAgent: Requesting shutdown +31:39.882 [INFO] TestAgent.server: shutting down server +31:39.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:39.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:39.882 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:39.882 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:39.882 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:39.882 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:39.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:39.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:39.883 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:39.883 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:39.887 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:39.896Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:39.896Z [INFO] agent.router.manager: shutting down +31:39.896 [INFO] TestAgent: consul server down +31:39.896 [INFO] TestAgent: shutdown complete +31:39.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30274 network=tcp +31:39.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30274 network=udp +31:39.896 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30275 network=tcp +31:40.396 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.396 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:38.662Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:38.710Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:38.770 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f7f29e51-5ba1-2a81-be08-f1e127231b53 Address:127.0.0.1:30214}]" +31:38.770 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30214 [Follower]" leader= +31:38.771 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f7f29e51-5ba1-2a81-be08-f1e127231b53.dc1 127.0.0.1 +31:38.772 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f7f29e51-5ba1-2a81-be08-f1e127231b53 127.0.0.1 +2021-10-05T16:31:38.772Z [INFO] agent.router: Initializing LAN area manager +31:38.772 [INFO] TestAgent.server: Adding LAN server: server="Node-f7f29e51-5ba1-2a81-be08-f1e127231b53 (Addr: tcp/127.0.0.1:30214) (DC: dc1)" +31:38.772 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30193 network=udp +31:38.772 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f7f29e51-5ba1-2a81-be08-f1e127231b53.dc1 area=wan +31:38.772 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30193 network=tcp +31:38.773 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30194 network=tcp +31:38.773 [INFO] TestAgent: started state syncer +31:38.773 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30224 network=tcp +31:38.817 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:38.817 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30214 [Candidate]" term=2 +31:38.847 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:38.847 [DEBUG] TestAgent.server.raft: vote granted: from=f7f29e51-5ba1-2a81-be08-f1e127231b53 term=2 tally=1 +31:38.847 [INFO] TestAgent.server.raft: election won: tally=1 +31:38.847 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30214 [Leader]" +31:38.848 [INFO] TestAgent.server: cluster leadership acquired +31:38.848 [INFO] TestAgent.server: New leader elected: payload=Node-f7f29e51-5ba1-2a81-be08-f1e127231b53 +31:38.858 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30214 +31:38.871 [INFO] TestAgent: Requesting shutdown +31:38.872 [INFO] TestAgent.server: shutting down server +31:38.872 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:38.872 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:38.879 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:38.886 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:31:38.890Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:38.890Z [INFO] agent.router.manager: shutting down +31:38.899 [INFO] TestAgent: consul server down +31:38.899 [INFO] TestAgent: shutdown complete +31:38.899 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30193 network=tcp +31:38.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30193 network=udp +31:38.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30194 network=tcp +31:38.900 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:38.900 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:38.900 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:39.400 [INFO] TestAgent: Waiting for endpoints to shut down +31:39.400 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_GetCoordinate (1.78s) +=== CONT TestAgent_PersistCheck +2021-10-05T16:31:39.587Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.633Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.740 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:90547f71-4a03-5221-a677-693c8a577657 Address:127.0.0.1:30265}]" +31:39.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-90547f71-4a03-5221-a677-693c8a577657.dc1 127.0.0.1 +31:39.742 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-90547f71-4a03-5221-a677-693c8a577657 127.0.0.1 +2021-10-05T16:31:39.742Z [INFO] agent.router: Initializing LAN area manager +31:39.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30260 network=udp +31:39.742 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30265 [Follower]" leader= +31:39.742 [INFO] TestAgent.server: Adding LAN server: server="Node-90547f71-4a03-5221-a677-693c8a577657 (Addr: tcp/127.0.0.1:30265) (DC: dc1)" +31:39.742 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-90547f71-4a03-5221-a677-693c8a577657.dc1 area=wan +31:39.743 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30260 network=tcp +31:39.743 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30261 network=tcp +31:39.743 [INFO] TestAgent: started state syncer +31:39.743 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30115 network=tcp +31:39.809 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.809 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30265 [Candidate]" term=2 +31:39.824 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.824 [DEBUG] TestAgent.server.raft: vote granted: from=90547f71-4a03-5221-a677-693c8a577657 term=2 tally=1 +31:39.824 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.824 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30265 [Leader]" +31:39.825 [INFO] TestAgent.server: cluster leadership acquired +31:39.825 [INFO] TestAgent.server: New leader elected: payload=Node-90547f71-4a03-5221-a677-693c8a577657 +31:39.840 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30265 +31:39.856 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:39.880 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:39.880 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:39.880 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:39.880 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:39.880 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:39.880 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90547f71-4a03-5221-a677-693c8a577657 +31:39.880 [INFO] TestAgent.server: member joined, marking health alive: member=Node-90547f71-4a03-5221-a677-693c8a577657 +31:39.895 [INFO] TestAgent.server: federation state anti-entropy synced +31:40.016 [INFO] TestAgent: Requesting shutdown +31:40.016 [INFO] TestAgent.server: shutting down server +31:40.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:40.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:40.016 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:40.016 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:40.016 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:40.016 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:40.016 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:40.016 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:40.016 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:40.016 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:40.016 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:40.028 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:40.040Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:40.040Z [INFO] agent.router.manager: shutting down +31:40.040 [INFO] TestAgent: consul server down +31:40.040 [INFO] TestAgent: shutdown complete +31:40.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30260 network=tcp +31:40.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30260 network=udp +31:40.040 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30261 network=tcp +31:40.544 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.544 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_restoresSnapshot (0.99s) +=== CONT TestAgent_updateTTLCheck +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:39.728Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.793Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.876 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:71e3adc3-57fa-1c29-25a2-dad958bf6eb2 Address:127.0.0.1:30121}]" +31:39.876 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30121 [Follower]" leader= +31:39.876 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2.dc1 127.0.0.1 +31:39.878 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2 127.0.0.1 +2021-10-05T16:31:39.878Z [INFO] agent.router: Initializing LAN area manager +31:39.878 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30116 network=udp +31:39.878 [INFO] TestAgent.server: Adding LAN server: server="Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2 (Addr: tcp/127.0.0.1:30121) (DC: dc1)" +31:39.878 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2.dc1 area=wan +31:39.878 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30116 network=tcp +31:39.879 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30117 network=tcp +31:39.879 [INFO] TestAgent: started state syncer +31:39.879 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30280 network=tcp +31:39.939 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.939 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30121 [Candidate]" term=2 +31:39.966 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:39.966 [DEBUG] TestAgent.server.raft: vote granted: from=71e3adc3-57fa-1c29-25a2-dad958bf6eb2 term=2 tally=1 +31:39.966 [INFO] TestAgent.server.raft: election won: tally=1 +31:39.966 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30121 [Leader]" +31:39.966 [INFO] TestAgent.server: cluster leadership acquired +31:39.966 [INFO] TestAgent.server: New leader elected: payload=Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2 +31:39.988 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30121 +31:39.988 [INFO] TestAgent: Synced node info +31:40.041 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:40.066 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:40.066 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:40.066 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:40.066 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:40.066 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:40.066 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2 +31:40.066 [INFO] TestAgent.server: member joined, marking health alive: member=Node-71e3adc3-57fa-1c29-25a2-dad958bf6eb2 +31:40.087 [INFO] TestAgent.server: federation state anti-entropy synced +31:40.312 [INFO] TestAgent: Service entered maintenance mode: service=redis +31:40.312 [DEBUG] TestAgent: removed check: check=_service_maintenance:redis +31:40.312 [INFO] TestAgent: Service left maintenance mode: service=redis +31:40.332 [INFO] TestAgent: Service entered maintenance mode: service=redis +31:40.332 [INFO] TestAgent: Requesting shutdown +31:40.332 [INFO] TestAgent.server: shutting down server +31:40.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:40.332 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:40.332 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:40.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:40.332 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:40.332 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:40.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:40.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:40.332 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:40.332 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:40.352 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:40.364Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:40.364Z [INFO] agent.router.manager: shutting down +31:40.365 [INFO] TestAgent: consul server down +31:40.365 [INFO] TestAgent: shutdown complete +31:40.365 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30116 network=tcp +31:40.365 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30116 network=udp +31:40.365 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30117 network=tcp +31:40.868 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.868 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Service_MaintenanceMode (1.21s) +=== CONT TestAgent_HTTPCheck_TLSSkipVerify +2021-10-05T16:31:40.111Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:40.146Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:40.217 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5fe98851-0473-cd54-e322-1452659f54c4 Address:127.0.0.1:30300}]" +31:40.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5fe98851-0473-cd54-e322-1452659f54c4.dc1 127.0.0.1 +31:40.218 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5fe98851-0473-cd54-e322-1452659f54c4 127.0.0.1 +2021-10-05T16:31:40.218Z [INFO] agent.router: Initializing LAN area manager +31:40.219 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30295 network=udp +31:40.219 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30300 [Follower]" leader= +31:40.219 [INFO] TestAgent.server: Adding LAN server: server="Node-5fe98851-0473-cd54-e322-1452659f54c4 (Addr: tcp/127.0.0.1:30300) (DC: dc1)" +31:40.219 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5fe98851-0473-cd54-e322-1452659f54c4.dc1 area=wan +31:40.219 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30295 network=tcp +31:40.220 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30296 network=tcp +31:40.220 [INFO] TestAgent: started state syncer +31:40.220 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30301 network=tcp +31:40.272 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:40.272 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30300 [Candidate]" term=2 +31:40.317 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:40.317 [DEBUG] TestAgent.server.raft: vote granted: from=5fe98851-0473-cd54-e322-1452659f54c4 term=2 tally=1 +31:40.317 [INFO] TestAgent.server.raft: election won: tally=1 +31:40.317 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30300 [Leader]" +31:40.317 [INFO] TestAgent.server: cluster leadership acquired +31:40.317 [INFO] TestAgent.server: New leader elected: payload=Node-5fe98851-0473-cd54-e322-1452659f54c4 +31:40.359 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30300 +31:40.408 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:40.455 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:40.455 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:40.455 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:40.455 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:40.455 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:40.455 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5fe98851-0473-cd54-e322-1452659f54c4 +31:40.455 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5fe98851-0473-cd54-e322-1452659f54c4 +31:40.460 [INFO] TestAgent: Requesting shutdown +31:40.460 [INFO] TestAgent.server: shutting down server +31:40.460 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:40.460 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:40.460 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:40.460 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:40.460 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:40.460 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:40.460 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:40.460 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:40.460 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:40.465 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:40.465 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:40.472 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:40.484Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:40.484Z [INFO] agent.router.manager: shutting down +31:40.484 [INFO] TestAgent: consul server down +31:40.484 [INFO] TestAgent: shutdown complete +31:40.484 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30295 network=tcp +31:40.484 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30295 network=udp +31:40.484 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30296 network=tcp +31:40.988 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.988 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_loadChecks_token (0.98s) +=== CONT TestAgent_RemoveCheck +2021-10-05T16:31:40.010Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:40.098Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:40.154 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02dff127-2beb-741d-d046-6e8232ef052b Address:127.0.0.1:30293}]" +31:40.155 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02dff127-2beb-741d-d046-6e8232ef052b.dc1 127.0.0.1 +31:40.156 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30293 [Follower]" leader= +31:40.156 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02dff127-2beb-741d-d046-6e8232ef052b 127.0.0.1 +2021-10-05T16:31:40.156Z [INFO] agent.router: Initializing LAN area manager +31:40.156 [INFO] TestAgent.server: Adding LAN server: server="Node-02dff127-2beb-741d-d046-6e8232ef052b (Addr: tcp/127.0.0.1:30293) (DC: dc1)" +31:40.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30288 network=udp +31:40.156 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02dff127-2beb-741d-d046-6e8232ef052b.dc1 area=wan +31:40.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30288 network=tcp +31:40.157 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30289 network=tcp +31:40.157 [INFO] TestAgent: started state syncer +31:40.157 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30294 network=tcp +31:40.222 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:40.222 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30293 [Candidate]" term=2 +31:40.235 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:40.236 [DEBUG] TestAgent.server.raft: vote granted: from=02dff127-2beb-741d-d046-6e8232ef052b term=2 tally=1 +31:40.236 [INFO] TestAgent.server.raft: election won: tally=1 +31:40.236 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30293 [Leader]" +31:40.236 [INFO] TestAgent.server: cluster leadership acquired +31:40.236 [INFO] TestAgent.server: New leader elected: payload=Node-02dff127-2beb-741d-d046-6e8232ef052b +31:40.264 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30293 +31:40.312 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:40.404 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:40.404 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:40.404 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:40.404 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:40.404 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:40.404 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02dff127-2beb-741d-d046-6e8232ef052b +31:40.404 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02dff127-2beb-741d-d046-6e8232ef052b +31:40.427 [INFO] TestAgent.server: federation state anti-entropy synced +31:40.512 [DEBUG] TestAgent: removed check: check=service:redis +31:40.512 [INFO] TestAgent: Requesting shutdown +31:40.512 [INFO] TestAgent.server: shutting down server +31:40.512 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:40.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:40.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:40.512 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:40.512 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:40.512 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:40.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:40.512 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:40.512 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:40.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:40.512 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:40.520 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:40.536Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:40.536Z [INFO] agent.router.manager: shutting down +31:40.536 [INFO] TestAgent: consul server down +31:40.536 [INFO] TestAgent: shutdown complete +31:40.536 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30288 network=tcp +31:40.536 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30288 network=udp +31:40.536 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30289 network=tcp +31:41.038 [INFO] TestAgent: Waiting for endpoints to shut down +31:41.038 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_unloadChecks (1.05s) +=== CONT TestAgent_AddCheck_Alias_userAndSetToken +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:40.367Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:40.394Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:40.468 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b29c8b3-89e0-acea-9a43-56bdf03eb95c Address:127.0.0.1:30321}]" +31:40.468 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30321 [Follower]" leader= +31:40.469 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c.dc1 127.0.0.1 +31:40.469 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c 127.0.0.1 +2021-10-05T16:31:40.469Z [INFO] agent.router: Initializing LAN area manager +31:40.469 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30316 network=udp +31:40.470 [INFO] TestAgent.server: Adding LAN server: server="Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c (Addr: tcp/127.0.0.1:30321) (DC: dc1)" +31:40.470 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c.dc1 area=wan +31:40.470 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30316 network=tcp +31:40.470 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30317 network=tcp +31:40.470 [INFO] TestAgent: started state syncer +31:40.470 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30322 network=tcp +31:40.540 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:40.540 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30321 [Candidate]" term=2 +31:40.588 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:40.588 [DEBUG] TestAgent.server.raft: vote granted: from=2b29c8b3-89e0-acea-9a43-56bdf03eb95c term=2 tally=1 +31:40.588 [INFO] TestAgent.server.raft: election won: tally=1 +31:40.588 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30321 [Leader]" +31:40.588 [INFO] TestAgent.server: cluster leadership acquired +31:40.588 [INFO] TestAgent.server: New leader elected: payload=Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c +31:40.596 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30321 +31:40.614 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:40.648 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:40.649 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:40.649 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:40.649 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:40.649 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:40.649 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c +31:40.649 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b29c8b3-89e0-acea-9a43-56bdf03eb95c +31:40.660 [INFO] TestAgent.server: federation state anti-entropy synced +31:40.716 [DEBUG] TestAgent: removed check: check=mem +31:40.716 [DEBUG] TestAgent: removed check: check=mem +31:40.716 [INFO] TestAgent: Requesting shutdown +31:40.716 [INFO] TestAgent.server: shutting down server +31:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:40.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:40.716 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:40.716 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:40.721 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:40.728Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:40.728Z [INFO] agent.router.manager: shutting down +31:40.728 [INFO] TestAgent: consul server down +31:40.728 [INFO] TestAgent: shutdown complete +31:40.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30316 network=tcp +31:40.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30316 network=udp +31:40.728 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30317 network=tcp +31:41.232 [INFO] TestAgent: Waiting for endpoints to shut down +31:41.232 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_PurgeCheck (0.88s) +=== CONT TestAgent_AddCheck_Alias_userToken +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:40.915Z [WARN] agent: Node name "Node 08b983a0-dc37-f7e8-635d-dc3152ee884c" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.915Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:40.983Z [WARN] agent.auto_config: Node name "Node 08b983a0-dc37-f7e8-635d-dc3152ee884c" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.983Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:40.985 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node 08b983a0-dc37-f7e8-635d-dc3152ee884c 127.0.0.1 +2021-10-05T16:31:40.985Z [INFO] agent.router: Initializing LAN area manager +31:40.986 [DEBUG] Agent2: check exists, not restoring from file: check=mem file=/tmp/consul-test/TestAgent_PurgeCheckOnDuplicate-TestAgent-agent123976933/checks/afc4fc7e48a0710a1dc94ef3e8bc5764 +31:40.986 [INFO] Agent2: Started DNS server: address=127.0.0.1:30358 network=udp +31:40.986 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node +31:40.986 [INFO] Agent2: Started DNS server: address=127.0.0.1:30358 network=tcp +31:40.987 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30359 network=tcp +31:40.987 [INFO] Agent2: started state syncer +31:40.988 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30364 network=tcp +31:40.988 [INFO] Agent2: Requesting shutdown +31:40.988 [INFO] Agent2.client: shutting down client +31:40.988 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave +2021-10-05T16:31:40.988Z [WARN] agent.router.manager: No servers available +31:40.988 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:41.010 [INFO] Agent2: consul client down +31:41.010 [INFO] Agent2: shutdown complete +31:41.010 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30358 network=tcp +31:41.010 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30358 network=udp +31:41.010 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30359 network=tcp +31:41.510 [INFO] Agent2: Waiting for endpoints to shut down +31:41.510 [INFO] Agent2: Endpoints down +2021-10-05T16:31:40.303Z [WARN] agent: Node name "Node 08b983a0-dc37-f7e8-635d-dc3152ee884c" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.303Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:40.339Z [WARN] agent.auto_config: Node name "Node 08b983a0-dc37-f7e8-635d-dc3152ee884c" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.339Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:40.340 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node 08b983a0-dc37-f7e8-635d-dc3152ee884c 127.0.0.1 +2021-10-05T16:31:40.340Z [INFO] agent.router: Initializing LAN area manager +31:40.344 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30309 network=tcp +31:40.344 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30309 network=udp +31:40.344 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30310 network=tcp +31:40.345 [INFO] TestAgent: started state syncer +31:40.345 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30315 network=tcp +2021-10-05T16:31:40.345Z [WARN] agent.router.manager: No servers available +31:40.345 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:40.356 [INFO] TestAgent: Requesting shutdown +31:40.356 [INFO] TestAgent.client: shutting down client +31:40.356 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:40.372 [INFO] TestAgent: consul client down +31:40.372 [INFO] TestAgent: shutdown complete +31:40.372 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30309 network=tcp +31:40.372 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30309 network=udp +31:40.372 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30310 network=tcp +31:40.876 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.876 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_PurgeCheckOnDuplicate (1.23s) +=== CONT TestAgent_AddCheck_Alias_setToken +2021-10-05T16:31:40.908Z [WARN] agent: Node name "Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.957Z [WARN] agent.auto_config: Node name "Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +31:40.958 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5 127.0.0.1 +2021-10-05T16:31:40.958Z [INFO] agent.router: Initializing LAN area manager +31:40.959 [DEBUG] Agent2: restored service definition from file: service=svc-sidecar-proxy file=/tmp/consul-test/TestAgent_DeregisterPersistedSidecarAfterRestart-TestAgent-agent632262686/services/18ed17ea501b78b6980ecfa920fcf57a +31:40.959 [DEBUG] Agent2: restored service definition from file: service=svc file=/tmp/consul-test/TestAgent_DeregisterPersistedSidecarAfterRestart-TestAgent-agent632262686/services/961e38b5146c0b07d078f53dec9b699a +31:40.959 [INFO] Agent2: Started DNS server: address=127.0.0.1:30337 network=udp +31:40.965 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node +31:40.966 [INFO] Agent2: Started DNS server: address=127.0.0.1:30337 network=tcp +31:40.968 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30338 network=tcp +31:40.968 [INFO] Agent2: started state syncer +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.970Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.971Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.972Z [WARN] agent.router.manager: No servers available +31:40.972 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30350 network=tcp +2021-10-05T16:31:40.972Z [WARN] agent.router.manager: No servers available +31:40.972 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" +31:40.973 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:40.973Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.973Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.973Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.973Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.974Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.974Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.974Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.974Z [WARN] agent.router.manager: No servers available +31:40.975 [DEBUG] Agent2: removed service: service=svc +2021-10-05T16:31:40.975Z [WARN] agent.router.manager: No servers available +31:40.975 [DEBUG] Agent2: removed service: service=svc-sidecar-proxy +2021-10-05T16:31:40.975Z [WARN] agent.router.manager: No servers available +31:40.975 [INFO] Agent2: Requesting shutdown +2021-10-05T16:31:40.975Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.975Z [WARN] agent.router.manager: No servers available +31:40.975 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:40.975 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:40.975 [INFO] Agent2.client: shutting down client +31:40.975 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave +31:41.016 [INFO] Agent2: consul client down +31:41.016 [INFO] Agent2: shutdown complete +31:41.016 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30337 network=tcp +31:41.016 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30337 network=udp +31:41.016 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30338 network=tcp +31:41.517 [INFO] Agent2: Waiting for endpoints to shut down +31:41.517 [INFO] Agent2: Endpoints down +2021-10-05T16:31:40.277Z [WARN] agent: Node name "Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:31:40.313Z [WARN] agent.auto_config: Node name "Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +31:40.313 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node 2e44cd71-37f1-abc0-0ce5-03c84e7203f5 127.0.0.1 +2021-10-05T16:31:40.313Z [INFO] agent.router: Initializing LAN area manager +31:40.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30302 network=udp +31:40.314 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30302 network=tcp +31:40.314 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30303 network=tcp +31:40.314 [INFO] TestAgent: started state syncer +31:40.315 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30308 network=tcp +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +31:40.325 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:40.325 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:40.325 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:40.325Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +2021-10-05T16:31:40.326Z [WARN] agent.router.manager: No servers available +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" +31:40.326 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" +31:40.336 [INFO] TestAgent: Requesting shutdown +31:40.336 [INFO] TestAgent.client: shutting down client +31:40.336 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:40.352 [INFO] TestAgent: consul client down +31:40.352 [INFO] TestAgent: shutdown complete +31:40.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30302 network=tcp +31:40.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30302 network=udp +31:40.352 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30303 network=tcp +31:40.854 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.854 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_DeregisterPersistedSidecarAfterRestart (1.26s) +=== CONT TestAgent_AddCheck_Alias +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:40.570Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:40.604Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:40.685 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b Address:127.0.0.1:30349}]" +31:40.685 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30349 [Follower]" leader= +31:40.686 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b.dc1 127.0.0.1 +31:40.686 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b 127.0.0.1 +2021-10-05T16:31:40.686Z [INFO] agent.router: Initializing LAN area manager +31:40.687 [INFO] TestAgent.server: Adding LAN server: server="Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b (Addr: tcp/127.0.0.1:30349) (DC: dc1)" +31:40.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30344 network=udp +31:40.687 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b.dc1 area=wan +31:40.687 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30344 network=tcp +31:40.687 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30345 network=tcp +31:40.687 [INFO] TestAgent: started state syncer +31:40.687 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30336 network=tcp +31:40.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:40.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30349 [Candidate]" term=2 +31:40.783 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:40.783 [DEBUG] TestAgent.server.raft: vote granted: from=03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b term=2 tally=1 +31:40.783 [INFO] TestAgent.server.raft: election won: tally=1 +31:40.783 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30349 [Leader]" +31:40.784 [INFO] TestAgent.server: cluster leadership acquired +31:40.784 [INFO] TestAgent.server: New leader elected: payload=Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b +31:40.797 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30349 +31:40.818 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:40.836 [INFO] TestAgent: Synced node info +31:40.853 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:40.853 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:40.853 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:40.853 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:40.853 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:40.853 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b +31:40.853 [INFO] TestAgent.server: member joined, marking health alive: member=Node-03ed785f-9f4e-c85c-ef2c-f4a0fd22fe3b +31:40.890 [INFO] TestAgent.server: federation state anti-entropy synced +31:41.100 [DEBUG] TestAgent: Check status updated: check=mem status=passing +31:41.101 [DEBUG] TestAgent: Check status updated: check=mem status=critical +31:41.101 [INFO] TestAgent: Requesting shutdown +31:41.101 [INFO] TestAgent.server: shutting down server +31:41.101 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:41.101 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:41.101 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:41.101 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:41.101 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:41.101 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:41.101 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:41.101 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:41.101 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:41.101 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:41.118 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:41.119Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:41.119Z [INFO] agent.router.manager: shutting down +31:41.119 [INFO] TestAgent: consul server down +31:41.119 [INFO] TestAgent: shutdown complete +31:41.119 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30344 network=tcp +31:41.119 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30344 network=udp +31:41.119 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30345 network=tcp +31:41.620 [INFO] TestAgent: Waiting for endpoints to shut down +31:41.620 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_updateTTLCheck (1.08s) +=== CONT TestAgent_AddCheck_GRPC +2021-10-05T16:31:41.033Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:41.114Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:41.115 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-0be06c17-199b-8258-9b3c-8bd4eba8431c 127.0.0.1 +2021-10-05T16:31:41.115Z [INFO] agent.router: Initializing LAN area manager +31:41.115 [DEBUG] Agent2: restored health check from file: check=mem file=/tmp/consul-test/TestAgent_PersistCheck-TestAgent-agent844221023/checks/afc4fc7e48a0710a1dc94ef3e8bc5764 +31:41.115 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-36971cda-bf1e-5e57-f91f-cb635f12f30b: 127.0.0.1:30326 +31:41.115 [INFO] Agent2: Started DNS server: address=127.0.0.1:30365 network=udp +31:41.116 [INFO] Agent2: Started DNS server: address=127.0.0.1:30365 network=tcp +31:41.116 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:30326: connect: connection refused +31:41.116 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node +31:41.128 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30366 network=tcp +31:41.129 [INFO] Agent2: started state syncer +31:41.131 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30371 network=tcp +2021-10-05T16:31:41.131Z [WARN] agent.router.manager: No servers available +31:41.131 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:41.136 [INFO] Agent2: Requesting shutdown +31:41.136 [INFO] Agent2.client: shutting down client +31:41.136 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave +31:41.139 [INFO] Agent2: consul client down +31:41.139 [INFO] Agent2: shutdown complete +31:41.139 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30365 network=tcp +31:41.139 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30365 network=udp +31:41.139 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30366 network=tcp +31:41.640 [INFO] Agent2: Waiting for endpoints to shut down +31:41.640 [INFO] Agent2: Endpoints down +2021-10-05T16:31:40.418Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:40.441Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:40.442 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-36971cda-bf1e-5e57-f91f-cb635f12f30b 127.0.0.1 +2021-10-05T16:31:40.442Z [INFO] agent.router: Initializing LAN area manager +31:40.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30323 network=udp +31:40.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30323 network=tcp +31:40.443 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30324 network=tcp +31:40.443 [INFO] TestAgent: started state syncer +31:40.444 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30343 network=tcp +2021-10-05T16:31:40.444Z [WARN] agent.router.manager: No servers available +31:40.444 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:40.458 [INFO] TestAgent: Requesting shutdown +31:40.458 [INFO] TestAgent.client: shutting down client +31:40.458 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:40.466 [INFO] TestAgent: consul client down +31:40.466 [INFO] TestAgent: shutdown complete +31:40.466 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30323 network=tcp +31:40.466 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30323 network=udp +31:40.466 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30324 network=tcp +31:40.966 [INFO] TestAgent: Waiting for endpoints to shut down +31:40.966 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_PersistCheck (1.25s) +=== CONT TestAgent_AddCheck_ExecRemoteDisable +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:41.247Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.263Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f69a916f-cce2-6f69-6484-4dedeb327c4f Address:127.0.0.1:30405}]" +31:41.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30405 [Follower]" leader= +31:41.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f69a916f-cce2-6f69-6484-4dedeb327c4f.dc1 127.0.0.1 +31:41.321 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f69a916f-cce2-6f69-6484-4dedeb327c4f 127.0.0.1 +2021-10-05T16:31:41.321Z [INFO] agent.router: Initializing LAN area manager +31:41.321 [INFO] TestAgent.server: Adding LAN server: server="Node-f69a916f-cce2-6f69-6484-4dedeb327c4f (Addr: tcp/127.0.0.1:30405) (DC: dc1)" +31:41.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30400 network=udp +31:41.321 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f69a916f-cce2-6f69-6484-4dedeb327c4f.dc1 area=wan +31:41.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30400 network=tcp +31:41.322 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30401 network=tcp +31:41.322 [INFO] TestAgent: started state syncer +31:41.322 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30392 network=tcp +31:41.364 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.364 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30405 [Candidate]" term=2 +31:41.393 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.393 [DEBUG] TestAgent.server.raft: vote granted: from=f69a916f-cce2-6f69-6484-4dedeb327c4f term=2 tally=1 +31:41.393 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.393 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30405 [Leader]" +31:41.393 [INFO] TestAgent.server: cluster leadership acquired +31:41.393 [INFO] TestAgent.server: New leader elected: payload=Node-f69a916f-cce2-6f69-6484-4dedeb327c4f +31:41.415 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30405 +31:41.435 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.452 [INFO] TestAgent: Requesting shutdown +31:41.452 [INFO] TestAgent.server: shutting down server +31:41.452 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:41.452 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:41.468 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:41.468Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:41.468Z [INFO] agent.router.manager: shutting down +31:41.468 [INFO] TestAgent: consul server down +31:41.468 [INFO] TestAgent: shutdown complete +31:41.468 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30400 network=tcp +31:41.468 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30400 network=udp +31:41.468 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30401 network=tcp +31:41.469 [ERROR] TestAgent.server.connect: Raft apply failed: error="raft is already shutdown" +31:41.469 [ERROR] TestAgent.server: failed to establish leadership: error="raft is already shutdown" +31:41.970 [INFO] TestAgent: Waiting for endpoints to shut down +31:41.970 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_Alias_userToken (0.74s) +=== CONT TestAgent_AddCheck_ExecDisable +2021-10-05T16:31:41.059Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.059Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:41.101Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.101Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:41.142 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eb4b0b24-8e78-c0d1-05a2-494594ab07f4 Address:127.0.0.1:30377}]" +31:41.142 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30377 [Follower]" leader= +31:41.143 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4.dc1 127.0.0.1 +31:41.144 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4 127.0.0.1 +2021-10-05T16:31:41.144Z [INFO] agent.router: Initializing LAN area manager +31:41.144 [INFO] TestAgent.server: Adding LAN server: server="Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4 (Addr: tcp/127.0.0.1:30377) (DC: dc1)" +31:41.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30372 network=udp +31:41.144 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4.dc1 area=wan +31:41.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30372 network=tcp +31:41.145 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30373 network=tcp +31:41.145 [INFO] TestAgent: started state syncer +31:41.145 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30378 network=tcp +31:41.198 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.198 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30377 [Candidate]" term=2 +31:41.216 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.216 [DEBUG] TestAgent.server.raft: vote granted: from=eb4b0b24-8e78-c0d1-05a2-494594ab07f4 term=2 tally=1 +31:41.216 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.216 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30377 [Leader]" +31:41.216 [INFO] TestAgent.server: cluster leadership acquired +31:41.216 [INFO] TestAgent.server: New leader elected: payload=Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4 +31:41.245 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30377 +31:41.265 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.344 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.344 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.344 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.344 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.344 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.344 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4 +31:41.344 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eb4b0b24-8e78-c0d1-05a2-494594ab07f4 +31:41.376 [INFO] TestAgent.server: federation state anti-entropy synced +31:41.430 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:41.437 [INFO] TestAgent: Synced node info +31:41.496 [DEBUG] TestAgent: removed check: check=mem +31:41.496 [DEBUG] TestAgent: removed check: check=mem +31:41.496 [INFO] TestAgent: Requesting shutdown +31:41.496 [INFO] TestAgent.server: shutting down server +31:41.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:41.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:41.496 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:41.496 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:41.496 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:41.496 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:41.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:41.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:41.496 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:41.496 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:41.508 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:41.510Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:41.510Z [INFO] agent.router.manager: shutting down +31:41.510 [INFO] TestAgent: consul server down +31:41.510 [INFO] TestAgent: shutdown complete +31:41.510 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30372 network=tcp +31:41.510 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30372 network=udp +31:41.510 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30373 network=tcp +31:42.010 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.010 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RemoveCheck (1.00s) +=== CONT TestAgent_AddCheck_RestoreState +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:41.069Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.116Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.216 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d10021f9-b4ed-5523-1234-8b9d753a97a4 Address:127.0.0.1:30384}]" +31:41.217 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30384 [Follower]" leader= +31:41.217 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d10021f9-b4ed-5523-1234-8b9d753a97a4.dc1 127.0.0.1 +31:41.218 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d10021f9-b4ed-5523-1234-8b9d753a97a4 127.0.0.1 +2021-10-05T16:31:41.218Z [INFO] agent.router: Initializing LAN area manager +31:41.218 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30379 network=udp +31:41.218 [INFO] TestAgent.server: Adding LAN server: server="Node-d10021f9-b4ed-5523-1234-8b9d753a97a4 (Addr: tcp/127.0.0.1:30384) (DC: dc1)" +31:41.218 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d10021f9-b4ed-5523-1234-8b9d753a97a4.dc1 area=wan +31:41.218 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30379 network=tcp +31:41.219 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30380 network=tcp +31:41.219 [INFO] TestAgent: started state syncer +31:41.219 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30399 network=tcp +31:41.277 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.277 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30384 [Candidate]" term=2 +31:41.295 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.295 [DEBUG] TestAgent.server.raft: vote granted: from=d10021f9-b4ed-5523-1234-8b9d753a97a4 term=2 tally=1 +31:41.295 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30384 [Leader]" +31:41.295 [INFO] TestAgent.server: cluster leadership acquired +31:41.295 [INFO] TestAgent.server: New leader elected: payload=Node-d10021f9-b4ed-5523-1234-8b9d753a97a4 +31:41.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30384 +31:41.343 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.404 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.404 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.404 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.404 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.404 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.404 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d10021f9-b4ed-5523-1234-8b9d753a97a4 +31:41.404 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d10021f9-b4ed-5523-1234-8b9d753a97a4 +31:41.420 [INFO] TestAgent.server: federation state anti-entropy synced +31:41.532 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:41.544 [INFO] TestAgent: Synced node info +31:41.621 [INFO] TestAgent: Requesting shutdown +31:41.621 [INFO] TestAgent.server: shutting down server +31:41.621 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:41.621 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:41.621 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:41.621 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:41.621 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:41.621 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:41.621 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:41.621 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:41.621 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:41.621 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:41.636 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:41.637Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:41.637Z [INFO] agent.router.manager: shutting down +31:41.637 [INFO] TestAgent: consul server down +31:41.637 [INFO] TestAgent: shutdown complete +31:41.637 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30379 network=tcp +31:41.637 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30379 network=udp +31:41.637 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30380 network=tcp +31:42.138 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.138 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_Alias_userAndSetToken (1.10s) +=== CONT TestAgent_AddCheck_MissingService +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:41.663Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.695Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.775 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3752e28a-8717-c2ad-51f0-35e7a174ce99 Address:127.0.0.1:30419}]" +31:41.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30419 [Follower]" leader= +31:41.777 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3752e28a-8717-c2ad-51f0-35e7a174ce99.dc1 127.0.0.1 +31:41.777 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3752e28a-8717-c2ad-51f0-35e7a174ce99 127.0.0.1 +2021-10-05T16:31:41.777Z [INFO] agent.router: Initializing LAN area manager +31:41.777 [INFO] TestAgent.server: Adding LAN server: server="Node-3752e28a-8717-c2ad-51f0-35e7a174ce99 (Addr: tcp/127.0.0.1:30419) (DC: dc1)" +31:41.778 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3752e28a-8717-c2ad-51f0-35e7a174ce99.dc1 area=wan +31:41.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30414 network=udp +31:41.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30414 network=tcp +31:41.778 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30415 network=tcp +31:41.778 [INFO] TestAgent: started state syncer +31:41.778 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29477 network=tcp +31:41.837 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.837 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30419 [Candidate]" term=2 +31:41.899 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.899 [DEBUG] TestAgent.server.raft: vote granted: from=3752e28a-8717-c2ad-51f0-35e7a174ce99 term=2 tally=1 +31:41.899 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.899 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30419 [Leader]" +31:41.899 [INFO] TestAgent.server: cluster leadership acquired +31:41.899 [INFO] TestAgent.server: New leader elected: payload=Node-3752e28a-8717-c2ad-51f0-35e7a174ce99 +31:41.909 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30419 +31:41.927 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.988 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.988 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.988 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.988 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.988 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.988 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3752e28a-8717-c2ad-51f0-35e7a174ce99 +31:41.988 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3752e28a-8717-c2ad-51f0-35e7a174ce99 +31:42.014 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.020 [INFO] TestAgent: Requesting shutdown +31:42.020 [INFO] TestAgent.server: shutting down server +31:42.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.020 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.020 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.020 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:42.020 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.020 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.032 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.036Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.036Z [INFO] agent.router.manager: shutting down +31:42.036 [INFO] TestAgent: consul server down +31:42.036 [INFO] TestAgent: shutdown complete +31:42.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30414 network=tcp +31:42.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30414 network=udp +31:42.036 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30415 network=tcp +31:42.536 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.536 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_ExecRemoteDisable (0.89s) +=== CONT TestAgent_AddCheck_MinInterval +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:41.530Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.585Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.673 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38346855-12b6-bf8b-18ca-db4dffce4488 Address:127.0.0.1:30412}]" +31:41.674 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-38346855-12b6-bf8b-18ca-db4dffce4488.dc1 127.0.0.1 +31:41.674 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-38346855-12b6-bf8b-18ca-db4dffce4488 127.0.0.1 +2021-10-05T16:31:41.674Z [INFO] agent.router: Initializing LAN area manager +31:41.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30407 network=udp +31:41.675 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30412 [Follower]" leader= +31:41.675 [INFO] TestAgent.server: Adding LAN server: server="Node-38346855-12b6-bf8b-18ca-db4dffce4488 (Addr: tcp/127.0.0.1:30412) (DC: dc1)" +31:41.675 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-38346855-12b6-bf8b-18ca-db4dffce4488.dc1 area=wan +31:41.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30407 network=tcp +31:41.680 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30408 network=tcp +31:41.680 [INFO] TestAgent: started state syncer +31:41.680 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29400 network=tcp +31:41.716 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.716 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30412 [Candidate]" term=2 +31:41.729 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.729 [DEBUG] TestAgent.server.raft: vote granted: from=38346855-12b6-bf8b-18ca-db4dffce4488 term=2 tally=1 +31:41.729 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.729 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30412 [Leader]" +31:41.730 [INFO] TestAgent.server: cluster leadership acquired +31:41.730 [INFO] TestAgent.server: New leader elected: payload=Node-38346855-12b6-bf8b-18ca-db4dffce4488 +31:41.748 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30412 +31:41.784 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.837 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.837 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.837 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.837 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.837 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.837 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38346855-12b6-bf8b-18ca-db4dffce4488 +31:41.837 [INFO] TestAgent.server: member joined, marking health alive: member=Node-38346855-12b6-bf8b-18ca-db4dffce4488 +31:41.896 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.076 [INFO] TestAgent: Requesting shutdown +31:42.076 [INFO] TestAgent.server: shutting down server +31:42.076 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.076 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.076 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.076 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.076 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.076 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.076 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.076 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:42.076 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.076 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.076 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.088 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.089Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.089Z [INFO] agent.router.manager: shutting down +31:42.089 [INFO] TestAgent: consul server down +31:42.089 [INFO] TestAgent: shutdown complete +31:42.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30407 network=tcp +31:42.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30407 network=udp +31:42.089 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30408 network=tcp +31:42.596 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.596 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_Alias (1.08s) +=== CONT TestAgent_AddCheck_StartPassing +2021-10-05T16:31:41.642Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.674Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.723 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:35ef45cd-b172-17cc-740c-8d1b38b7b1b9 Address:127.0.0.1:29406}]" +31:41.723 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29406 [Follower]" leader= +31:41.724 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9.dc1 127.0.0.1 +31:41.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9 127.0.0.1 +2021-10-05T16:31:41.725Z [INFO] agent.router: Initializing LAN area manager +31:41.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29401 network=udp +31:41.726 [INFO] TestAgent.server: Adding LAN server: server="Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9 (Addr: tcp/127.0.0.1:29406) (DC: dc1)" +31:41.726 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9.dc1 area=wan +31:41.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29401 network=tcp +31:41.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29402 network=tcp +31:41.727 [INFO] TestAgent: started state syncer +31:41.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30413 network=tcp +31:41.762 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.762 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29406 [Candidate]" term=2 +31:41.792 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.792 [DEBUG] TestAgent.server.raft: vote granted: from=35ef45cd-b172-17cc-740c-8d1b38b7b1b9 term=2 tally=1 +31:41.792 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.792 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29406 [Leader]" +31:41.794 [INFO] TestAgent.server: cluster leadership acquired +31:41.794 [INFO] TestAgent.server: New leader elected: payload=Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9 +31:41.801 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29406 +31:41.866 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.904 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.904 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.904 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.904 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.904 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.904 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9 +31:41.904 [INFO] TestAgent.server: member joined, marking health alive: member=Node-35ef45cd-b172-17cc-740c-8d1b38b7b1b9 +31:41.922 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.090 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:42.096 [INFO] TestAgent: Synced node info +31:42.096 [DEBUG] TestAgent: Node info in sync +31:42.099 [INFO] TestAgent: Requesting shutdown +31:42.099 [INFO] TestAgent.server: shutting down server +31:42.099 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.099 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.099 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.099 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.099 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.099 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.099 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.099 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.099 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.099 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.104 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.107Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.107Z [INFO] agent.router.manager: shutting down +31:42.107 [INFO] TestAgent: consul server down +31:42.107 [INFO] TestAgent: shutdown complete +31:42.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29401 network=tcp +31:42.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29401 network=udp +31:42.107 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29402 network=tcp +31:42.612 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.612 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_GRPC (0.99s) +=== CONT TestAgent_AddCheck +2021-10-05T16:31:41.536Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:41.593Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.692 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46a90208-f038-86e5-e768-5bbd9a8f8093 Address:127.0.0.1:30398}]" +31:41.693 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46a90208-f038-86e5-e768-5bbd9a8f8093.dc1 127.0.0.1 +31:41.693 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46a90208-f038-86e5-e768-5bbd9a8f8093 127.0.0.1 +2021-10-05T16:31:41.693Z [INFO] agent.router: Initializing LAN area manager +31:41.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30393 network=udp +31:41.694 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30398 [Follower]" leader= +31:41.694 [INFO] TestAgent.server: Adding LAN server: server="Node-46a90208-f038-86e5-e768-5bbd9a8f8093 (Addr: tcp/127.0.0.1:30398) (DC: dc1)" +31:41.694 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46a90208-f038-86e5-e768-5bbd9a8f8093.dc1 area=wan +31:41.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30393 network=tcp +31:41.695 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30394 network=tcp +31:41.695 [INFO] TestAgent: started state syncer +31:41.695 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30406 network=tcp +31:41.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30398 [Candidate]" term=2 +31:41.761 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.762 [DEBUG] TestAgent.server.raft: vote granted: from=46a90208-f038-86e5-e768-5bbd9a8f8093 term=2 tally=1 +31:41.762 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.762 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30398 [Leader]" +31:41.762 [INFO] TestAgent.server: cluster leadership acquired +31:41.762 [INFO] TestAgent.server: New leader elected: payload=Node-46a90208-f038-86e5-e768-5bbd9a8f8093 +31:41.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30398 +31:41.797 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.842 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.842 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.842 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.842 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.842 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.842 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46a90208-f038-86e5-e768-5bbd9a8f8093 +31:41.842 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46a90208-f038-86e5-e768-5bbd9a8f8093 +31:41.873 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.032 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:42.048 [INFO] TestAgent: Synced node info +31:42.116 [INFO] TestAgent: Requesting shutdown +31:42.116 [INFO] TestAgent.server: shutting down server +31:42.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.116 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.116 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.116 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.116 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.116 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.116 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.116 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.116 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.124 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.128Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.128Z [INFO] agent.router.manager: shutting down +31:42.128 [INFO] TestAgent: consul server down +31:42.128 [INFO] TestAgent: shutdown complete +31:42.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30393 network=tcp +31:42.128 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30393 network=udp +31:42.128 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30394 network=tcp +31:42.632 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.632 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_Alias_setToken (1.12s) +=== CONT TestAgent_IndexChurn +=== RUN TestAgent_IndexChurn/no_tags +2021-10-05T16:31:41.996Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.010Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:42.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3a71674-1519-3158-5206-b4867a45f6ab Address:127.0.0.1:29448}]" +31:42.080 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3a71674-1519-3158-5206-b4867a45f6ab.dc1 127.0.0.1 +31:42.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3a71674-1519-3158-5206-b4867a45f6ab 127.0.0.1 +2021-10-05T16:31:42.081Z [INFO] agent.router: Initializing LAN area manager +31:42.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29478 network=udp +31:42.081 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29448 [Follower]" leader= +31:42.084 [INFO] TestAgent.server: Adding LAN server: server="Node-f3a71674-1519-3158-5206-b4867a45f6ab (Addr: tcp/127.0.0.1:29448) (DC: dc1)" +31:42.084 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3a71674-1519-3158-5206-b4867a45f6ab.dc1 area=wan +31:42.084 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29478 network=tcp +31:42.085 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29444 network=tcp +31:42.085 [INFO] TestAgent: started state syncer +31:42.085 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29407 network=tcp +31:42.131 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.131 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29448 [Candidate]" term=2 +31:42.161 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.161 [DEBUG] TestAgent.server.raft: vote granted: from=f3a71674-1519-3158-5206-b4867a45f6ab term=2 tally=1 +31:42.161 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.161 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29448 [Leader]" +31:42.164 [INFO] TestAgent.server: cluster leadership acquired +31:42.164 [INFO] TestAgent.server: New leader elected: payload=Node-f3a71674-1519-3158-5206-b4867a45f6ab +31:42.178 [INFO] TestAgent: Requesting shutdown +31:42.178 [INFO] TestAgent.server: shutting down server +31:42.178 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.178 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:42.179 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29448 +31:42.188 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.197Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.197Z [INFO] agent.router.manager: shutting down +31:42.200 [INFO] TestAgent: consul server down +31:42.200 [INFO] TestAgent: shutdown complete +31:42.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29478 network=tcp +31:42.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29478 network=udp +31:42.200 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29444 network=tcp +31:42.200 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:42.200 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:42.200 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:42.200 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:42.200 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:42.700 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.700 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_ExecDisable (0.73s) +=== CONT TestAgent_ReconnectConfigWanDisabled +2021-10-05T16:31:40.890Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:40.920Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:41.040 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c35ae4e3-2e5d-5348-d12c-373ef69a5d80 Address:127.0.0.1:30356}]" +31:41.041 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80.dc1 127.0.0.1 +31:41.041 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80 127.0.0.1 +2021-10-05T16:31:41.041Z [INFO] agent.router: Initializing LAN area manager +31:41.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30351 network=udp +31:41.042 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30356 [Follower]" leader= +31:41.042 [INFO] TestAgent.server: Adding LAN server: server="Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80 (Addr: tcp/127.0.0.1:30356) (DC: dc1)" +31:41.042 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80.dc1 area=wan +31:41.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30351 network=tcp +31:41.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30352 network=tcp +31:41.043 [INFO] TestAgent: started state syncer +31:41.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30357 network=tcp +31:41.093 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:41.093 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30356 [Candidate]" term=2 +31:41.109 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:41.109 [DEBUG] TestAgent.server.raft: vote granted: from=c35ae4e3-2e5d-5348-d12c-373ef69a5d80 term=2 tally=1 +31:41.109 [INFO] TestAgent.server.raft: election won: tally=1 +31:41.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30356 [Leader]" +31:41.109 [INFO] TestAgent.server: cluster leadership acquired +31:41.110 [INFO] TestAgent.server: New leader elected: payload=Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80 +31:41.118 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30356 +31:41.147 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:41.183 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:41.183 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:41.183 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:41.183 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:41.183 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:41.183 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80 +31:41.183 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c35ae4e3-2e5d-5348-d12c-373ef69a5d80 +31:41.208 [INFO] TestAgent: Synced node info +31:41.209 [DEBUG] TestAgent: Node info in sync +31:41.216 [INFO] TestAgent.server: federation state anti-entropy synced +31:41.458 [WARN] TestAgent: check has interval below minimum: check=tls minimum_interval=1s +31:42.220 [DEBUG] TestAgent: Check status updated: check=tls status=passing +31:42.229 [INFO] TestAgent: Requesting shutdown +31:42.229 [INFO] TestAgent.server: shutting down server +31:42.229 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.229 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.229 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.229 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.229 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.229 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.229 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.229 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.238 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.242Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.242Z [INFO] agent.router.manager: shutting down +31:42.242 [INFO] TestAgent: consul server down +31:42.242 [INFO] TestAgent: shutdown complete +31:42.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30351 network=tcp +31:42.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30351 network=udp +31:42.242 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30352 network=tcp +31:42.743 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.743 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_HTTPCheck_TLSSkipVerify (1.88s) +=== CONT TestAgent_ReconnectConfigSettings +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:42.038Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.054Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:42.087 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 Address:127.0.0.1:29413}]" +31:42.088 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01.dc1 127.0.0.1 +31:42.088 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29413 [Follower]" leader= +31:42.088 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 127.0.0.1 +2021-10-05T16:31:42.088Z [INFO] agent.router: Initializing LAN area manager +31:42.088 [INFO] TestAgent.server: Adding LAN server: server="Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 (Addr: tcp/127.0.0.1:29413) (DC: dc1)" +31:42.088 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01.dc1 area=wan +31:42.088 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29408 network=udp +31:42.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29408 network=tcp +31:42.089 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29409 network=tcp +31:42.089 [INFO] TestAgent: started state syncer +31:42.089 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29414 network=tcp +31:42.127 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.127 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29413 [Candidate]" term=2 +31:42.154 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.154 [DEBUG] TestAgent.server.raft: vote granted: from=a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 term=2 tally=1 +31:42.154 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.154 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29413 [Leader]" +31:42.154 [INFO] TestAgent.server: cluster leadership acquired +31:42.154 [INFO] TestAgent.server: New leader elected: payload=Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 +31:42.167 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29413 +31:42.192 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:42.230 [INFO] TestAgent: Synced node info +31:42.243 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:42.243 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:42.243 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:42.243 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:42.243 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:42.243 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 +31:42.243 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a19e5f7b-47a7-a7f6-6d45-1d17666f8b01 +31:42.258 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.456 [INFO] TestAgent: Requesting shutdown +31:42.456 [INFO] TestAgent.server: shutting down server +31:42.457 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.457 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.457 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.457 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.457 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.457 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.457 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.457 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.457 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.457 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.484 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.488Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.488Z [INFO] agent.router.manager: shutting down +31:42.488 [INFO] TestAgent: consul server down +31:42.488 [INFO] TestAgent: shutdown complete +31:42.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29408 network=tcp +31:42.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29408 network=udp +31:42.488 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29409 network=tcp +31:42.990 [INFO] TestAgent: Waiting for endpoints to shut down +31:42.990 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_RestoreState (0.98s) +=== CONT TestAgent_TokenStore +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:42.153Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.153Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:42.166Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.166Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:42.243 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d60020e9-7ff5-f2d7-078b-ad428f8771a6 Address:127.0.0.1:29490}]" +31:42.244 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29490 [Follower]" leader= +31:42.244 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6.dc1 127.0.0.1 +31:42.244 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6 127.0.0.1 +2021-10-05T16:31:42.244Z [INFO] agent.router: Initializing LAN area manager +31:42.245 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29415 network=udp +31:42.245 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6.dc1 area=wan +31:42.245 [INFO] TestAgent.server: Adding LAN server: server="Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6 (Addr: tcp/127.0.0.1:29490) (DC: dc1)" +31:42.245 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29415 network=tcp +31:42.245 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29486 network=tcp +31:42.245 [INFO] TestAgent: started state syncer +31:42.245 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29435 network=tcp +31:42.298 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.298 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29490 [Candidate]" term=2 +31:42.313 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.313 [DEBUG] TestAgent.server.raft: vote granted: from=d60020e9-7ff5-f2d7-078b-ad428f8771a6 term=2 tally=1 +31:42.313 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.313 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29490 [Leader]" +31:42.314 [INFO] TestAgent.server: cluster leadership acquired +31:42.314 [INFO] TestAgent.server: New leader elected: payload=Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6 +31:42.319 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29490 +31:42.360 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:42.464 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:42.464 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:42.464 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:42.464 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:42.464 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:42.464 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6 +31:42.464 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d60020e9-7ff5-f2d7-078b-ad428f8771a6 +31:42.560 [INFO] TestAgent.server: federation state anti-entropy synced +31:42.580 [INFO] TestAgent: Requesting shutdown +31:42.580 [INFO] TestAgent.server: shutting down server +31:42.580 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:42.580 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:42.580 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:42.580 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:42.580 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.580 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:42.580 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:42.580 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:42.580 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:42.580 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:42.580 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:42.600 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.604Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.604Z [INFO] agent.router.manager: shutting down +31:42.604 [INFO] TestAgent: consul server down +31:42.604 [INFO] TestAgent: shutdown complete +31:42.604 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29415 network=tcp +31:42.604 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29415 network=udp +31:42.604 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29486 network=tcp +31:43.106 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.106 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_MissingService (0.97s) +=== CONT TestAgent_RPCPing +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:42.670Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.670Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:42.702Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.702Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:42.768 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:49f0226b-eda9-fb08-7734-df6bda4f0ad5 Address:127.0.0.1:29427}]" +31:42.768 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29427 [Follower]" leader= +31:42.768 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49f0226b-eda9-fb08-7734-df6bda4f0ad5.dc1 127.0.0.1 +31:42.770 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-49f0226b-eda9-fb08-7734-df6bda4f0ad5 127.0.0.1 +2021-10-05T16:31:42.770Z [INFO] agent.router: Initializing LAN area manager +31:42.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29471 network=udp +31:42.771 [INFO] TestAgent.server: Adding LAN server: server="Node-49f0226b-eda9-fb08-7734-df6bda4f0ad5 (Addr: tcp/127.0.0.1:29427) (DC: dc1)" +31:42.771 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49f0226b-eda9-fb08-7734-df6bda4f0ad5.dc1 area=wan +31:42.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29471 network=tcp +31:42.772 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29423 network=tcp +31:42.772 [INFO] TestAgent: started state syncer +31:42.772 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29484 network=tcp +31:42.833 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.833 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29427 [Candidate]" term=2 +31:42.853 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.853 [DEBUG] TestAgent.server.raft: vote granted: from=49f0226b-eda9-fb08-7734-df6bda4f0ad5 term=2 tally=1 +31:42.853 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.853 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29427 [Leader]" +31:42.854 [INFO] TestAgent.server: cluster leadership acquired +31:42.854 [INFO] TestAgent.server: New leader elected: payload=Node-49f0226b-eda9-fb08-7734-df6bda4f0ad5 +31:42.866 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29427 +31:42.898 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:42.913 [INFO] TestAgent: Requesting shutdown +31:42.914 [INFO] TestAgent.server: shutting down server +31:42.914 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:42.914 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:42.925 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:42.928Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:42.928Z [INFO] agent.router.manager: shutting down +31:42.932 [INFO] TestAgent: consul server down +31:42.932 [INFO] TestAgent: shutdown complete +31:42.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29471 network=tcp +31:42.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29471 network=udp +31:42.932 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29423 network=tcp +31:42.932 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:42.933 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:43.433 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.433 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_StartPassing (0.84s) +=== CONT TestAgent_StartStop +2021-10-05T16:31:42.559Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.559Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:42.629Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.629Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:42.715 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:87dc4074-109d-905d-b873-4ecafcb42356 Address:127.0.0.1:29455}]" +31:42.715 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29455 [Follower]" leader= +31:42.716 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-87dc4074-109d-905d-b873-4ecafcb42356.dc1 127.0.0.1 +31:42.726 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-87dc4074-109d-905d-b873-4ecafcb42356 127.0.0.1 +2021-10-05T16:31:42.726Z [INFO] agent.router: Initializing LAN area manager +31:42.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29436 network=udp +31:42.727 [INFO] TestAgent.server: Adding LAN server: server="Node-87dc4074-109d-905d-b873-4ecafcb42356 (Addr: tcp/127.0.0.1:29455) (DC: dc1)" +31:42.727 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-87dc4074-109d-905d-b873-4ecafcb42356.dc1 area=wan +31:42.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29436 network=tcp +31:42.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29451 network=tcp +31:42.727 [INFO] TestAgent: started state syncer +31:42.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29470 network=tcp +31:42.767 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.767 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29455 [Candidate]" term=2 +31:42.781 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.781 [DEBUG] TestAgent.server.raft: vote granted: from=87dc4074-109d-905d-b873-4ecafcb42356 term=2 tally=1 +31:42.781 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.781 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29455 [Leader]" +31:42.782 [INFO] TestAgent.server: cluster leadership acquired +31:42.782 [INFO] TestAgent.server: New leader elected: payload=Node-87dc4074-109d-905d-b873-4ecafcb42356 +31:42.790 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29455 +31:42.820 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:42.885 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:42.885 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:42.885 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:42.885 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:42.885 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:42.885 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-87dc4074-109d-905d-b873-4ecafcb42356 +31:42.885 [INFO] TestAgent.server: member joined, marking health alive: member=Node-87dc4074-109d-905d-b873-4ecafcb42356 +31:42.931 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.034 [WARN] TestAgent: check has interval below minimum: check=mem minimum_interval=1s +31:43.034 [INFO] TestAgent: Requesting shutdown +31:43.035 [INFO] TestAgent.server: shutting down server +31:43.035 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.035 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.035 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.035 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.035 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.035 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:43.035 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.035 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:43.035 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.035 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.035 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.038 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.042Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.042Z [INFO] agent.router.manager: shutting down +31:43.042 [INFO] TestAgent: consul server down +31:43.042 [INFO] TestAgent: shutdown complete +31:43.042 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29436 network=tcp +31:43.042 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29436 network=udp +31:43.042 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29451 network=tcp +31:43.544 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.544 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_AddCheck_MinInterval (1.01s) +=== CONT TestAgent_Services_ExposeConfig +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:42.724Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.740Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:42.826 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08fb5da1-9a8b-c3dc-c341-1a438807e80a Address:127.0.0.1:29476}]" +31:42.827 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08fb5da1-9a8b-c3dc-c341-1a438807e80a 127.0.0.1 +2021-10-05T16:31:42.827Z [INFO] agent.router: Initializing LAN area manager +31:42.827 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29450 network=udp +31:42.827 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29476 [Follower]" leader= +31:42.828 [INFO] TestAgent.server: Adding LAN server: server="Node-08fb5da1-9a8b-c3dc-c341-1a438807e80a (Addr: tcp/127.0.0.1:29476) (DC: dc1)" +31:42.828 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29450 network=tcp +31:42.828 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29472 network=tcp +31:42.828 [INFO] TestAgent: started state syncer +31:42.829 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30385 network=tcp +31:42.888 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29476 [Candidate]" term=2 +31:42.925 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.925 [DEBUG] TestAgent.server.raft: vote granted: from=08fb5da1-9a8b-c3dc-c341-1a438807e80a term=2 tally=1 +31:42.925 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.925 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29476 [Leader]" +31:42.925 [INFO] TestAgent.server: cluster leadership acquired +31:42.925 [INFO] TestAgent.server: New leader elected: payload=Node-08fb5da1-9a8b-c3dc-c341-1a438807e80a +31:42.952 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29476 +31:42.990 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.052 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.052 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.052 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.052 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.052 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.052 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08fb5da1-9a8b-c3dc-c341-1a438807e80a +31:43.052 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08fb5da1-9a8b-c3dc-c341-1a438807e80a +31:43.100 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.176 [INFO] TestAgent: Requesting shutdown +31:43.176 [INFO] TestAgent.server: shutting down server +31:43.176 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.176 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.176 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.176 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.176 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.176 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:43.176 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.176 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:43.176 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.176 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.176 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +2021-10-05T16:31:43.188Z [INFO] agent.router.manager: shutting down +31:43.188 [INFO] TestAgent: consul server down +31:43.188 [INFO] TestAgent: shutdown complete +31:43.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29450 network=tcp +31:43.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29450 network=udp +31:43.188 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29472 network=tcp +31:43.692 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.692 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:42.647Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.647Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:42.689Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.689Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:42.755 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76e452d7-51c0-528e-b122-274aa452d7d9 Address:127.0.0.1:29469}]" +31:42.755 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29469 [Follower]" leader= +31:42.757 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76e452d7-51c0-528e-b122-274aa452d7d9.dc1 127.0.0.1 +31:42.757 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76e452d7-51c0-528e-b122-274aa452d7d9 127.0.0.1 +2021-10-05T16:31:42.757Z [INFO] agent.router: Initializing LAN area manager +31:42.758 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29485 network=udp +31:42.758 [INFO] TestAgent.server: Adding LAN server: server="Node-76e452d7-51c0-528e-b122-274aa452d7d9 (Addr: tcp/127.0.0.1:29469) (DC: dc1)" +31:42.758 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76e452d7-51c0-528e-b122-274aa452d7d9.dc1 area=wan +31:42.758 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29485 network=tcp +31:42.759 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29465 network=tcp +31:42.759 [INFO] TestAgent: started state syncer +31:42.759 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29463 network=tcp +31:42.799 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.799 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29469 [Candidate]" term=2 +31:42.826 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.826 [DEBUG] TestAgent.server.raft: vote granted: from=76e452d7-51c0-528e-b122-274aa452d7d9 term=2 tally=1 +31:42.826 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.826 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29469 [Leader]" +31:42.828 [INFO] TestAgent.server: cluster leadership acquired +31:42.828 [INFO] TestAgent.server: New leader elected: payload=Node-76e452d7-51c0-528e-b122-274aa452d7d9 +31:42.843 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29469 +31:42.864 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:42.928 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:42.928 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:42.928 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:42.928 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:42.928 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:42.928 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76e452d7-51c0-528e-b122-274aa452d7d9 +31:42.928 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76e452d7-51c0-528e-b122-274aa452d7d9 +31:42.960 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.037 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.053 [INFO] TestAgent: Synced node info +31:43.183 [INFO] TestAgent: Requesting shutdown +31:43.183 [INFO] TestAgent.server: shutting down server +31:43.183 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.183 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.183 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.183 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.183 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.183 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:43.183 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.183 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.183 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.184 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.191 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.192Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.192Z [INFO] agent.router.manager: shutting down +31:43.192 [INFO] TestAgent: consul server down +31:43.192 [INFO] TestAgent: shutdown complete +31:43.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29485 network=tcp +31:43.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29485 network=udp +31:43.192 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29465 network=tcp +31:43.692 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.692 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReconnectConfigWanDisabled (0.99s) +=== CONT TestAgent_HostBadACL +--- PASS: TestAgent_AddCheck (1.08s) +=== CONT TestAgent_Host +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:43.132Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.146Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.186 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07a53f81-95d3-3371-372a-dc30dde7ad5e Address:127.0.0.1:29441}]" +31:43.186 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07a53f81-95d3-3371-372a-dc30dde7ad5e.dc1 127.0.0.1 +31:43.187 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29441 [Follower]" leader= +31:43.187 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07a53f81-95d3-3371-372a-dc30dde7ad5e 127.0.0.1 +2021-10-05T16:31:43.187Z [INFO] agent.router: Initializing LAN area manager +31:43.187 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29457 network=udp +31:43.188 [INFO] TestAgent.server: Adding LAN server: server="Node-07a53f81-95d3-3371-372a-dc30dde7ad5e (Addr: tcp/127.0.0.1:29441) (DC: dc1)" +31:43.188 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07a53f81-95d3-3371-372a-dc30dde7ad5e.dc1 area=wan +31:43.188 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29457 network=tcp +31:43.188 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29437 network=tcp +31:43.188 [INFO] TestAgent: started state syncer +31:43.189 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29442 network=tcp +31:43.233 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.233 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29441 [Candidate]" term=2 +31:43.256 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.256 [DEBUG] TestAgent.server.raft: vote granted: from=07a53f81-95d3-3371-372a-dc30dde7ad5e term=2 tally=1 +31:43.256 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.256 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29441 [Leader]" +31:43.257 [INFO] TestAgent.server: cluster leadership acquired +31:43.257 [INFO] TestAgent.server: New leader elected: payload=Node-07a53f81-95d3-3371-372a-dc30dde7ad5e +31:43.289 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29441 +31:43.308 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.336 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.336 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.336 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.336 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.336 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.336 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07a53f81-95d3-3371-372a-dc30dde7ad5e +31:43.336 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07a53f81-95d3-3371-372a-dc30dde7ad5e +31:43.362 [INFO] TestAgent: Requesting shutdown +31:43.362 [INFO] TestAgent.server: shutting down server +31:43.362 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.362 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.362 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.362 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.362 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.362 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:43.362 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.362 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.362 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.363 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.368 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.376Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.376Z [INFO] agent.router.manager: shutting down +31:43.376 [INFO] TestAgent: consul server down +31:43.376 [INFO] TestAgent: shutdown complete +31:43.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29457 network=tcp +31:43.377 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29457 network=udp +31:43.377 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29437 network=tcp +31:43.882 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.882 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RPCPing (0.78s) +=== CONT TestAgentConnectAuthorize_defaultAllow +2021-10-05T16:31:43.002Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.016Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.082 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:afa34dfe-ca21-fc5e-9780-ce347e129d38 Address:127.0.0.1:30335}]" +31:43.083 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-afa34dfe-ca21-fc5e-9780-ce347e129d38.dc1 127.0.0.1 +31:43.083 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-afa34dfe-ca21-fc5e-9780-ce347e129d38 127.0.0.1 +2021-10-05T16:31:43.083Z [INFO] agent.router: Initializing LAN area manager +31:43.084 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30330 network=udp +31:43.084 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30335 [Follower]" leader= +31:43.084 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-afa34dfe-ca21-fc5e-9780-ce347e129d38.dc1 area=wan +31:43.084 [INFO] TestAgent.server: Adding LAN server: server="Node-afa34dfe-ca21-fc5e-9780-ce347e129d38 (Addr: tcp/127.0.0.1:30335) (DC: dc1)" +31:43.084 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30330 network=tcp +31:43.085 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30331 network=tcp +31:43.085 [INFO] TestAgent: started state syncer +31:43.085 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29456 network=tcp +31:43.125 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.125 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30335 [Candidate]" term=2 +31:43.138 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.138 [DEBUG] TestAgent.server.raft: vote granted: from=afa34dfe-ca21-fc5e-9780-ce347e129d38 term=2 tally=1 +31:43.138 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30335 [Leader]" +31:43.139 [INFO] TestAgent.server: cluster leadership acquired +31:43.139 [INFO] TestAgent.server: New leader elected: payload=Node-afa34dfe-ca21-fc5e-9780-ce347e129d38 +31:43.148 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30335 +31:43.172 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.211 [INFO] TestAgent: Synced node info +31:43.226 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.226 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.226 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.226 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.226 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.226 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-afa34dfe-ca21-fc5e-9780-ce347e129d38 +31:43.226 [INFO] TestAgent.server: member joined, marking health alive: member=Node-afa34dfe-ca21-fc5e-9780-ce347e129d38 +31:43.240 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.464 [INFO] TestAgent: Requesting shutdown +31:43.464 [INFO] TestAgent.server: shutting down server +31:43.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.464 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.464 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.464 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.464 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:43.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.464 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.464 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.471 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.476Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.476Z [INFO] agent.router.manager: shutting down +31:43.476 [INFO] TestAgent: consul server down +31:43.476 [INFO] TestAgent: shutdown complete +31:43.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30330 network=tcp +31:43.476 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30330 network=udp +31:43.476 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30331 network=tcp +31:43.977 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.977 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_TokenStore (0.99s) +=== CONT TestAgentConnectAuthorize_defaultDeny +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:42.698Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.731Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:42.844 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b881c3eb-a650-e8ca-411d-5edd3ca0de41 Address:127.0.0.1:29518}]" +31:42.844 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29518 [Follower]" leader= +31:42.845 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41.dc1 127.0.0.1 +31:42.845 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41 127.0.0.1 +2021-10-05T16:31:42.845Z [INFO] agent.router: Initializing LAN area manager +31:42.845 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29464 network=udp +31:42.846 [INFO] TestAgent.server: Adding LAN server: server="Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41 (Addr: tcp/127.0.0.1:29518) (DC: dc1)" +31:42.846 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41.dc1 area=wan +31:42.846 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29464 network=tcp +31:42.846 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29514 network=tcp +31:42.846 [INFO] TestAgent: started state syncer +31:42.847 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29449 network=tcp +31:42.894 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.894 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29518 [Candidate]" term=2 +31:42.926 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.926 [DEBUG] TestAgent.server.raft: vote granted: from=b881c3eb-a650-e8ca-411d-5edd3ca0de41 term=2 tally=1 +31:42.926 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.926 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29518 [Leader]" +31:42.926 [INFO] TestAgent.server: cluster leadership acquired +31:42.926 [INFO] TestAgent.server: New leader elected: payload=Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41 +31:42.940 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29518 +31:42.980 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.041 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.041 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.041 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.041 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.041 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.041 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41 +31:43.041 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b881c3eb-a650-e8ca-411d-5edd3ca0de41 +31:43.069 [INFO] TestAgent.server: federation state anti-entropy synced +31:43.108 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.124 [INFO] TestAgent: Synced node info +31:43.136 [INFO] TestAgent: Synced service: service=redis +31:43.136 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.141 [INFO] TestAgent: Synced check: check=node-check +31:43.216 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.216 [DEBUG] TestAgent: Node info in sync +31:43.216 [DEBUG] TestAgent: Service in sync: service=redis +31:43.216 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.216 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.643 [DEBUG] TestAgent: Registered node: node="Node-level check" +31:43.643 [DEBUG] TestAgent: Registered node: node="Serf Health Status" +31:43.643 [DEBUG] TestAgent: Registered node: node="Service-level check" +31:43.643 [INFO] TestAgent: Sync in progress: iteration=1 +31:43.643 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.643 [DEBUG] TestAgent: Node info in sync +31:43.643 [DEBUG] TestAgent: Service in sync: service=redis +31:43.643 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.643 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.643 [INFO] TestAgent: Sync in progress: iteration=2 +31:43.643 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.643 [DEBUG] TestAgent: Node info in sync +31:43.643 [DEBUG] TestAgent: Service in sync: service=redis +31:43.643 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.643 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.643 [INFO] TestAgent: Sync in progress: iteration=3 +31:43.643 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.643 [DEBUG] TestAgent: Node info in sync +31:43.643 [DEBUG] TestAgent: Service in sync: service=redis +31:43.643 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.643 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.643 [INFO] TestAgent: Sync in progress: iteration=4 +31:43.643 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.643 [DEBUG] TestAgent: Node info in sync +31:43.643 [DEBUG] TestAgent: Service in sync: service=redis +31:43.643 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.643 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.643 [INFO] TestAgent: Sync in progress: iteration=5 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [INFO] TestAgent: Sync in progress: iteration=6 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [INFO] TestAgent: Sync in progress: iteration=7 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [INFO] TestAgent: Sync in progress: iteration=8 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [INFO] TestAgent: Sync in progress: iteration=9 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [INFO] TestAgent: Sync in progress: iteration=10 +31:43.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:43.644 [DEBUG] TestAgent: Node info in sync +31:43.644 [DEBUG] TestAgent: Service in sync: service=redis +31:43.644 [DEBUG] TestAgent: Check in sync: check=redis-check +31:43.644 [DEBUG] TestAgent: Check in sync: check=node-check +31:43.644 [INFO] TestAgent: Requesting shutdown +31:43.644 [INFO] TestAgent.server: shutting down server +31:43.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.644 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.644 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.644 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.644 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:43.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.645 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.645 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.645 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.652 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.653Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.653Z [INFO] agent.router.manager: shutting down +31:43.653 [INFO] TestAgent: consul server down +31:43.653 [INFO] TestAgent: shutdown complete +31:43.653 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29464 network=tcp +31:43.653 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29464 network=udp +31:43.653 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29514 network=tcp +31:44.154 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.154 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_IndexChurn/with_tags +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:43.591Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.627Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.764 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e4a95f7-5ff3-251e-11a6-508f475723e0 Address:127.0.0.1:29483}]" +31:43.764 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29483 [Follower]" leader= +31:43.765 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2e4a95f7-5ff3-251e-11a6-508f475723e0.dc1 127.0.0.1 +31:43.765 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2e4a95f7-5ff3-251e-11a6-508f475723e0 127.0.0.1 +2021-10-05T16:31:43.765Z [INFO] agent.router: Initializing LAN area manager +31:43.766 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29492 network=udp +31:43.766 [INFO] TestAgent.server: Adding LAN server: server="Node-2e4a95f7-5ff3-251e-11a6-508f475723e0 (Addr: tcp/127.0.0.1:29483) (DC: dc1)" +31:43.766 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2e4a95f7-5ff3-251e-11a6-508f475723e0.dc1 area=wan +31:43.766 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29492 network=tcp +31:43.767 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29479 network=tcp +31:43.767 [INFO] TestAgent: started state syncer +31:43.767 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29428 network=tcp +31:43.821 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.821 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29483 [Candidate]" term=2 +31:43.864 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.864 [DEBUG] TestAgent.server.raft: vote granted: from=2e4a95f7-5ff3-251e-11a6-508f475723e0 term=2 tally=1 +31:43.864 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.864 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29483 [Leader]" +31:43.864 [INFO] TestAgent.server: cluster leadership acquired +31:43.864 [INFO] TestAgent.server: New leader elected: payload=Node-2e4a95f7-5ff3-251e-11a6-508f475723e0 +31:43.885 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29483 +31:43.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.968 [INFO] TestAgent: Synced node info +31:43.999 [INFO] TestAgent: Requesting shutdown +31:43.999 [INFO] TestAgent.server: shutting down server +31:43.999 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.008 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.008Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.008Z [INFO] agent.router.manager: shutting down +31:44.008 [INFO] TestAgent: consul server down +31:44.008 [INFO] TestAgent: shutdown complete +31:44.008 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29492 network=tcp +31:44.008 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29492 network=udp +31:44.008 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29479 network=tcp +31:44.008 [ERROR] TestAgent.server.connect: Raft apply failed: error="raft is already shutdown" +31:44.008 [ERROR] TestAgent.server: failed to establish leadership: error="raft is already shutdown" +31:44.509 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.509 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:42.764Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:42.778Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:42.831 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6345a528-8311-bbf2-d0d3-aa746d068da0 Address:127.0.0.1:30391}]" +31:42.831 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6345a528-8311-bbf2-d0d3-aa746d068da0.dc1 127.0.0.1 +31:42.832 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30391 [Follower]" leader= +31:42.832 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6345a528-8311-bbf2-d0d3-aa746d068da0 127.0.0.1 +2021-10-05T16:31:42.832Z [INFO] agent.router: Initializing LAN area manager +31:42.832 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30386 network=udp +31:42.833 [INFO] TestAgent.server: Adding LAN server: server="Node-6345a528-8311-bbf2-d0d3-aa746d068da0 (Addr: tcp/127.0.0.1:30391) (DC: dc1)" +31:42.833 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6345a528-8311-bbf2-d0d3-aa746d068da0.dc1 area=wan +31:42.833 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30386 network=tcp +31:42.834 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30387 network=tcp +31:42.834 [INFO] TestAgent: started state syncer +31:42.834 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30329 network=tcp +31:42.888 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:42.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30391 [Candidate]" term=2 +31:42.916 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:42.916 [DEBUG] TestAgent.server.raft: vote granted: from=6345a528-8311-bbf2-d0d3-aa746d068da0 term=2 tally=1 +31:42.916 [INFO] TestAgent.server.raft: election won: tally=1 +31:42.916 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30391 [Leader]" +31:42.916 [INFO] TestAgent.server: cluster leadership acquired +31:42.916 [INFO] TestAgent.server: New leader elected: payload=Node-6345a528-8311-bbf2-d0d3-aa746d068da0 +31:42.934 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30391 +31:42.947 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.023 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.023 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.023 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.023 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.023 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.023 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6345a528-8311-bbf2-d0d3-aa746d068da0 +31:43.023 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6345a528-8311-bbf2-d0d3-aa746d068da0 +31:43.026 [INFO] TestAgent: Requesting shutdown +31:43.026 [INFO] TestAgent.server: shutting down server +31:43.026 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:43.026 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:43.026 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:43.026 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:43.026 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:43.026 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:43.026 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:43.026 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:43.026 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:43.037 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:43.039Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:43.039Z [INFO] agent.router.manager: shutting down +31:43.052 [INFO] TestAgent: consul server down +31:43.052 [INFO] TestAgent: shutdown complete +31:43.052 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:43.053 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30386 network=tcp +31:43.053 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:43.052 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-6345a528-8311-bbf2-d0d3-aa746d068da0 127.0.0.1 30389 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:6345a528-8311-bbf2-d0d3-aa746d068da0 port:30391 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:30390] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:43.053 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30386 network=udp +31:43.053 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30387 network=tcp +31:43.556 [INFO] TestAgent: Waiting for endpoints to shut down +31:43.556 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ReconnectConfigSettings (1.76s) +=== CONT TestAgentConnectAuthorize_serviceWrite +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:43.576Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.699Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.852 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b65ce6a-d7d9-659f-404e-c991f9ba136a Address:127.0.0.1:29504}]" +31:43.853 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a.dc1 127.0.0.1 +31:43.853 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29504 [Follower]" leader= +31:43.853 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a 127.0.0.1 +2021-10-05T16:31:43.853Z [INFO] agent.router: Initializing LAN area manager +31:43.854 [INFO] TestAgent.server: Adding LAN server: server="Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a (Addr: tcp/127.0.0.1:29504) (DC: dc1)" +31:43.854 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a.dc1 area=wan +31:43.854 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29506 network=udp +31:43.854 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29506 network=tcp +31:43.855 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29500 network=tcp +31:43.855 [INFO] TestAgent: started state syncer +31:43.855 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29491 network=tcp +31:43.917 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.917 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29504 [Candidate]" term=2 +31:43.944 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.944 [DEBUG] TestAgent.server.raft: vote granted: from=8b65ce6a-d7d9-659f-404e-c991f9ba136a term=2 tally=1 +31:43.944 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.944 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29504 [Leader]" +31:43.945 [INFO] TestAgent.server: cluster leadership acquired +31:43.945 [INFO] TestAgent.server: New leader elected: payload=Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a +31:43.958 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29504 +31:44.004 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.076 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.076 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.076 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.076 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.076 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.076 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a +31:44.076 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b65ce6a-d7d9-659f-404e-c991f9ba136a +31:44.122 [INFO] TestAgent.server: federation state anti-entropy synced +31:44.124 [INFO] TestAgent: Requesting shutdown +31:44.125 [INFO] TestAgent.server: shutting down server +31:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:44.125 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.125 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:44.125 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:44.125 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:44.125 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:44.125 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:44.125 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:44.132 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.133Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.133Z [INFO] agent.router.manager: shutting down +31:44.133 [INFO] TestAgent: consul server down +31:44.133 [INFO] TestAgent: shutdown complete +31:44.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29506 network=tcp +31:44.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29506 network=udp +31:44.133 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29500 network=tcp +31:44.633 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.633 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_ExposeConfig (1.09s) +=== CONT TestAgentConnectAuthorize_denyWildcard +2021-10-05T16:31:43.747Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.747Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.835Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.835Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.900 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ebafa759-d9f4-61d2-19f3-bf6a320b3d21 Address:127.0.0.1:29462}]" +31:43.900 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 127.0.0.1 +31:43.901 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 127.0.0.1 +2021-10-05T16:31:43.901Z [INFO] agent.router: Initializing LAN area manager +31:43.901 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29429 network=udp +31:43.901 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29462 [Follower]" leader= +31:43.902 [INFO] TestAgent.server: Adding LAN server: server="Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 (Addr: tcp/127.0.0.1:29462) (DC: dc1)" +31:43.902 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 area=wan +31:43.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29429 network=tcp +31:43.902 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29458 network=tcp +31:43.902 [INFO] TestAgent: started state syncer +31:43.902 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29498 network=tcp +31:43.952 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:43.968 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.968 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29462 [Candidate]" term=2 +31:43.987 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.987 [DEBUG] TestAgent.server.raft: vote granted: from=ebafa759-d9f4-61d2-19f3-bf6a320b3d21 term=2 tally=1 +31:43.987 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.987 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29462 [Leader]" +31:43.988 [INFO] TestAgent.server: cluster leadership acquired +31:43.988 [INFO] TestAgent.server: New leader elected: payload=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.016 [INFO] TestAgent.server: initializing acls +31:44.028 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:44.028 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.042 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:44.052 [INFO] TestAgent.server: initializing acls +31:44.052 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.078 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.078 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:44.078 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:44.078 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.078 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 +31:44.078 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 area=wan +31:44.078 [INFO] TestAgent.server: Updating LAN server: server="Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 (Addr: tcp/127.0.0.1:29462) (DC: dc1)" +31:44.107 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.107 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:44.107 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.107 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 +31:44.107 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21.dc1 area=wan +31:44.107 [INFO] TestAgent.server: Updating LAN server: server="Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 (Addr: tcp/127.0.0.1:29462) (DC: dc1)" +31:44.130 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.164 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.164 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.164 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.164 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.164 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.164 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.164 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.176 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:44.179 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.179 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.180 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ebafa759-d9f4-61d2-19f3-bf6a320b3d21 +31:44.180 [INFO] TestAgent: Requesting shutdown +31:44.180 [INFO] TestAgent.server: shutting down server +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:44.180 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:44.180 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.180 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:44.180 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:44.180 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:44.180 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:44.180 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:44.188 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.188Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.188Z [INFO] agent.router.manager: shutting down +31:44.195 [INFO] TestAgent: consul server down +31:44.195 [INFO] TestAgent: shutdown complete +31:44.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29429 network=tcp +31:44.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29429 network=udp +31:44.195 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29458 network=tcp +31:44.195 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:44.195 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:44.695 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.695 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_HostBadACL (1.00s) +=== CONT TestAgentConnectAuthorize_allowTrustDomain +2021-10-05T16:31:43.743Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.743Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.804Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.804Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.890 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ce69ccf-469d-a377-5c10-2b8289be8b88 Address:127.0.0.1:29539}]" +31:43.890 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29539 [Follower]" leader= +31:43.891 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1ce69ccf-469d-a377-5c10-2b8289be8b88.dc1 127.0.0.1 +31:43.891 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 127.0.0.1 +2021-10-05T16:31:43.891Z [INFO] agent.router: Initializing LAN area manager +31:43.892 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88.dc1 area=wan +31:43.892 [INFO] TestAgent.server: Adding LAN server: server="Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 (Addr: tcp/127.0.0.1:29539) (DC: dc1)" +31:43.892 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29499 network=tcp +31:43.894 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29499 network=udp +31:43.895 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29535 network=tcp +31:43.895 [INFO] TestAgent: started state syncer +31:43.895 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29421 network=tcp +31:43.942 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:43.946 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.946 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29539 [Candidate]" term=2 +31:43.959 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.959 [DEBUG] TestAgent.server.raft: vote granted: from=1ce69ccf-469d-a377-5c10-2b8289be8b88 term=2 tally=1 +31:43.959 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.959 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29539 [Leader]" +31:43.959 [INFO] TestAgent.server: cluster leadership acquired +31:43.960 [INFO] TestAgent.server: New leader elected: payload=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 +31:43.970 [INFO] TestAgent.server: initializing acls +31:43.975 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:43.975 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:43.992 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:44.024 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.024 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:44.024 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:44.024 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 +31:44.024 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1ce69ccf-469d-a377-5c10-2b8289be8b88.dc1 +31:44.026 [INFO] TestAgent.server: Updating LAN server: server="Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 (Addr: tcp/127.0.0.1:29539) (DC: dc1)" +31:44.026 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88.dc1 area=wan +31:44.044 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.089 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.089 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.089 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.089 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.090 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.090 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 +31:44.090 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 +31:44.097 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ce69ccf-469d-a377-5c10-2b8289be8b88 +31:44.105 [INFO] TestAgent.server: federation state anti-entropy synced +31:44.188 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:44.243 [INFO] TestAgent: Requesting shutdown +31:44.243 [INFO] TestAgent.server: shutting down server +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:44.243 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.243 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:44.251 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.252Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.252Z [INFO] agent.router.manager: shutting down +31:44.252 [INFO] TestAgent: consul server down +31:44.253 [INFO] TestAgent: shutdown complete +31:44.253 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29499 network=tcp +31:44.253 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29499 network=udp +31:44.253 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29535 network=tcp +31:44.753 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.753 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Host (1.06s) +=== CONT TestAgentConnectAuthorize_deny +2021-10-05T16:31:43.907Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.907Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.921Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:43.921Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.989 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55847d09-662b-8b57-6b42-beec79a72db1 Address:127.0.0.1:29420}]" +31:43.989 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 127.0.0.1 +31:43.990 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55847d09-662b-8b57-6b42-beec79a72db1 127.0.0.1 +2021-10-05T16:31:43.990Z [INFO] agent.router: Initializing LAN area manager +31:43.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29422 network=udp +31:43.990 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29420 [Follower]" leader= +31:43.991 [INFO] TestAgent.server: Adding LAN server: server="Node-55847d09-662b-8b57-6b42-beec79a72db1 (Addr: tcp/127.0.0.1:29420) (DC: dc1)" +31:43.991 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 area=wan +31:43.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29422 network=tcp +31:43.991 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29416 network=tcp +31:43.991 [INFO] TestAgent: started state syncer +31:43.991 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29540 network=tcp +31:44.038 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.038 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29420 [Candidate]" term=2 +31:44.041 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:44.065 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.065 [DEBUG] TestAgent.server.raft: vote granted: from=55847d09-662b-8b57-6b42-beec79a72db1 term=2 tally=1 +31:44.065 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.065 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29420 [Leader]" +31:44.065 [INFO] TestAgent.server: cluster leadership acquired +31:44.065 [INFO] TestAgent.server: New leader elected: payload=Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.108 [INFO] TestAgent.server: initializing acls +31:44.119 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:44.124 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:44.124 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.134 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:44.142 [INFO] TestAgent.server: initializing acls +31:44.142 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.162 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.162 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:44.162 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:44.162 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:44.162 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.162 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 +31:44.162 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 area=wan +31:44.162 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.162 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.163 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 +31:44.164 [INFO] TestAgent.server: Updating LAN server: server="Node-55847d09-662b-8b57-6b42-beec79a72db1 (Addr: tcp/127.0.0.1:29420) (DC: dc1)" +31:44.164 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-55847d09-662b-8b57-6b42-beec79a72db1.dc1 area=wan +31:44.164 [INFO] TestAgent.server: Updating LAN server: server="Node-55847d09-662b-8b57-6b42-beec79a72db1 (Addr: tcp/127.0.0.1:29420) (DC: dc1)" +31:44.198 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.230 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.230 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.230 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.230 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.230 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.230 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.230 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.238 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.238 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55847d09-662b-8b57-6b42-beec79a72db1 +31:44.243 [INFO] TestAgent: Requesting shutdown +31:44.243 [INFO] TestAgent.server: shutting down server +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:44.243 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:44.249 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.252Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.252Z [INFO] agent.router.manager: shutting down +31:44.256 [INFO] TestAgent: consul server down +31:44.256 [INFO] TestAgent: shutdown complete +31:44.256 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29422 network=tcp +31:44.256 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29422 network=udp +31:44.256 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29416 network=tcp +31:44.256 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:44.256 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:44.756 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.756 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_defaultAllow (0.87s) +=== CONT TestAgentConnectAuthorize_allow +2021-10-05T16:31:44.002Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:44.002Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.022Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:44.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.071 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 Address:127.0.0.1:29497}]" +31:44.071 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7.dc1 127.0.0.1 +31:44.072 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29497 [Follower]" leader= +31:44.072 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 127.0.0.1 +2021-10-05T16:31:44.072Z [INFO] agent.router: Initializing LAN area manager +31:44.072 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7.dc1 area=wan +31:44.072 [INFO] TestAgent.server: Adding LAN server: server="Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 (Addr: tcp/127.0.0.1:29497) (DC: dc1)" +31:44.072 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29541 network=tcp +31:44.072 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29541 network=udp +31:44.073 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29493 network=tcp +31:44.073 [INFO] TestAgent: started state syncer +31:44.073 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29512 network=tcp +31:44.122 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:44.124 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.124 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29497 [Candidate]" term=2 +31:44.147 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.147 [DEBUG] TestAgent.server.raft: vote granted: from=0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 term=2 tally=1 +31:44.147 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.147 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29497 [Leader]" +31:44.148 [INFO] TestAgent.server: cluster leadership acquired +31:44.148 [INFO] TestAgent.server: New leader elected: payload=Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 +31:44.157 [INFO] TestAgent.server: initializing acls +31:44.163 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:44.163 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.188 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:44.211 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.211 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:44.211 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:44.211 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 +31:44.211 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7.dc1 +31:44.212 [INFO] TestAgent.server: Updating LAN server: server="Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 (Addr: tcp/127.0.0.1:29497) (DC: dc1)" +31:44.212 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7.dc1 area=wan +31:44.244 [INFO] TestAgent: Synced node info +31:44.263 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.272 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 +31:44.272 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0916cdaa-72e1-d8b3-d3ba-56a0f22440e7 +31:44.273 [INFO] TestAgent: Requesting shutdown +31:44.273 [INFO] TestAgent.server: shutting down server +31:44.273 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:44.273 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:44.273 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.273 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:44.273 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:44.284 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.284Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.284Z [INFO] agent.router.manager: shutting down +31:44.284 [INFO] TestAgent: consul server down +31:44.284 [INFO] TestAgent: shutdown complete +31:44.284 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29541 network=tcp +31:44.284 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29541 network=udp +31:44.284 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29493 network=tcp +31:44.286 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" +31:44.286 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:44.286 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:44.785 [INFO] TestAgent: Waiting for endpoints to shut down +31:44.785 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_defaultDeny (0.81s) +=== CONT TestAgentConnectAuthorize_idNotService +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc2 +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc3 +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc1 +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc3 +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc1 +=== RUN TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc2 +2021-10-05T16:31:44.531Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:44.531Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.550Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:44.550Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.641 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0ab2120-ac40-6097-3473-299af6c60c84 Address:127.0.0.1:29525}]" +31:44.642 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29525 [Follower]" leader= +31:44.642 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0ab2120-ac40-6097-3473-299af6c60c84.dc1 127.0.0.1 +31:44.643 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0ab2120-ac40-6097-3473-299af6c60c84 127.0.0.1 +2021-10-05T16:31:44.643Z [INFO] agent.router: Initializing LAN area manager +31:44.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29534 network=udp +31:44.643 [INFO] TestAgent.server: Adding LAN server: server="Node-b0ab2120-ac40-6097-3473-299af6c60c84 (Addr: tcp/127.0.0.1:29525) (DC: dc1)" +31:44.643 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0ab2120-ac40-6097-3473-299af6c60c84.dc1 area=wan +31:44.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29534 network=tcp +31:44.644 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29521 network=tcp +31:44.644 [INFO] TestAgent: started state syncer +31:44.644 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29526 network=tcp +31:44.683 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.683 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29525 [Candidate]" term=2 +31:44.693 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:44.701 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.701 [DEBUG] TestAgent.server.raft: vote granted: from=b0ab2120-ac40-6097-3473-299af6c60c84 term=2 tally=1 +31:44.701 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.701 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29525 [Leader]" +31:44.701 [INFO] TestAgent.server: cluster leadership acquired +31:44.701 [INFO] TestAgent.server: New leader elected: payload=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.719 [INFO] TestAgent.server: initializing acls +31:44.724 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:44.724 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:44.732 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:44.747 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:44.747 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:44.747 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:44.747 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.747 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b0ab2120-ac40-6097-3473-299af6c60c84.dc1 +31:44.748 [INFO] TestAgent.server: Updating LAN server: server="Node-b0ab2120-ac40-6097-3473-299af6c60c84 (Addr: tcp/127.0.0.1:29525) (DC: dc1)" +31:44.748 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b0ab2120-ac40-6097-3473-299af6c60c84.dc1 area=wan +31:44.773 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.821 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.821 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.821 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.821 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.821 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.821 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.835 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.860 [INFO] TestAgent.server: federation state anti-entropy synced +31:44.902 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:44.937 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.937 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b0ab2120-ac40-6097-3473-299af6c60c84 +31:44.945 [INFO] TestAgent: Synced node info +31:44.972 [INFO] TestAgent: Requesting shutdown +31:44.972 [INFO] TestAgent.server: shutting down server +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:44.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:44.972 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:44.973 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:44.973 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:44.979 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:44.992Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:44.992Z [INFO] agent.router.manager: shutting down +31:44.992 [INFO] TestAgent: consul server down +31:44.992 [INFO] TestAgent: shutdown complete +31:44.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29534 network=tcp +31:44.992 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29534 network=udp +31:44.992 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29521 network=tcp +31:45.494 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.494 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_serviceWrite (0.99s) +=== CONT TestAgentConnectAuthorize_idInvalidFormat +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:44.651Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.665Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.701 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a048a145-350a-35e8-5546-7c671c304499 Address:127.0.0.1:29546}]" +31:44.702 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29546 [Follower]" leader= +31:44.702 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a048a145-350a-35e8-5546-7c671c304499.dc1 127.0.0.1 +31:44.703 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a048a145-350a-35e8-5546-7c671c304499 127.0.0.1 +2021-10-05T16:31:44.703Z [INFO] agent.router: Initializing LAN area manager +31:44.703 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29527 network=udp +31:44.703 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a048a145-350a-35e8-5546-7c671c304499.dc1 area=wan +31:44.704 [INFO] TestAgent.server: Adding LAN server: server="Node-a048a145-350a-35e8-5546-7c671c304499 (Addr: tcp/127.0.0.1:29546) (DC: dc1)" +31:44.704 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29527 network=tcp +31:44.704 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29542 network=tcp +31:44.704 [INFO] TestAgent: started state syncer +31:44.704 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29547 network=tcp +31:44.751 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.751 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29546 [Candidate]" term=2 +31:44.783 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.783 [DEBUG] TestAgent.server.raft: vote granted: from=a048a145-350a-35e8-5546-7c671c304499 term=2 tally=1 +31:44.783 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.783 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29546 [Leader]" +31:44.783 [INFO] TestAgent.server: cluster leadership acquired +31:44.783 [INFO] TestAgent.server: New leader elected: payload=Node-a048a145-350a-35e8-5546-7c671c304499 +31:44.805 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29546 +31:44.838 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.901 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.901 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.901 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.901 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.901 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.901 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a048a145-350a-35e8-5546-7c671c304499 +31:44.901 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a048a145-350a-35e8-5546-7c671c304499 +31:44.974 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.069 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.088 [INFO] TestAgent: Synced node info +31:45.088 [DEBUG] TestAgent: Node info in sync +31:45.091 [INFO] TestAgent: Requesting shutdown +31:45.091 [INFO] TestAgent.server: shutting down server +31:45.091 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.091 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.091 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.091 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.091 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.091 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.091 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.091 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.091 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.091 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.097 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.104Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.104Z [INFO] agent.router.manager: shutting down +31:45.104 [INFO] TestAgent: consul server down +31:45.104 [INFO] TestAgent: shutdown complete +31:45.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29527 network=tcp +31:45.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29527 network=udp +31:45.104 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29542 network=tcp +31:45.604 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.604 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_denyWildcard (0.97s) +=== CONT TestAgentConnectAuthorize_noTarget +2021-10-05T16:31:44.796Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.807Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.862 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fa4496ff-49e6-184f-dac9-e1a7f24a4198 Address:127.0.0.1:29574}]" +31:44.863 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198.dc1 127.0.0.1 +31:44.863 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198 127.0.0.1 +2021-10-05T16:31:44.863Z [INFO] agent.router: Initializing LAN area manager +31:44.863 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29590 network=udp +31:44.864 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29574 [Follower]" leader= +31:44.864 [INFO] TestAgent.server: Adding LAN server: server="Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198 (Addr: tcp/127.0.0.1:29574) (DC: dc1)" +31:44.864 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198.dc1 area=wan +31:44.864 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29590 network=tcp +31:44.864 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29570 network=tcp +31:44.864 [INFO] TestAgent: started state syncer +31:44.865 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29561 network=tcp +31:44.912 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.912 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29574 [Candidate]" term=2 +31:44.959 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.959 [DEBUG] TestAgent.server.raft: vote granted: from=fa4496ff-49e6-184f-dac9-e1a7f24a4198 term=2 tally=1 +31:44.959 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.959 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29574 [Leader]" +31:44.967 [INFO] TestAgent.server: cluster leadership acquired +31:44.967 [INFO] TestAgent.server: New leader elected: payload=Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198 +31:44.996 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29574 +31:45.036 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.103 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.103 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.103 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.103 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.103 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.103 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198 +31:45.103 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fa4496ff-49e6-184f-dac9-e1a7f24a4198 +31:45.128 [INFO] TestAgent: Requesting shutdown +31:45.128 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.129 [INFO] TestAgent.server: shutting down server +31:45.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.129 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.129 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.129 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.129 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.129 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.129 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:45.129 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.129 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.129 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.134 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.142Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.142Z [INFO] agent.router.manager: shutting down +31:45.142 [INFO] TestAgent: consul server down +31:45.142 [INFO] TestAgent: shutdown complete +31:45.142 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29590 network=tcp +31:45.142 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29590 network=udp +31:45.142 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29570 network=tcp +31:45.642 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.642 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_idNotService (0.86s) +=== CONT TestRPC_HTTPSMaxConnsPerClient +=== RUN TestRPC_HTTPSMaxConnsPerClient/HTTP +2021-10-05T16:31:44.181Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.194Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.234 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:25b9ce52-c9a4-e47a-84d3-dac23c8007cb Address:127.0.0.1:29532}]" +31:44.235 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb.dc1 127.0.0.1 +31:44.236 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb 127.0.0.1 +2021-10-05T16:31:44.236Z [INFO] agent.router: Initializing LAN area manager +31:44.236 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29532 [Follower]" leader= +31:44.236 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29513 network=udp +31:44.236 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb.dc1 area=wan +31:44.236 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29513 network=tcp +31:44.236 [INFO] TestAgent.server: Adding LAN server: server="Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb (Addr: tcp/127.0.0.1:29532) (DC: dc1)" +31:44.237 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29528 network=tcp +31:44.237 [INFO] TestAgent: started state syncer +31:44.237 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29533 network=tcp +31:44.284 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.284 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29532 [Candidate]" term=2 +31:44.292 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.292 [DEBUG] TestAgent.server.raft: vote granted: from=25b9ce52-c9a4-e47a-84d3-dac23c8007cb term=2 tally=1 +31:44.292 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.292 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29532 [Leader]" +31:44.294 [INFO] TestAgent.server: cluster leadership acquired +31:44.294 [INFO] TestAgent.server: New leader elected: payload=Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb +31:44.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29532 +31:44.362 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:44.415 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:44.415 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:44.415 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:44.415 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:44.415 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:44.415 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb +31:44.415 [INFO] TestAgent.server: member joined, marking health alive: member=Node-25b9ce52-c9a4-e47a-84d3-dac23c8007cb +31:44.431 [INFO] TestAgent.server: federation state anti-entropy synced +31:44.577 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:44.582 [INFO] TestAgent: Synced node info +31:44.582 [DEBUG] TestAgent: Node info in sync +31:44.632 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:44.632 [DEBUG] TestAgent: Node info in sync +31:44.643 [INFO] TestAgent: Synced service: service=redis +31:44.643 [DEBUG] TestAgent: Check in sync: check=redis-check +31:44.648 [INFO] TestAgent: Synced check: check=node-check +31:45.148 [DEBUG] TestAgent: Registered node: node="Node-level check" +31:45.148 [DEBUG] TestAgent: Registered node: node="Serf Health Status" +31:45.148 [DEBUG] TestAgent: Registered node: node="Service-level check" +31:45.148 [INFO] TestAgent: Sync in progress: iteration=1 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=2 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=3 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=4 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=5 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=6 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=7 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.148 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.148 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.148 [INFO] TestAgent: Sync in progress: iteration=8 +31:45.148 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.148 [DEBUG] TestAgent: Node info in sync +31:45.148 [DEBUG] TestAgent: Service in sync: service=redis +31:45.149 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.149 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.149 [INFO] TestAgent: Sync in progress: iteration=9 +31:45.149 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.149 [DEBUG] TestAgent: Node info in sync +31:45.149 [DEBUG] TestAgent: Service in sync: service=redis +31:45.149 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.149 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.149 [INFO] TestAgent: Sync in progress: iteration=10 +31:45.149 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.149 [DEBUG] TestAgent: Node info in sync +31:45.149 [DEBUG] TestAgent: Service in sync: service=redis +31:45.149 [DEBUG] TestAgent: Check in sync: check=redis-check +31:45.149 [DEBUG] TestAgent: Check in sync: check=node-check +31:45.149 [INFO] TestAgent: Requesting shutdown +31:45.149 [INFO] TestAgent.server: shutting down server +31:45.149 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.149 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.149 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.149 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.149 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.149 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.149 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.149 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.149 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.149 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.161 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.163Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.163Z [INFO] agent.router.manager: shutting down +31:45.163 [INFO] TestAgent: consul server down +31:45.163 [INFO] TestAgent: shutdown complete +31:45.163 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29513 network=tcp +31:45.163 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29513 network=udp +31:45.163 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29528 network=tcp +31:45.664 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.664 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:44.776Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.791Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.880 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:359d8597-b871-4105-b012-421690b0ed9d Address:127.0.0.1:29567}]" +31:44.880 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29567 [Follower]" leader= +31:44.881 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-359d8597-b871-4105-b012-421690b0ed9d.dc1 127.0.0.1 +31:44.881 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-359d8597-b871-4105-b012-421690b0ed9d 127.0.0.1 +2021-10-05T16:31:44.881Z [INFO] agent.router: Initializing LAN area manager +31:44.881 [INFO] TestAgent.server: Adding LAN server: server="Node-359d8597-b871-4105-b012-421690b0ed9d (Addr: tcp/127.0.0.1:29567) (DC: dc1)" +31:44.881 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-359d8597-b871-4105-b012-421690b0ed9d.dc1 area=wan +31:44.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29555 network=udp +31:44.889 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29555 network=tcp +31:44.900 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29563 network=tcp +31:44.900 [INFO] TestAgent: started state syncer +31:44.901 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29568 network=tcp +31:44.930 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.930 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29567 [Candidate]" term=2 +31:44.972 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.972 [DEBUG] TestAgent.server.raft: vote granted: from=359d8597-b871-4105-b012-421690b0ed9d term=2 tally=1 +31:44.972 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.972 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29567 [Leader]" +31:44.975 [INFO] TestAgent.server: cluster leadership acquired +31:44.975 [INFO] TestAgent.server: New leader elected: payload=Node-359d8597-b871-4105-b012-421690b0ed9d +31:45.000 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29567 +31:45.039 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.117 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.117 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.117 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.117 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.117 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.117 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-359d8597-b871-4105-b012-421690b0ed9d +31:45.117 [INFO] TestAgent.server: member joined, marking health alive: member=Node-359d8597-b871-4105-b012-421690b0ed9d +31:45.144 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.151 [INFO] TestAgent: Requesting shutdown +31:45.151 [INFO] TestAgent.server: shutting down server +31:45.151 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.151 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.151 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.151 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.151 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.151 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.151 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.151 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:45.151 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.151 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.151 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.156 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.164Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.164Z [INFO] agent.router.manager: shutting down +31:45.164 [INFO] TestAgent: consul server down +31:45.164 [INFO] TestAgent: shutdown complete +31:45.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29555 network=tcp +31:45.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29555 network=udp +31:45.164 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29563 network=tcp +31:45.664 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.664 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_IndexChurn (3.03s) + --- PASS: TestAgent_IndexChurn/no_tags (1.52s) + --- PASS: TestAgent_IndexChurn/with_tags (1.51s) +=== CONT TestIntentionsCheck_basic +--- PASS: TestAgentConnectAuthorize_deny (0.91s) +=== CONT TestIntentionsMatch_noName +2021-10-05T16:31:44.718Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.733Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.806 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c96ee2e-93a6-856d-a5a3-a79b5d263d7d Address:127.0.0.1:29553}]" +31:44.807 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29553 [Follower]" leader= +31:44.810 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d.dc1 127.0.0.1 +31:44.811 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d 127.0.0.1 +2021-10-05T16:31:44.811Z [INFO] agent.router: Initializing LAN area manager +31:44.811 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29548 network=udp +31:44.812 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d.dc1 area=wan +31:44.812 [INFO] TestAgent.server: Adding LAN server: server="Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d (Addr: tcp/127.0.0.1:29553) (DC: dc1)" +31:44.812 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29548 network=tcp +31:44.813 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29549 network=tcp +31:44.813 [INFO] TestAgent: started state syncer +31:44.813 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29554 network=tcp +31:44.847 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.847 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29553 [Candidate]" term=2 +31:44.872 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.872 [DEBUG] TestAgent.server.raft: vote granted: from=0c96ee2e-93a6-856d-a5a3-a79b5d263d7d term=2 tally=1 +31:44.872 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.872 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29553 [Leader]" +31:44.872 [INFO] TestAgent.server: cluster leadership acquired +31:44.872 [INFO] TestAgent.server: New leader elected: payload=Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d +31:44.894 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29553 +31:44.978 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.019 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.019 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.019 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.019 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.019 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.020 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d +31:45.020 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c96ee2e-93a6-856d-a5a3-a79b5d263d7d +31:45.048 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.141 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.147 [INFO] TestAgent: Synced node info +31:45.228 [INFO] TestAgent: Requesting shutdown +31:45.228 [INFO] TestAgent.server: shutting down server +31:45.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.228 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.228 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.228 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.228 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.228 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.228 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.236 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.248Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.248Z [INFO] agent.router.manager: shutting down +31:45.248 [INFO] TestAgent: consul server down +31:45.248 [INFO] TestAgent: shutdown complete +31:45.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29548 network=tcp +31:45.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29548 network=udp +31:45.248 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29549 network=tcp +31:45.750 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.750 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_allowTrustDomain (1.06s) +=== CONT TestIntentionsMatch_byInvalid +2021-10-05T16:31:44.770Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.782Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.839 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:392dda34-c3af-05bf-0ffd-be4b2d6c5f50 Address:127.0.0.1:29560}]" +31:44.839 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29560 [Follower]" leader= +31:44.839 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50.dc1 127.0.0.1 +31:44.841 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50 127.0.0.1 +2021-10-05T16:31:44.841Z [INFO] agent.router: Initializing LAN area manager +31:44.841 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50.dc1 area=wan +31:44.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29569 network=udp +31:44.841 [INFO] TestAgent.server: Adding LAN server: server="Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50 (Addr: tcp/127.0.0.1:29560) (DC: dc1)" +31:44.841 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29569 network=tcp +31:44.842 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29556 network=tcp +31:44.842 [INFO] TestAgent: started state syncer +31:44.842 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29589 network=tcp +31:44.875 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.875 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29560 [Candidate]" term=2 +31:44.930 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.930 [DEBUG] TestAgent.server.raft: vote granted: from=392dda34-c3af-05bf-0ffd-be4b2d6c5f50 term=2 tally=1 +31:44.930 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.930 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29560 [Leader]" +31:44.930 [INFO] TestAgent.server: cluster leadership acquired +31:44.930 [INFO] TestAgent.server: New leader elected: payload=Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50 +31:44.970 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29560 +31:45.003 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.044 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.045 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.045 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.045 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.045 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.045 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50 +31:45.045 [INFO] TestAgent.server: member joined, marking health alive: member=Node-392dda34-c3af-05bf-0ffd-be4b2d6c5f50 +31:45.092 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.125 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.146 [INFO] TestAgent: Synced node info +31:45.146 [DEBUG] TestAgent: Node info in sync +31:45.254 [INFO] TestAgent: Requesting shutdown +31:45.254 [INFO] TestAgent.server: shutting down server +31:45.254 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.254 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.254 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.254 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.254 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.254 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.254 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.254 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.254 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.254 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.275 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.281Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.281Z [INFO] agent.router.manager: shutting down +31:45.281 [INFO] TestAgent: consul server down +31:45.281 [INFO] TestAgent: shutdown complete +31:45.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29569 network=tcp +31:45.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29569 network=udp +31:45.281 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29556 network=tcp +31:45.781 [INFO] TestAgent: Waiting for endpoints to shut down +31:45.781 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_allow (1.03s) +=== CONT TestIntentionsMatch_noBy [WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== CONT TestAgent_JoinWAN_viaMeshGateway - agent_test.go:4395: routing {bob.,}server.dc1.consul => 127.0.0.1:11252 - agent_test.go:4396: routing {betty.,}server.dc2.consul => 127.0.0.1:11231 - agent_test.go:4397: routing {bonnie.,}server.dc3.consul => 127.0.0.1:11274 -2022-11-07T22:46:15.081Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.091Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.139 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:50aeb223-ca83-a6ee-07a4-dc35736099ef Address:127.0.0.1:11186}]" -46:15.139 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11186 [Follower]" leader= -46:15.140 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-50aeb223-ca83-a6ee-07a4-dc35736099ef.dc1 127.0.0.1 -46:15.141 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-50aeb223-ca83-a6ee-07a4-dc35736099ef 127.0.0.1 -2022-11-07T22:46:15.141Z [INFO] agent.router: Initializing LAN area manager -46:15.141 [INFO] TestAgent.server: Adding LAN server: server="Node-50aeb223-ca83-a6ee-07a4-dc35736099ef (Addr: tcp/127.0.0.1:11186) (DC: dc1)" -46:15.141 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-50aeb223-ca83-a6ee-07a4-dc35736099ef.dc1 area=wan -46:15.141 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11195 network=tcp -46:15.141 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11195 network=udp -46:15.142 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11196 network=tcp -46:15.142 [INFO] TestAgent: started state syncer -46:15.142 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11187 network=tcp -46:15.205 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.205 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11186 [Candidate]" term=2 -46:15.214 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.214 [DEBUG] TestAgent.server.raft: vote granted: from=50aeb223-ca83-a6ee-07a4-dc35736099ef term=2 tally=1 -46:15.214 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.214 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11186 [Leader]" -46:15.214 [INFO] TestAgent.server: cluster leadership acquired -46:15.214 [INFO] TestAgent.server: New leader elected: payload=Node-50aeb223-ca83-a6ee-07a4-dc35736099ef -46:15.224 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11186 -46:15.238 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.259 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.259 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.259 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.259 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.259 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.259 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-50aeb223-ca83-a6ee-07a4-dc35736099ef -46:15.259 [INFO] TestAgent.server: member joined, marking health alive: member=Node-50aeb223-ca83-a6ee-07a4-dc35736099ef -46:15.265 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.305 [INFO] TestAgent: Requesting shutdown -46:15.305 [INFO] TestAgent.server: shutting down server -46:15.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.305 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.305 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.305 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.305 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.305 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.305 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.305 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.319 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.322Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.322Z [INFO] agent.router.manager: shutting down -46:15.322 [INFO] TestAgent: consul server down -46:15.322 [INFO] TestAgent: shutdown complete -46:15.322 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11195 network=tcp -46:15.322 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11195 network=udp -46:15.322 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11196 network=tcp -46:15.822 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.822 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_Alias_setToken (0.75s) -=== CONT TestAgent_RerouteExistingHTTPChecks +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:45.659Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.674Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.719 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7da4193b-8da6-ee55-e506-b659744dc4b3 Address:127.0.0.1:29609}]" +31:45.719 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29609 [Follower]" leader= +31:45.720 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7da4193b-8da6-ee55-e506-b659744dc4b3.dc1 127.0.0.1 +31:45.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7da4193b-8da6-ee55-e506-b659744dc4b3 127.0.0.1 +2021-10-05T16:31:45.721Z [INFO] agent.router: Initializing LAN area manager +31:45.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29520 network=udp +31:45.721 [INFO] TestAgent.server: Adding LAN server: server="Node-7da4193b-8da6-ee55-e506-b659744dc4b3 (Addr: tcp/127.0.0.1:29609) (DC: dc1)" +31:45.721 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7da4193b-8da6-ee55-e506-b659744dc4b3.dc1 area=wan +31:45.721 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29520 network=tcp +31:45.722 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29605 network=tcp +31:45.722 [INFO] TestAgent: started state syncer +31:45.722 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29610 network=tcp +31:45.772 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.772 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29609 [Candidate]" term=2 +31:45.791 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.791 [DEBUG] TestAgent.server.raft: vote granted: from=7da4193b-8da6-ee55-e506-b659744dc4b3 term=2 tally=1 +31:45.791 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.791 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29609 [Leader]" +31:45.791 [INFO] TestAgent.server: cluster leadership acquired +31:45.791 [INFO] TestAgent.server: New leader elected: payload=Node-7da4193b-8da6-ee55-e506-b659744dc4b3 +31:45.802 [INFO] TestAgent: Requesting shutdown +31:45.802 [INFO] TestAgent.server: shutting down server +31:45.802 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.803 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:45.816 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.827Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.827Z [INFO] agent.router.manager: shutting down +31:45.828 [INFO] TestAgent: consul server down +31:45.828 [INFO] TestAgent: shutdown complete +31:45.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29520 network=tcp +31:45.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29520 network=udp +31:45.828 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29605 network=tcp +31:45.828 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +31:45.829 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:45.829 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:45.829 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:45.829 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:46.329 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.329 [INFO] TestAgent: Endpoints down +=== RUN TestRPC_HTTPSMaxConnsPerClient/HTTPS +2021-10-05T16:31:45.529Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.544Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.575 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1040edc7-aea6-f5de-5573-7cec38da3a2c Address:127.0.0.1:29588}]" +31:45.576 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29588 [Follower]" leader= +31:45.576 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1040edc7-aea6-f5de-5573-7cec38da3a2c.dc1 127.0.0.1 +31:45.577 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1040edc7-aea6-f5de-5573-7cec38da3a2c 127.0.0.1 +2021-10-05T16:31:45.577Z [INFO] agent.router: Initializing LAN area manager +31:45.577 [INFO] TestAgent.server: Adding LAN server: server="Node-1040edc7-aea6-f5de-5573-7cec38da3a2c (Addr: tcp/127.0.0.1:29588) (DC: dc1)" +31:45.577 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1040edc7-aea6-f5de-5573-7cec38da3a2c.dc1 area=wan +31:45.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29562 network=udp +31:45.577 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29562 network=tcp +31:45.577 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29584 network=tcp +31:45.578 [INFO] TestAgent: started state syncer +31:45.578 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29582 network=tcp +31:45.620 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.620 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29588 [Candidate]" term=2 +31:45.644 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.644 [DEBUG] TestAgent.server.raft: vote granted: from=1040edc7-aea6-f5de-5573-7cec38da3a2c term=2 tally=1 +31:45.644 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.644 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29588 [Leader]" +31:45.645 [INFO] TestAgent.server: cluster leadership acquired +31:45.645 [INFO] TestAgent.server: New leader elected: payload=Node-1040edc7-aea6-f5de-5573-7cec38da3a2c +31:45.667 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29588 +31:45.693 [INFO] TestAgent: Synced node info +31:45.693 [DEBUG] TestAgent: Node info in sync +31:45.709 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.737 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.737 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.737 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.737 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.737 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.737 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1040edc7-aea6-f5de-5573-7cec38da3a2c +31:45.737 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1040edc7-aea6-f5de-5573-7cec38da3a2c +31:45.754 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.858 [INFO] TestAgent: Requesting shutdown +31:45.858 [INFO] TestAgent.server: shutting down server +31:45.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.858 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.858 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.858 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.858 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.858 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.858 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.863 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.872Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.872Z [INFO] agent.router.manager: shutting down +31:45.872 [INFO] TestAgent: consul server down +31:45.872 [INFO] TestAgent: shutdown complete +31:45.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29562 network=tcp +31:45.872 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29562 network=udp +31:45.872 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29584 network=tcp +31:46.373 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.373 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_idInvalidFormat (0.88s) +=== CONT TestIntentionsMatch_basic +2021-10-05T16:31:45.686Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.705Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.738 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8b4c47e0-44e2-43ae-4995-e92663ace5ec Address:127.0.0.1:29630}]" +31:45.738 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29630 [Follower]" leader= +31:45.739 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec.dc1 127.0.0.1 +31:45.744 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec 127.0.0.1 +2021-10-05T16:31:45.744Z [INFO] agent.router: Initializing LAN area manager +31:45.744 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29625 network=udp +31:45.744 [INFO] TestAgent.server: Adding LAN server: server="Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec (Addr: tcp/127.0.0.1:29630) (DC: dc1)" +31:45.745 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec.dc1 area=wan +31:45.745 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29625 network=tcp +31:45.745 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29626 network=tcp +31:45.745 [INFO] TestAgent: started state syncer +31:45.745 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29617 network=tcp +31:45.788 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.788 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29630 [Candidate]" term=2 +31:45.829 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.829 [DEBUG] TestAgent.server.raft: vote granted: from=8b4c47e0-44e2-43ae-4995-e92663ace5ec term=2 tally=1 +31:45.829 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.829 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29630 [Leader]" +31:45.829 [INFO] TestAgent.server: cluster leadership acquired +31:45.829 [INFO] TestAgent.server: New leader elected: payload=Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec +31:45.840 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29630 +31:45.866 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.900 [INFO] TestAgent: Requesting shutdown +31:45.900 [INFO] TestAgent.server: shutting down server +31:45.900 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.900 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:45.912 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:45.917 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.917 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.917 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.917 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.917 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.917 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec +31:45.917 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec +2021-10-05T16:31:45.924Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.924Z [INFO] agent.router.manager: shutting down +31:45.927 [INFO] TestAgent: consul server down +31:45.927 [INFO] TestAgent: shutdown complete +31:45.927 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29625 network=tcp +31:45.927 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29625 network=udp +31:45.927 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29626 network=tcp +31:45.927 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-8b4c47e0-44e2-43ae-4995-e92663ace5ec 127.0.0.1 29628 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:8b4c47e0-44e2-43ae-4995-e92663ace5ec port:29630 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29629] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:45.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.927 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.927 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.927 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.927 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.927 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.927 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.440 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.440 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsMatch_noName (0.78s) +=== CONT TestIntentionsList_values +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:45.773Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.795Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.834 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:85900922-67a0-29f2-0d98-d5795f0100f4 Address:127.0.0.1:29602}]" +31:45.835 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-85900922-67a0-29f2-0d98-d5795f0100f4.dc1 127.0.0.1 +31:45.836 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-85900922-67a0-29f2-0d98-d5795f0100f4 127.0.0.1 +2021-10-05T16:31:45.836Z [INFO] agent.router: Initializing LAN area manager +31:45.836 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29602 [Follower]" leader= +31:45.836 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29618 network=udp +31:45.836 [INFO] TestAgent.server: Adding LAN server: server="Node-85900922-67a0-29f2-0d98-d5795f0100f4 (Addr: tcp/127.0.0.1:29602) (DC: dc1)" +31:45.836 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-85900922-67a0-29f2-0d98-d5795f0100f4.dc1 area=wan +31:45.836 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29618 network=tcp +31:45.837 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29598 network=tcp +31:45.837 [INFO] TestAgent: started state syncer +31:45.837 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29575 network=tcp +31:45.901 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.901 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29602 [Candidate]" term=2 +31:45.929 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.929 [DEBUG] TestAgent.server.raft: vote granted: from=85900922-67a0-29f2-0d98-d5795f0100f4 term=2 tally=1 +31:45.929 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.929 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29602 [Leader]" +31:45.929 [INFO] TestAgent.server: cluster leadership acquired +31:45.929 [INFO] TestAgent.server: New leader elected: payload=Node-85900922-67a0-29f2-0d98-d5795f0100f4 +31:45.949 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29602 +31:45.969 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.016 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.016 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.016 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.016 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.016 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.016 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-85900922-67a0-29f2-0d98-d5795f0100f4 +31:46.016 [INFO] TestAgent.server: member joined, marking health alive: member=Node-85900922-67a0-29f2-0d98-d5795f0100f4 +31:46.025 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.062 [INFO] TestAgent: Requesting shutdown +31:46.062 [INFO] TestAgent.server: shutting down server +31:46.062 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.062 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.062 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.062 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.062 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.062 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.062 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.062 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:46.062 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.062 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.062 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.072 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.077Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.077Z [INFO] agent.router.manager: shutting down +31:46.077 [INFO] TestAgent: consul server down +31:46.077 [INFO] TestAgent: shutdown complete +31:46.077 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29618 network=tcp +31:46.077 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29618 network=udp +31:46.078 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29598 network=tcp +31:46.578 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.578 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsMatch_byInvalid (0.83s) +=== CONT TestIntentionsList_empty +2021-10-05T16:31:45.691Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.726Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.816 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04eee9ab-8e68-0ced-37c1-dbab66d99f32 Address:127.0.0.1:29616}]" +31:45.816 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29616 [Follower]" leader= +31:45.817 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32.dc1 127.0.0.1 +31:45.817 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32 127.0.0.1 +2021-10-05T16:31:45.817Z [INFO] agent.router: Initializing LAN area manager +31:45.817 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29611 network=udp +31:45.818 [INFO] TestAgent.server: Adding LAN server: server="Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32 (Addr: tcp/127.0.0.1:29616) (DC: dc1)" +31:45.818 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32.dc1 area=wan +31:45.818 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29611 network=tcp +31:45.819 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29612 network=tcp +31:45.819 [INFO] TestAgent: started state syncer +31:45.820 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29624 network=tcp +31:45.868 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.868 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29616 [Candidate]" term=2 +31:45.886 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.886 [DEBUG] TestAgent.server.raft: vote granted: from=04eee9ab-8e68-0ced-37c1-dbab66d99f32 term=2 tally=1 +31:45.886 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.886 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29616 [Leader]" +31:45.886 [INFO] TestAgent.server: cluster leadership acquired +31:45.886 [INFO] TestAgent.server: New leader elected: payload=Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32 +31:45.894 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29616 +31:45.921 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.962 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.962 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.962 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.962 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.962 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.962 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32 +31:45.962 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04eee9ab-8e68-0ced-37c1-dbab66d99f32 +31:45.983 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.036 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:46.055 [INFO] TestAgent: Synced node info +31:46.073 [INFO] TestAgent: Requesting shutdown +31:46.073 [INFO] TestAgent.server: shutting down server +31:46.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.073 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.073 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.073 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.073 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.073 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.073 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.078 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.084Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.084Z [INFO] agent.router.manager: shutting down +31:46.084 [INFO] TestAgent: consul server down +31:46.084 [INFO] TestAgent: shutdown complete +31:46.084 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29611 network=tcp +31:46.084 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29611 network=udp +31:46.084 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29612 network=tcp +31:46.585 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.585 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsCheck_basic (0.92s) +=== CONT TestAgent_InmemKeyrings +=== RUN TestAgent_InmemKeyrings/no_keys +2021/10/05 16:31:46 http: TLS handshake error from 127.0.0.1:40764: read tcp 127.0.0.1:29634->127.0.0.1:40764: use of closed network connection +2021-10-05T16:31:45.625Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.643Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.690 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 Address:127.0.0.1:29511}]" +31:45.690 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1.dc1 127.0.0.1 +31:45.691 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 127.0.0.1 +2021-10-05T16:31:45.691Z [INFO] agent.router: Initializing LAN area manager +31:45.691 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29511 [Follower]" leader= +31:45.691 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1.dc1 area=wan +31:45.691 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29583 network=tcp +31:45.691 [INFO] TestAgent.server: Adding LAN server: server="Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 (Addr: tcp/127.0.0.1:29511) (DC: dc1)" +31:45.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29583 network=udp +31:45.692 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29507 network=tcp +31:45.692 [INFO] TestAgent: started state syncer +31:45.692 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29519 network=tcp +31:45.730 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.730 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29511 [Candidate]" term=2 +31:45.742 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.742 [DEBUG] TestAgent.server.raft: vote granted: from=a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 term=2 tally=1 +31:45.742 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.742 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29511 [Leader]" +31:45.742 [INFO] TestAgent.server: cluster leadership acquired +31:45.742 [INFO] TestAgent.server: New leader elected: payload=Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 +31:45.750 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29511 +31:45.764 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.809 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.809 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.809 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.809 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.809 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.809 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 +31:45.809 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a47e9b90-a38b-a47f-3a29-a0f4ba1d4ee1 +31:45.836 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.013 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:46.019 [INFO] TestAgent: Synced node info +31:46.019 [DEBUG] TestAgent: Node info in sync +31:46.123 [INFO] TestAgent: Requesting shutdown +31:46.123 [INFO] TestAgent.server: shutting down server +31:46.123 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.123 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.123 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.123 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.123 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.123 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.123 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.123 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.123 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.123 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.130 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.133Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.133Z [INFO] agent.router.manager: shutting down +31:46.133 [INFO] TestAgent: consul server down +31:46.133 [INFO] TestAgent: shutdown complete +31:46.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29583 network=tcp +31:46.133 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29583 network=udp +31:46.133 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29507 network=tcp +31:46.635 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.635 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectAuthorize_noTarget (1.03s) +=== CONT TestKVSEndpoint_GET_Raw +2021/10/05 16:31:46 http: TLS handshake error from 127.0.0.1:40760: EOF +2021/10/05 16:31:46 http: TLS handshake error from 127.0.0.1:40766: EOF +2021/10/05 16:31:46 http: TLS handshake error from 127.0.0.1:40762: EOF +2021-10-05T16:31:45.804Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.831Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.879 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:29463f02-33e3-a243-eb2d-a53f537dcf4c Address:127.0.0.1:29581}]" +31:45.880 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29581 [Follower]" leader= +31:45.880 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-29463f02-33e3-a243-eb2d-a53f537dcf4c.dc1 127.0.0.1 +31:45.881 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-29463f02-33e3-a243-eb2d-a53f537dcf4c 127.0.0.1 +2021-10-05T16:31:45.881Z [INFO] agent.router: Initializing LAN area manager +31:45.881 [INFO] TestAgent.server: Adding LAN server: server="Node-29463f02-33e3-a243-eb2d-a53f537dcf4c (Addr: tcp/127.0.0.1:29581) (DC: dc1)" +31:45.881 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-29463f02-33e3-a243-eb2d-a53f537dcf4c.dc1 area=wan +31:45.881 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29576 network=udp +31:45.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29576 network=tcp +31:45.882 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29577 network=tcp +31:45.882 [INFO] TestAgent: started state syncer +31:45.882 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29638 network=tcp +31:45.938 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.938 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29581 [Candidate]" term=2 +31:45.960 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.960 [DEBUG] TestAgent.server.raft: vote granted: from=29463f02-33e3-a243-eb2d-a53f537dcf4c term=2 tally=1 +31:45.960 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.960 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29581 [Leader]" +31:45.960 [INFO] TestAgent.server: cluster leadership acquired +31:45.961 [INFO] TestAgent.server: New leader elected: payload=Node-29463f02-33e3-a243-eb2d-a53f537dcf4c +31:45.968 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29581 +31:45.988 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.018 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.018 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.018 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.018 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.018 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.018 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-29463f02-33e3-a243-eb2d-a53f537dcf4c +31:46.018 [INFO] TestAgent.server: member joined, marking health alive: member=Node-29463f02-33e3-a243-eb2d-a53f537dcf4c +31:46.027 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.192 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:46.197 [INFO] TestAgent: Synced node info +31:46.270 [INFO] TestAgent: Requesting shutdown +31:46.270 [INFO] TestAgent.server: shutting down server +31:46.270 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.270 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.270 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.270 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.270 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.270 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.270 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.270 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.270 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.270 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.274 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.279Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.279Z [INFO] agent.router.manager: shutting down +31:46.280 [INFO] TestAgent: consul server down +31:46.280 [INFO] TestAgent: shutdown complete +31:46.280 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29576 network=tcp +31:46.280 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29576 network=udp +31:46.280 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29577 network=tcp +31:46.780 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.780 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsMatch_noBy (1.00s) +=== CONT TestKVSEndpoint_AcquireRelease +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:39.930Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.944Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.945 [INFO] bonnie.server.gateway_locator: will dial the primary datacenter through its mesh gateways +31:39.978 [INFO] bonnie.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba0fdd7a-de15-7320-ed49-c12f977c54c5 Address:127.0.0.1:30129}]" +31:39.979 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 +31:39.979 [INFO] bonnie.server.serf.lan: serf: EventMemberJoin: bonnie 127.0.0.1 +2021-10-05T16:31:39.979Z [INFO] agent.router: Initializing LAN area manager +31:39.980 [INFO] bonnie: Started DNS server: address=127.0.0.1:30281 network=udp +31:39.980 [INFO] bonnie.server.raft: entering follower state: follower="Node at 127.0.0.1:30129 [Follower]" leader= +31:39.980 [INFO] bonnie.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan +31:39.980 [INFO] bonnie: Started DNS server: address=127.0.0.1:30281 network=tcp +31:39.980 [INFO] bonnie.server: Adding LAN server: server="bonnie (Addr: tcp/127.0.0.1:30129) (DC: dc3)" +31:39.980 [INFO] bonnie: Started HTTP server: address=127.0.0.1:30282 network=tcp +31:39.980 [INFO] bonnie: started state syncer +31:39.981 [INFO] bonnie: Started gRPC server: address=127.0.0.1:30287 network=tcp +31:39.981 [INFO] bonnie: Refreshing mesh gateways is supported for the following discovery methods: discovery_methods="aliyun aws gce mdns os packet" +31:39.981 [INFO] bonnie: Refreshing mesh gateways... +31:39.981 [INFO] bonnie.server.gateway_locator: updated fallback list of primary mesh gateways: mesh_gateways=[127.0.0.1:30127] +31:39.981 [INFO] bonnie: Refreshing mesh gateways completed +31:39.981 [INFO] bonnie: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" +31:39.981 [INFO] bonnie: Joining cluster...: cluster=WAN +31:39.981 [INFO] bonnie: (WAN) joining: wan_addresses=[*.dc1/192.0.2.2] +31:40.047 [WARN] bonnie.server.raft: heartbeat timeout reached, starting election: last-leader= +31:40.047 [INFO] bonnie.server.raft: entering candidate state: node="Node at 127.0.0.1:30129 [Candidate]" term=2 +31:40.072 [DEBUG] bonnie.server.raft: votes: needed=1 +31:40.072 [DEBUG] bonnie.server.raft: vote granted: from=ba0fdd7a-de15-7320-ed49-c12f977c54c5 term=2 tally=1 +31:40.072 [INFO] bonnie.server.raft: election won: tally=1 +31:40.072 [INFO] bonnie.server.raft: entering leader state: leader="Node at 127.0.0.1:30129 [Leader]" +31:40.072 [INFO] bonnie.server: cluster leadership acquired +31:40.072 [INFO] bonnie.server: New leader elected: payload=bonnie +31:40.087 [DEBUG] bonnie.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false +31:40.095 [WARN] bonnie.server.connect: primary datacenter is configured but unreachable - deferring initialization of the secondary datacenter CA +31:40.095 [INFO] bonnie.leader: started routine: routine="config entry replication" +31:40.095 [INFO] bonnie.leader: started routine: routine="federation state replication" +31:40.095 [INFO] bonnie.leader: started routine: routine="federation state anti-entropy" +31:40.095 [INFO] bonnie.leader: started routine: routine="secondary CA roots watch" +31:40.095 [INFO] bonnie.leader: started routine: routine="intention replication" +31:40.095 [INFO] bonnie.leader: started routine: routine="intermediate cert renew watch" +31:40.095 [INFO] bonnie.leader: started routine: routine="CA root pruning" +31:40.095 [DEBUG] bonnie.server: Skipping self join check for node since the cluster is too small: node=bonnie +31:40.095 [INFO] bonnie.server: member joined, marking health alive: member=bonnie +31:40.096 [DEBUG] bonnie.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.096 [WARN] bonnie.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConnectCA.Roots +31:40.096 [ERROR] bonnie.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: No path to datacenter" +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.096 [WARN] bonnie.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConfigEntry.ListAll +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.096 [DEBUG] bonnie.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 +31:40.096 [WARN] bonnie.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=Intention.List +31:40.096 [ERROR] bonnie.server.connect: error replicating intentions: routine="intention replication" error="No path to datacenter" +31:40.096 [DEBUG] bonnie.server: federation states are not enabled in the primary dc +31:40.130 [DEBUG] bonnie.server.memberlist.wan: memberlist: Initiating push/pull sync with: *.dc1 127.0.0.1:30127 +31:40.132 [INFO] bonnie: Synced node info +31:40.137 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 +31:40.137 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 +31:40.137 [INFO] bonnie: (WAN) joined: number_of_nodes=1 +31:40.137 [INFO] bonnie: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 +31:40.137 [INFO] bonnie.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan +31:40.137 [INFO] bonnie.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan +31:40.176 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.176 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.176 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.176 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.177 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:40.183 [DEBUG] bonnie: Node info in sync +31:40.192 [INFO] bonnie: Synced service: service=mesh-gateway +31:40.391 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: betty.dc2 +31:40.479 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:30204: Remote DC has no server currently reachable +31:40.777 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: betty.dc2 +31:40.992 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:30204: Remote DC has no server currently reachable +31:41.136 [DEBUG] bonnie.server.replication.config_entry: finished fetching config entries: amount=0 +31:41.136 [DEBUG] bonnie.server.replication.config_entry: Config Entry replication: local=0 remote=0 +31:41.136 [DEBUG] bonnie.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 +31:41.136 [DEBUG] bonnie.server.replication.config_entry: replication completed through remote index: index=1 +31:41.177 [INFO] bonnie.server.gateway_locator: will dial the primary datacenter using our local mesh gateways if possible +31:41.177 [DEBUG] bonnie.server: finished fetching remote objects: amount=2 +31:41.177 [DEBUG] bonnie.server: diffing replication state: local_amount=0 remote_amount=2 +31:41.177 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=2 +31:41.177 [DEBUG] bonnie.server: performing updates: updates=2 +31:41.203 [DEBUG] bonnie.server: finished updates +31:41.203 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=18 +31:41.274 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.610 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.776 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.964 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.981 [INFO] bonnie.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:30127] local=[] +31:42.104 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false +31:42.136 [INFO] bonnie.server.connect: received new intermediate certificate from primary datacenter +31:42.156 [INFO] bonnie.server.connect: updated root certificates from primary datacenter +31:42.236 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:43.010 [DEBUG] bonnie: Skipping remote check since it is managed automatically: check=serfHealth +31:43.010 [DEBUG] bonnie: Node info in sync +31:43.010 [DEBUG] bonnie: Service in sync: service=mesh-gateway +31:43.010 [DEBUG] bonnie: Node info in sync +31:43.010 [DEBUG] bonnie: Service in sync: service=mesh-gateway +31:43.149 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:43.407 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:43.894 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.058 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.058 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.058 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.058 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.058 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30128 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" +31:44.979 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable +31:45.117 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 +31:45.118 [DEBUG] bonnie.server: diffing replication state: local_amount=2 remote_amount=3 +31:45.118 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 +31:45.118 [DEBUG] bonnie.server: performing updates: updates=1 +31:45.120 [INFO] bonnie.server: federation state anti-entropy synced +31:45.144 [DEBUG] bonnie.server: finished updates +31:45.144 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=20 +31:45.144 [INFO] bonnie.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:30127] local=[127.0.0.1:30127] +31:45.588 [INFO] bonnie: Requesting shutdown +31:45.588 [INFO] bonnie.server: shutting down server +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="config entry replication" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="federation state replication" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="federation state anti-entropy" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="secondary CA roots watch" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="intention replication" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="intermediate cert renew watch" +31:45.588 [DEBUG] bonnie.leader: stopping routine: routine="CA root pruning" +31:45.588 [WARN] bonnie.server.serf.lan: serf: Shutdown without a Leave +31:45.589 [DEBUG] bonnie.leader: stopped routine: routine="federation state anti-entropy" +31:45.589 [DEBUG] bonnie.leader: stopped routine: routine="intermediate cert renew watch" +31:45.589 [DEBUG] bonnie.leader: stopped routine: routine="CA root pruning" +31:45.600 [WARN] bonnie.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.612Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.612Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.612Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.612Z [INFO] agent.router.manager: shutting down +31:45.612 [INFO] bonnie: consul server down +31:45.612 [INFO] bonnie: shutdown complete +31:45.612 [INFO] bonnie: Stopping server: protocol=DNS address=127.0.0.1:30281 network=tcp +31:45.612 [INFO] bonnie: Stopping server: protocol=DNS address=127.0.0.1:30281 network=udp +31:45.612 [INFO] bonnie: Stopping server: protocol=HTTP address=127.0.0.1:30282 network=tcp +31:45.612 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=Internal.ServiceDump error="rpc error making call: EOF" +31:45.612 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" +31:45.612 [INFO] bonnie.server.replication.config_entry: stopped replication +31:45.612 [DEBUG] bonnie.leader: stopped routine: routine="config entry replication" +31:45.612 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" +31:45.612 [ERROR] bonnie.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" +31:45.612 [DEBUG] bonnie.leader: stopped routine: routine="secondary CA roots watch" +31:45.612 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" +31:45.612 [ERROR] bonnie.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" +31:45.612 [DEBUG] bonnie.leader: stopped routine: routine="intention replication" +31:45.612 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" +31:45.612 [WARN] bonnie.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" +31:45.612 [INFO] bonnie.replication.federation_state: stopped replication +31:45.612 [DEBUG] bonnie.leader: stopped routine: routine="federation state replication" +31:46.113 [INFO] bonnie: Waiting for endpoints to shut down +31:46.113 [INFO] bonnie: Endpoints down +2021-10-05T16:31:39.330Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:39.376Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:39.377 [INFO] betty.server.gateway_locator: will dial the primary datacenter through its mesh gateways +31:39.455 [INFO] betty.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f6c9c8e-117b-32c8-34da-63dfd926a124 Address:127.0.0.1:30128}]" +31:39.456 [INFO] betty.server.raft: entering follower state: follower="Node at 127.0.0.1:30128 [Follower]" leader= +31:39.456 [INFO] betty.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 +31:39.457 [INFO] betty.server.serf.lan: serf: EventMemberJoin: betty 127.0.0.1 +2021-10-05T16:31:39.457Z [INFO] agent.router: Initializing LAN area manager +31:39.457 [INFO] betty: Started DNS server: address=127.0.0.1:30200 network=udp +31:39.457 [INFO] betty.server: Adding LAN server: server="betty (Addr: tcp/127.0.0.1:30128) (DC: dc2)" +31:39.457 [INFO] betty.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan +31:39.458 [INFO] betty: Started DNS server: address=127.0.0.1:30200 network=tcp +31:39.458 [INFO] betty: Started HTTP server: address=127.0.0.1:30201 network=tcp +31:39.458 [INFO] betty: started state syncer +31:39.458 [INFO] betty: Started gRPC server: address=127.0.0.1:30266 network=tcp +31:39.458 [INFO] betty: Refreshing mesh gateways is supported for the following discovery methods: discovery_methods="aliyun aws gce mdns os packet" +31:39.458 [INFO] betty: Refreshing mesh gateways... +31:39.458 [INFO] betty.server.gateway_locator: updated fallback list of primary mesh gateways: mesh_gateways=[127.0.0.1:30127] +31:39.458 [INFO] betty: Refreshing mesh gateways completed +31:39.458 [INFO] betty: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" +31:39.458 [INFO] betty: Joining cluster...: cluster=WAN +31:39.458 [INFO] betty: (WAN) joining: wan_addresses=[*.dc1/192.0.2.2] +31:39.516 [WARN] betty.server.raft: heartbeat timeout reached, starting election: last-leader= +31:39.516 [INFO] betty.server.raft: entering candidate state: node="Node at 127.0.0.1:30128 [Candidate]" term=2 +31:39.560 [DEBUG] betty.server.raft: votes: needed=1 +31:39.560 [DEBUG] betty.server.raft: vote granted: from=0f6c9c8e-117b-32c8-34da-63dfd926a124 term=2 tally=1 +31:39.560 [INFO] betty.server.raft: election won: tally=1 +31:39.560 [INFO] betty.server.raft: entering leader state: leader="Node at 127.0.0.1:30128 [Leader]" +31:39.564 [INFO] betty.server: cluster leadership acquired +31:39.564 [INFO] betty.server: New leader elected: payload=betty +31:39.567 [DEBUG] betty.server.memberlist.wan: memberlist: Initiating push/pull sync with: *.dc1 127.0.0.1:30127 +31:39.577 [INFO] betty.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 +31:39.577 [INFO] betty: (WAN) joined: number_of_nodes=1 +31:39.577 [INFO] betty: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 +31:39.577 [INFO] betty.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan +31:39.628 [INFO] betty: Synced node info +31:39.628 [DEBUG] betty.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true +31:39.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false +31:39.865 [INFO] betty.server.connect: received new intermediate certificate from primary datacenter +31:39.880 [INFO] betty.server.connect: updated root certificates from primary datacenter +31:39.880 [INFO] betty.server.connect: initialized secondary datacenter CA with provider: provider=consul +31:39.880 [INFO] betty.leader: started routine: routine="config entry replication" +31:39.880 [INFO] betty.leader: started routine: routine="federation state replication" +31:39.880 [INFO] betty.leader: started routine: routine="federation state anti-entropy" +31:39.880 [INFO] betty.leader: started routine: routine="secondary CA roots watch" +31:39.880 [INFO] betty.leader: started routine: routine="intention replication" +31:39.880 [INFO] betty.leader: started routine: routine="intermediate cert renew watch" +31:39.880 [INFO] betty.leader: started routine: routine="CA root pruning" +31:39.880 [DEBUG] betty.server: Skipping self join check for node since the cluster is too small: node=betty +31:39.880 [INFO] betty.server: member joined, marking health alive: member=betty +31:39.881 [DEBUG] betty.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 +31:39.881 [DEBUG] betty.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 +31:39.882 [DEBUG] betty.server.replication.config_entry: finished fetching config entries: amount=0 +31:39.882 [DEBUG] betty.server.replication.config_entry: Config Entry replication: local=0 remote=0 +31:39.882 [DEBUG] betty.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 +31:39.882 [DEBUG] betty.server.replication.config_entry: replication completed through remote index: index=1 +31:39.882 [INFO] betty.server.gateway_locator: will dial the primary datacenter using our local mesh gateways if possible +31:39.882 [DEBUG] betty.server: finished fetching remote objects: amount=1 +31:39.882 [DEBUG] betty.server: diffing replication state: local_amount=0 remote_amount=1 +31:39.882 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 +31:39.882 [DEBUG] betty.server: performing updates: updates=1 +31:39.913 [INFO] betty.server.connect: updated root certificates from primary datacenter +31:39.914 [DEBUG] betty.server: finished updates +31:39.914 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=14 +31:39.920 [DEBUG] betty.server: finished fetching remote objects: amount=2 +31:39.920 [DEBUG] betty.server: diffing replication state: local_amount=1 remote_amount=2 +31:39.920 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 +31:39.920 [DEBUG] betty.server: performing updates: updates=1 +31:39.921 [INFO] betty.server: federation state anti-entropy synced +31:39.932 [DEBUG] betty.server: finished updates +31:39.932 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=16 +31:39.944 [INFO] betty.server: federation state anti-entropy synced +31:39.944 [DEBUG] betty.server: finished fetching remote objects: amount=2 +31:39.944 [DEBUG] betty.server: diffing replication state: local_amount=2 remote_amount=2 +31:39.944 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 +31:39.944 [DEBUG] betty.server: performing updates: updates=1 +31:39.964 [DEBUG] betty.server: finished updates +31:39.964 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=17 +31:40.167 [DEBUG] betty: Node info in sync +31:40.173 [INFO] betty: Synced service: service=mesh-gateway +31:40.196 [INFO] betty.server: federation state anti-entropy synced +31:40.200 [DEBUG] betty.server: finished fetching remote objects: amount=2 +31:40.200 [DEBUG] betty.server: diffing replication state: local_amount=2 remote_amount=2 +31:40.200 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 +31:40.200 [DEBUG] betty.server: performing updates: updates=1 +31:40.213 [DEBUG] betty.server: finished updates +31:40.213 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=18 +31:40.276 [INFO] betty.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 +31:40.276 [DEBUG] betty.server.serf.wan: serf: messageJoinType: betty.dc2 +31:40.276 [INFO] betty.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan +31:40.457 [ERROR] betty.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:30285: Remote DC has no server currently reachable +31:40.777 [DEBUG] betty.server.serf.wan: serf: messageJoinType: betty.dc2 +31:40.997 [ERROR] betty.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:30285: Remote DC has no server currently reachable +31:41.273 [DEBUG] betty.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.457 [INFO] betty.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:30127] local=[127.0.0.1:30127] +31:41.776 [DEBUG] betty.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:42.328 [DEBUG] betty: Skipping remote check since it is managed automatically: check=serfHealth +31:42.328 [DEBUG] betty: Node info in sync +31:42.328 [DEBUG] betty: Service in sync: service=mesh-gateway +31:42.328 [DEBUG] betty: Node info in sync +31:42.328 [DEBUG] betty: Service in sync: service=mesh-gateway +31:45.120 [DEBUG] betty.server: finished fetching remote objects: amount=3 +31:45.120 [DEBUG] betty.server: diffing replication state: local_amount=2 remote_amount=3 +31:45.120 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 +31:45.120 [DEBUG] betty.server: performing updates: updates=1 +31:45.144 [DEBUG] betty.server: finished updates +31:45.144 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=20 +31:46.113 [INFO] betty: Requesting shutdown +31:46.113 [INFO] betty.server: shutting down server +31:46.113 [DEBUG] betty.leader: stopping routine: routine="secondary CA roots watch" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="intention replication" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="intermediate cert renew watch" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="CA root pruning" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="config entry replication" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="federation state replication" +31:46.113 [DEBUG] betty.leader: stopping routine: routine="federation state anti-entropy" +31:46.113 [WARN] betty.server.serf.lan: serf: Shutdown without a Leave +31:46.113 [DEBUG] betty.leader: stopped routine: routine="intermediate cert renew watch" +31:46.113 [DEBUG] betty.leader: stopped routine: routine="CA root pruning" +31:46.113 [DEBUG] betty.leader: stopped routine: routine="federation state anti-entropy" +31:46.126 [WARN] betty.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.127Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.127Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.127Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.127Z [INFO] agent.router.manager: shutting down +31:46.128 [INFO] betty: consul server down +31:46.128 [INFO] betty: shutdown complete +31:46.128 [INFO] betty: Stopping server: protocol=DNS address=127.0.0.1:30200 network=tcp +31:46.128 [INFO] betty: Stopping server: protocol=DNS address=127.0.0.1:30200 network=udp +31:46.128 [INFO] betty: Stopping server: protocol=HTTP address=127.0.0.1:30201 network=tcp +31:46.128 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" +31:46.128 [ERROR] betty.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" +31:46.128 [DEBUG] betty.leader: stopped routine: routine="intention replication" +31:46.128 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" +31:46.128 [WARN] betty.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" +31:46.128 [INFO] betty.replication.federation_state: stopped replication +31:46.128 [DEBUG] betty.leader: stopped routine: routine="federation state replication" +31:46.128 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" +31:46.128 [INFO] betty.server.replication.config_entry: stopped replication +31:46.128 [DEBUG] betty.leader: stopped routine: routine="config entry replication" +31:46.128 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" +31:46.128 [ERROR] betty.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" +31:46.128 [DEBUG] betty.leader: stopped routine: routine="secondary CA roots watch" +31:46.128 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:30142 datacenter=dc1 method=Internal.ServiceDump error="rpc error making call: EOF" +31:46.628 [INFO] betty: Waiting for endpoints to shut down +31:46.628 [INFO] betty: Endpoints down +2021-10-05T16:31:37.149Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:37.195Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:37.272 [INFO] bob.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:11efacca-32a2-3715-a864-5022a8475258 Address:127.0.0.1:30142}]" +31:37.272 [INFO] bob.server.raft: entering follower state: follower="Node at 127.0.0.1:30142 [Follower]" leader= +31:37.273 [INFO] bob.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 +31:37.273 [INFO] bob.server.serf.lan: serf: EventMemberJoin: bob 127.0.0.1 +2021-10-05T16:31:37.273Z [INFO] agent.router: Initializing LAN area manager +31:37.273 [INFO] bob: Started DNS server: address=127.0.0.1:30130 network=udp +31:37.274 [INFO] bob.server: Adding LAN server: server="bob (Addr: tcp/127.0.0.1:30142) (DC: dc1)" +31:37.274 [INFO] bob.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan +31:37.274 [INFO] bob: Started DNS server: address=127.0.0.1:30130 network=tcp +31:37.274 [INFO] bob: Started HTTP server: address=127.0.0.1:30131 network=tcp +31:37.274 [INFO] bob: started state syncer +31:37.274 [INFO] bob: Started gRPC server: address=127.0.0.1:30073 network=tcp +31:37.317 [WARN] bob.server.raft: heartbeat timeout reached, starting election: last-leader= +31:37.317 [INFO] bob.server.raft: entering candidate state: node="Node at 127.0.0.1:30142 [Candidate]" term=2 +31:37.352 [DEBUG] bob.server.raft: votes: needed=1 +31:37.352 [DEBUG] bob.server.raft: vote granted: from=11efacca-32a2-3715-a864-5022a8475258 term=2 tally=1 +31:37.352 [INFO] bob.server.raft: election won: tally=1 +31:37.352 [INFO] bob.server.raft: entering leader state: leader="Node at 127.0.0.1:30142 [Leader]" +31:37.352 [INFO] bob.server: cluster leadership acquired +31:37.352 [INFO] bob.server: New leader elected: payload=bob +31:37.381 [DEBUG] bob.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30142 +31:37.429 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:37.462 [INFO] bob.server.connect: initialized primary datacenter CA with provider: provider=consul +31:37.462 [INFO] bob.leader: started routine: routine="federation state anti-entropy" +31:37.462 [INFO] bob.leader: started routine: routine="federation state pruning" +31:37.462 [INFO] bob.leader: started routine: routine="intermediate cert renew watch" +31:37.462 [INFO] bob.leader: started routine: routine="CA root pruning" +31:37.462 [DEBUG] bob.server: Skipping self join check for node since the cluster is too small: node=bob +31:37.462 [INFO] bob.server: member joined, marking health alive: member=bob +31:37.478 [INFO] bob.server: federation state anti-entropy synced +31:37.554 [DEBUG] bob: Skipping remote check since it is managed automatically: check=serfHealth +31:37.558 [INFO] bob: Synced node info +31:37.736 [DEBUG] bob: Node info in sync +31:37.748 [INFO] bob: Synced service: service=mesh-gateway +31:37.763 [INFO] bob.server: federation state anti-entropy synced +31:38.138 [DEBUG] bob: Skipping remote check since it is managed automatically: check=serfHealth +31:38.138 [DEBUG] bob: Node info in sync +31:38.138 [DEBUG] bob: Service in sync: service=mesh-gateway +31:38.138 [DEBUG] bob: Node info in sync +31:38.138 [DEBUG] bob: Service in sync: service=mesh-gateway +31:39.276 [INFO] bob.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:30127] local=[127.0.0.1:30127] +31:39.573 [DEBUG] bob.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:39496 +31:39.574 [INFO] bob.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 +31:39.574 [INFO] bob.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan +31:40.066 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 +31:40.133 [DEBUG] bob.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:39514 +31:40.134 [INFO] bob.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 +31:40.134 [INFO] bob.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan +31:40.460 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 +31:41.110 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.460 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.480 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:41.964 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 +31:46.628 [INFO] bob: Requesting shutdown +31:46.628 [INFO] bob.server: shutting down server +31:46.628 [DEBUG] bob.leader: stopping routine: routine="federation state anti-entropy" +31:46.628 [DEBUG] bob.leader: stopping routine: routine="federation state pruning" +31:46.628 [DEBUG] bob.leader: stopping routine: routine="intermediate cert renew watch" +31:46.628 [DEBUG] bob.leader: stopping routine: routine="CA root pruning" +31:46.628 [WARN] bob.server.serf.lan: serf: Shutdown without a Leave +31:46.629 [ERROR] bob.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.629 [DEBUG] bob.leader: stopped routine: routine="federation state anti-entropy" +31:46.629 [DEBUG] bob.leader: stopped routine: routine="federation state pruning" +31:46.629 [DEBUG] bob.leader: stopped routine: routine="intermediate cert renew watch" +31:46.629 [DEBUG] bob.leader: stopped routine: routine="CA root pruning" +31:46.645 [WARN] bob.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.650Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.650Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.650Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.650Z [INFO] agent.router.manager: shutting down +31:46.651 [INFO] bob: consul server down +31:46.651 [INFO] bob: shutdown complete +31:46.651 [INFO] bob: Stopping server: protocol=DNS address=127.0.0.1:30130 network=tcp +31:46.651 [INFO] bob: Stopping server: protocol=DNS address=127.0.0.1:30130 network=udp +31:46.651 [INFO] bob: Stopping server: protocol=HTTP address=127.0.0.1:30131 network=tcp +31:47.151 [INFO] bob: Waiting for endpoints to shut down +31:47.151 [INFO] bob: Endpoints down +--- PASS: TestAgent_JoinWAN_viaMeshGateway (10.04s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc2 (0.07s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc3 (0.14s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc1 (0.01s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc3 (0.11s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc1 (0.01s) + --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc2 (0.10s) +=== CONT TestKVSEndpoint_ListKeys +2021-10-05T16:31:46.345Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.363Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.448 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1eb32792-a765-7db5-bc09-010f2986e680 Address:127.0.0.1:29637}]" +31:46.449 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1eb32792-a765-7db5-bc09-010f2986e680.dc1 127.0.0.1 +31:46.450 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1eb32792-a765-7db5-bc09-010f2986e680 127.0.0.1 +2021-10-05T16:31:46.450Z [INFO] agent.router: Initializing LAN area manager +31:46.450 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29639 network=udp +31:46.450 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29637 [Follower]" leader= +31:46.451 [INFO] TestAgent.server: Adding LAN server: server="Node-1eb32792-a765-7db5-bc09-010f2986e680 (Addr: tcp/127.0.0.1:29637) (DC: dc1)" +31:46.451 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1eb32792-a765-7db5-bc09-010f2986e680.dc1 area=wan +31:46.452 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29639 network=tcp +31:46.453 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:29634 network=tcp +31:46.453 [INFO] TestAgent: started state syncer +31:46.453 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29603 network=tcp +31:46.519 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.519 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29637 [Candidate]" term=2 +31:46.529 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.529 [DEBUG] TestAgent.server.raft: vote granted: from=1eb32792-a765-7db5-bc09-010f2986e680 term=2 tally=1 +31:46.529 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.529 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29637 [Leader]" +31:46.530 [INFO] TestAgent.server: cluster leadership acquired +31:46.530 [INFO] TestAgent.server: New leader elected: payload=Node-1eb32792-a765-7db5-bc09-010f2986e680 +31:46.541 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29637 +31:46.572 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.625 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.625 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.625 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.625 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.625 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.625 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1eb32792-a765-7db5-bc09-010f2986e680 +31:46.625 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1eb32792-a765-7db5-bc09-010f2986e680 +31:46.636 [INFO] TestAgent: Requesting shutdown +31:46.636 [INFO] TestAgent.server: shutting down server +31:46.636 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.636 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:46.644 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.645 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.654Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.654Z [INFO] agent.router.manager: shutting down +31:46.654 [INFO] TestAgent: consul server down +31:46.654 [INFO] TestAgent: shutdown complete +31:46.654 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29639 network=tcp +31:46.654 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29639 network=udp +31:46.654 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:29634 network=tcp +31:47.154 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.154 [INFO] TestAgent: Endpoints down +--- PASS: TestRPC_HTTPSMaxConnsPerClient (1.51s) + --- PASS: TestRPC_HTTPSMaxConnsPerClient/HTTP (0.69s) + --- PASS: TestRPC_HTTPSMaxConnsPerClient/HTTPS (0.82s) +=== CONT TestKVSEndpoint_CAS +[WARN] freeport: 10 out of 41 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:46.400Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.414Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.512 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:35c950ab-f93f-0fdd-3fc9-659432c8fe30 Address:127.0.0.1:29623}]" +31:46.513 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30.dc1 127.0.0.1 +31:46.513 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30 127.0.0.1 +2021-10-05T16:31:46.513Z [INFO] agent.router: Initializing LAN area manager +31:46.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29604 network=udp +31:46.514 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29623 [Follower]" leader= +31:46.514 [INFO] TestAgent.server: Adding LAN server: server="Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30 (Addr: tcp/127.0.0.1:29623) (DC: dc1)" +31:46.514 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30.dc1 area=wan +31:46.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29604 network=tcp +31:46.515 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29619 network=tcp +31:46.515 [INFO] TestAgent: started state syncer +31:46.515 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29645 network=tcp +31:46.572 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.572 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29623 [Candidate]" term=2 +31:46.599 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.599 [DEBUG] TestAgent.server.raft: vote granted: from=35c950ab-f93f-0fdd-3fc9-659432c8fe30 term=2 tally=1 +31:46.599 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.599 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29623 [Leader]" +31:46.600 [INFO] TestAgent.server: cluster leadership acquired +31:46.600 [INFO] TestAgent.server: New leader elected: payload=Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30 +31:46.624 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29623 +31:46.643 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.684 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.684 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.684 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.684 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.684 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.684 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30 +31:46.684 [INFO] TestAgent.server: member joined, marking health alive: member=Node-35c950ab-f93f-0fdd-3fc9-659432c8fe30 +31:46.706 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.902 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:46.924 [INFO] TestAgent: Synced node info +31:46.925 [DEBUG] TestAgent: Node info in sync +31:46.925 [INFO] TestAgent: Requesting shutdown +31:46.925 [INFO] TestAgent.server: shutting down server +31:46.925 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.925 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.925 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.925 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.925 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.925 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.925 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.925 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.925 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.925 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.932 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.935Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.935Z [INFO] agent.router.manager: shutting down +31:46.936 [INFO] TestAgent: consul server down +31:46.936 [INFO] TestAgent: shutdown complete +31:46.936 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29604 network=tcp +31:46.936 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29604 network=udp +31:46.936 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29619 network=tcp +31:47.436 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.436 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsMatch_basic (1.06s) +=== CONT TestKVSEndpoint_DELETE_CAS +2021-10-05T16:31:46.621Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.642Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.691 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:de2daa55-212a-aef8-f32d-640e13fd1b9b Address:127.0.0.1:29595}]" +31:46.692 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29595 [Follower]" leader= +31:46.692 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-de2daa55-212a-aef8-f32d-640e13fd1b9b.dc1 127.0.0.1 +31:46.693 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-de2daa55-212a-aef8-f32d-640e13fd1b9b 127.0.0.1 +2021-10-05T16:31:46.693Z [INFO] agent.router: Initializing LAN area manager +31:46.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29632 network=udp +31:46.693 [INFO] TestAgent.server: Adding LAN server: server="Node-de2daa55-212a-aef8-f32d-640e13fd1b9b (Addr: tcp/127.0.0.1:29595) (DC: dc1)" +31:46.694 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-de2daa55-212a-aef8-f32d-640e13fd1b9b.dc1 area=wan +31:46.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29632 network=tcp +31:46.694 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29591 network=tcp +31:46.694 [INFO] TestAgent: started state syncer +31:46.694 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29680 network=tcp +31:46.751 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.751 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29595 [Candidate]" term=2 +31:46.773 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.773 [DEBUG] TestAgent.server.raft: vote granted: from=de2daa55-212a-aef8-f32d-640e13fd1b9b term=2 tally=1 +31:46.773 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.773 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29595 [Leader]" +31:46.773 [INFO] TestAgent.server: cluster leadership acquired +31:46.773 [INFO] TestAgent.server: New leader elected: payload=Node-de2daa55-212a-aef8-f32d-640e13fd1b9b +31:46.788 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29595 +31:46.808 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.880 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.880 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.880 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.880 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.880 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.881 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-de2daa55-212a-aef8-f32d-640e13fd1b9b +31:46.881 [INFO] TestAgent.server: member joined, marking health alive: member=Node-de2daa55-212a-aef8-f32d-640e13fd1b9b +31:46.904 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:46.921 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.921 [INFO] TestAgent: Synced node info +31:46.954 [INFO] TestAgent: Requesting shutdown +31:46.954 [INFO] TestAgent.server: shutting down server +31:46.954 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.954 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.954 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.954 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.954 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.954 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.954 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.954 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.954 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.954 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.959 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.965Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.965Z [INFO] agent.router.manager: shutting down +31:46.965 [INFO] TestAgent: consul server down +31:46.965 [INFO] TestAgent: shutdown complete +31:46.965 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29632 network=tcp +31:46.965 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29632 network=udp +31:46.965 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29591 network=tcp +31:47.467 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.467 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_InmemKeyrings/server_with_keys +2021-10-05T16:31:46.651Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.668Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.756 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:78ee195b-b7f0-bd9f-db4d-8582c4585f37 Address:127.0.0.1:29665}]" +31:46.756 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29665 [Follower]" leader= +31:46.757 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37.dc1 127.0.0.1 +31:46.757 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37 127.0.0.1 +2021-10-05T16:31:46.757Z [INFO] agent.router: Initializing LAN area manager +31:46.757 [INFO] TestAgent.server: Adding LAN server: server="Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37 (Addr: tcp/127.0.0.1:29665) (DC: dc1)" +31:46.757 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37.dc1 area=wan +31:46.757 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29681 network=udp +31:46.757 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29681 network=tcp +31:46.758 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29661 network=tcp +31:46.758 [INFO] TestAgent: started state syncer +31:46.758 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29687 network=tcp +31:46.820 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.824 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29665 [Candidate]" term=2 +31:46.846 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.846 [DEBUG] TestAgent.server.raft: vote granted: from=78ee195b-b7f0-bd9f-db4d-8582c4585f37 term=2 tally=1 +31:46.846 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.846 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29665 [Leader]" +31:46.846 [INFO] TestAgent.server: cluster leadership acquired +31:46.847 [INFO] TestAgent.server: New leader elected: payload=Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37 +31:46.858 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29665 +31:46.883 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.924 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.924 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.924 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.924 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.924 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.924 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37 +31:46.924 [INFO] TestAgent.server: member joined, marking health alive: member=Node-78ee195b-b7f0-bd9f-db4d-8582c4585f37 +31:46.956 [INFO] TestAgent.server: federation state anti-entropy synced +31:46.981 [INFO] TestAgent: Requesting shutdown +31:46.981 [INFO] TestAgent.server: shutting down server +31:46.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.981 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.981 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.981 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.981 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.981 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:46.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.981 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.981 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.989 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.996Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.996Z [INFO] agent.router.manager: shutting down +31:46.996 [INFO] TestAgent: consul server down +31:46.996 [INFO] TestAgent: shutdown complete +31:46.996 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29681 network=tcp +31:46.996 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29681 network=udp +31:46.996 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29661 network=tcp +31:47.500 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.500 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_GET_Raw (0.86s) +=== CONT TestKVSEndpoint_Recurse +2021-10-05T16:31:46.481Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.528Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.621 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:44b8c83b-d92d-4a63-f97d-1b0e3763898b Address:127.0.0.1:29658}]" +31:46.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b.dc1 127.0.0.1 +31:46.623 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b 127.0.0.1 +2021-10-05T16:31:46.623Z [INFO] agent.router: Initializing LAN area manager +31:46.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29646 network=udp +31:46.623 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29658 [Follower]" leader= +31:46.623 [INFO] TestAgent.server: Adding LAN server: server="Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b (Addr: tcp/127.0.0.1:29658) (DC: dc1)" +31:46.624 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b.dc1 area=wan +31:46.624 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29646 network=tcp +31:46.624 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29654 network=tcp +31:46.624 [INFO] TestAgent: started state syncer +31:46.624 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29652 network=tcp +31:46.685 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.685 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29658 [Candidate]" term=2 +31:46.703 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.703 [DEBUG] TestAgent.server.raft: vote granted: from=44b8c83b-d92d-4a63-f97d-1b0e3763898b term=2 tally=1 +31:46.703 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.703 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29658 [Leader]" +31:46.703 [INFO] TestAgent.server: cluster leadership acquired +31:46.704 [INFO] TestAgent.server: New leader elected: payload=Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b +31:46.711 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29658 +31:46.725 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.751 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.751 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.751 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.751 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.751 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.751 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b +31:46.751 [INFO] TestAgent.server: member joined, marking health alive: member=Node-44b8c83b-d92d-4a63-f97d-1b0e3763898b +31:46.772 [INFO] TestAgent: Synced node info +31:46.772 [DEBUG] TestAgent: Node info in sync +31:46.777 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.004 [INFO] TestAgent: Requesting shutdown +31:47.004 [INFO] TestAgent.server: shutting down server +31:47.004 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.004 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.004 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.004 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.004 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.004 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.011 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.023Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.023Z [INFO] agent.router.manager: shutting down +31:47.023 [INFO] TestAgent: consul server down +31:47.023 [INFO] TestAgent: shutdown complete +31:47.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29646 network=tcp +31:47.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29646 network=udp +31:47.024 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29654 network=tcp +31:47.524 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.524 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsList_values (1.08s) +=== CONT TestKVSEndpoint_PUT_GET_DELETE +2021-10-05T16:31:46.631Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.649Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.720 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b3bcc6c-6522-e0a5-5300-ca21a6b4630c Address:127.0.0.1:29679}]" +31:46.720 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29679 [Follower]" leader= +31:46.724 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c.dc1 127.0.0.1 +31:46.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c 127.0.0.1 +2021-10-05T16:31:46.725Z [INFO] agent.router: Initializing LAN area manager +31:46.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29653 network=udp +31:46.725 [INFO] TestAgent.server: Adding LAN server: server="Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c (Addr: tcp/127.0.0.1:29679) (DC: dc1)" +31:46.725 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c.dc1 area=wan +31:46.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29653 network=tcp +31:46.726 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29675 network=tcp +31:46.726 [INFO] TestAgent: started state syncer +31:46.726 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29631 network=tcp +31:46.786 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.786 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29679 [Candidate]" term=2 +31:46.800 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.800 [DEBUG] TestAgent.server.raft: vote granted: from=4b3bcc6c-6522-e0a5-5300-ca21a6b4630c term=2 tally=1 +31:46.800 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.800 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29679 [Leader]" +31:46.800 [INFO] TestAgent.server: cluster leadership acquired +31:46.800 [INFO] TestAgent.server: New leader elected: payload=Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c +31:46.811 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29679 +31:46.845 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:46.888 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:46.888 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:46.888 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:46.888 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:46.888 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:46.888 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c +31:46.888 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b3bcc6c-6522-e0a5-5300-ca21a6b4630c +31:46.924 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.038 [INFO] TestAgent: Requesting shutdown +31:47.038 [INFO] TestAgent.server: shutting down server +31:47.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.038 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.038 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.038 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.038 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.039 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.039 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:47.039 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.039 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.039 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.056Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.056Z [INFO] agent.router.manager: shutting down +31:47.056 [INFO] TestAgent: consul server down +31:47.056 [INFO] TestAgent: shutdown complete +31:47.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29653 network=tcp +31:47.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29653 network=udp +31:47.056 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29675 network=tcp +31:47.559 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.559 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsList_empty (0.98s) +=== CONT TestAgentKeyring_ACL +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:46.797Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.815Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.900 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9fe1b291-6618-3398-79ce-cdf7d43e0bde Address:127.0.0.1:29644}]" +31:46.900 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29644 [Follower]" leader= +31:46.900 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde.dc1 127.0.0.1 +31:46.901 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde 127.0.0.1 +2021-10-05T16:31:46.901Z [INFO] agent.router: Initializing LAN area manager +31:46.901 [INFO] TestAgent.server: Adding LAN server: server="Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde (Addr: tcp/127.0.0.1:29644) (DC: dc1)" +31:46.901 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde.dc1 area=wan +31:46.901 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29688 network=udp +31:46.901 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29688 network=tcp +31:46.902 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29640 network=tcp +31:46.902 [INFO] TestAgent: started state syncer +31:46.902 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29694 network=tcp +31:46.965 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.965 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29644 [Candidate]" term=2 +31:46.983 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:46.983 [DEBUG] TestAgent.server.raft: vote granted: from=9fe1b291-6618-3398-79ce-cdf7d43e0bde term=2 tally=1 +31:46.983 [INFO] TestAgent.server.raft: election won: tally=1 +31:46.983 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29644 [Leader]" +31:46.983 [INFO] TestAgent.server: cluster leadership acquired +31:46.983 [INFO] TestAgent.server: New leader elected: payload=Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde +31:46.996 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29644 +31:47.022 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde +31:47.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9fe1b291-6618-3398-79ce-cdf7d43e0bde +31:47.076 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.142 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:47.145 [INFO] TestAgent: Synced node info +31:47.145 [DEBUG] TestAgent: Node info in sync +31:47.323 [INFO] TestAgent: Requesting shutdown +31:47.323 [INFO] TestAgent.server: shutting down server +31:47.323 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.323 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.323 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.323 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.323 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.323 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.323 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.323 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.323 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.323 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.332 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.340Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.340Z [INFO] agent.router.manager: shutting down +31:47.340 [INFO] TestAgent: consul server down +31:47.340 [INFO] TestAgent: shutdown complete +31:47.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29688 network=tcp +31:47.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29688 network=udp +31:47.340 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29640 network=tcp +31:47.841 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.841 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_AcquireRelease (1.06s) +=== CONT TestAgent_InitKeyring +--- PASS: TestAgent_InitKeyring (0.00s) +=== CONT TestIntentionsSpecificGet_good +2021-10-05T16:31:47.170Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.185Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.242 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:512e7810-743a-29a5-706f-dd51462babed Address:127.0.0.1:29700}]" +31:47.242 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29700 [Follower]" leader= +31:47.243 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-512e7810-743a-29a5-706f-dd51462babed.dc1 127.0.0.1 +31:47.243 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-512e7810-743a-29a5-706f-dd51462babed 127.0.0.1 +2021-10-05T16:31:47.243Z [INFO] agent.router: Initializing LAN area manager +31:47.244 [INFO] TestAgent.server: Adding LAN server: server="Node-512e7810-743a-29a5-706f-dd51462babed (Addr: tcp/127.0.0.1:29700) (DC: dc1)" +31:47.244 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-512e7810-743a-29a5-706f-dd51462babed.dc1 area=wan +31:47.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29695 network=udp +31:47.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29695 network=tcp +31:47.244 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29696 network=tcp +31:47.245 [INFO] TestAgent: started state syncer +31:47.245 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29701 network=tcp +31:47.294 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.294 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29700 [Candidate]" term=2 +31:47.313 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.313 [DEBUG] TestAgent.server.raft: vote granted: from=512e7810-743a-29a5-706f-dd51462babed term=2 tally=1 +31:47.313 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.313 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29700 [Leader]" +31:47.313 [INFO] TestAgent.server: cluster leadership acquired +31:47.313 [INFO] TestAgent.server: New leader elected: payload=Node-512e7810-743a-29a5-706f-dd51462babed +31:47.323 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29700 +31:47.345 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.374 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.374 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.374 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.374 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.374 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.374 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-512e7810-743a-29a5-706f-dd51462babed +31:47.374 [INFO] TestAgent.server: member joined, marking health alive: member=Node-512e7810-743a-29a5-706f-dd51462babed +31:47.402 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.425 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:47.431 [INFO] TestAgent: Synced node info +31:47.431 [DEBUG] TestAgent: Node info in sync +31:47.458 [INFO] TestAgent: Requesting shutdown +31:47.458 [INFO] TestAgent.server: shutting down server +31:47.458 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.458 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.458 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.458 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.458 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.458 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.458 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.458 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.458 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.458 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.468 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.480Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.480Z [INFO] agent.router.manager: shutting down +31:47.480 [INFO] TestAgent: consul server down +31:47.480 [INFO] TestAgent: shutdown complete +31:47.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29695 network=tcp +31:47.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29695 network=udp +31:47.480 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29696 network=tcp +31:47.981 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.981 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_ListKeys (0.83s) +=== CONT TestAgent_LoadKeyrings +=== RUN TestAgent_LoadKeyrings/no_keys +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:47.171Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.185Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.231 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8fceaa5-b17b-d3cc-cb2f-4625a2219042 Address:127.0.0.1:29728}]" +31:47.231 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29728 [Follower]" leader= +31:47.231 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042.dc1 127.0.0.1 +31:47.232 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042 127.0.0.1 +2021-10-05T16:31:47.232Z [INFO] agent.router: Initializing LAN area manager +31:47.232 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29702 network=udp +31:47.233 [INFO] TestAgent.server: Adding LAN server: server="Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042 (Addr: tcp/127.0.0.1:29728) (DC: dc1)" +31:47.233 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042.dc1 area=wan +31:47.233 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29702 network=tcp +31:47.233 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29724 network=tcp +31:47.233 [INFO] TestAgent: started state syncer +31:47.234 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29596 network=tcp +31:47.276 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.276 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29728 [Candidate]" term=2 +31:47.288 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.288 [DEBUG] TestAgent.server.raft: vote granted: from=b8fceaa5-b17b-d3cc-cb2f-4625a2219042 term=2 tally=1 +31:47.288 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.288 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29728 [Leader]" +31:47.288 [INFO] TestAgent.server: cluster leadership acquired +31:47.288 [INFO] TestAgent.server: New leader elected: payload=Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042 +31:47.316 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29728 +31:47.364 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.417 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.417 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.417 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.417 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.417 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.417 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042 +31:47.417 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8fceaa5-b17b-d3cc-cb2f-4625a2219042 +31:47.440 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.440 [INFO] TestAgent: Synced node info +31:47.596 [INFO] TestAgent: Requesting shutdown +31:47.596 [INFO] TestAgent.server: shutting down server +31:47.596 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.596 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.596 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.596 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.596 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.596 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.596 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.597 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.597 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.597 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.603 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.608Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.608Z [INFO] agent.router.manager: shutting down +31:47.608 [INFO] TestAgent: consul server down +31:47.608 [INFO] TestAgent: shutdown complete +31:47.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29702 network=tcp +31:47.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29702 network=udp +31:47.608 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29724 network=tcp +31:48.109 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.109 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_CAS (0.96s) +=== CONT TestIntentionsSpecificDelete_good [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:15.015Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.030Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.058 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b3c2f8ff-da86-66c6-a28c-3f70d12258cb Address:127.0.0.1:11193}]" -46:15.058 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11193 [Follower]" leader= -46:15.059 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb.dc1 127.0.0.1 -46:15.059 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb 127.0.0.1 -2022-11-07T22:46:15.059Z [INFO] agent.router: Initializing LAN area manager -46:15.060 [INFO] TestAgent.server: Adding LAN server: server="Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb (Addr: tcp/127.0.0.1:11193) (DC: dc1)" -46:15.060 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb.dc1 area=wan -46:15.060 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11055 network=udp -46:15.060 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11055 network=tcp -46:15.060 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11056 network=tcp -46:15.060 [INFO] TestAgent: started state syncer -46:15.060 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11194 network=tcp -46:15.126 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.126 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11193 [Candidate]" term=2 -46:15.138 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.138 [DEBUG] TestAgent.server.raft: vote granted: from=b3c2f8ff-da86-66c6-a28c-3f70d12258cb term=2 tally=1 -46:15.138 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11193 [Leader]" -46:15.138 [INFO] TestAgent.server: cluster leadership acquired -46:15.138 [INFO] TestAgent.server: New leader elected: payload=Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb -46:15.143 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11193 -46:15.161 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.190 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.190 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.190 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.190 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.190 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.190 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb -46:15.190 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b3c2f8ff-da86-66c6-a28c-3f70d12258cb -46:15.197 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.328 [INFO] TestAgent: Requesting shutdown -46:15.328 [INFO] TestAgent.server: shutting down server -46:15.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.328 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.328 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.328 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.328 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.328 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.328 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.328 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.334 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.341Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.341Z [INFO] agent.router.manager: shutting down -46:15.341 [INFO] TestAgent: consul server down -46:15.341 [INFO] TestAgent: shutdown complete -46:15.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11055 network=tcp -46:15.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11055 network=udp -46:15.341 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11056 network=tcp -46:15.841 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.841 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_Decoding (0.84s) - --- PASS: TestConfig_Apply_Decoding/No_Kind (0.00s) - --- PASS: TestConfig_Apply_Decoding/Kind_Not_String (0.00s) - --- PASS: TestConfig_Apply_Decoding/Lowercase_kind (0.00s) -=== CONT TestConfig_Apply_TerminatingGateway -2022-11-07T22:46:14.930Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.940Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.975 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:59624fc3-647a-d116-3191-156a7771ec81 Address:127.0.0.1:11053}]" -46:14.975 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11053 [Follower]" leader= -46:14.976 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-59624fc3-647a-d116-3191-156a7771ec81.dc1 127.0.0.1 -46:14.976 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-59624fc3-647a-d116-3191-156a7771ec81 127.0.0.1 -2022-11-07T22:46:14.976Z [INFO] agent.router: Initializing LAN area manager -46:14.976 [INFO] TestAgent.server: Adding LAN server: server="Node-59624fc3-647a-d116-3191-156a7771ec81 (Addr: tcp/127.0.0.1:11053) (DC: dc1)" -46:14.976 [DEBUG] TestAgent.dns: recursor enabled -46:14.976 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-59624fc3-647a-d116-3191-156a7771ec81.dc1 area=wan -46:14.976 [DEBUG] TestAgent.dns: recursor enabled -46:14.976 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11076 network=udp -46:14.976 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11076 network=tcp -46:14.977 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11077 network=tcp -46:14.977 [INFO] TestAgent: started state syncer -46:14.977 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11054 network=tcp -46:15.032 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.032 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11053 [Candidate]" term=2 -46:15.045 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.045 [DEBUG] TestAgent.server.raft: vote granted: from=59624fc3-647a-d116-3191-156a7771ec81 term=2 tally=1 -46:15.045 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.045 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11053 [Leader]" -46:15.045 [INFO] TestAgent.server: cluster leadership acquired -46:15.045 [INFO] TestAgent.server: New leader elected: payload=Node-59624fc3-647a-d116-3191-156a7771ec81 -46:15.054 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11053 -46:15.065 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.086 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.086 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.086 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.086 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.086 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.086 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-59624fc3-647a-d116-3191-156a7771ec81 -46:15.086 [INFO] TestAgent.server: member joined, marking health alive: member=Node-59624fc3-647a-d116-3191-156a7771ec81 -46:15.104 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.161 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.164 [INFO] TestAgent: Synced node info -46:15.375 [DEBUG] TestAgent.dns: cname recurse RTT for name: name=www.google.com. rtt=155.198µs -46:15.375 [DEBUG] TestAgent.dns: request served from client: name=google.node.consul. type=ANY class=IN latency=302.958µs client=127.0.0.1:33352 client_network=udp -46:15.375 [INFO] TestAgent: Requesting shutdown -46:15.375 [INFO] TestAgent.server: shutting down server -46:15.375 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.375 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.375 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.375 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.375 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.375 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.375 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.375 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.375 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.375 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.382 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.393Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.393Z [INFO] agent.router.manager: shutting down -46:15.393 [INFO] TestAgent: consul server down -46:15.393 [INFO] TestAgent: shutdown complete -46:15.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11076 network=tcp -46:15.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11076 network=udp -46:15.393 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11077 network=tcp -46:15.893 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.893 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_CNAME (0.98s) +2021-10-05T16:31:47.521Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.536Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.573 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 Address:127.0.0.1:29721}]" +31:47.574 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29721 [Follower]" leader= +31:47.574 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659.dc1 127.0.0.1 +31:47.575 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 127.0.0.1 +2021-10-05T16:31:47.575Z [INFO] agent.router: Initializing LAN area manager +31:47.575 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29716 network=udp +31:47.575 [INFO] TestAgent.server: Adding LAN server: server="Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 (Addr: tcp/127.0.0.1:29721) (DC: dc1)" +31:47.575 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659.dc1 area=wan +31:47.575 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29716 network=tcp +31:47.576 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29717 network=tcp +31:47.577 [INFO] TestAgent: started state syncer +31:47.577 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29743 network=tcp +31:47.624 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.624 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29721 [Candidate]" term=2 +31:47.664 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.664 [DEBUG] TestAgent.server.raft: vote granted: from=e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 term=2 tally=1 +31:47.664 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.664 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29721 [Leader]" +31:47.665 [INFO] TestAgent.server: cluster leadership acquired +31:47.665 [INFO] TestAgent.server: New leader elected: payload=Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 +31:47.682 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29721 +31:47.736 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.861 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.861 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.861 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.861 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.861 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.861 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 +31:47.861 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 +31:47.876 [INFO] TestAgent: Requesting shutdown +31:47.876 [INFO] TestAgent.server: shutting down server +31:47.876 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.876 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.876 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.876 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.876 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.876 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:47.876 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.876 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.876 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.883 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.890Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.890Z [INFO] agent.router.manager: shutting down +31:47.890 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: raft is already shutdown" +31:47.890 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.896 [INFO] TestAgent: consul server down +31:47.896 [INFO] TestAgent: shutdown complete +31:47.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29716 network=tcp +31:47.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29716 network=udp +31:47.896 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29717 network=tcp +31:47.896 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 127.0.0.1 29719 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:e5c6dc1c-ae7b-a6d1-e07c-38e7fcfb1659 port:29721 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29720] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:48.400 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.400 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_Recurse (0.90s) +=== CONT TestIntentionsSpecificUpdate_good +2021-10-05T16:31:47.485Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.503Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.557 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf768d51-ebaf-a2eb-4497-633fcc763c2b Address:127.0.0.1:29714}]" +31:47.558 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b.dc1 127.0.0.1 +31:47.558 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b 127.0.0.1 +2021-10-05T16:31:47.558Z [INFO] agent.router: Initializing LAN area manager +31:47.558 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29737 network=udp +31:47.558 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29714 [Follower]" leader= +31:47.559 [INFO] TestAgent.server: Adding LAN server: server="Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b (Addr: tcp/127.0.0.1:29714) (DC: dc1)" +31:47.559 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b.dc1 area=wan +31:47.559 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29737 network=tcp +31:47.559 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29710 network=tcp +31:47.559 [INFO] TestAgent: started state syncer +31:47.559 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29715 network=tcp +31:47.624 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.624 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29714 [Candidate]" term=2 +31:47.660 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.660 [DEBUG] TestAgent.server.raft: vote granted: from=bf768d51-ebaf-a2eb-4497-633fcc763c2b term=2 tally=1 +31:47.660 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.660 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29714 [Leader]" +31:47.672 [INFO] TestAgent.server: cluster leadership acquired +31:47.672 [INFO] TestAgent.server: New leader elected: payload=Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b +31:47.691 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29714 +31:47.739 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.784 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.784 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.784 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.784 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.784 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.784 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b +31:47.784 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bf768d51-ebaf-a2eb-4497-633fcc763c2b +31:47.808 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.920 [INFO] TestAgent: Requesting shutdown +31:47.920 [INFO] TestAgent.server: shutting down server +31:47.920 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.920 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.920 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.920 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.920 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:47.920 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.920 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.932 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.940Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.940Z [INFO] agent.router.manager: shutting down +31:47.940 [INFO] TestAgent: consul server down +31:47.940 [INFO] TestAgent: shutdown complete +31:47.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29737 network=tcp +31:47.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29737 network=udp +31:47.940 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29710 network=tcp +31:48.444 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.444 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_InmemKeyrings/client_with_keys +2021-10-05T16:31:47.456Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.467Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.525 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ac5a99f9-6868-075c-809d-497a86c27e50 Address:127.0.0.1:29693}]" +31:47.526 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29693 [Follower]" leader= +31:47.527 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ac5a99f9-6868-075c-809d-497a86c27e50.dc1 127.0.0.1 +31:47.528 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ac5a99f9-6868-075c-809d-497a86c27e50 127.0.0.1 +2021-10-05T16:31:47.528Z [INFO] agent.router: Initializing LAN area manager +31:47.528 [INFO] TestAgent.server: Adding LAN server: server="Node-ac5a99f9-6868-075c-809d-497a86c27e50 (Addr: tcp/127.0.0.1:29693) (DC: dc1)" +31:47.528 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29597 network=udp +31:47.528 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ac5a99f9-6868-075c-809d-497a86c27e50.dc1 area=wan +31:47.528 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29597 network=tcp +31:47.529 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29689 network=tcp +31:47.529 [INFO] TestAgent: started state syncer +31:47.529 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29736 network=tcp +31:47.563 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.563 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29693 [Candidate]" term=2 +31:47.594 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.594 [DEBUG] TestAgent.server.raft: vote granted: from=ac5a99f9-6868-075c-809d-497a86c27e50 term=2 tally=1 +31:47.594 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.594 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29693 [Leader]" +31:47.594 [INFO] TestAgent.server: cluster leadership acquired +31:47.594 [INFO] TestAgent.server: New leader elected: payload=Node-ac5a99f9-6868-075c-809d-497a86c27e50 +31:47.604 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29693 +31:47.657 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.727 [INFO] TestAgent: Synced node info +31:47.727 [DEBUG] TestAgent: Node info in sync +31:47.748 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.748 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.748 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.748 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.748 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.748 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ac5a99f9-6868-075c-809d-497a86c27e50 +31:47.748 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ac5a99f9-6868-075c-809d-497a86c27e50 +31:47.780 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.947 [INFO] TestAgent: Requesting shutdown +31:47.947 [INFO] TestAgent.server: shutting down server +31:47.947 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.947 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.947 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.947 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.947 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.947 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.947 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.947 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.947 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.947 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.955 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.965Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.965Z [INFO] agent.router.manager: shutting down +31:47.965 [INFO] TestAgent: consul server down +31:47.965 [INFO] TestAgent: shutdown complete +31:47.965 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29597 network=tcp +31:47.965 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29597 network=udp +31:47.965 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29689 network=tcp +31:48.466 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.466 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_DELETE_CAS (1.03s) +=== CONT TestIntentionsSpecificGet_invalidId +2021-10-05T16:31:47.598Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:47.598Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.663Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:47.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.742 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:453fa430-bd5a-9d86-c54e-6b39b7c75719 Address:127.0.0.1:29749}]" +31:47.743 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 127.0.0.1 +31:47.743 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29749 [Follower]" leader= +31:47.745 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 127.0.0.1 +2021-10-05T16:31:47.745Z [INFO] agent.router: Initializing LAN area manager +31:47.745 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29730 network=udp +31:47.745 [INFO] TestAgent.server: Adding LAN server: server="Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 (Addr: tcp/127.0.0.1:29749) (DC: dc1)" +31:47.745 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 area=wan +31:47.745 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29730 network=tcp +31:47.746 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29745 network=tcp +31:47.746 [INFO] TestAgent: started state syncer +31:47.746 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29757 network=tcp +31:47.796 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:47.810 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.811 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29749 [Candidate]" term=2 +31:47.826 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.826 [DEBUG] TestAgent.server.raft: vote granted: from=453fa430-bd5a-9d86-c54e-6b39b7c75719 term=2 tally=1 +31:47.826 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.826 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29749 [Leader]" +31:47.826 [INFO] TestAgent.server: cluster leadership acquired +31:47.826 [INFO] TestAgent.server: New leader elected: payload=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.832 [INFO] TestAgent.server: initializing acls +31:47.835 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:47.835 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:47.841 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:47.856 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:47.856 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:47.856 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:47.856 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.856 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 +31:47.856 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 area=wan +31:47.857 [INFO] TestAgent.server: Updating LAN server: server="Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 (Addr: tcp/127.0.0.1:29749) (DC: dc1)" +31:47.868 [INFO] TestAgent: Synced node info +31:47.881 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.916 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.916 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.916 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.916 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.916 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.916 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.916 [INFO] TestAgent.server: member joined, marking health alive: member=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.943 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.948 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.949 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +31:47.949 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 +31:47.949 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:47.949 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.949 [INFO] TestAgent.server.serf.wan: serf: Received install-key query +31:47.950 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 +31:47.950 [INFO] TestAgent.server.serf.lan: serf: Received install-key query +31:47.950 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.950 [INFO] TestAgent.server.serf.wan: serf: Received use-key query +31:47.951 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 +31:47.951 [INFO] TestAgent.server.serf.lan: serf: Received use-key query +31:47.951 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.951 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query +31:47.953 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719.dc1 +31:47.953 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query +31:47.953 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-453fa430-bd5a-9d86-c54e-6b39b7c75719 +31:47.953 [INFO] TestAgent: Requesting shutdown +31:47.953 [INFO] TestAgent.server: shutting down server +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.953 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.953 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.953 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:47.953 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:47.953 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.953 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.953 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.956 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.956 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.962 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.972Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.972Z [INFO] agent.router.manager: shutting down +31:47.972 [INFO] TestAgent: consul server down +31:47.972 [INFO] TestAgent: shutdown complete +31:47.972 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29730 network=tcp +31:47.972 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29730 network=udp +31:47.972 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29745 network=tcp +31:48.476 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.476 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentKeyring_ACL (0.91s) +=== CONT TestParseConsistency_Invalid +2021-10-05T16:31:47.565Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.584Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.644 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:801ac18c-4393-70a8-4068-576ea898ae01 Address:127.0.0.1:29756}]" +31:47.645 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-801ac18c-4393-70a8-4068-576ea898ae01.dc1 127.0.0.1 +31:47.646 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-801ac18c-4393-70a8-4068-576ea898ae01 127.0.0.1 +2021-10-05T16:31:47.646Z [INFO] agent.router: Initializing LAN area manager +31:47.646 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29744 network=udp +31:47.646 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29756 [Follower]" leader= +31:47.648 [INFO] TestAgent.server: Adding LAN server: server="Node-801ac18c-4393-70a8-4068-576ea898ae01 (Addr: tcp/127.0.0.1:29756) (DC: dc1)" +31:47.649 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-801ac18c-4393-70a8-4068-576ea898ae01.dc1 area=wan +31:47.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29744 network=tcp +31:47.652 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29752 network=tcp +31:47.652 [INFO] TestAgent: started state syncer +31:47.654 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29729 network=tcp +31:47.712 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.712 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29756 [Candidate]" term=2 +31:47.760 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.760 [DEBUG] TestAgent.server.raft: vote granted: from=801ac18c-4393-70a8-4068-576ea898ae01 term=2 tally=1 +31:47.760 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.760 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29756 [Leader]" +31:47.760 [INFO] TestAgent.server: cluster leadership acquired +31:47.760 [INFO] TestAgent.server: New leader elected: payload=Node-801ac18c-4393-70a8-4068-576ea898ae01 +31:47.792 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29756 +31:47.878 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.940 [INFO] TestAgent: Synced node info +31:47.961 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.961 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.961 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.961 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.961 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.961 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-801ac18c-4393-70a8-4068-576ea898ae01 +31:47.961 [INFO] TestAgent.server: member joined, marking health alive: member=Node-801ac18c-4393-70a8-4068-576ea898ae01 +31:47.996 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.004 [INFO] TestAgent: Requesting shutdown +31:48.004 [INFO] TestAgent.server: shutting down server +31:48.004 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.004 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.004 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.004 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.004 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.004 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.012 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.020Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.020Z [INFO] agent.router.manager: shutting down +31:48.020 [INFO] TestAgent: consul server down +31:48.020 [INFO] TestAgent: shutdown complete +31:48.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29744 network=tcp +31:48.020 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29744 network=udp +31:48.020 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29752 network=tcp +31:48.520 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.520 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_PUT_GET_DELETE (1.00s) +=== CONT TestHTTPServer_HandshakeTimeout +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:47.856Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:47.869Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:47.932 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ad15b60-363c-8559-e1d5-bfe5f06f1983 Address:127.0.0.1:29707}]" +31:47.932 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29707 [Follower]" leader= +31:47.933 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983.dc1 127.0.0.1 +31:47.933 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983 127.0.0.1 +2021-10-05T16:31:47.933Z [INFO] agent.router: Initializing LAN area manager +31:47.933 [INFO] TestAgent.server: Adding LAN server: server="Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983 (Addr: tcp/127.0.0.1:29707) (DC: dc1)" +31:47.933 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983.dc1 area=wan +31:47.934 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29758 network=udp +31:47.934 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29758 network=tcp +31:47.934 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29703 network=tcp +31:47.934 [INFO] TestAgent: started state syncer +31:47.934 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29673 network=tcp +31:47.980 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:47.980 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29707 [Candidate]" term=2 +31:48.028 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.028 [DEBUG] TestAgent.server.raft: vote granted: from=4ad15b60-363c-8559-e1d5-bfe5f06f1983 term=2 tally=1 +31:48.028 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.028 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29707 [Leader]" +31:48.028 [INFO] TestAgent.server: cluster leadership acquired +31:48.028 [INFO] TestAgent.server: New leader elected: payload=Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983 +31:48.064 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29707 +31:48.108 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.146 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.146 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.146 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.146 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.146 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.146 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983 +31:48.146 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4ad15b60-363c-8559-e1d5-bfe5f06f1983 +31:48.166 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.344 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:48.356 [INFO] TestAgent: Synced node info +31:48.356 [DEBUG] TestAgent: Node info in sync +31:48.373 [INFO] TestAgent: Requesting shutdown +31:48.373 [INFO] TestAgent.server: shutting down server +31:48.373 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.373 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.373 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.373 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.373 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.373 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.373 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.373 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.373 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.373 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.384 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.391Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.391Z [INFO] agent.router.manager: shutting down +31:48.391 [INFO] TestAgent: consul server down +31:48.391 [INFO] TestAgent: shutdown complete +31:48.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29758 network=tcp +31:48.391 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29758 network=udp +31:48.391 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29703 network=tcp +31:48.891 [INFO] TestAgent: Waiting for endpoints to shut down +31:48.891 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsSpecificGet_good (1.05s) +=== CONT TestEnableWebUI +2021/10/05 16:31:48 http: TLS handshake error from 127.0.0.1:59194: read tcp 127.0.0.1:29823->127.0.0.1:59194: i/o timeout +31:48.518 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-a619c316-37a3-ae1c-971b-5cab01522175 127.0.0.1 +2021-10-05T16:31:48.518Z [INFO] agent.router: Initializing LAN area manager +31:48.519 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29751 network=udp +31:48.519 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29751 network=tcp +31:48.519 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29759 network=tcp +31:48.519 [INFO] TestAgent: started state syncer +31:48.520 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29785 network=tcp +2021-10-05T16:31:48.520Z [WARN] agent.router.manager: No servers available +31:48.520 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:48.533 [INFO] TestAgent: Requesting shutdown +31:48.533 [INFO] TestAgent.client: shutting down client +31:48.533 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:48.540 [INFO] TestAgent: consul client down +31:48.540 [INFO] TestAgent: shutdown complete +31:48.540 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29751 network=tcp +31:48.540 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29751 network=udp +31:48.540 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29759 network=tcp +31:49.040 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.040 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_InmemKeyrings/ignore_files +2021-10-05T16:31:48.143Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.178Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.215 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 Address:127.0.0.1:29672}]" +31:48.215 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05.dc1 127.0.0.1 +31:48.216 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29672 [Follower]" leader= +31:48.216 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 127.0.0.1 +2021-10-05T16:31:48.216Z [INFO] agent.router: Initializing LAN area manager +31:48.216 [INFO] TestAgent.server: Adding LAN server: server="Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 (Addr: tcp/127.0.0.1:29672) (DC: dc1)" +31:48.216 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05.dc1 area=wan +31:48.216 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29709 network=udp +31:48.216 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29709 network=tcp +31:48.217 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29668 network=tcp +31:48.217 [INFO] TestAgent: started state syncer +31:48.217 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29764 network=tcp +31:48.284 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.284 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29672 [Candidate]" term=2 +31:48.304 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.304 [DEBUG] TestAgent.server.raft: vote granted: from=4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 term=2 tally=1 +31:48.304 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.304 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29672 [Leader]" +31:48.304 [INFO] TestAgent.server: cluster leadership acquired +31:48.304 [INFO] TestAgent.server: New leader elected: payload=Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 +31:48.320 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29672 +31:48.354 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.409 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.409 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.409 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.409 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.409 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.409 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 +31:48.409 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a8cfc0a-1284-1f7f-a99b-438b89e1ae05 +31:48.449 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.534 [INFO] TestAgent: Requesting shutdown +31:48.534 [INFO] TestAgent.server: shutting down server +31:48.534 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.534 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.534 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.534 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.534 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:48.534 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.534 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.544 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.565Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.565Z [INFO] agent.router.manager: shutting down +31:48.565 [INFO] TestAgent: consul server down +31:48.565 [INFO] TestAgent: shutdown complete +31:48.565 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29709 network=tcp +31:48.565 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29709 network=udp +31:48.565 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29668 network=tcp +31:49.068 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.068 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsSpecificDelete_good (0.96s) +=== CONT TestACLResolution +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:48.027Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.065Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.128 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:575edd4a-8125-3caa-2e3e-169842ab026d Address:127.0.0.1:29742}]" +31:48.128 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29742 [Follower]" leader= +31:48.129 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-575edd4a-8125-3caa-2e3e-169842ab026d.dc1 127.0.0.1 +31:48.129 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-575edd4a-8125-3caa-2e3e-169842ab026d 127.0.0.1 +2021-10-05T16:31:48.129Z [INFO] agent.router: Initializing LAN area manager +31:48.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29674 network=udp +31:48.130 [INFO] TestAgent.server: Adding LAN server: server="Node-575edd4a-8125-3caa-2e3e-169842ab026d (Addr: tcp/127.0.0.1:29742) (DC: dc1)" +31:48.130 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-575edd4a-8125-3caa-2e3e-169842ab026d.dc1 area=wan +31:48.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29674 network=tcp +31:48.131 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29738 network=tcp +31:48.131 [INFO] TestAgent: started state syncer +31:48.131 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29708 network=tcp +31:48.182 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.182 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29742 [Candidate]" term=2 +31:48.201 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.201 [DEBUG] TestAgent.server.raft: vote granted: from=575edd4a-8125-3caa-2e3e-169842ab026d term=2 tally=1 +31:48.201 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.201 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29742 [Leader]" +31:48.202 [INFO] TestAgent.server: cluster leadership acquired +31:48.202 [INFO] TestAgent.server: New leader elected: payload=Node-575edd4a-8125-3caa-2e3e-169842ab026d +31:48.211 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29742 +31:48.240 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.304 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.304 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.304 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.304 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.304 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.304 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-575edd4a-8125-3caa-2e3e-169842ab026d +31:48.304 [INFO] TestAgent.server: member joined, marking health alive: member=Node-575edd4a-8125-3caa-2e3e-169842ab026d +31:48.332 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.550 [INFO] TestAgent: Requesting shutdown +31:48.550 [INFO] TestAgent.server: shutting down server +31:48.550 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.550 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.550 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.550 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.550 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.550 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:48.550 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.550 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.550 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.576 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.594Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.594Z [INFO] agent.router.manager: shutting down +31:48.594 [INFO] TestAgent: consul server down +31:48.594 [INFO] TestAgent: shutdown complete +31:48.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29674 network=tcp +31:48.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29674 network=udp +31:48.594 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29738 network=tcp +31:49.104 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.104 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_LoadKeyrings/server_with_keys +2021-10-05T16:31:48.521Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.556Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.617 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6ec16b21-d3ff-4a98-df91-0142e10f9a98 Address:127.0.0.1:29791}]" +31:48.618 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29791 [Follower]" leader= +31:48.618 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6ec16b21-d3ff-4a98-df91-0142e10f9a98.dc1 127.0.0.1 +31:48.619 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6ec16b21-d3ff-4a98-df91-0142e10f9a98 127.0.0.1 +2021-10-05T16:31:48.619Z [INFO] agent.router: Initializing LAN area manager +31:48.619 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29786 network=udp +31:48.619 [INFO] TestAgent.server: Adding LAN server: server="Node-6ec16b21-d3ff-4a98-df91-0142e10f9a98 (Addr: tcp/127.0.0.1:29791) (DC: dc1)" +31:48.619 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6ec16b21-d3ff-4a98-df91-0142e10f9a98.dc1 area=wan +31:48.620 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29786 network=tcp +31:48.620 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29787 network=tcp +31:48.620 [INFO] TestAgent: started state syncer +31:48.620 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29778 network=tcp +31:48.668 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.668 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29791 [Candidate]" term=2 +31:48.701 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.701 [DEBUG] TestAgent.server.raft: vote granted: from=6ec16b21-d3ff-4a98-df91-0142e10f9a98 term=2 tally=1 +31:48.701 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.701 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29791 [Leader]" +31:48.702 [INFO] TestAgent.server: cluster leadership acquired +31:48.702 [INFO] TestAgent.server: New leader elected: payload=Node-6ec16b21-d3ff-4a98-df91-0142e10f9a98 +31:48.724 [INFO] TestAgent: Synced node info +31:48.724 [DEBUG] TestAgent: Node info in sync +31:48.724 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29791 +31:48.749 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.752 [INFO] TestAgent: Requesting shutdown +31:48.752 [INFO] TestAgent.server: shutting down server +31:48.752 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.756 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.763Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.763Z [INFO] agent.router.manager: shutting down +31:48.763 [INFO] TestAgent: consul server down +31:48.763 [INFO] TestAgent: shutdown complete +31:48.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29786 network=tcp +31:48.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29786 network=udp +31:48.763 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29787 network=tcp +31:48.766 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" +31:48.766 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:49.264 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.264 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsSpecificGet_invalidId (0.80s) +=== CONT TestOperator_Keyring_InvalidRelayFactor +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:48.413Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.434Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.497 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cca88a19-1937-1861-30fb-023ed7e2cb8f Address:127.0.0.1:29784}]" +31:48.498 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cca88a19-1937-1861-30fb-023ed7e2cb8f.dc1 127.0.0.1 +31:48.499 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cca88a19-1937-1861-30fb-023ed7e2cb8f 127.0.0.1 +2021-10-05T16:31:48.499Z [INFO] agent.router: Initializing LAN area manager +31:48.499 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29765 network=udp +31:48.499 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29784 [Follower]" leader= +31:48.499 [INFO] TestAgent.server: Adding LAN server: server="Node-cca88a19-1937-1861-30fb-023ed7e2cb8f (Addr: tcp/127.0.0.1:29784) (DC: dc1)" +31:48.499 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cca88a19-1937-1861-30fb-023ed7e2cb8f.dc1 area=wan +31:48.500 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29765 network=tcp +31:48.500 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29780 network=tcp +31:48.500 [INFO] TestAgent: started state syncer +31:48.500 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29750 network=tcp +31:48.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29784 [Candidate]" term=2 +31:48.591 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.591 [DEBUG] TestAgent.server.raft: vote granted: from=cca88a19-1937-1861-30fb-023ed7e2cb8f term=2 tally=1 +31:48.591 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.591 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29784 [Leader]" +31:48.592 [INFO] TestAgent.server: cluster leadership acquired +31:48.592 [INFO] TestAgent.server: New leader elected: payload=Node-cca88a19-1937-1861-30fb-023ed7e2cb8f +31:48.601 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29784 +31:48.622 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.680 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.680 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.680 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.680 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.680 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.680 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cca88a19-1937-1861-30fb-023ed7e2cb8f +31:48.680 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cca88a19-1937-1861-30fb-023ed7e2cb8f +31:48.715 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.846 [INFO] TestAgent: Requesting shutdown +31:48.846 [INFO] TestAgent.server: shutting down server +31:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.846 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.846 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.847 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.847 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:48.847 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.847 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.847 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.851 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.855Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.855Z [INFO] agent.router.manager: shutting down +31:48.855 [INFO] TestAgent: consul server down +31:48.855 [INFO] TestAgent: shutdown complete +31:48.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29765 network=tcp +31:48.855 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29765 network=udp +31:48.855 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29780 network=tcp +31:49.356 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.356 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsSpecificUpdate_good (0.96s) +=== CONT TestPreparedQuery_Create +2021-10-05T16:31:48.540Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.572Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.641 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:96ac7dbd-bea3-e031-aa97-d6203df3a28b Address:127.0.0.1:29840}]" +31:48.641 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29840 [Follower]" leader= +31:48.641 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b.dc1 127.0.0.1 +31:48.642 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b 127.0.0.1 +2021-10-05T16:31:48.642Z [INFO] agent.router: Initializing LAN area manager +31:48.642 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29779 network=udp +31:48.642 [INFO] TestAgent.server: Adding LAN server: server="Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b (Addr: tcp/127.0.0.1:29840) (DC: dc1)" +31:48.642 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29779 network=tcp +31:48.642 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b.dc1 area=wan +31:48.643 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29836 network=tcp +31:48.643 [INFO] TestAgent: started state syncer +31:48.643 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29820 network=tcp +31:48.696 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.696 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29840 [Candidate]" term=2 +31:48.727 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.727 [DEBUG] TestAgent.server.raft: vote granted: from=96ac7dbd-bea3-e031-aa97-d6203df3a28b term=2 tally=1 +31:48.727 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.727 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29840 [Leader]" +31:48.727 [INFO] TestAgent.server: cluster leadership acquired +31:48.727 [INFO] TestAgent.server: New leader elected: payload=Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b +31:48.739 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29840 +31:48.762 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.803 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.803 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.803 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.803 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.803 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.804 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b +31:48.804 [INFO] TestAgent.server: member joined, marking health alive: member=Node-96ac7dbd-bea3-e031-aa97-d6203df3a28b +31:48.820 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.939 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:48.947 [INFO] TestAgent: Synced node info +31:48.947 [DEBUG] TestAgent: Node info in sync +31:48.961 [INFO] TestAgent: Requesting shutdown +31:48.961 [INFO] TestAgent.server: shutting down server +31:48.961 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:48.961 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:48.961 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:48.961 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:48.961 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:48.961 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:48.961 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:48.961 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:48.961 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:48.961 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:48.966 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:48.972Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:48.972Z [INFO] agent.router.manager: shutting down +31:48.972 [INFO] TestAgent: consul server down +31:48.972 [INFO] TestAgent: shutdown complete +31:48.972 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29779 network=tcp +31:48.972 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29779 network=udp +31:48.972 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29836 network=tcp +31:49.474 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.474 [INFO] TestAgent: Endpoints down +--- PASS: TestParseConsistency_Invalid (1.00s) +=== CONT TestOperator_ServerHealth_Unhealthy +2021-10-05T16:31:48.534Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.546Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.632 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b2e4a8a8-1b65-d449-abd5-028de13e025e Address:127.0.0.1:29826}]" +31:48.633 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b2e4a8a8-1b65-d449-abd5-028de13e025e.dc1 127.0.0.1 +31:48.633 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b2e4a8a8-1b65-d449-abd5-028de13e025e 127.0.0.1 +2021-10-05T16:31:48.633Z [INFO] agent.router: Initializing LAN area manager +31:48.633 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29821 network=udp +31:48.633 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29826 [Follower]" leader= +31:48.634 [INFO] TestAgent.server: Adding LAN server: server="Node-b2e4a8a8-1b65-d449-abd5-028de13e025e (Addr: tcp/127.0.0.1:29826) (DC: dc1)" +31:48.634 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b2e4a8a8-1b65-d449-abd5-028de13e025e.dc1 area=wan +31:48.634 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29821 network=tcp +31:48.635 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:29823 network=tcp +31:48.635 [INFO] TestAgent: started state syncer +31:48.635 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29813 network=tcp +31:48.677 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:48.677 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29826 [Candidate]" term=2 +31:48.702 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:48.702 [DEBUG] TestAgent.server.raft: vote granted: from=b2e4a8a8-1b65-d449-abd5-028de13e025e term=2 tally=1 +31:48.702 [INFO] TestAgent.server.raft: election won: tally=1 +31:48.702 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29826 [Leader]" +31:48.703 [INFO] TestAgent.server: cluster leadership acquired +31:48.703 [INFO] TestAgent.server: New leader elected: payload=Node-b2e4a8a8-1b65-d449-abd5-028de13e025e +31:48.736 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29826 +31:48.776 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:48.844 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:48.844 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:48.844 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:48.844 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:48.844 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:48.844 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b2e4a8a8-1b65-d449-abd5-028de13e025e +31:48.844 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b2e4a8a8-1b65-d449-abd5-028de13e025e +31:48.870 [INFO] TestAgent.server: federation state anti-entropy synced +31:48.910 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:48.915 [INFO] TestAgent: Synced node info +31:48.915 [DEBUG] TestAgent: Node info in sync +31:49.032 [INFO] TestAgent: Requesting shutdown +31:49.032 [INFO] TestAgent.server: shutting down server +31:49.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:49.032 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:49.032 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:49.032 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:49.032 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.032 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:49.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:49.032 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:49.032 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:49.032 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:49.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.052Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.052Z [INFO] agent.router.manager: shutting down +31:49.052 [INFO] TestAgent: consul server down +31:49.052 [INFO] TestAgent: shutdown complete +31:49.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29821 network=tcp +31:49.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29821 network=udp +31:49.052 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:29823 network=tcp +31:49.552 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.552 [INFO] TestAgent: Endpoints down +--- PASS: TestHTTPServer_HandshakeTimeout (1.03s) +=== CONT TestOperator_ServerHealth +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:48.910Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:48.924Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:48.959 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:501bed91-a5f3-740f-5d2d-c7af94c0a4c4 Address:127.0.0.1:29819}]" +31:48.960 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4.dc1 127.0.0.1 +31:48.960 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29819 [Follower]" leader= +31:48.960 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4 127.0.0.1 +2021-10-05T16:31:48.961Z [INFO] agent.router: Initializing LAN area manager +31:48.961 [INFO] TestAgent.server: Adding LAN server: server="Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4 (Addr: tcp/127.0.0.1:29819) (DC: dc1)" +31:48.961 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4.dc1 area=wan +31:48.961 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29814 network=udp +31:48.961 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29814 network=tcp +31:48.962 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29815 network=tcp +31:48.962 [INFO] TestAgent: started state syncer +31:48.962 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29827 network=tcp +31:49.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29819 [Candidate]" term=2 +31:49.015 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.015 [DEBUG] TestAgent.server.raft: vote granted: from=501bed91-a5f3-740f-5d2d-c7af94c0a4c4 term=2 tally=1 +31:49.015 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.015 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29819 [Leader]" +31:49.016 [INFO] TestAgent.server: cluster leadership acquired +31:49.016 [INFO] TestAgent.server: New leader elected: payload=Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4 +31:49.048 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29819 +31:49.100 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.160 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.160 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.160 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.160 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.160 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.160 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4 +31:49.160 [INFO] TestAgent.server: member joined, marking health alive: member=Node-501bed91-a5f3-740f-5d2d-c7af94c0a4c4 +31:49.194 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.313 [INFO] TestAgent: Requesting shutdown +31:49.313 [INFO] TestAgent.server: shutting down server +31:49.313 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:49.313 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:49.313 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:49.313 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:49.313 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.313 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:49.313 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:49.313 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:49.313 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:49.313 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:49.313 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:49.332 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.344Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.344Z [INFO] agent.router.manager: shutting down +31:49.344 [INFO] TestAgent: consul server down +31:49.344 [INFO] TestAgent: shutdown complete +31:49.344 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29814 network=tcp +31:49.344 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29814 network=udp +31:49.344 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29815 network=tcp +31:49.844 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.844 [INFO] TestAgent: Endpoints down +--- PASS: TestEnableWebUI (0.95s) +=== CONT TestOperator_AutopilotCASConfiguration +2021-10-05T16:31:49.058Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.135Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.209 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:77536064-012f-f850-6ec5-034fc000452c Address:127.0.0.1:29798}]" +31:49.209 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29798 [Follower]" leader= +31:49.210 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77536064-012f-f850-6ec5-034fc000452c.dc1 127.0.0.1 +31:49.210 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-77536064-012f-f850-6ec5-034fc000452c 127.0.0.1 +2021-10-05T16:31:49.210Z [INFO] agent.router: Initializing LAN area manager +31:49.211 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29828 network=udp +31:49.211 [INFO] TestAgent.server: Adding LAN server: server="Node-77536064-012f-f850-6ec5-034fc000452c (Addr: tcp/127.0.0.1:29798) (DC: dc1)" +31:49.211 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77536064-012f-f850-6ec5-034fc000452c.dc1 area=wan +31:49.211 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29828 network=tcp +31:49.212 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29794 network=tcp +31:49.212 [INFO] TestAgent: started state syncer +31:49.212 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29834 network=tcp +31:49.261 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.261 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29798 [Candidate]" term=2 +31:49.277 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.277 [DEBUG] TestAgent.server.raft: vote granted: from=77536064-012f-f850-6ec5-034fc000452c term=2 tally=1 +31:49.277 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.277 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29798 [Leader]" +31:49.277 [INFO] TestAgent.server: cluster leadership acquired +31:49.277 [INFO] TestAgent.server: New leader elected: payload=Node-77536064-012f-f850-6ec5-034fc000452c +31:49.288 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29798 +31:49.312 [INFO] TestAgent: Requesting shutdown +31:49.312 [INFO] TestAgent.server: shutting down server +31:49.312 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.312 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:49.323 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.328 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.340Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.340Z [INFO] agent.router.manager: shutting down +31:49.362 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:49.363 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:49.363 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:49.363 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +31:49.363 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +31:49.363 [INFO] TestAgent: consul server down +31:49.363 [INFO] TestAgent: shutdown complete +31:49.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29828 network=tcp +31:49.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29828 network=udp +31:49.363 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29794 network=tcp +31:49.864 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.864 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_InmemKeyrings (3.28s) + --- PASS: TestAgent_InmemKeyrings/no_keys (0.88s) + --- PASS: TestAgent_InmemKeyrings/server_with_keys (0.98s) + --- PASS: TestAgent_InmemKeyrings/client_with_keys (0.60s) + --- PASS: TestAgent_InmemKeyrings/ignore_files (0.82s) +=== CONT TestOperator_AutopilotSetConfiguration +2021-10-05T16:31:49.162Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.199Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.256 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b9f88975-1d69-4a55-27df-a24fc7e34dbf Address:127.0.0.1:29833}]" +31:49.257 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b9f88975-1d69-4a55-27df-a24fc7e34dbf.dc1 127.0.0.1 +31:49.258 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b9f88975-1d69-4a55-27df-a24fc7e34dbf 127.0.0.1 +2021-10-05T16:31:49.258Z [INFO] agent.router: Initializing LAN area manager +31:49.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29800 network=udp +31:49.258 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29833 [Follower]" leader= +31:49.258 [INFO] TestAgent.server: Adding LAN server: server="Node-b9f88975-1d69-4a55-27df-a24fc7e34dbf (Addr: tcp/127.0.0.1:29833) (DC: dc1)" +31:49.258 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b9f88975-1d69-4a55-27df-a24fc7e34dbf.dc1 area=wan +31:49.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29800 network=tcp +31:49.259 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29829 network=tcp +31:49.259 [INFO] TestAgent: started state syncer +31:49.259 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29792 network=tcp +31:49.304 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.304 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29833 [Candidate]" term=2 +31:49.326 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.326 [DEBUG] TestAgent.server.raft: vote granted: from=b9f88975-1d69-4a55-27df-a24fc7e34dbf term=2 tally=1 +31:49.326 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.326 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29833 [Leader]" +31:49.328 [INFO] TestAgent.server: cluster leadership acquired +31:49.328 [INFO] TestAgent.server: New leader elected: payload=Node-b9f88975-1d69-4a55-27df-a24fc7e34dbf +31:49.351 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29833 +31:49.352 [INFO] TestAgent: Synced node info +31:49.352 [DEBUG] TestAgent: Node info in sync +31:49.388 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.412 [INFO] TestAgent: Requesting shutdown +31:49.412 [INFO] TestAgent.server: shutting down server +31:49.412 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.419 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.427Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.427Z [INFO] agent.router.manager: shutting down +31:49.431 [INFO] TestAgent: consul server down +31:49.431 [INFO] TestAgent: shutdown complete +31:49.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29800 network=tcp +31:49.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29800 network=udp +31:49.431 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29829 network=tcp +31:49.431 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +31:49.431 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:49.931 [INFO] TestAgent: Waiting for endpoints to shut down +31:49.931 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_LoadKeyrings/client_with_keys +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:49.138Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.197Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.251 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:41a59a80-737a-af66-4263-c293fd906d62 Address:127.0.0.1:29770}]" +31:49.251 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-41a59a80-737a-af66-4263-c293fd906d62.dc1 127.0.0.1 +31:49.252 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29770 [Follower]" leader= +31:49.254 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-41a59a80-737a-af66-4263-c293fd906d62 127.0.0.1 +2021-10-05T16:31:49.254Z [INFO] agent.router: Initializing LAN area manager +31:49.254 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29835 network=udp +31:49.254 [INFO] TestAgent.server: Adding LAN server: server="Node-41a59a80-737a-af66-4263-c293fd906d62 (Addr: tcp/127.0.0.1:29770) (DC: dc1)" +31:49.254 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-41a59a80-737a-af66-4263-c293fd906d62.dc1 area=wan +31:49.254 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29835 network=tcp +31:49.255 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29766 network=tcp +31:49.255 [INFO] TestAgent: started state syncer +31:49.255 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29799 network=tcp +31:49.316 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.316 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29770 [Candidate]" term=2 +31:49.348 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.348 [DEBUG] TestAgent.server.raft: vote granted: from=41a59a80-737a-af66-4263-c293fd906d62 term=2 tally=1 +31:49.348 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.348 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29770 [Leader]" +31:49.350 [INFO] TestAgent.server: cluster leadership acquired +31:49.350 [INFO] TestAgent.server: New leader elected: payload=Node-41a59a80-737a-af66-4263-c293fd906d62 +31:49.361 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29770 +31:49.402 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.439 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.439 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.439 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.439 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.439 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.439 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-41a59a80-737a-af66-4263-c293fd906d62 +31:49.439 [INFO] TestAgent.server: member joined, marking health alive: member=Node-41a59a80-737a-af66-4263-c293fd906d62 +31:49.456 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.569 [INFO] TestAgent: Requesting shutdown +31:49.569 [INFO] TestAgent.server: shutting down server +31:49.569 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:49.569 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:49.569 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:49.569 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:49.569 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.569 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:49.569 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:49.569 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:49.569 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:49.569 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:49.570 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:49.579 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.592Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.592Z [INFO] agent.router.manager: shutting down +31:49.592 [INFO] TestAgent: consul server down +31:49.592 [INFO] TestAgent: shutdown complete +31:49.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29835 network=tcp +31:49.592 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29835 network=udp +31:49.592 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29766 network=tcp +31:50.096 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.096 [INFO] TestAgent: Endpoints down +--- PASS: TestACLResolution (1.03s) +=== CONT TestOperator_AutopilotGetConfiguration +2021-10-05T16:31:49.292Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.317Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.414 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8cf1f7cf-5741-dc40-95fe-652c50789c47 Address:127.0.0.1:29805}]" +31:49.415 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29805 [Follower]" leader= +31:49.415 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8cf1f7cf-5741-dc40-95fe-652c50789c47.dc1 127.0.0.1 +31:49.417 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8cf1f7cf-5741-dc40-95fe-652c50789c47 127.0.0.1 +2021-10-05T16:31:49.417Z [INFO] agent.router: Initializing LAN area manager +31:49.417 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29793 network=udp +31:49.417 [INFO] TestAgent.server: Adding LAN server: server="Node-8cf1f7cf-5741-dc40-95fe-652c50789c47 (Addr: tcp/127.0.0.1:29805) (DC: dc1)" +31:49.417 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8cf1f7cf-5741-dc40-95fe-652c50789c47.dc1 area=wan +31:49.417 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29793 network=tcp +31:49.418 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29801 network=tcp +31:49.418 [INFO] TestAgent: started state syncer +31:49.418 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29806 network=tcp +31:49.465 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.465 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29805 [Candidate]" term=2 +31:49.475 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.475 [DEBUG] TestAgent.server.raft: vote granted: from=8cf1f7cf-5741-dc40-95fe-652c50789c47 term=2 tally=1 +31:49.475 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.475 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29805 [Leader]" +31:49.476 [INFO] TestAgent.server: cluster leadership acquired +31:49.476 [INFO] TestAgent.server: New leader elected: payload=Node-8cf1f7cf-5741-dc40-95fe-652c50789c47 +31:49.486 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29805 +31:49.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.560 [INFO] TestAgent: Synced node info +31:49.560 [DEBUG] TestAgent: Node info in sync +31:49.604 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.604 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.604 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.604 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.604 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.604 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8cf1f7cf-5741-dc40-95fe-652c50789c47 +31:49.604 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8cf1f7cf-5741-dc40-95fe-652c50789c47 +31:49.624 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.739 [INFO] TestAgent: Requesting shutdown +31:49.739 [INFO] TestAgent.server: shutting down server +31:49.739 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:49.739 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:49.739 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:49.739 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:49.739 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.739 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:49.739 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:49.739 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:49.739 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:49.739 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:49.744 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.749Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.749Z [INFO] agent.router.manager: shutting down +31:49.749 [INFO] TestAgent: consul server down +31:49.749 [INFO] TestAgent: shutdown complete +31:49.749 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29793 network=tcp +31:49.749 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29793 network=udp +31:49.749 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29801 network=tcp +31:50.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.252 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_Keyring_InvalidRelayFactor (0.99s) +=== CONT TestOperator_Keyring_LocalOnly +2021-10-05T16:31:49.380Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.402Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.455 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:382460e2-b022-4c6d-3cb8-4d9094ecb691 Address:127.0.0.1:29812}]" +31:49.456 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29812 [Follower]" leader= +31:49.456 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-382460e2-b022-4c6d-3cb8-4d9094ecb691.dc1 127.0.0.1 +31:49.457 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-382460e2-b022-4c6d-3cb8-4d9094ecb691 127.0.0.1 +2021-10-05T16:31:49.457Z [INFO] agent.router: Initializing LAN area manager +31:49.457 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29807 network=udp +31:49.457 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-382460e2-b022-4c6d-3cb8-4d9094ecb691.dc1 area=wan +31:49.457 [INFO] TestAgent.server: Adding LAN server: server="Node-382460e2-b022-4c6d-3cb8-4d9094ecb691 (Addr: tcp/127.0.0.1:29812) (DC: dc1)" +31:49.457 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29807 network=tcp +31:49.458 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29808 network=tcp +31:49.458 [INFO] TestAgent: started state syncer +31:49.458 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29855 network=tcp +31:49.525 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.525 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29812 [Candidate]" term=2 +31:49.546 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.546 [DEBUG] TestAgent.server.raft: vote granted: from=382460e2-b022-4c6d-3cb8-4d9094ecb691 term=2 tally=1 +31:49.546 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.546 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29812 [Leader]" +31:49.546 [INFO] TestAgent.server: cluster leadership acquired +31:49.547 [INFO] TestAgent.server: New leader elected: payload=Node-382460e2-b022-4c6d-3cb8-4d9094ecb691 +31:49.576 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29812 +31:49.659 [INFO] TestAgent: Synced node info +31:49.659 [DEBUG] TestAgent: Node info in sync +31:49.659 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.700 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.700 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.700 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.700 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.700 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.701 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-382460e2-b022-4c6d-3cb8-4d9094ecb691 +31:49.701 [INFO] TestAgent.server: member joined, marking health alive: member=Node-382460e2-b022-4c6d-3cb8-4d9094ecb691 +31:49.732 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.764 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +31:49.764 [INFO] TestAgent: Requesting shutdown +31:49.764 [INFO] TestAgent.server: shutting down server +31:49.764 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:49.764 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:49.764 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:49.764 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:49.764 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:49.764 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:49.764 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:49.764 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:49.764 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:49.764 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:49.768 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:49.774Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:49.774Z [INFO] agent.router.manager: shutting down +31:49.774 [INFO] TestAgent: consul server down +31:49.774 [INFO] TestAgent: shutdown complete +31:49.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29807 network=tcp +31:49.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29807 network=udp +31:49.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29808 network=tcp +31:50.276 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.276 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Create (0.92s) +=== CONT TestOperator_KeyringList +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +31:49.975 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-7d68848b-426e-7177-f650-c540b659da5d 127.0.0.1 +2021-10-05T16:31:49.975Z [INFO] agent.router: Initializing LAN area manager +31:49.975 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29660 network=udp +31:49.975 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29660 network=tcp +31:49.976 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29647 network=tcp +31:49.976 [INFO] TestAgent: started state syncer +31:49.976 [INFO] TestAgent: Requesting shutdown +31:49.976 [INFO] TestAgent.client: shutting down client +31:49.976 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:49.977 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29841 network=tcp +31:49.977 [ERROR] TestAgent: gRPC server failed: error="grpc: the server has been stopped" +2021-10-05T16:31:49.977Z [WARN] agent.router.manager: No servers available +31:49.977 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:49.986 [INFO] TestAgent: consul client down +31:49.986 [INFO] TestAgent: shutdown complete +31:49.986 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29660 network=tcp +31:49.986 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29660 network=udp +31:49.986 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29647 network=tcp +31:50.488 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.488 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_LoadKeyrings (2.51s) + --- PASS: TestAgent_LoadKeyrings/no_keys (1.13s) + --- PASS: TestAgent_LoadKeyrings/server_with_keys (0.82s) + --- PASS: TestAgent_LoadKeyrings/client_with_keys (0.56s) +=== CONT TestOperator_KeyringUse +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:49.865Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.877Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.945 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:31ad9173-4622-42a6-7cbd-9704e49fd97f Address:127.0.0.1:29889}]" +31:49.945 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29889 [Follower]" leader= +31:49.946 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-31ad9173-4622-42a6-7cbd-9704e49fd97f.dc1 127.0.0.1 +31:49.947 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-31ad9173-4622-42a6-7cbd-9704e49fd97f 127.0.0.1 +2021-10-05T16:31:49.947Z [INFO] agent.router: Initializing LAN area manager +31:49.947 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29884 network=udp +31:49.947 [INFO] TestAgent.server: Adding LAN server: server="Node-31ad9173-4622-42a6-7cbd-9704e49fd97f (Addr: tcp/127.0.0.1:29889) (DC: dc1)" +31:49.947 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-31ad9173-4622-42a6-7cbd-9704e49fd97f.dc1 area=wan +31:49.947 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29884 network=tcp +31:49.948 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29885 network=tcp +31:49.948 [INFO] TestAgent: started state syncer +31:49.948 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29666 network=tcp +31:50.011 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.011 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29889 [Candidate]" term=2 +31:50.022 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.022 [DEBUG] TestAgent.server.raft: vote granted: from=31ad9173-4622-42a6-7cbd-9704e49fd97f term=2 tally=1 +31:50.022 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.022 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29889 [Leader]" +31:50.022 [INFO] TestAgent.server: cluster leadership acquired +31:50.022 [INFO] TestAgent.server: New leader elected: payload=Node-31ad9173-4622-42a6-7cbd-9704e49fd97f +31:50.040 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29889 +31:50.076 [INFO] TestAgent: Synced node info +31:50.076 [DEBUG] TestAgent: Node info in sync +31:50.097 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:50.135 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:50.135 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:50.135 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:50.135 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:50.135 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:50.135 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-31ad9173-4622-42a6-7cbd-9704e49fd97f +31:50.136 [INFO] TestAgent.server: member joined, marking health alive: member=Node-31ad9173-4622-42a6-7cbd-9704e49fd97f +31:50.161 [INFO] TestAgent.server: federation state anti-entropy synced +31:50.220 [INFO] TestAgent: Requesting shutdown +31:50.220 [INFO] TestAgent.server: shutting down server +31:50.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:50.220 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:50.220 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:50.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:50.220 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.220 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:50.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:50.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:50.220 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:50.220 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:50.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.236Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.236Z [INFO] agent.router.manager: shutting down +31:50.236 [INFO] TestAgent: consul server down +31:50.236 [INFO] TestAgent: shutdown complete +31:50.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29884 network=tcp +31:50.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29884 network=udp +31:50.236 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29885 network=tcp +31:50.740 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.740 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_AutopilotCASConfiguration (0.90s) +=== CONT TestOperator_KeyringRemove +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:49.885Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.898Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.968 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a6b831f3-df57-5fde-789a-efb156424fc9 Address:127.0.0.1:29686}]" +31:49.968 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29686 [Follower]" leader= +31:49.969 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a6b831f3-df57-5fde-789a-efb156424fc9.dc1 127.0.0.1 +31:49.970 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a6b831f3-df57-5fde-789a-efb156424fc9 127.0.0.1 +2021-10-05T16:31:49.970Z [INFO] agent.router: Initializing LAN area manager +31:49.970 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29667 network=udp +31:49.970 [INFO] TestAgent.server: Adding LAN server: server="Node-a6b831f3-df57-5fde-789a-efb156424fc9 (Addr: tcp/127.0.0.1:29686) (DC: dc1)" +31:49.970 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a6b831f3-df57-5fde-789a-efb156424fc9.dc1 area=wan +31:49.970 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29667 network=tcp +31:49.971 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29682 network=tcp +31:49.971 [INFO] TestAgent: started state syncer +31:49.971 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29659 network=tcp +31:50.039 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.039 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29686 [Candidate]" term=2 +31:50.051 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.051 [DEBUG] TestAgent.server.raft: vote granted: from=a6b831f3-df57-5fde-789a-efb156424fc9 term=2 tally=1 +31:50.051 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.051 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29686 [Leader]" +31:50.051 [INFO] TestAgent.server: cluster leadership acquired +31:50.051 [INFO] TestAgent.server: New leader elected: payload=Node-a6b831f3-df57-5fde-789a-efb156424fc9 +31:50.080 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29686 +31:50.118 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:50.149 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:50.149 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:50.149 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:50.149 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:50.149 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:50.149 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a6b831f3-df57-5fde-789a-efb156424fc9 +31:50.149 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a6b831f3-df57-5fde-789a-efb156424fc9 +31:50.180 [INFO] TestAgent.server: federation state anti-entropy synced +31:50.312 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:50.318 [INFO] TestAgent: Synced node info +31:50.457 [INFO] TestAgent: Requesting shutdown +31:50.457 [INFO] TestAgent.server: shutting down server +31:50.457 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:50.457 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:50.457 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:50.457 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:50.457 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.457 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:50.457 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:50.457 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:50.457 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:50.457 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:50.467 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.490Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.490Z [INFO] agent.router.manager: shutting down +31:50.490 [INFO] TestAgent: consul server down +31:50.490 [INFO] TestAgent: shutdown complete +31:50.490 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29667 network=tcp +31:50.490 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29667 network=udp +31:50.490 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29682 network=tcp +31:50.992 [INFO] TestAgent: Waiting for endpoints to shut down +31:50.992 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_AutopilotSetConfiguration (1.13s) +=== CONT TestOperator_KeyringListLocalOnly +2021-10-05T16:31:50.297Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:50.321Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:50.396 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a2129af-e775-65c4-75fe-89f8c114e68d Address:127.0.0.1:29909}]" +31:50.397 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a2129af-e775-65c4-75fe-89f8c114e68d.dc1 127.0.0.1 +31:50.398 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a2129af-e775-65c4-75fe-89f8c114e68d 127.0.0.1 +2021-10-05T16:31:50.398Z [INFO] agent.router: Initializing LAN area manager +31:50.398 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29877 network=udp +31:50.398 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29909 [Follower]" leader= +31:50.399 [INFO] TestAgent.server: Adding LAN server: server="Node-4a2129af-e775-65c4-75fe-89f8c114e68d (Addr: tcp/127.0.0.1:29909) (DC: dc1)" +31:50.399 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a2129af-e775-65c4-75fe-89f8c114e68d.dc1 area=wan +31:50.399 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29877 network=tcp +31:50.399 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29905 network=tcp +31:50.399 [INFO] TestAgent: started state syncer +31:50.400 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29862 network=tcp +31:50.447 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.447 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29909 [Candidate]" term=2 +31:50.496 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.496 [DEBUG] TestAgent.server.raft: vote granted: from=4a2129af-e775-65c4-75fe-89f8c114e68d term=2 tally=1 +31:50.496 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.496 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29909 [Leader]" +31:50.497 [INFO] TestAgent.server: cluster leadership acquired +31:50.497 [INFO] TestAgent.server: New leader elected: payload=Node-4a2129af-e775-65c4-75fe-89f8c114e68d +31:50.528 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +31:50.530 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29909 +31:50.532 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4a2129af-e775-65c4-75fe-89f8c114e68d.dc1 +31:50.532 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:50.532 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4a2129af-e775-65c4-75fe-89f8c114e68d +31:50.532 [INFO] TestAgent: Requesting shutdown +31:50.532 [INFO] TestAgent.server: shutting down server +31:50.532 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.533 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:50.546 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.560Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.560Z [INFO] agent.router.manager: shutting down +31:50.567 [INFO] TestAgent: consul server down +31:50.567 [INFO] TestAgent: shutdown complete +31:50.567 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29877 network=tcp +31:50.567 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29877 network=udp +31:50.567 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29905 network=tcp +31:50.567 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:50.568 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:51.068 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.068 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_KeyringList (0.79s) +=== CONT TestIntentionsCreate_good +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:50.278Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:50.311Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:50.347 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 Address:127.0.0.1:29875}]" +31:50.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29875 [Follower]" leader= +31:50.348 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192.dc1 127.0.0.1 +31:50.349 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 127.0.0.1 +2021-10-05T16:31:50.351Z [INFO] agent.router: Initializing LAN area manager +31:50.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29849 network=udp +31:50.351 [INFO] TestAgent.server: Adding LAN server: server="Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 (Addr: tcp/127.0.0.1:29875) (DC: dc1)" +31:50.351 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192.dc1 area=wan +31:50.352 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29849 network=tcp +31:50.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29871 network=tcp +31:50.353 [INFO] TestAgent: started state syncer +31:50.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29876 network=tcp +31:50.417 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.418 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29875 [Candidate]" term=2 +31:50.452 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.452 [DEBUG] TestAgent.server.raft: vote granted: from=9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 term=2 tally=1 +31:50.452 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.452 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29875 [Leader]" +31:50.452 [INFO] TestAgent.server: cluster leadership acquired +31:50.452 [INFO] TestAgent.server: New leader elected: payload=Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 +31:50.501 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29875 +31:50.548 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:50.602 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:50.603 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 +31:50.603 [INFO] TestAgent: Requesting shutdown +31:50.603 [INFO] TestAgent.server: shutting down server +31:50.603 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.603 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:50.609 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.618Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.618Z [INFO] agent.router.manager: shutting down +31:50.618 [INFO] TestAgent: consul server down +31:50.618 [INFO] TestAgent: shutdown complete +31:50.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29849 network=tcp +31:50.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29849 network=udp +31:50.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29871 network=tcp +31:50.619 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:50.619 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:50.619 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:50.619 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:50.619 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:50.619 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 +31:50.619 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 +31:50.619 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 127.0.0.1 29873 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:9d27f4c8-c9e2-a7a9-d3b1-6bd86eaa1192 port:29875 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29874] alive 1 5 2 2 5 4}" error="raft is already shutdown" +31:50.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:50.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:50.619 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:50.619 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:50.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:50.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:50.619 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:50.619 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:51.120 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.120 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_Keyring_LocalOnly (0.87s) +=== CONT TestIntentionsCreate_noBody +2021-10-05T16:31:50.132Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:50.167Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:50.268 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:405c4068-6d40-8ef3-c593-3d9743efa5d1 Address:127.0.0.1:29847}]" +31:50.269 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-405c4068-6d40-8ef3-c593-3d9743efa5d1.dc1 127.0.0.1 +31:50.269 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-405c4068-6d40-8ef3-c593-3d9743efa5d1 127.0.0.1 +2021-10-05T16:31:50.269Z [INFO] agent.router: Initializing LAN area manager +31:50.269 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29847 [Follower]" leader= +31:50.270 [INFO] TestAgent.server: Adding LAN server: server="Node-405c4068-6d40-8ef3-c593-3d9743efa5d1 (Addr: tcp/127.0.0.1:29847) (DC: dc1)" +31:50.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-405c4068-6d40-8ef3-c593-3d9743efa5d1.dc1 area=wan +31:50.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29842 network=tcp +31:50.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29842 network=udp +31:50.271 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29843 network=tcp +31:50.271 [INFO] TestAgent: started state syncer +31:50.271 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29848 network=tcp +31:50.328 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.328 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29847 [Candidate]" term=2 +31:50.380 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.380 [DEBUG] TestAgent.server.raft: vote granted: from=405c4068-6d40-8ef3-c593-3d9743efa5d1 term=2 tally=1 +31:50.380 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.380 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29847 [Leader]" +31:50.380 [INFO] TestAgent.server: cluster leadership acquired +31:50.380 [INFO] TestAgent.server: New leader elected: payload=Node-405c4068-6d40-8ef3-c593-3d9743efa5d1 +31:50.406 [INFO] TestAgent: Synced node info +31:50.406 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29847 +31:50.463 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:50.530 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:50.530 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:50.530 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:50.530 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:50.530 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:50.530 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-405c4068-6d40-8ef3-c593-3d9743efa5d1 +31:50.530 [INFO] TestAgent.server: member joined, marking health alive: member=Node-405c4068-6d40-8ef3-c593-3d9743efa5d1 +31:50.560 [INFO] TestAgent.server: federation state anti-entropy synced +31:50.665 [INFO] TestAgent: Requesting shutdown +31:50.665 [INFO] TestAgent.server: shutting down server +31:50.665 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:50.665 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:50.665 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:50.665 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:50.665 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.666 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:50.666 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:50.666 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:50.666 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:50.666 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:50.669 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.675Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.675Z [INFO] agent.router.manager: shutting down +31:50.675 [INFO] TestAgent: consul server down +31:50.675 [INFO] TestAgent: shutdown complete +31:50.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29842 network=tcp +31:50.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29842 network=udp +31:50.675 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29843 network=tcp +31:51.176 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.176 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_AutopilotGetConfiguration (1.08s) +=== CONT TestStatusLeaderSecondary +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:50.542Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:50.569Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:50.656 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d8385449-a9e5-8413-26fc-942f7000e811 Address:127.0.0.1:29861}]" +31:50.656 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29861 [Follower]" leader= +31:50.657 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 127.0.0.1 +31:50.657 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d8385449-a9e5-8413-26fc-942f7000e811 127.0.0.1 +2021-10-05T16:31:50.657Z [INFO] agent.router: Initializing LAN area manager +31:50.657 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29863 network=udp +31:50.657 [INFO] TestAgent.server: Adding LAN server: server="Node-d8385449-a9e5-8413-26fc-942f7000e811 (Addr: tcp/127.0.0.1:29861) (DC: dc1)" +31:50.657 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 area=wan +31:50.657 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29863 network=tcp +31:50.658 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29857 network=tcp +31:50.658 [INFO] TestAgent: started state syncer +31:50.658 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29910 network=tcp +31:50.716 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.716 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29861 [Candidate]" term=2 +31:50.764 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:50.764 [DEBUG] TestAgent.server.raft: vote granted: from=d8385449-a9e5-8413-26fc-942f7000e811 term=2 tally=1 +31:50.764 [INFO] TestAgent.server.raft: election won: tally=1 +31:50.764 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29861 [Leader]" +31:50.764 [INFO] TestAgent.server: cluster leadership acquired +31:50.764 [INFO] TestAgent.server: New leader elected: payload=Node-d8385449-a9e5-8413-26fc-942f7000e811 +31:50.796 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29861 +31:50.816 [INFO] TestAgent.server.serf.wan: serf: Received install-key query +31:50.820 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 +31:50.820 [INFO] TestAgent.server.serf.lan: serf: Received install-key query +31:50.824 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811 +31:50.824 [INFO] TestAgent.server.serf.wan: serf: Received use-key query +31:50.828 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 +31:50.828 [INFO] TestAgent.server.serf.lan: serf: Received use-key query +31:50.828 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811 +31:50.828 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query +31:50.829 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 +31:50.829 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query +31:50.829 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811 +31:50.829 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +31:50.829 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811.dc1 +31:50.829 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:50.830 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-d8385449-a9e5-8413-26fc-942f7000e811 +31:50.830 [INFO] TestAgent: Requesting shutdown +31:50.830 [INFO] TestAgent.server: shutting down server +31:50.830 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:50.830 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:50.840 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:50.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:50.852Z [INFO] agent.router.manager: shutting down +31:50.860 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +31:50.862 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:50.862 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +31:50.862 [INFO] TestAgent: consul server down +31:50.862 [INFO] TestAgent: shutdown complete +31:50.862 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29863 network=tcp +31:50.862 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29863 network=udp +31:50.862 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29857 network=tcp +31:51.364 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.364 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_KeyringUse (0.88s) +=== CONT TestStatusPeersSecondary +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:51.018Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.049Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.128 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07609ce0-a3a9-0001-8de3-e354057bb352 Address:127.0.0.1:29930}]" +31:51.129 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07609ce0-a3a9-0001-8de3-e354057bb352.dc1 127.0.0.1 +31:51.129 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07609ce0-a3a9-0001-8de3-e354057bb352 127.0.0.1 +2021-10-05T16:31:51.129Z [INFO] agent.router: Initializing LAN area manager +31:51.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29904 network=udp +31:51.130 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29930 [Follower]" leader= +31:51.130 [INFO] TestAgent.server: Adding LAN server: server="Node-07609ce0-a3a9-0001-8de3-e354057bb352 (Addr: tcp/127.0.0.1:29930) (DC: dc1)" +31:51.130 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07609ce0-a3a9-0001-8de3-e354057bb352.dc1 area=wan +31:51.130 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29904 network=tcp +31:51.131 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29926 network=tcp +31:51.131 [INFO] TestAgent: started state syncer +31:51.131 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29917 network=tcp +31:51.172 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.172 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29930 [Candidate]" term=2 +31:51.208 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.208 [DEBUG] TestAgent.server.raft: vote granted: from=07609ce0-a3a9-0001-8de3-e354057bb352 term=2 tally=1 +31:51.208 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.208 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29930 [Leader]" +31:51.208 [INFO] TestAgent.server: cluster leadership acquired +31:51.208 [INFO] TestAgent.server: New leader elected: payload=Node-07609ce0-a3a9-0001-8de3-e354057bb352 +31:51.227 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29930 +31:51.259 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:51.259 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-07609ce0-a3a9-0001-8de3-e354057bb352 +31:51.259 [INFO] TestAgent: Requesting shutdown +31:51.260 [INFO] TestAgent.server: shutting down server +31:51.260 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.260 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:51.276 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:51.277 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:31:51.284Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.284Z [INFO] agent.router.manager: shutting down +31:51.292 [INFO] TestAgent: consul server down +31:51.292 [INFO] TestAgent: shutdown complete +31:51.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29904 network=tcp +31:51.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29904 network=udp +31:51.292 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29926 network=tcp +31:51.292 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +31:51.796 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.796 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_KeyringListLocalOnly (0.80s) +=== CONT TestStatusPeers +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:50.777Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:50.825Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:50.896 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8ab7c864-a39a-efb6-0949-9ca1f012d99e Address:127.0.0.1:29923}]" +31:50.896 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29923 [Follower]" leader= +31:50.896 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 127.0.0.1 +31:50.897 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e 127.0.0.1 +2021-10-05T16:31:50.897Z [INFO] agent.router: Initializing LAN area manager +31:50.897 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29897 network=udp +31:50.898 [INFO] TestAgent.server: Adding LAN server: server="Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e (Addr: tcp/127.0.0.1:29923) (DC: dc1)" +31:50.898 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 area=wan +31:50.898 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29897 network=tcp +31:50.898 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29919 network=tcp +31:50.898 [INFO] TestAgent: started state syncer +31:50.898 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29892 network=tcp +31:50.952 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:50.952 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29923 [Candidate]" term=2 +31:51.000 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.000 [DEBUG] TestAgent.server.raft: vote granted: from=8ab7c864-a39a-efb6-0949-9ca1f012d99e term=2 tally=1 +31:51.000 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.000 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29923 [Leader]" +31:51.000 [INFO] TestAgent.server: cluster leadership acquired +31:51.000 [INFO] TestAgent.server: New leader elected: payload=Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.020 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29923 +31:51.040 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.104 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:51.104 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:51.104 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:51.104 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:51.104 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:51.104 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.104 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.148 [INFO] TestAgent.server: federation state anti-entropy synced +31:51.188 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:51.208 [INFO] TestAgent: Synced node info +31:51.336 [INFO] TestAgent.server.serf.wan: serf: Received install-key query +31:51.344 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 +31:51.344 [INFO] TestAgent.server.serf.lan: serf: Received install-key query +31:51.345 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.345 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +31:51.345 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 +31:51.345 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:51.345 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.345 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query +31:51.346 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 +31:51.346 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query +31:51.346 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.346 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +31:51.346 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e.dc1 +31:51.346 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +31:51.347 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-8ab7c864-a39a-efb6-0949-9ca1f012d99e +31:51.347 [INFO] TestAgent: Requesting shutdown +31:51.347 [INFO] TestAgent.server: shutting down server +31:51.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:51.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:51.347 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:51.347 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:51.347 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.347 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:51.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:51.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:51.347 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:51.347 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:51.360 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.376Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.376Z [INFO] agent.router.manager: shutting down +31:51.376 [INFO] TestAgent: consul server down +31:51.376 [INFO] TestAgent: shutdown complete +31:51.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29897 network=tcp +31:51.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29897 network=udp +31:51.376 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29919 network=tcp +31:51.876 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.876 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_KeyringRemove (1.14s) +=== CONT TestIntentionsCheck_noDestination +2021-10-05T16:31:51.098Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.129Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.203 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f87021de-be1b-bfa2-4d6b-2e9932a55106 Address:127.0.0.1:29902}]" +31:51.203 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29902 [Follower]" leader= +31:51.205 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f87021de-be1b-bfa2-4d6b-2e9932a55106.dc1 127.0.0.1 +31:51.206 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f87021de-be1b-bfa2-4d6b-2e9932a55106 127.0.0.1 +2021-10-05T16:31:51.206Z [INFO] agent.router: Initializing LAN area manager +31:51.207 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29918 network=udp +31:51.207 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f87021de-be1b-bfa2-4d6b-2e9932a55106.dc1 area=wan +31:51.207 [INFO] TestAgent.server: Adding LAN server: server="Node-f87021de-be1b-bfa2-4d6b-2e9932a55106 (Addr: tcp/127.0.0.1:29902) (DC: dc1)" +31:51.207 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29918 network=tcp +31:51.207 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29898 network=tcp +31:51.207 [INFO] TestAgent: started state syncer +31:51.207 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29931 network=tcp +31:51.260 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.260 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29902 [Candidate]" term=2 +31:51.294 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.294 [DEBUG] TestAgent.server.raft: vote granted: from=f87021de-be1b-bfa2-4d6b-2e9932a55106 term=2 tally=1 +31:51.294 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.294 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29902 [Leader]" +31:51.295 [INFO] TestAgent.server: cluster leadership acquired +31:51.295 [INFO] TestAgent.server: New leader elected: payload=Node-f87021de-be1b-bfa2-4d6b-2e9932a55106 +31:51.302 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29902 +31:51.352 [INFO] TestAgent: Requesting shutdown +31:51.352 [INFO] TestAgent.server: shutting down server +31:51.352 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.352 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.352 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:51.362 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.369Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.369Z [INFO] agent.router.manager: shutting down +31:51.380 [INFO] TestAgent: consul server down +31:51.380 [INFO] TestAgent: shutdown complete +31:51.380 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29918 network=tcp +31:51.380 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +31:51.380 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:51.380 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29918 network=udp +31:51.380 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:51.380 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29898 network=tcp +31:51.880 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.880 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsCreate_good (0.81s) +=== CONT TestStatusLeader +2021-10-05T16:31:51.148Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.179Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.273 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0294718b-8e83-083f-77ee-d7b7c77da585 Address:127.0.0.1:29916}]" +31:51.274 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0294718b-8e83-083f-77ee-d7b7c77da585.dc1 127.0.0.1 +31:51.274 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0294718b-8e83-083f-77ee-d7b7c77da585 127.0.0.1 +2021-10-05T16:31:51.274Z [INFO] agent.router: Initializing LAN area manager +31:51.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29932 network=udp +31:51.275 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29916 [Follower]" leader= +31:51.275 [INFO] TestAgent.server: Adding LAN server: server="Node-0294718b-8e83-083f-77ee-d7b7c77da585 (Addr: tcp/127.0.0.1:29916) (DC: dc1)" +31:51.275 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0294718b-8e83-083f-77ee-d7b7c77da585.dc1 area=wan +31:51.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29932 network=tcp +31:51.276 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29912 network=tcp +31:51.276 [INFO] TestAgent: started state syncer +31:51.276 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29903 network=tcp +31:51.320 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.320 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29916 [Candidate]" term=2 +31:51.358 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.358 [DEBUG] TestAgent.server.raft: vote granted: from=0294718b-8e83-083f-77ee-d7b7c77da585 term=2 tally=1 +31:51.358 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.358 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29916 [Leader]" +31:51.360 [INFO] TestAgent.server: cluster leadership acquired +31:51.360 [INFO] TestAgent.server: New leader elected: payload=Node-0294718b-8e83-083f-77ee-d7b7c77da585 +31:51.370 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29916 +31:51.404 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.422 [INFO] TestAgent: Requesting shutdown +31:51.422 [INFO] TestAgent.server: shutting down server +31:51.422 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.422 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:51.428 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.436Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.436Z [INFO] agent.router.manager: shutting down +31:51.441 [INFO] TestAgent: consul server down +31:51.441 [INFO] TestAgent: shutdown complete +31:51.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29932 network=tcp +31:51.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29932 network=udp +31:51.441 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29912 network=tcp +31:51.441 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +31:51.441 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +31:51.441 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +31:51.942 [INFO] TestAgent: Waiting for endpoints to shut down +31:51.943 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsCreate_noBody (0.82s) +=== CONT TestSnapshot_Options +=== RUN TestSnapshot_Options/GET +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:49.491Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.506Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.550 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58357369-b5be-54e3-9419-baa3577b326c Address:127.0.0.1:29777}]" +31:49.551 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29777 [Follower]" leader= +31:49.556 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-58357369-b5be-54e3-9419-baa3577b326c.dc1 127.0.0.1 +31:49.564 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-58357369-b5be-54e3-9419-baa3577b326c 127.0.0.1 +2021-10-05T16:31:49.564Z [INFO] agent.router: Initializing LAN area manager +31:49.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29856 network=udp +31:49.565 [INFO] TestAgent.server: Adding LAN server: server="Node-58357369-b5be-54e3-9419-baa3577b326c (Addr: tcp/127.0.0.1:29777) (DC: dc1)" +31:49.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29856 network=tcp +31:49.565 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-58357369-b5be-54e3-9419-baa3577b326c.dc1 area=wan +31:49.566 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29773 network=tcp +31:49.566 [INFO] TestAgent: started state syncer +31:49.566 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29771 network=tcp +31:49.592 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.592 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29777 [Candidate]" term=2 +31:49.632 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.632 [DEBUG] TestAgent.server.raft: vote granted: from=58357369-b5be-54e3-9419-baa3577b326c term=2 tally=1 +31:49.632 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.632 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29777 [Leader]" +31:49.633 [INFO] TestAgent.server: cluster leadership acquired +31:49.633 [INFO] TestAgent.server: New leader elected: payload=Node-58357369-b5be-54e3-9419-baa3577b326c +31:49.655 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29777 +31:49.680 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.728 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.728 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.728 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.728 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.728 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.728 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58357369-b5be-54e3-9419-baa3577b326c +31:49.728 [INFO] TestAgent.server: member joined, marking health alive: member=Node-58357369-b5be-54e3-9419-baa3577b326c +31:49.756 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.878 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:49.891 [INFO] TestAgent: Synced node info +31:49.891 [DEBUG] TestAgent: Node info in sync +31:51.676 [INFO] TestAgent: Requesting shutdown +31:51.676 [INFO] TestAgent.server: shutting down server +31:51.676 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:51.676 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:51.676 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:51.676 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:51.676 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.676 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:51.676 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:51.676 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:51.676 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:51.676 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:51.688 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.696Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.696Z [INFO] agent.router.manager: shutting down +31:51.696 [INFO] TestAgent: consul server down +31:51.696 [INFO] TestAgent: shutdown complete +31:51.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29856 network=tcp +31:51.696 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29856 network=udp +31:51.696 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29773 network=tcp +31:52.199 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.199 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_ServerHealth_Unhealthy (2.72s) +=== CONT TestAgent_Join +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:49.589Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:49.618Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:49.700 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:41891d93-afb4-f98e-f7fc-7fcca1cdc75e Address:127.0.0.1:29854}]" +31:49.701 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e.dc1 127.0.0.1 +31:49.701 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29854 [Follower]" leader= +31:49.701 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e 127.0.0.1 +2021-10-05T16:31:49.701Z [INFO] agent.router: Initializing LAN area manager +31:49.702 [INFO] TestAgent.server: Adding LAN server: server="Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e (Addr: tcp/127.0.0.1:29854) (DC: dc1)" +31:49.702 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e.dc1 area=wan +31:49.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29772 network=udp +31:49.702 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29772 network=tcp +31:49.714 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29850 network=tcp +31:49.715 [INFO] TestAgent: started state syncer +31:49.715 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29883 network=tcp +31:49.752 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:49.752 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29854 [Candidate]" term=2 +31:49.784 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:49.784 [DEBUG] TestAgent.server.raft: vote granted: from=41891d93-afb4-f98e-f7fc-7fcca1cdc75e term=2 tally=1 +31:49.784 [INFO] TestAgent.server.raft: election won: tally=1 +31:49.784 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29854 [Leader]" +31:49.784 [INFO] TestAgent.server: cluster leadership acquired +31:49.784 [INFO] TestAgent.server: New leader elected: payload=Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e +31:49.796 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29854 +31:49.822 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:49.872 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:49.872 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:49.872 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:49.872 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:49.872 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:49.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e +31:49.872 [INFO] TestAgent.server: member joined, marking health alive: member=Node-41891d93-afb4-f98e-f7fc-7fcca1cdc75e +31:49.898 [INFO] TestAgent.server: federation state anti-entropy synced +31:49.927 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:49.937 [INFO] TestAgent: Synced node info +31:51.837 [INFO] TestAgent: Requesting shutdown +31:51.837 [INFO] TestAgent.server: shutting down server +31:51.837 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:51.837 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:51.837 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:51.837 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:51.837 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.838 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:51.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:51.838 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:51.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:51.838 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:51.844 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.852Z [INFO] agent.router.manager: shutting down +31:51.852 [INFO] TestAgent: consul server down +31:51.852 [INFO] TestAgent: shutdown complete +31:51.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29772 network=tcp +31:51.852 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29772 network=udp +31:51.852 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29850 network=tcp +31:52.353 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.353 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_ServerHealth (2.80s) +=== CONT TestAgentConnectCALeafCert_secondaryDC_good +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:51.814Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.828Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.873 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e186114a-78b4-eb51-9ce4-559e6d12760d Address:127.0.0.1:29951}]" +31:51.873 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29951 [Follower]" leader= +31:51.879 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e186114a-78b4-eb51-9ce4-559e6d12760d.dc1 127.0.0.1 +31:51.881 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e186114a-78b4-eb51-9ce4-559e6d12760d 127.0.0.1 +2021-10-05T16:31:51.881Z [INFO] agent.router: Initializing LAN area manager +31:51.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29946 network=udp +31:51.882 [INFO] TestAgent.server: Adding LAN server: server="Node-e186114a-78b4-eb51-9ce4-559e6d12760d (Addr: tcp/127.0.0.1:29951) (DC: dc1)" +31:51.882 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e186114a-78b4-eb51-9ce4-559e6d12760d.dc1 area=wan +31:51.882 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29946 network=tcp +31:51.882 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29947 network=tcp +31:51.882 [INFO] TestAgent: started state syncer +31:51.883 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30001 network=tcp +31:51.940 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.940 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29951 [Candidate]" term=2 +31:51.974 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.974 [DEBUG] TestAgent.server.raft: vote granted: from=e186114a-78b4-eb51-9ce4-559e6d12760d term=2 tally=1 +31:51.974 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.974 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29951 [Leader]" +31:51.975 [INFO] TestAgent.server: cluster leadership acquired +31:51.975 [INFO] TestAgent.server: New leader elected: payload=Node-e186114a-78b4-eb51-9ce4-559e6d12760d +31:51.984 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29951 +31:52.010 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.058 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.058 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.058 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.058 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.058 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.059 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e186114a-78b4-eb51-9ce4-559e6d12760d +31:52.059 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e186114a-78b4-eb51-9ce4-559e6d12760d +31:52.075 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.138 [INFO] TestAgent: Requesting shutdown +31:52.138 [INFO] TestAgent.server: shutting down server +31:52.138 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.138 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.138 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.138 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.138 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.138 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.138 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.138 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.138 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.143 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.156Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.156Z [INFO] agent.router.manager: shutting down +31:52.156 [INFO] TestAgent: consul server down +31:52.156 [INFO] TestAgent: shutdown complete +31:52.156 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29946 network=tcp +31:52.156 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29946 network=udp +31:52.156 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29947 network=tcp +31:52.657 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.657 [INFO] TestAgent: Endpoints down +--- PASS: TestStatusPeers (0.86s) === CONT TestAgentConnectCALeafCert_aclServiceReadDeny -2022-11-07T22:46:15.262Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.270Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.304 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:48c90f29-2a39-7315-597d-5a672f56fda5 Address:127.0.0.1:11228}]" -46:15.304 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11228 [Follower]" leader= -46:15.305 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-48c90f29-2a39-7315-597d-5a672f56fda5.dc1 127.0.0.1 -46:15.305 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-48c90f29-2a39-7315-597d-5a672f56fda5 127.0.0.1 -2022-11-07T22:46:15.305Z [INFO] agent.router: Initializing LAN area manager -46:15.306 [INFO] TestAgent.server: Adding LAN server: server="Node-48c90f29-2a39-7315-597d-5a672f56fda5 (Addr: tcp/127.0.0.1:11228) (DC: dc1)" -46:15.306 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-48c90f29-2a39-7315-597d-5a672f56fda5.dc1 area=wan -46:15.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11181 network=tcp -46:15.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11181 network=udp -46:15.306 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11182 network=tcp -46:15.306 [INFO] TestAgent: started state syncer -46:15.306 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11229 network=tcp -46:15.349 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.349 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11228 [Candidate]" term=2 -46:15.364 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.364 [DEBUG] TestAgent.server.raft: vote granted: from=48c90f29-2a39-7315-597d-5a672f56fda5 term=2 tally=1 -46:15.364 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.364 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11228 [Leader]" -46:15.364 [INFO] TestAgent.server: cluster leadership acquired -46:15.364 [INFO] TestAgent.server: New leader elected: payload=Node-48c90f29-2a39-7315-597d-5a672f56fda5 -46:15.373 [INFO] TestAgent: Requesting shutdown -46:15.373 [INFO] TestAgent.server: shutting down server -46:15.373 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.373 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.375 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11228 -46:15.378 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.387Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.387Z [INFO] agent.router.manager: shutting down -46:15.394 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" -46:15.394 [INFO] TestAgent: consul server down -46:15.394 [INFO] TestAgent: shutdown complete -46:15.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11181 network=tcp -46:15.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11181 network=udp -46:15.394 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11182 network=tcp -46:15.894 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.894 [INFO] TestAgent: Endpoints down ---- PASS: TestPreparedQuery_Execute (6.59s) - --- PASS: TestPreparedQuery_Execute/#00 (0.99s) - --- PASS: TestPreparedQuery_Execute/#01 (0.81s) - --- PASS: TestPreparedQuery_Execute/#02 (0.94s) - --- PASS: TestPreparedQuery_Execute/#03 (0.68s) - --- PASS: TestPreparedQuery_Execute/#04 (0.67s) - --- PASS: TestPreparedQuery_Execute/#05 (0.95s) - --- PASS: TestPreparedQuery_Execute/#06 (0.93s) - --- PASS: TestPreparedQuery_Execute/#07 (0.64s) +2021-10-05T16:31:51.964Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.993Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.046 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e4196fa9-8879-a445-8e0d-a0e6e79c3225 Address:127.0.0.1:29993}]" +31:52.046 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225.dc1 127.0.0.1 +31:52.047 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225 127.0.0.1 +2021-10-05T16:31:52.047Z [INFO] agent.router: Initializing LAN area manager +31:52.047 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29988 network=udp +31:52.047 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29993 [Follower]" leader= +31:52.048 [INFO] TestAgent.server: Adding LAN server: server="Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225 (Addr: tcp/127.0.0.1:29993) (DC: dc1)" +31:52.048 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225.dc1 area=wan +31:52.048 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29988 network=tcp +31:52.048 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29989 network=tcp +31:52.048 [INFO] TestAgent: started state syncer +31:52.048 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29973 network=tcp +31:52.113 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.113 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29993 [Candidate]" term=2 +31:52.138 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.138 [DEBUG] TestAgent.server.raft: vote granted: from=e4196fa9-8879-a445-8e0d-a0e6e79c3225 term=2 tally=1 +31:52.138 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29993 [Leader]" +31:52.138 [INFO] TestAgent.server: cluster leadership acquired +31:52.138 [INFO] TestAgent.server: New leader elected: payload=Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225 +31:52.149 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29993 +31:52.167 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.204 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.204 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.204 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.204 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.204 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.204 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225 +31:52.204 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e4196fa9-8879-a445-8e0d-a0e6e79c3225 +31:52.228 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.251 [INFO] TestAgent: Requesting shutdown +31:52.251 [INFO] TestAgent.server: shutting down server +31:52.251 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.251 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.251 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.251 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.251 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.251 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.251 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.251 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.251 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.251 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.251 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.266 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.271Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.271Z [INFO] agent.router.manager: shutting down +31:52.271 [INFO] TestAgent: consul server down +31:52.271 [INFO] TestAgent: shutdown complete +31:52.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29988 network=tcp +31:52.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29988 network=udp +31:52.271 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29989 network=tcp +31:52.771 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.771 [INFO] TestAgent: Endpoints down +--- PASS: TestIntentionsCheck_noDestination (0.89s) === CONT TestAgentConnectCALeafCert_aclServiceWrite -2022-11-07T22:46:15.253Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.264Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.298 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:96ba6913-554b-6f97-c290-48a7917bc9d6 Address:127.0.0.1:11179}]" -46:15.298 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11179 [Follower]" leader= -46:15.298 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-96ba6913-554b-6f97-c290-48a7917bc9d6.dc1 127.0.0.1 -46:15.299 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-96ba6913-554b-6f97-c290-48a7917bc9d6 127.0.0.1 -2022-11-07T22:46:15.299Z [INFO] agent.router: Initializing LAN area manager -46:15.299 [INFO] TestAgent.server: Adding LAN server: server="Node-96ba6913-554b-6f97-c290-48a7917bc9d6 (Addr: tcp/127.0.0.1:11179) (DC: dc1)" -46:15.299 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-96ba6913-554b-6f97-c290-48a7917bc9d6.dc1 area=wan -46:15.299 [DEBUG] TestAgent.dns: recursor enabled -46:15.299 [DEBUG] TestAgent.dns: recursor enabled -46:15.299 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11188 network=tcp -46:15.299 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11188 network=udp -46:15.299 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11189 network=tcp -46:15.299 [INFO] TestAgent: started state syncer -46:15.299 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11180 network=tcp -46:15.337 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.337 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11179 [Candidate]" term=2 -46:15.351 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.351 [DEBUG] TestAgent.server.raft: vote granted: from=96ba6913-554b-6f97-c290-48a7917bc9d6 term=2 tally=1 -46:15.351 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.351 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11179 [Leader]" -46:15.351 [INFO] TestAgent.server: cluster leadership acquired -46:15.351 [INFO] TestAgent.server: New leader elected: payload=Node-96ba6913-554b-6f97-c290-48a7917bc9d6 -46:15.359 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11179 -46:15.376 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.408 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.408 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.408 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.408 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.408 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.408 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-96ba6913-554b-6f97-c290-48a7917bc9d6 -46:15.408 [INFO] TestAgent.server: member joined, marking health alive: member=Node-96ba6913-554b-6f97-c290-48a7917bc9d6 -46:15.418 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.429 [INFO] TestAgent: Synced node info -46:15.499 [DEBUG] TestAgent.dns: recurse failed for question: question="{google.com. 1 1}" rtt=113.724µs recursor=127.0.0.1:44158 rcode=SERVFAIL -46:15.499 [DEBUG] TestAgent.dns: recurse succeeded for question: question="{google.com. 1 1}" rtt=82.691µs recursor=127.0.0.1:45959 -46:15.499 [DEBUG] TestAgent.dns: request served from client: question="{google.com. 1 1}" network=udp latency=358.136µs client=127.0.0.1:40762 client_network=udp -46:15.499 [INFO] TestAgent: Requesting shutdown -46:15.499 [INFO] TestAgent.server: shutting down server -46:15.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.499 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.499 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.499 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.499 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.499 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.499 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.504 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.515Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.515Z [INFO] agent.router.manager: shutting down -46:15.515 [INFO] TestAgent: consul server down -46:15.515 [INFO] TestAgent: shutdown complete -46:15.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11188 network=tcp -46:15.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11188 network=udp -46:15.516 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11189 network=tcp -46:16.016 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.016 [INFO] TestAgent: Endpoints down ---- PASS: TestDNSCycleRecursorCheck (0.77s) -=== CONT TestAgent_AddCheck_ExecRemoteDisable -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:15.379Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.388Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.425 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a79f756-ed6d-f1fa-893c-ab635d54e55d Address:127.0.0.1:11217}]" -46:15.425 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11217 [Follower]" leader= -46:15.425 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d.dc1 127.0.0.1 -46:15.426 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d 127.0.0.1 -2022-11-07T22:46:15.426Z [INFO] agent.router: Initializing LAN area manager -46:15.426 [INFO] TestAgent.server: Adding LAN server: server="Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d (Addr: tcp/127.0.0.1:11217) (DC: dc1)" -46:15.426 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d.dc1 area=wan -46:15.426 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11184 network=udp -46:15.426 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11184 network=tcp -46:15.427 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11213 network=tcp -46:15.427 [INFO] TestAgent: started state syncer -46:15.427 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11232 network=tcp -46:15.484 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.484 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11217 [Candidate]" term=2 -46:15.496 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.496 [DEBUG] TestAgent.server.raft: vote granted: from=1a79f756-ed6d-f1fa-893c-ab635d54e55d term=2 tally=1 -46:15.496 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.496 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11217 [Leader]" -46:15.496 [INFO] TestAgent.server: cluster leadership acquired -46:15.496 [INFO] TestAgent.server: New leader elected: payload=Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d -46:15.503 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11217 -46:15.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.557 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.557 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.557 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.557 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.557 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.557 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d -46:15.557 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a79f756-ed6d-f1fa-893c-ab635d54e55d -46:15.574 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.605 [WARN] TestAgent: check has interval below minimum: check=http minimum_interval=1s -46:15.605 [WARN] TestAgent: check has interval below minimum: check=grpc minimum_interval=1s -46:15.605 [INFO] TestAgent: Requesting shutdown -46:15.606 [INFO] TestAgent.server: shutting down server -46:15.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.606 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.606 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.606 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.606 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.606 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.606 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.606 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.618 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.624Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.624Z [INFO] agent.router.manager: shutting down -46:15.624 [INFO] TestAgent: consul server down -46:15.624 [INFO] TestAgent: shutdown complete -46:15.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11184 network=tcp -46:15.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11184 network=udp -46:15.624 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11213 network=tcp -46:16.124 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.124 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RerouteNewHTTPChecks (0.76s) -=== CONT TestAgent_consulConfig_RaftTrailingLogs -2022-11-07T22:46:15.304Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.315Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.352 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 Address:127.0.0.1:11168}]" -46:15.352 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11168 [Follower]" leader= -46:15.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3.dc1 127.0.0.1 -46:15.353 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 127.0.0.1 -2022-11-07T22:46:15.353Z [INFO] agent.router: Initializing LAN area manager -46:15.353 [INFO] TestAgent.server: Adding LAN server: server="Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 (Addr: tcp/127.0.0.1:11168) (DC: dc1)" -46:15.353 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3.dc1 area=wan -46:15.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11170 network=udp -46:15.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11170 network=tcp -46:15.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11164 network=tcp -46:15.353 [INFO] TestAgent: started state syncer -46:15.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11267 network=tcp -46:15.414 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.414 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11168 [Candidate]" term=2 -46:15.427 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.427 [DEBUG] TestAgent.server.raft: vote granted: from=a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 term=2 tally=1 -46:15.427 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.427 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11168 [Leader]" -46:15.427 [INFO] TestAgent.server: cluster leadership acquired -46:15.427 [INFO] TestAgent.server: New leader elected: payload=Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 -46:15.438 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11168 -46:15.463 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.503 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.503 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.503 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 -46:15.503 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a11ecb43-b36e-e5c1-186a-9d4d4e2e11f3 -46:15.522 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.559 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.568 [INFO] TestAgent: Synced node info -46:15.568 [DEBUG] TestAgent: Node info in sync -46:15.649 [DEBUG] TestAgent.dns: request served from client: name=bar.node.consul. type=AAAA class=IN latency=79.112µs client=127.0.0.1:60355 client_network=udp -46:15.649 [INFO] TestAgent: Requesting shutdown -46:15.650 [INFO] TestAgent.server: shutting down server -46:15.650 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.650 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.650 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.650 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.650 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.650 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.650 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.650 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.650 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.650 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.656 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.663Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.663Z [INFO] agent.router.manager: shutting down -46:15.663 [INFO] TestAgent: consul server down -46:15.663 [INFO] TestAgent: shutdown complete -46:15.663 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11170 network=tcp -46:15.663 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11170 network=udp -46:15.663 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11164 network=tcp -46:16.163 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.163 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_AAAA (0.87s) -=== CONT TestConfig_Apply -2022-11-07T22:46:15.326Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.335Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.377 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:86ab60dd-9674-75b0-ea5d-835c5034c297 Address:127.0.0.1:11175}]" -46:15.377 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11175 [Follower]" leader= -46:15.378 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-86ab60dd-9674-75b0-ea5d-835c5034c297.dc1 127.0.0.1 -46:15.378 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-86ab60dd-9674-75b0-ea5d-835c5034c297 127.0.0.1 -2022-11-07T22:46:15.378Z [INFO] agent.router: Initializing LAN area manager -46:15.378 [INFO] TestAgent.server: Adding LAN server: server="Node-86ab60dd-9674-75b0-ea5d-835c5034c297 (Addr: tcp/127.0.0.1:11175) (DC: dc1)" -46:15.378 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-86ab60dd-9674-75b0-ea5d-835c5034c297.dc1 area=wan -46:15.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11268 network=udp -46:15.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11268 network=tcp -46:15.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11171 network=tcp -46:15.379 [INFO] TestAgent: started state syncer -46:15.379 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11218 network=tcp -46:15.420 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.420 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11175 [Candidate]" term=2 -46:15.434 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.434 [DEBUG] TestAgent.server.raft: vote granted: from=86ab60dd-9674-75b0-ea5d-835c5034c297 term=2 tally=1 -46:15.434 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.434 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11175 [Leader]" -46:15.434 [INFO] TestAgent.server: cluster leadership acquired -46:15.435 [INFO] TestAgent.server: New leader elected: payload=Node-86ab60dd-9674-75b0-ea5d-835c5034c297 -46:15.447 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11175 -46:15.471 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.509 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.509 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.509 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.509 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.509 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.509 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-86ab60dd-9674-75b0-ea5d-835c5034c297 -46:15.509 [INFO] TestAgent.server: member joined, marking health alive: member=Node-86ab60dd-9674-75b0-ea5d-835c5034c297 -46:15.520 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.597 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.613 [INFO] TestAgent: Synced node info -46:15.664 [INFO] TestAgent: Requesting shutdown -46:15.664 [INFO] TestAgent.server: shutting down server -46:15.664 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.664 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.664 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.664 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.664 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.664 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.664 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.664 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.664 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.664 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.670 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.681Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.681Z [INFO] agent.router.manager: shutting down -46:15.681 [INFO] TestAgent: consul server down -46:15.681 [INFO] TestAgent: shutdown complete -46:15.681 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11268 network=tcp -46:15.681 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11268 network=udp -46:15.681 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11171 network=tcp -46:16.182 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.182 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_CAS (0.87s) -=== CONT TestAgent_AddCheck_ExecDisable -2022-11-07T22:46:15.438Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.448Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.484 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:961f1510-83ea-ca05-e014-5baa0718bc73 Address:127.0.0.1:11259}]" -46:15.484 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11259 [Follower]" leader= -46:15.484 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-961f1510-83ea-ca05-e014-5baa0718bc73.dc1 127.0.0.1 -46:15.485 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-961f1510-83ea-ca05-e014-5baa0718bc73 127.0.0.1 -2022-11-07T22:46:15.485Z [INFO] agent.router: Initializing LAN area manager -46:15.485 [INFO] TestAgent.server: Adding LAN server: server="Node-961f1510-83ea-ca05-e014-5baa0718bc73 (Addr: tcp/127.0.0.1:11259) (DC: dc1)" -46:15.485 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-961f1510-83ea-ca05-e014-5baa0718bc73.dc1 area=wan -46:15.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11289 network=udp -46:15.485 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11289 network=tcp -46:15.486 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11255 network=tcp -46:15.486 [INFO] TestAgent: started state syncer -46:15.486 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11253 network=tcp -46:15.536 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.536 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11259 [Candidate]" term=2 -46:15.550 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.550 [DEBUG] TestAgent.server.raft: vote granted: from=961f1510-83ea-ca05-e014-5baa0718bc73 term=2 tally=1 -46:15.550 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.550 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11259 [Leader]" -46:15.550 [INFO] TestAgent.server: cluster leadership acquired -46:15.551 [INFO] TestAgent.server: New leader elected: payload=Node-961f1510-83ea-ca05-e014-5baa0718bc73 -46:15.568 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11259 -46:15.585 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.632 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.632 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.632 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.632 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.632 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.632 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-961f1510-83ea-ca05-e014-5baa0718bc73 -46:15.632 [INFO] TestAgent.server: member joined, marking health alive: member=Node-961f1510-83ea-ca05-e014-5baa0718bc73 -46:15.652 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.733 [INFO] TestAgent: Requesting shutdown -46:15.734 [INFO] TestAgent.server: shutting down server -46:15.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.734 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.734 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.734 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.734 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.734 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.734 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.734 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.736 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.741Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.741Z [INFO] agent.router.manager: shutting down -46:15.741 [INFO] TestAgent: consul server down -46:15.741 [INFO] TestAgent: shutdown complete -46:15.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11289 network=tcp -46:15.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11289 network=udp -46:15.741 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11255 network=tcp -46:16.241 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.241 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_ProxyDefaultsMeshGateway (0.81s) +2021-10-05T16:31:51.988Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:51.988Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.011Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.011Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.068 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8c4f9841-f84c-07cd-ca94-303cb1917dd6 Address:127.0.0.1:30021}]" +31:52.069 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30021 [Follower]" leader= +31:52.069 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6.dc1 127.0.0.1 +31:52.070 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 127.0.0.1 +2021-10-05T16:31:52.070Z [INFO] agent.router: Initializing LAN area manager +31:52.070 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29995 network=udp +31:52.070 [INFO] TestAgent.server: Adding LAN server: server="Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 (Addr: tcp/127.0.0.1:30021) (DC: dc1)" +31:52.070 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6.dc1 area=wan +31:52.070 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29995 network=tcp +31:52.071 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30017 network=tcp +31:52.071 [INFO] TestAgent: started state syncer +31:52.071 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29987 network=tcp +31:52.109 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.109 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30021 [Candidate]" term=2 +31:52.124 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:52.128 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.128 [DEBUG] TestAgent.server.raft: vote granted: from=8c4f9841-f84c-07cd-ca94-303cb1917dd6 term=2 tally=1 +31:52.128 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.128 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30021 [Leader]" +31:52.128 [INFO] TestAgent.server: cluster leadership acquired +31:52.128 [INFO] TestAgent.server: New leader elected: payload=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.142 [INFO] TestAgent.server: initializing acls +31:52.148 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:52.148 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:52.159 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:52.174 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:52.174 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:52.174 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:52.174 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.174 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6.dc1 +31:52.175 [INFO] TestAgent.server: Updating LAN server: server="Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 (Addr: tcp/127.0.0.1:30021) (DC: dc1)" +31:52.175 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6.dc1 area=wan +31:52.194 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.234 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.234 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.234 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.234 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.234 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.234 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.234 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.256 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8c4f9841-f84c-07cd-ca94-303cb1917dd6 +31:52.256 [INFO] TestAgent: Requesting shutdown +31:52.256 [INFO] TestAgent.server: shutting down server +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:52.256 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.256 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.256 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.256 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.256 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.256 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.256 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:52.256 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:52.264 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.272Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.272Z [INFO] agent.router.manager: shutting down +31:52.272 [INFO] TestAgent: consul server down +31:52.272 [INFO] TestAgent: shutdown complete +31:52.272 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29995 network=tcp +31:52.272 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29995 network=udp +31:52.272 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30017 network=tcp +31:52.272 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +31:52.272 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.773 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.773 [INFO] TestAgent: Endpoints down +=== RUN TestSnapshot_Options/GET#01 +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:51.928Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.963Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.034 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0283a109-d299-8bc1-083f-e199e329a5d8 Address:127.0.0.1:30000}]" +31:52.034 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30000 [Follower]" leader= +31:52.035 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0283a109-d299-8bc1-083f-e199e329a5d8.dc1 127.0.0.1 +31:52.035 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0283a109-d299-8bc1-083f-e199e329a5d8 127.0.0.1 +2021-10-05T16:31:52.035Z [INFO] agent.router: Initializing LAN area manager +31:52.035 [INFO] TestAgent.server: Adding LAN server: server="Node-0283a109-d299-8bc1-083f-e199e329a5d8 (Addr: tcp/127.0.0.1:30000) (DC: dc1)" +31:52.035 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0283a109-d299-8bc1-083f-e199e329a5d8.dc1 area=wan +31:52.035 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29974 network=udp +31:52.035 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29974 network=tcp +31:52.036 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29996 network=tcp +31:52.036 [INFO] TestAgent: started state syncer +31:52.036 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29980 network=tcp +31:52.071 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.071 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30000 [Candidate]" term=2 +31:52.087 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.087 [DEBUG] TestAgent.server.raft: vote granted: from=0283a109-d299-8bc1-083f-e199e329a5d8 term=2 tally=1 +31:52.087 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.087 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30000 [Leader]" +31:52.087 [INFO] TestAgent.server: cluster leadership acquired +31:52.087 [INFO] TestAgent.server: New leader elected: payload=Node-0283a109-d299-8bc1-083f-e199e329a5d8 +31:52.103 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30000 +31:52.120 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.159 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.159 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.159 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.159 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.159 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.159 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0283a109-d299-8bc1-083f-e199e329a5d8 +31:52.159 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0283a109-d299-8bc1-083f-e199e329a5d8 +31:52.185 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.336 [INFO] TestAgent: Requesting shutdown +31:52.337 [INFO] TestAgent.server: shutting down server +31:52.337 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.337 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.337 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.337 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.337 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.337 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.337 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.337 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.337 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.337 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.337 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.348 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.360Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.360Z [INFO] agent.router.manager: shutting down +31:52.360 [INFO] TestAgent: consul server down +31:52.360 [INFO] TestAgent: shutdown complete +31:52.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29974 network=tcp +31:52.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29974 network=udp +31:52.360 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29996 network=tcp +31:52.864 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.864 [INFO] TestAgent: Endpoints down +--- PASS: TestStatusLeader (0.98s) === CONT TestAgentConnectCALeafCert_aclDefaultDeny -2022-11-07T22:46:15.331Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.340Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.380 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9aa5d476-be57-68ec-773e-3bdaf50b5d1a Address:127.0.0.1:11266}]" -46:15.380 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11266 [Follower]" leader= -46:15.381 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a.dc1 127.0.0.1 -46:15.381 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a 127.0.0.1 -2022-11-07T22:46:15.381Z [INFO] agent.router: Initializing LAN area manager -46:15.381 [INFO] TestAgent.server: Adding LAN server: server="Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a (Addr: tcp/127.0.0.1:11266) (DC: dc1)" -46:15.381 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a.dc1 area=wan -46:15.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11219 network=udp -46:15.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11219 network=tcp -46:15.382 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11262 network=tcp -46:15.382 [INFO] TestAgent: started state syncer -46:15.382 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11183 network=tcp -46:15.424 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.424 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11266 [Candidate]" term=2 -46:15.438 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.438 [DEBUG] TestAgent.server.raft: vote granted: from=9aa5d476-be57-68ec-773e-3bdaf50b5d1a term=2 tally=1 -46:15.438 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.438 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11266 [Leader]" -46:15.438 [INFO] TestAgent.server: cluster leadership acquired -46:15.438 [INFO] TestAgent.server: New leader elected: payload=Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a -46:15.447 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11266 -46:15.465 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.508 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.508 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.508 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.508 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.508 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.508 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a -46:15.508 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9aa5d476-be57-68ec-773e-3bdaf50b5d1a -46:15.525 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.603 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.622 [INFO] TestAgent: Synced node info -46:15.622 [DEBUG] TestAgent: Node info in sync -46:15.749 [INFO] TestAgent: Requesting shutdown -46:15.749 [INFO] TestAgent.server: shutting down server -46:15.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.749 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.749 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.749 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.749 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.749 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.749 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.753 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.758Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.758Z [INFO] agent.router.manager: shutting down -46:15.758 [INFO] TestAgent: consul server down -46:15.758 [INFO] TestAgent: shutdown complete -46:15.758 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11219 network=tcp -46:15.758 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11219 network=udp -46:15.758 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11262 network=tcp -46:16.259 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.259 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_Alias (0.94s) -=== CONT TestAgent_consulConfig_AutoEncryptAllowTLS -2022-11-07T22:46:15.488Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.499Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.548 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c8c1b18a-a426-8f38-6597-d4e891c09cc6 Address:127.0.0.1:11308}]" -46:15.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6.dc1 127.0.0.1 -46:15.549 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11308 [Follower]" leader= -46:15.550 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6 127.0.0.1 -2022-11-07T22:46:15.550Z [INFO] agent.router: Initializing LAN area manager -46:15.550 [INFO] TestAgent.server: Adding LAN server: server="Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6 (Addr: tcp/127.0.0.1:11308) (DC: dc1)" -46:15.550 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6.dc1 area=wan -46:15.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11240 network=udp -46:15.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11240 network=tcp -46:15.550 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11304 network=tcp -46:15.551 [INFO] TestAgent: started state syncer -46:15.551 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11337 network=tcp -46:15.600 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.600 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11308 [Candidate]" term=2 -46:15.625 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.625 [DEBUG] TestAgent.server.raft: vote granted: from=c8c1b18a-a426-8f38-6597-d4e891c09cc6 term=2 tally=1 -46:15.625 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.625 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11308 [Leader]" -46:15.625 [INFO] TestAgent.server: cluster leadership acquired -46:15.625 [INFO] TestAgent.server: New leader elected: payload=Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6 -46:15.635 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11308 -46:15.654 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6 -46:15.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c8c1b18a-a426-8f38-6597-d4e891c09cc6 -46:15.691 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.767 [INFO] TestAgent: Requesting shutdown -46:15.767 [INFO] TestAgent.server: shutting down server -46:15.767 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.767 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.767 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.767 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.767 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.767 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.767 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.767 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.767 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.767 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.767 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.770 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.774Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.774Z [INFO] agent.router.manager: shutting down -46:15.774 [INFO] TestAgent: consul server down -46:15.774 [INFO] TestAgent: shutdown complete -46:15.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11240 network=tcp -46:15.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11240 network=udp -46:15.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11304 network=tcp -46:16.274 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.274 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_GRPC (0.80s) +2021-10-05T16:31:51.471Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.511Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.604 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04f6becf-87f7-acdf-b927-9d2050c5b397 Address:127.0.0.1:29868}]" +31:51.604 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29868 [Follower]" leader= +31:51.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04f6becf-87f7-acdf-b927-9d2050c5b397.secondary 127.0.0.1 +31:51.605 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04f6becf-87f7-acdf-b927-9d2050c5b397 127.0.0.1 +2021-10-05T16:31:51.605Z [INFO] agent.router: Initializing LAN area manager +31:51.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29870 network=udp +31:51.606 [INFO] TestAgent.server: Adding LAN server: server="Node-04f6becf-87f7-acdf-b927-9d2050c5b397 (Addr: tcp/127.0.0.1:29868) (DC: secondary)" +31:51.606 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04f6becf-87f7-acdf-b927-9d2050c5b397.secondary area=wan +31:51.606 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29870 network=tcp +31:51.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29864 network=tcp +31:51.607 [INFO] TestAgent: started state syncer +31:51.607 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29938 network=tcp +31:51.664 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.664 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29868 [Candidate]" term=2 +31:51.704 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.704 [DEBUG] TestAgent.server.raft: vote granted: from=04f6becf-87f7-acdf-b927-9d2050c5b397 term=2 tally=1 +31:51.704 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.704 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29868 [Leader]" +31:51.705 [INFO] TestAgent.server: cluster leadership acquired +31:51.705 [INFO] TestAgent.server: New leader elected: payload=Node-04f6becf-87f7-acdf-b927-9d2050c5b397 +31:51.732 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29868 +31:51.786 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.850 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:51.850 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:51.850 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:51.850 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:51.850 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:51.850 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04f6becf-87f7-acdf-b927-9d2050c5b397 +31:51.850 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04f6becf-87f7-acdf-b927-9d2050c5b397 +31:51.876 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29936] +31:51.880 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29936 +31:51.893 [INFO] TestAgent.server: federation state anti-entropy synced +31:51.899 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1fd99831-7a92-b5d3-c201-f834d2196c7f.primary 127.0.0.1 +31:51.899 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:51.900 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1fd99831-7a92-b5d3-c201-f834d2196c7f.primary area=wan +31:51.904 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:51.907 [INFO] TestAgent: Requesting shutdown +31:51.907 [INFO] TestAgent.server: shutting down server +31:51.907 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:51.907 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:51.907 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:51.907 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:51.907 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:51.912 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:51.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:51.912 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:51.912 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:51.912 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:51.928 [INFO] TestAgent: Synced node info +31:51.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:51.956Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.956Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:51.956Z [INFO] agent.router.manager: shutting down +31:51.957 [INFO] TestAgent: consul server down +31:51.957 [INFO] TestAgent: shutdown complete +31:51.957 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29870 network=tcp +31:51.957 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29870 network=udp +31:51.957 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29864 network=tcp +31:52.461 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.461 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:51.207Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.251Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.336 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1fd99831-7a92-b5d3-c201-f834d2196c7f Address:127.0.0.1:29937}]" +31:51.337 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1fd99831-7a92-b5d3-c201-f834d2196c7f.primary 127.0.0.1 +31:51.337 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1fd99831-7a92-b5d3-c201-f834d2196c7f 127.0.0.1 +2021-10-05T16:31:51.337Z [INFO] agent.router: Initializing LAN area manager +31:51.338 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29911 network=udp +31:51.338 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29937 [Follower]" leader= +31:51.338 [INFO] TestAgent.server: Adding LAN server: server="Node-1fd99831-7a92-b5d3-c201-f834d2196c7f (Addr: tcp/127.0.0.1:29937) (DC: primary)" +31:51.338 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1fd99831-7a92-b5d3-c201-f834d2196c7f.primary area=wan +31:51.338 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29911 network=tcp +31:51.339 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29933 network=tcp +31:51.339 [INFO] TestAgent: started state syncer +31:51.339 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29924 network=tcp +31:51.400 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.400 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29937 [Candidate]" term=2 +31:51.427 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.427 [DEBUG] TestAgent.server.raft: vote granted: from=1fd99831-7a92-b5d3-c201-f834d2196c7f term=2 tally=1 +31:51.427 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.427 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29937 [Leader]" +31:51.428 [INFO] TestAgent.server: cluster leadership acquired +31:51.428 [INFO] TestAgent.server: New leader elected: payload=Node-1fd99831-7a92-b5d3-c201-f834d2196c7f +31:51.437 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29937 +31:51.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.536 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:51.536 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:51.536 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:51.536 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:51.536 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:51.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1fd99831-7a92-b5d3-c201-f834d2196c7f +31:51.536 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1fd99831-7a92-b5d3-c201-f834d2196c7f +31:51.566 [INFO] TestAgent.server: federation state anti-entropy synced +31:51.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:51.656 [INFO] TestAgent: Synced node info +31:51.656 [DEBUG] TestAgent: Node info in sync +31:51.880 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:58902 +31:51.896 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04f6becf-87f7-acdf-b927-9d2050c5b397.secondary 127.0.0.1 +31:51.897 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04f6becf-87f7-acdf-b927-9d2050c5b397.secondary area=wan +31:52.461 [INFO] TestAgent: Requesting shutdown +31:52.461 [INFO] TestAgent.server: shutting down server +31:52.461 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.461 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.461 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.461 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.461 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.461 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.461 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.461 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.461 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.461 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.470 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.474Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.474Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.474Z [INFO] agent.router.manager: shutting down +31:52.474 [INFO] TestAgent: consul server down +31:52.474 [INFO] TestAgent: shutdown complete +31:52.474 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29911 network=tcp +31:52.475 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29911 network=udp +31:52.475 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29933 network=tcp +31:52.975 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.975 [INFO] TestAgent: Endpoints down +--- PASS: TestStatusLeaderSecondary (1.80s) === CONT TestAgentConnectCARoots_list -2022-11-07T22:46:15.556Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.566Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.595 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8254b4bc-b97c-5727-14c9-775c8507b07f Address:127.0.0.1:11364}]" -46:15.595 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11364 [Follower]" leader= -46:15.596 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8254b4bc-b97c-5727-14c9-775c8507b07f.dc1 127.0.0.1 -46:15.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8254b4bc-b97c-5727-14c9-775c8507b07f 127.0.0.1 -2022-11-07T22:46:15.597Z [INFO] agent.router: Initializing LAN area manager -46:15.597 [INFO] TestAgent.server: Adding LAN server: server="Node-8254b4bc-b97c-5727-14c9-775c8507b07f (Addr: tcp/127.0.0.1:11364) (DC: dc1)" -46:15.597 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8254b4bc-b97c-5727-14c9-775c8507b07f.dc1 area=wan -46:15.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11338 network=udp -46:15.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11338 network=tcp -46:15.600 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11360 network=tcp -46:15.600 [INFO] TestAgent: started state syncer -46:15.600 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11365 network=tcp -46:15.663 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.663 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11364 [Candidate]" term=2 -46:15.675 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.675 [DEBUG] TestAgent.server.raft: vote granted: from=8254b4bc-b97c-5727-14c9-775c8507b07f term=2 tally=1 -46:15.675 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.675 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11364 [Leader]" -46:15.675 [INFO] TestAgent.server: cluster leadership acquired -46:15.675 [INFO] TestAgent.server: New leader elected: payload=Node-8254b4bc-b97c-5727-14c9-775c8507b07f -46:15.683 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11364 -46:15.700 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.718 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.718 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.718 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.718 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.718 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.718 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8254b4bc-b97c-5727-14c9-775c8507b07f -46:15.718 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8254b4bc-b97c-5727-14c9-775c8507b07f -46:15.731 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.761 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.764 [INFO] TestAgent: Synced node info -46:15.764 [DEBUG] TestAgent: Node info in sync -46:15.777 [INFO] TestAgent: Requesting shutdown -46:15.777 [INFO] TestAgent.server: shutting down server -46:15.777 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.777 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.777 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.777 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.777 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.777 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.777 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.777 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.777 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.777 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.781 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.784Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.784Z [INFO] agent.router.manager: shutting down -46:15.784 [INFO] TestAgent: consul server down -46:15.784 [INFO] TestAgent: shutdown complete -46:15.784 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11338 network=tcp -46:15.784 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11338 network=udp -46:15.784 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11360 network=tcp -46:16.285 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.285 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_IngressGateway (0.74s) -=== CONT TestAgent_AddCheck_RestoreState -2022-11-07T22:46:15.472Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.480Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.523 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fe96630-9bed-7136-6d21-4f78d4632d5e Address:127.0.0.1:11245}]" -46:15.524 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11245 [Follower]" leader= -46:15.524 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fe96630-9bed-7136-6d21-4f78d4632d5e.dc1 127.0.0.1 -46:15.524 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fe96630-9bed-7136-6d21-4f78d4632d5e 127.0.0.1 -2022-11-07T22:46:15.524Z [INFO] agent.router: Initializing LAN area manager -46:15.524 [INFO] TestAgent.server: Adding LAN server: server="Node-2fe96630-9bed-7136-6d21-4f78d4632d5e (Addr: tcp/127.0.0.1:11245) (DC: dc1)" -46:15.524 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fe96630-9bed-7136-6d21-4f78d4632d5e.dc1 area=wan -46:15.524 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11254 network=udp -46:15.525 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11254 network=tcp -46:15.525 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11241 network=tcp -46:15.525 [INFO] TestAgent: started state syncer -46:15.525 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11239 network=tcp -46:15.568 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.568 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11245 [Candidate]" term=2 -46:15.580 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.580 [DEBUG] TestAgent.server.raft: vote granted: from=2fe96630-9bed-7136-6d21-4f78d4632d5e term=2 tally=1 -46:15.580 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.580 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11245 [Leader]" -46:15.580 [INFO] TestAgent.server: cluster leadership acquired -46:15.580 [INFO] TestAgent.server: New leader elected: payload=Node-2fe96630-9bed-7136-6d21-4f78d4632d5e -46:15.589 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11245 -46:15.620 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.659 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.659 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.659 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.659 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.659 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.659 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fe96630-9bed-7136-6d21-4f78d4632d5e -46:15.659 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2fe96630-9bed-7136-6d21-4f78d4632d5e -46:15.678 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.799 [INFO] TestAgent.server.connect: CA provider config updated -46:15.799 [INFO] TestAgent: Requesting shutdown -46:15.799 [INFO] TestAgent.server: shutting down server -46:15.799 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.799 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.799 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.799 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.799 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.799 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.799 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.799 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.799 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.799 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.799 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.802 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.806Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.806Z [INFO] agent.router.manager: shutting down -46:15.806 [INFO] TestAgent: consul server down -46:15.806 [INFO] TestAgent: shutdown complete -46:15.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11254 network=tcp -46:15.806 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11254 network=udp -46:15.806 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11241 network=tcp -46:16.306 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.306 [INFO] TestAgent: Endpoints down -=== RUN TestConnectCAConfig/force_without_cross_sign_snake_case -2022-11-07T22:46:15.403Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.412Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.458 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f304df61-b50c-01da-31a9-5b81ee595673 Address:127.0.0.1:11203}]" -46:15.458 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11203 [Follower]" leader= -46:15.459 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f304df61-b50c-01da-31a9-5b81ee595673.dc1 127.0.0.1 -46:15.459 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f304df61-b50c-01da-31a9-5b81ee595673 127.0.0.1 -2022-11-07T22:46:15.459Z [INFO] agent.router: Initializing LAN area manager -46:15.459 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f304df61-b50c-01da-31a9-5b81ee595673.dc1 area=wan -46:15.459 [INFO] TestAgent.server: Adding LAN server: server="Node-f304df61-b50c-01da-31a9-5b81ee595673 (Addr: tcp/127.0.0.1:11203) (DC: dc1)" -46:15.459 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11233 network=udp -46:15.459 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11233 network=tcp -46:15.460 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11199 network=tcp -46:15.460 [INFO] TestAgent: started state syncer -46:15.460 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11260 network=tcp -46:15.525 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.525 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11203 [Candidate]" term=2 -46:15.542 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.542 [DEBUG] TestAgent.server.raft: vote granted: from=f304df61-b50c-01da-31a9-5b81ee595673 term=2 tally=1 -46:15.542 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.542 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11203 [Leader]" -46:15.542 [INFO] TestAgent.server: cluster leadership acquired -46:15.542 [INFO] TestAgent.server: New leader elected: payload=Node-f304df61-b50c-01da-31a9-5b81ee595673 -46:15.554 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11203 -46:15.574 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.611 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.611 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.611 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.611 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.611 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.612 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f304df61-b50c-01da-31a9-5b81ee595673 -46:15.612 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f304df61-b50c-01da-31a9-5b81ee595673 -46:15.632 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.829 [DEBUG] TestAgent.dns: request served from client: name=foo.bar.node.consul. type=ANY class=IN latency=75.495µs client=127.0.0.1:40055 client_network=udp -46:15.829 [INFO] TestAgent: Requesting shutdown -46:15.829 [INFO] TestAgent.server: shutting down server -46:15.829 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.829 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.829 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.829 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.829 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.829 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.829 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.829 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.829 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.829 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.829 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.833 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.837Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.837Z [INFO] agent.router.manager: shutting down -46:15.837 [INFO] TestAgent: consul server down -46:15.837 [INFO] TestAgent: shutdown complete -46:15.837 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11233 network=tcp -46:15.837 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11233 network=udp -46:15.837 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11199 network=tcp -46:16.338 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.338 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_NodeLookup_PeriodName (0.94s) +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:51.692Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.723Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.816 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:678f51a7-51ef-ef7b-2cea-d9f98839e1ce Address:127.0.0.1:29944}]" +31:51.816 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29944 [Follower]" leader= +31:51.817 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce.secondary 127.0.0.1 +31:51.817 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce 127.0.0.1 +2021-10-05T16:31:51.817Z [INFO] agent.router: Initializing LAN area manager +31:51.818 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29939 network=udp +31:51.818 [INFO] TestAgent.server: Adding LAN server: server="Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce (Addr: tcp/127.0.0.1:29944) (DC: secondary)" +31:51.818 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce.secondary area=wan +31:51.818 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29939 network=tcp +31:51.819 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29940 network=tcp +31:51.819 [INFO] TestAgent: started state syncer +31:51.819 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29945 network=tcp +31:51.856 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.856 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29944 [Candidate]" term=2 +31:51.883 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.883 [DEBUG] TestAgent.server.raft: vote granted: from=678f51a7-51ef-ef7b-2cea-d9f98839e1ce term=2 tally=1 +31:51.883 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.883 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29944 [Leader]" +31:51.884 [INFO] TestAgent.server: cluster leadership acquired +31:51.884 [INFO] TestAgent.server: New leader elected: payload=Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce +31:51.917 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29944 +31:51.954 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.011 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.011 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.011 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.011 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.011 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.012 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce +31:52.012 [INFO] TestAgent.server: member joined, marking health alive: member=Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce +31:52.030 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29881] +31:52.030 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29881 +31:52.031 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7d1b16ab-753b-8942-3fe8-3906e061001e.primary 127.0.0.1 +31:52.031 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:52.031 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7d1b16ab-753b-8942-3fe8-3906e061001e.primary area=wan +31:52.039 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.041 [INFO] TestAgent: Requesting shutdown +31:52.041 [INFO] TestAgent.server: shutting down server +31:52.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.042 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.042 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.042 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.042 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.042 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.042 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.042 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.052 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.059Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.059Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.059Z [INFO] agent.router.manager: shutting down +31:52.059 [INFO] TestAgent: consul server down +31:52.060 [INFO] TestAgent: shutdown complete +31:52.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29939 network=tcp +31:52.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29939 network=udp +31:52.060 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29940 network=tcp +31:52.564 [INFO] TestAgent: Waiting for endpoints to shut down +31:52.564 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:51.405Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:51.449Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:51.548 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7d1b16ab-753b-8942-3fe8-3906e061001e Address:127.0.0.1:29882}]" +31:51.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7d1b16ab-753b-8942-3fe8-3906e061001e.primary 127.0.0.1 +31:51.549 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7d1b16ab-753b-8942-3fe8-3906e061001e 127.0.0.1 +2021-10-05T16:31:51.549Z [INFO] agent.router: Initializing LAN area manager +31:51.549 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29925 network=udp +31:51.549 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29882 [Follower]" leader= +31:51.550 [INFO] TestAgent.server: Adding LAN server: server="Node-7d1b16ab-753b-8942-3fe8-3906e061001e (Addr: tcp/127.0.0.1:29882) (DC: primary)" +31:51.550 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7d1b16ab-753b-8942-3fe8-3906e061001e.primary area=wan +31:51.550 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29925 network=tcp +31:51.550 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29878 network=tcp +31:51.551 [INFO] TestAgent: started state syncer +31:51.551 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29869 network=tcp +31:51.596 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:51.596 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29882 [Candidate]" term=2 +31:51.644 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:51.644 [DEBUG] TestAgent.server.raft: vote granted: from=7d1b16ab-753b-8942-3fe8-3906e061001e term=2 tally=1 +31:51.644 [INFO] TestAgent.server.raft: election won: tally=1 +31:51.644 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29882 [Leader]" +31:51.644 [INFO] TestAgent.server: cluster leadership acquired +31:51.644 [INFO] TestAgent.server: New leader elected: payload=Node-7d1b16ab-753b-8942-3fe8-3906e061001e +31:51.679 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29882 +31:51.705 [INFO] TestAgent: Synced node info +31:51.738 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:51.782 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:51.782 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:51.782 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:51.782 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:51.782 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:51.782 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7d1b16ab-753b-8942-3fe8-3906e061001e +31:51.782 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7d1b16ab-753b-8942-3fe8-3906e061001e +31:51.797 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.030 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:49370 +31:52.031 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce.secondary 127.0.0.1 +31:52.031 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-678f51a7-51ef-ef7b-2cea-d9f98839e1ce.secondary area=wan +31:52.564 [INFO] TestAgent: Requesting shutdown +31:52.564 [INFO] TestAgent.server: shutting down server +31:52.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.564 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.564 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.564 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:52.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:52.564 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.564 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.564 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.572 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:52.576Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.576Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.576Z [INFO] agent.router.manager: shutting down +31:52.576 [INFO] TestAgent: consul server down +31:52.576 [INFO] TestAgent: shutdown complete +31:52.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29925 network=tcp +31:52.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29925 network=udp +31:52.576 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29878 network=tcp +31:53.076 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.076 [INFO] TestAgent: Endpoints down +--- PASS: TestStatusPeersSecondary (1.71s) === CONT TestAgentConnectCARoots_empty -[WARN] freeport: 10 out of 73 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:15.425Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.435Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.479 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a6717484-fd3f-92f6-00b0-967058fe05fb Address:127.0.0.1:11238}]" -46:15.479 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11238 [Follower]" leader= -46:15.479 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a6717484-fd3f-92f6-00b0-967058fe05fb.dc1 127.0.0.1 -46:15.480 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a6717484-fd3f-92f6-00b0-967058fe05fb 127.0.0.1 -2022-11-07T22:46:15.480Z [INFO] agent.router: Initializing LAN area manager -46:15.480 [INFO] TestAgent.server: Adding LAN server: server="Node-a6717484-fd3f-92f6-00b0-967058fe05fb (Addr: tcp/127.0.0.1:11238) (DC: dc1)" -46:15.480 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a6717484-fd3f-92f6-00b0-967058fe05fb.dc1 area=wan -46:15.480 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11261 network=udp -46:15.480 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11261 network=tcp -46:15.480 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11234 network=tcp -46:15.480 [INFO] TestAgent: started state syncer -46:15.480 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11288 network=tcp -46:15.521 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.521 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11238 [Candidate]" term=2 -46:15.535 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.535 [DEBUG] TestAgent.server.raft: vote granted: from=a6717484-fd3f-92f6-00b0-967058fe05fb term=2 tally=1 -46:15.535 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.535 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11238 [Leader]" -46:15.536 [INFO] TestAgent.server: cluster leadership acquired -46:15.536 [INFO] TestAgent.server: New leader elected: payload=Node-a6717484-fd3f-92f6-00b0-967058fe05fb -46:15.549 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11238 -46:15.589 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.589 [INFO] TestAgent: Synced node info -46:15.622 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.622 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.622 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.622 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.622 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.622 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a6717484-fd3f-92f6-00b0-967058fe05fb -46:15.622 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a6717484-fd3f-92f6-00b0-967058fe05fb -46:15.631 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.910 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=121.934µs client=127.0.0.1:52790 client_network=udp -46:15.910 [INFO] TestAgent: Requesting shutdown -46:15.910 [INFO] TestAgent.server: shutting down server -46:15.910 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.910 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.910 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.910 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.910 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.910 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.910 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.910 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.913 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.920Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.920Z [INFO] agent.router.manager: shutting down -46:15.920 [INFO] TestAgent: consul server down -46:15.920 [INFO] TestAgent: shutdown complete -46:15.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11261 network=tcp -46:15.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11261 network=udp -46:15.920 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11234 network=tcp -46:16.420 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.420 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:14.572Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.581Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.616 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83af7446-64fa-2920-202c-32c69ae24325 Address:127.0.0.1:11158}]" -46:14.616 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11158 [Follower]" leader= -46:14.617 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83af7446-64fa-2920-202c-32c69ae24325.dc1 127.0.0.1 -46:14.617 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83af7446-64fa-2920-202c-32c69ae24325 127.0.0.1 -2022-11-07T22:46:14.617Z [INFO] agent.router: Initializing LAN area manager -46:14.617 [INFO] TestAgent.server: Adding LAN server: server="Node-83af7446-64fa-2920-202c-32c69ae24325 (Addr: tcp/127.0.0.1:11158) (DC: dc1)" -46:14.617 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83af7446-64fa-2920-202c-32c69ae24325.dc1 area=wan -46:14.617 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11146 network=udp -46:14.617 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11146 network=tcp -46:14.618 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11147 network=tcp -46:14.618 [INFO] TestAgent: started state syncer -46:14.618 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11159 network=tcp -46:14.655 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.655 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11158 [Candidate]" term=2 -46:14.673 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.673 [DEBUG] TestAgent.server.raft: vote granted: from=83af7446-64fa-2920-202c-32c69ae24325 term=2 tally=1 -46:14.673 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.673 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11158 [Leader]" -46:14.673 [INFO] TestAgent.server: cluster leadership acquired -46:14.673 [INFO] TestAgent.server: New leader elected: payload=Node-83af7446-64fa-2920-202c-32c69ae24325 -46:14.685 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11158 -46:14.714 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.746 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.746 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.746 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.746 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.746 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.746 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83af7446-64fa-2920-202c-32c69ae24325 -46:14.746 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83af7446-64fa-2920-202c-32c69ae24325 -46:14.752 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.760 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:14.763 [INFO] TestAgent: Synced node info -46:14.904 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=134.138µs client=127.0.0.1:55500 client_network=udp -46:14.904 [INFO] TestAgent: Requesting shutdown -46:14.904 [INFO] TestAgent.server: shutting down server -46:14.904 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.904 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.904 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.904 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.904 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.904 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.904 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.904 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.904 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.904 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.910 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.914Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.914Z [INFO] agent.router.manager: shutting down -46:14.914 [INFO] TestAgent: consul server down -46:14.914 [INFO] TestAgent: shutdown complete -46:14.915 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11146 network=tcp -46:14.915 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11146 network=udp -46:14.915 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11147 network=tcp -46:15.415 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.415 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:13.854Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.870Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.908 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8b3127b-b56b-7eb7-3f98-d3a4181747e3 Address:127.0.0.1:11025}]" -46:13.908 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11025 [Follower]" leader= -46:13.908 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3.dc1 127.0.0.1 -46:13.909 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 127.0.0.1 -2022-11-07T22:46:13.909Z [INFO] agent.router: Initializing LAN area manager -46:13.909 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11111 network=udp -46:13.909 [INFO] TestAgent.server: Adding LAN server: server="Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 (Addr: tcp/127.0.0.1:11025) (DC: dc1)" -46:13.909 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3.dc1 area=wan -46:13.909 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11111 network=tcp -46:13.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11112 network=tcp -46:13.909 [INFO] TestAgent: started state syncer -46:13.910 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11026 network=tcp -46:13.950 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.950 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11025 [Candidate]" term=2 -46:13.970 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.970 [DEBUG] TestAgent.server.raft: vote granted: from=f8b3127b-b56b-7eb7-3f98-d3a4181747e3 term=2 tally=1 -46:13.970 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.970 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11025 [Leader]" -46:13.970 [INFO] TestAgent.server: cluster leadership acquired -46:13.970 [INFO] TestAgent.server: New leader elected: payload=Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 -46:13.985 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11025 -46:14.005 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.035 [INFO] TestAgent: Synced node info -46:14.042 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=111.601µs client=127.0.0.1:32874 client_network=udp -46:14.042 [INFO] TestAgent: Requesting shutdown -46:14.042 [INFO] TestAgent.server: shutting down server -46:14.042 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.042 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.042 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.042 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.042 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.042 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.042 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 -46:14.042 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 -46:14.050 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.057Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.057Z [INFO] agent.router.manager: shutting down -46:14.060 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:14.060 [INFO] TestAgent: consul server down -46:14.060 [INFO] TestAgent: shutdown complete -46:14.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11111 network=tcp -46:14.060 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-f8b3127b-b56b-7eb7-3f98-d3a4181747e3 127.0.0.1 11023 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:f8b3127b-b56b-7eb7-3f98-d3a4181747e3 port:11025 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11024] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:14.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.060 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.060 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.060 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11111 network=udp -46:14.060 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11112 network=tcp -46:14.060 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.060 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.560 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.560 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:13.108Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.118Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.151 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:071f5e20-a114-724b-e4e1-1437fd357f48 Address:127.0.0.1:11968}]" -46:13.151 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11968 [Follower]" leader= -46:13.152 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-071f5e20-a114-724b-e4e1-1437fd357f48.dc1 127.0.0.1 -46:13.152 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-071f5e20-a114-724b-e4e1-1437fd357f48 127.0.0.1 -2022-11-07T22:46:13.152Z [INFO] agent.router: Initializing LAN area manager -46:13.152 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-071f5e20-a114-724b-e4e1-1437fd357f48.dc1 area=wan -46:13.152 [INFO] TestAgent.server: Adding LAN server: server="Node-071f5e20-a114-724b-e4e1-1437fd357f48 (Addr: tcp/127.0.0.1:11968) (DC: dc1)" -46:13.153 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11921 network=udp -46:13.153 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11921 network=tcp -46:13.153 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11922 network=tcp -46:13.153 [INFO] TestAgent: started state syncer -46:13.153 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11969 network=tcp -46:13.199 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.199 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11968 [Candidate]" term=2 -46:13.220 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.220 [DEBUG] TestAgent.server.raft: vote granted: from=071f5e20-a114-724b-e4e1-1437fd357f48 term=2 tally=1 -46:13.220 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.220 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11968 [Leader]" -46:13.220 [INFO] TestAgent.server: cluster leadership acquired -46:13.220 [INFO] TestAgent.server: New leader elected: payload=Node-071f5e20-a114-724b-e4e1-1437fd357f48 -46:13.236 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11968 -46:13.261 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.293 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.293 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.293 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.293 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.293 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.293 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-071f5e20-a114-724b-e4e1-1437fd357f48 -46:13.293 [INFO] TestAgent.server: member joined, marking health alive: member=Node-071f5e20-a114-724b-e4e1-1437fd357f48 -46:13.303 [INFO] TestAgent.server: federation state anti-entropy synced -46:13.330 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:13.333 [DEBUG] TestAgent.dns: request served from client: name=nofoo.node.dc1.consul. type=ANY class=IN latency=101.469µs client=127.0.0.1:37844 client_network=udp -46:13.333 [INFO] TestAgent: Requesting shutdown -46:13.333 [INFO] TestAgent.server: shutting down server -46:13.333 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:13.333 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:13.333 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:13.333 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:13.333 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:13.333 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:13.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:13.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:13.333 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:13.333 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:13.334 [INFO] TestAgent: Synced node info -46:13.338 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:13.341Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:13.341Z [INFO] agent.router.manager: shutting down -46:13.342 [INFO] TestAgent: consul server down -46:13.342 [INFO] TestAgent: shutdown complete -46:13.342 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11921 network=tcp -46:13.342 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11921 network=udp -46:13.342 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11922 network=tcp -46:13.842 [INFO] TestAgent: Waiting for endpoints to shut down -46:13.842 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_SOA_Settings (3.32s) -=== CONT TestAgent_ReloadConfigTLSConfigFailure -2022-11-07T22:46:15.833Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.843Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.868 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:65129238-7b9b-cb83-dc18-7bdce395c238 Address:127.0.0.1:11315}]" -46:15.868 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11315 [Follower]" leader= -46:15.868 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-65129238-7b9b-cb83-dc18-7bdce395c238.dc1 127.0.0.1 -46:15.869 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-65129238-7b9b-cb83-dc18-7bdce395c238 127.0.0.1 -2022-11-07T22:46:15.869Z [INFO] agent.router: Initializing LAN area manager -46:15.869 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-65129238-7b9b-cb83-dc18-7bdce395c238.dc1 area=wan -46:15.869 [INFO] TestAgent.server: Adding LAN server: server="Node-65129238-7b9b-cb83-dc18-7bdce395c238 (Addr: tcp/127.0.0.1:11315) (DC: dc1)" -46:15.869 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11366 network=udp -46:15.869 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11366 network=tcp -46:15.869 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11311 network=tcp -46:15.869 [INFO] TestAgent: started state syncer -46:15.869 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11190 network=tcp -46:15.919 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.919 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11315 [Candidate]" term=2 -46:15.928 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.928 [DEBUG] TestAgent.server.raft: vote granted: from=65129238-7b9b-cb83-dc18-7bdce395c238 term=2 tally=1 -46:15.928 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.928 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11315 [Leader]" -46:15.928 [INFO] TestAgent.server: cluster leadership acquired -46:15.928 [INFO] TestAgent.server: New leader elected: payload=Node-65129238-7b9b-cb83-dc18-7bdce395c238 -46:15.935 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11315 -46:15.950 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.969 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.969 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.969 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.969 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.969 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.969 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-65129238-7b9b-cb83-dc18-7bdce395c238 -46:15.969 [INFO] TestAgent.server: member joined, marking health alive: member=Node-65129238-7b9b-cb83-dc18-7bdce395c238 -46:15.979 [WARN] TestAgent: check has interval below minimum: check=http minimum_interval=1s -46:15.979 [WARN] TestAgent: check has interval below minimum: check=grpc minimum_interval=1s -46:15.979 [INFO] TestAgent: Requesting shutdown -46:15.979 [INFO] TestAgent.server: shutting down server -46:15.979 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.979 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.979 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.979 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.979 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.980 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.980 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.980 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.980 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.981 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.985 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.990Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.990Z [INFO] agent.router.manager: shutting down -46:15.990 [INFO] TestAgent: consul server down -46:15.990 [INFO] TestAgent: shutdown complete -46:15.990 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11366 network=tcp -46:15.990 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11366 network=udp -46:15.990 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11311 network=tcp -46:16.490 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.490 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RerouteExistingHTTPChecks (0.67s) -=== CONT TestAgent_AddCheck_MissingService -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:15.852Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.861Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.887 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:947b0e67-d585-bcfe-222c-9ee40e54c6cc Address:127.0.0.1:11224}]" -46:15.887 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11224 [Follower]" leader= -46:15.888 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc.dc1 127.0.0.1 -46:15.888 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc 127.0.0.1 -2022-11-07T22:46:15.888Z [INFO] agent.router: Initializing LAN area manager -46:15.888 [INFO] TestAgent.server: Adding LAN server: server="Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc (Addr: tcp/127.0.0.1:11224) (DC: dc1)" -46:15.888 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc.dc1 area=wan -46:15.888 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11191 network=udp -46:15.888 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11191 network=tcp -46:15.889 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11220 network=tcp -46:15.889 [INFO] TestAgent: started state syncer -46:15.889 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11316 network=tcp -46:15.955 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.955 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11224 [Candidate]" term=2 -46:15.964 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.964 [DEBUG] TestAgent.server.raft: vote granted: from=947b0e67-d585-bcfe-222c-9ee40e54c6cc term=2 tally=1 -46:15.964 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.964 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11224 [Leader]" -46:15.964 [INFO] TestAgent.server: cluster leadership acquired -46:15.964 [INFO] TestAgent.server: New leader elected: payload=Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc -46:15.969 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11224 -46:15.983 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.005 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.005 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.005 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.005 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.005 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.005 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc -46:16.005 [INFO] TestAgent.server: member joined, marking health alive: member=Node-947b0e67-d585-bcfe-222c-9ee40e54c6cc -46:16.022 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.064 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.069 [INFO] TestAgent: Synced node info -46:16.069 [DEBUG] TestAgent: Node info in sync -46:16.163 [INFO] TestAgent: Requesting shutdown -46:16.163 [INFO] TestAgent.server: shutting down server -46:16.163 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.163 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.163 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.163 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.163 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.164 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.164 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.164 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.164 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.164 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.167 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.173Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.173Z [INFO] agent.router.manager: shutting down -46:16.174 [INFO] TestAgent: consul server down -46:16.174 [INFO] TestAgent: shutdown complete -46:16.174 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11191 network=tcp -46:16.174 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11191 network=udp -46:16.174 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11220 network=tcp -46:16.674 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.674 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply_TerminatingGateway (0.83s) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:52.678Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.678Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.704Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.704Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.795 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:94732591-0f18-5891-521b-5b326f6e4720 Address:127.0.0.1:29979}]" +31:52.795 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-94732591-0f18-5891-521b-5b326f6e4720.dc1 127.0.0.1 +31:52.796 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29979 [Follower]" leader= +31:52.796 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-94732591-0f18-5891-521b-5b326f6e4720 127.0.0.1 +2021-10-05T16:31:52.796Z [INFO] agent.router: Initializing LAN area manager +31:52.796 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30002 network=udp +31:52.796 [INFO] TestAgent.server: Adding LAN server: server="Node-94732591-0f18-5891-521b-5b326f6e4720 (Addr: tcp/127.0.0.1:29979) (DC: dc1)" +31:52.796 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-94732591-0f18-5891-521b-5b326f6e4720.dc1 area=wan +31:52.797 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30002 network=tcp +31:52.797 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29975 network=tcp +31:52.797 [INFO] TestAgent: started state syncer +31:52.797 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29959 network=tcp +31:52.836 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.836 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29979 [Candidate]" term=2 +31:52.847 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:52.877 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.877 [DEBUG] TestAgent.server.raft: vote granted: from=94732591-0f18-5891-521b-5b326f6e4720 term=2 tally=1 +31:52.877 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.877 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29979 [Leader]" +31:52.877 [INFO] TestAgent.server: cluster leadership acquired +31:52.877 [INFO] TestAgent.server: New leader elected: payload=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:52.884 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:52.891 [INFO] TestAgent.server: initializing acls +31:52.897 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:52.897 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:52.911 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:52.941 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:52.941 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:52.941 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:52.942 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-94732591-0f18-5891-521b-5b326f6e4720 +31:52.944 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-94732591-0f18-5891-521b-5b326f6e4720.dc1 +31:52.944 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-94732591-0f18-5891-521b-5b326f6e4720.dc1 area=wan +31:52.944 [INFO] TestAgent.server: Updating LAN server: server="Node-94732591-0f18-5891-521b-5b326f6e4720 (Addr: tcp/127.0.0.1:29979) (DC: dc1)" +31:52.996 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:53.057 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:53.057 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.057 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.057 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:53.057 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:53.057 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:53.057 [INFO] TestAgent.server: member joined, marking health alive: member=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:53.080 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:53.088 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:53.088 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-94732591-0f18-5891-521b-5b326f6e4720 +31:53.114 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.140 [INFO] TestAgent: Synced node info +31:53.157 [INFO] TestAgent: Synced service: service=test-id +31:53.157 [DEBUG] TestAgent: Check in sync: check=service:test-id +31:53.196 [INFO] TestAgent: Requesting shutdown +31:53.197 [INFO] TestAgent.server: shutting down server +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.197 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.197 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.197 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:53.197 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.204 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.212Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.212Z [INFO] agent.router.manager: shutting down +31:53.212 [INFO] TestAgent: consul server down +31:53.212 [INFO] TestAgent: shutdown complete +31:53.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30002 network=tcp +31:53.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30002 network=udp +31:53.212 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29975 network=tcp +31:53.715 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.715 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectCALeafCert_aclServiceReadDeny (1.08s) === CONT TestAgent_Token -2022-11-07T22:46:15.906Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:15.906Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.916Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:15.916Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.948 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4387561b-844b-b0cf-6378-3a4eb64ae953 Address:127.0.0.1:11301}]" -46:15.948 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11301 [Follower]" leader= -46:15.949 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4387561b-844b-b0cf-6378-3a4eb64ae953.dc1 127.0.0.1 -46:15.949 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4387561b-844b-b0cf-6378-3a4eb64ae953 127.0.0.1 -2022-11-07T22:46:15.949Z [INFO] agent.router: Initializing LAN area manager -46:15.949 [INFO] TestAgent.server: Adding LAN server: server="Node-4387561b-844b-b0cf-6378-3a4eb64ae953 (Addr: tcp/127.0.0.1:11301) (DC: dc1)" -46:15.949 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4387561b-844b-b0cf-6378-3a4eb64ae953.dc1 area=wan -46:15.949 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11317 network=tcp -46:15.949 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11317 network=udp -46:15.950 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11297 network=tcp -46:15.950 [INFO] TestAgent: started state syncer -46:15.950 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11295 network=tcp -46:15.991 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.991 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11301 [Candidate]" term=2 -46:16.000 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:16.001 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.001 [DEBUG] TestAgent.server.raft: vote granted: from=4387561b-844b-b0cf-6378-3a4eb64ae953 term=2 tally=1 -46:16.001 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.001 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11301 [Leader]" -46:16.001 [INFO] TestAgent.server: cluster leadership acquired -46:16.001 [INFO] TestAgent.server: New leader elected: payload=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.004 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:16.009 [INFO] TestAgent.server: initializing acls -46:16.015 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:16.015 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:16.023 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:16.035 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:16.035 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:16.035 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:16.036 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.036 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4387561b-844b-b0cf-6378-3a4eb64ae953.dc1 -46:16.036 [INFO] TestAgent.server: Updating LAN server: server="Node-4387561b-844b-b0cf-6378-3a4eb64ae953 (Addr: tcp/127.0.0.1:11301) (DC: dc1)" -46:16.036 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4387561b-844b-b0cf-6378-3a4eb64ae953.dc1 area=wan -46:16.048 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.074 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.074 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.074 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.074 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.074 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.074 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.074 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.077 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.083 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.206 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.206 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4387561b-844b-b0cf-6378-3a4eb64ae953 -46:16.233 [INFO] TestAgent: Synced node info -46:16.240 [INFO] TestAgent: Synced service: service=test-id -46:16.240 [DEBUG] TestAgent: Check in sync: check=service:test-id -46:16.245 [INFO] TestAgent: Requesting shutdown -46:16.245 [INFO] TestAgent.server: shutting down server -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.245 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.245 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:16.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.249 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.255Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.255Z [INFO] agent.router.manager: shutting down -46:16.255 [INFO] TestAgent: consul server down -46:16.255 [INFO] TestAgent: shutdown complete -46:16.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11317 network=tcp -46:16.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11317 network=udp -46:16.255 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11297 network=tcp -46:16.756 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.756 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectCALeafCert_aclServiceReadDeny (0.86s) -=== CONT TestAgent_ReloadConfigIncomingRPCConfig +2021-10-05T16:31:52.798Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.798Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.814Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.814Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.853 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:279fd5b9-4b89-4f43-ad54-466fb339bf93 Address:127.0.0.1:29972}]" +31:52.853 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 127.0.0.1 +31:52.854 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29972 [Follower]" leader= +31:52.854 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 127.0.0.1 +2021-10-05T16:31:52.854Z [INFO] agent.router: Initializing LAN area manager +31:52.854 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29953 network=udp +31:52.854 [INFO] TestAgent.server: Adding LAN server: server="Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 (Addr: tcp/127.0.0.1:29972) (DC: dc1)" +31:52.854 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 area=wan +31:52.855 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29953 network=tcp +31:52.855 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29968 network=tcp +31:52.855 [INFO] TestAgent: started state syncer +31:52.856 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30029 network=tcp +31:52.905 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:52.913 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.913 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29972 [Candidate]" term=2 +31:52.944 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.944 [DEBUG] TestAgent.server.raft: vote granted: from=279fd5b9-4b89-4f43-ad54-466fb339bf93 term=2 tally=1 +31:52.944 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.944 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29972 [Leader]" +31:52.948 [INFO] TestAgent.server: cluster leadership acquired +31:52.948 [INFO] TestAgent.server: New leader elected: payload=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:52.972 [INFO] TestAgent.server: initializing acls +31:52.982 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:52.982 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:53.004 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:53.007 [INFO] TestAgent.server: initializing acls +31:53.007 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:53.033 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:53.033 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:53.033 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:53.033 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.033 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 +31:53.033 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 area=wan +31:53.034 [INFO] TestAgent.server: Updating LAN server: server="Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 (Addr: tcp/127.0.0.1:29972) (DC: dc1)" +31:53.044 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:53.044 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:53.044 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.044 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 +31:53.044 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93.dc1 area=wan +31:53.044 [INFO] TestAgent.server: Updating LAN server: server="Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 (Addr: tcp/127.0.0.1:29972) (DC: dc1)" +31:53.089 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:53.200 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:53.200 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.200 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.200 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:53.200 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:53.200 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.200 [INFO] TestAgent.server: member joined, marking health alive: member=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.208 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.208 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.224 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.248 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:53.268 [INFO] TestAgent: Synced node info +31:53.294 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-279fd5b9-4b89-4f43-ad54-466fb339bf93 +31:53.294 [INFO] TestAgent: Requesting shutdown +31:53.294 [INFO] TestAgent.server: shutting down server +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.294 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:53.294 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.295 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.295 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:53.304 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.310Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.310Z [INFO] agent.router.manager: shutting down +31:53.310 [INFO] TestAgent: consul server down +31:53.310 [INFO] TestAgent: shutdown complete +31:53.310 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29953 network=tcp +31:53.310 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29953 network=udp +31:53.310 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29968 network=tcp +31:53.811 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.811 [INFO] TestAgent: Endpoints down +=== RUN TestSnapshot_Options/GET#02 [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:16.028Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.038Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.067 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 Address:127.0.0.1:11294}]" -46:16.067 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11294 [Follower]" leader= -46:16.068 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1.dc1 127.0.0.1 -46:16.069 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 127.0.0.1 -2022-11-07T22:46:16.069Z [INFO] agent.router: Initializing LAN area manager -46:16.069 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1.dc1 area=wan -46:16.069 [INFO] TestAgent.server: Adding LAN server: server="Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 (Addr: tcp/127.0.0.1:11294) (DC: dc1)" -46:16.069 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11310 network=tcp -46:16.069 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11310 network=udp -46:16.069 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11290 network=tcp -46:16.069 [INFO] TestAgent: started state syncer -46:16.069 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11358 network=tcp -46:16.128 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.128 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11294 [Candidate]" term=2 -46:16.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.137 [DEBUG] TestAgent.server.raft: vote granted: from=5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 term=2 tally=1 -46:16.137 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11294 [Leader]" -46:16.137 [INFO] TestAgent.server: cluster leadership acquired -46:16.137 [INFO] TestAgent.server: New leader elected: payload=Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 -46:16.143 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11294 -46:16.157 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.185 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.185 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.185 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.185 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.185 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.185 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 -46:16.185 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5143b632-b3bd-eb96-e7db-f6f7a05e9bf1 -46:16.202 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.202 [INFO] TestAgent: Synced node info -46:16.348 [INFO] TestAgent: Requesting shutdown -46:16.348 [INFO] TestAgent.server: shutting down server -46:16.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.348 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.348 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.348 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.348 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.348 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.348 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.348 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.348 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.363Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.363Z [INFO] agent.router.manager: shutting down -46:16.363 [INFO] TestAgent: consul server down -46:16.363 [INFO] TestAgent: shutdown complete -46:16.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11310 network=tcp -46:16.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11310 network=udp -46:16.363 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11290 network=tcp -46:16.863 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.863 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_ExecRemoteDisable (0.85s) -=== CONT TestAgent_AddCheck_MinInterval -2022-11-07T22:46:15.907Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:15.907Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.916Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:15.916Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.950 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0a6581ca-18a7-ece8-6bcd-46d85b263832 Address:127.0.0.1:11336}]" -46:15.950 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11336 [Follower]" leader= -46:15.950 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0a6581ca-18a7-ece8-6bcd-46d85b263832.dc1 127.0.0.1 -46:15.951 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 127.0.0.1 -2022-11-07T22:46:15.951Z [INFO] agent.router: Initializing LAN area manager -46:15.951 [INFO] TestAgent.server: Adding LAN server: server="Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 (Addr: tcp/127.0.0.1:11336) (DC: dc1)" -46:15.951 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832.dc1 area=wan -46:15.951 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11296 network=udp -46:15.951 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11296 network=tcp -46:15.951 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11332 network=tcp -46:15.951 [INFO] TestAgent: started state syncer -46:15.951 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11309 network=tcp -46:16.001 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:16.002 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.002 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11336 [Candidate]" term=2 -46:16.016 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.016 [DEBUG] TestAgent.server.raft: vote granted: from=0a6581ca-18a7-ece8-6bcd-46d85b263832 term=2 tally=1 -46:16.016 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.016 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11336 [Leader]" -46:16.016 [INFO] TestAgent.server: cluster leadership acquired -46:16.016 [INFO] TestAgent.server: New leader elected: payload=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.026 [INFO] TestAgent.server: initializing acls -46:16.029 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:16.029 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:16.040 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:16.052 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:16.052 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:16.052 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:16.052 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.052 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0a6581ca-18a7-ece8-6bcd-46d85b263832.dc1 -46:16.052 [INFO] TestAgent.server: Updating LAN server: server="Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 (Addr: tcp/127.0.0.1:11336) (DC: dc1)" -46:16.052 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832.dc1 area=wan -46:16.067 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.086 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.086 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.086 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.086 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.086 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.086 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.086 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.093 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.093 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.319 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.319 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0a6581ca-18a7-ece8-6bcd-46d85b263832 -46:16.344 [INFO] TestAgent: Synced node info -46:16.350 [INFO] TestAgent: Synced service: service=test-id -46:16.350 [DEBUG] TestAgent: Check in sync: check=service:test-id -46:16.365 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.365 [DEBUG] TestAgent: Node info in sync -46:16.365 [DEBUG] TestAgent: Service in sync: service=test-id -46:16.366 [DEBUG] TestAgent: Check in sync: check=service:test-id -46:16.366 [DEBUG] TestAgent: Node info in sync -46:16.366 [DEBUG] TestAgent: Service in sync: service=test-id -46:16.366 [DEBUG] TestAgent: Check in sync: check=service:test-id -46:16.367 [INFO] TestAgent: Requesting shutdown -46:16.367 [INFO] TestAgent.server: shutting down server -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:16.367 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:16.367 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.367 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.367 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:16.367 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.367 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.368 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.367 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:16.375 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.383Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.383Z [INFO] agent.router.manager: shutting down -46:16.383 [INFO] TestAgent: consul server down -46:16.383 [INFO] TestAgent: shutdown complete -46:16.383 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11296 network=tcp -46:16.383 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11296 network=udp -46:16.383 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11332 network=tcp -46:16.884 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.884 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectCALeafCert_aclServiceWrite (0.99s) +2021-10-05T16:31:52.811Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.811Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.836Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.836Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.886 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:36e1da33-7e4f-d50b-e24b-11de5dc1200d Address:127.0.0.1:30007}]" +31:52.887 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d.dc1 127.0.0.1 +31:52.887 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d 127.0.0.1 +2021-10-05T16:31:52.887Z [INFO] agent.router: Initializing LAN area manager +31:52.887 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30009 network=udp +31:52.887 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30007 [Follower]" leader= +31:52.888 [INFO] TestAgent.server: Adding LAN server: server="Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d (Addr: tcp/127.0.0.1:30007) (DC: dc1)" +31:52.888 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d.dc1 area=wan +31:52.888 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30009 network=tcp +31:52.889 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30003 network=tcp +31:52.889 [INFO] TestAgent: started state syncer +31:52.889 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29966 network=tcp +31:52.940 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:52.956 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.956 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30007 [Candidate]" term=2 +31:52.983 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.983 [DEBUG] TestAgent.server.raft: vote granted: from=36e1da33-7e4f-d50b-e24b-11de5dc1200d term=2 tally=1 +31:52.983 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.983 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30007 [Leader]" +31:52.983 [INFO] TestAgent.server: cluster leadership acquired +31:52.983 [INFO] TestAgent.server: New leader elected: payload=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:52.993 [INFO] TestAgent.server: initializing acls +31:52.998 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:52.998 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:53.005 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:53.028 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:53.028 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:53.028 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:53.029 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.029 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d.dc1 +31:53.029 [INFO] TestAgent.server: Updating LAN server: server="Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d (Addr: tcp/127.0.0.1:30007) (DC: dc1)" +31:53.029 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d.dc1 area=wan +31:53.070 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:53.167 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:53.167 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.167 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.167 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:53.167 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:53.167 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.167 [INFO] TestAgent.server: member joined, marking health alive: member=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.180 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.180 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.196 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.196 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-36e1da33-7e4f-d50b-e24b-11de5dc1200d +31:53.232 [INFO] TestAgent: Synced node info +31:53.248 [INFO] TestAgent: Synced service: service=test-id +31:53.248 [DEBUG] TestAgent: Check in sync: check=service:test-id +31:53.269 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:53.269 [DEBUG] TestAgent: Node info in sync +31:53.269 [DEBUG] TestAgent: Service in sync: service=test-id +31:53.269 [DEBUG] TestAgent: Check in sync: check=service:test-id +31:53.269 [DEBUG] TestAgent: Node info in sync +31:53.269 [DEBUG] TestAgent: Service in sync: service=test-id +31:53.269 [DEBUG] TestAgent: Check in sync: check=service:test-id +31:53.315 [INFO] TestAgent: Requesting shutdown +31:53.315 [INFO] TestAgent.server: shutting down server +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.315 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.315 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.315 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:53.315 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.326 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.332Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.332Z [INFO] agent.router.manager: shutting down +31:53.332 [INFO] TestAgent: consul server down +31:53.332 [INFO] TestAgent: shutdown complete +31:53.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30009 network=tcp +31:53.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30009 network=udp +31:53.332 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30003 network=tcp +31:53.832 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.832 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectCALeafCert_aclServiceWrite (1.06s) === CONT TestAgent_TokenTriggersFullSync === RUN TestAgent_TokenTriggersFullSync/acl_agent_token -2022-11-07T22:46:16.192Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.205Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.241 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a978aea9-1b3f-5166-f079-48f35b6f8a8d Address:127.0.0.1:11357}]" -46:16.241 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11357 [Follower]" leader= -46:16.242 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d.dc1 127.0.0.1 -46:16.242 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d 127.0.0.1 -2022-11-07T22:46:16.242Z [INFO] agent.router: Initializing LAN area manager -46:16.242 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d.dc1 area=wan -46:16.242 [INFO] TestAgent.server: Adding LAN server: server="Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d (Addr: tcp/127.0.0.1:11357) (DC: dc1)" -46:16.242 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11303 network=tcp -46:16.242 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11303 network=udp -46:16.243 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11353 network=tcp -46:16.243 [INFO] TestAgent: started state syncer -46:16.243 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11386 network=tcp -46:16.308 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.308 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11357 [Candidate]" term=2 -46:16.322 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.322 [DEBUG] TestAgent.server.raft: vote granted: from=a978aea9-1b3f-5166-f079-48f35b6f8a8d term=2 tally=1 -46:16.322 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.322 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11357 [Leader]" -46:16.322 [INFO] TestAgent.server: cluster leadership acquired -46:16.323 [INFO] TestAgent.server: New leader elected: payload=Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d -46:16.336 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11357 -46:16.355 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.390 [INFO] TestAgent: Requesting shutdown -46:16.390 [INFO] TestAgent.server: shutting down server -46:16.390 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.390 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.396 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.396 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.396 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.396 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.396 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d -46:16.396 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d -46:16.397 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.405Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.405Z [INFO] agent.router.manager: shutting down -46:16.414 [INFO] TestAgent: consul server down -46:16.414 [INFO] TestAgent: shutdown complete -46:16.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11303 network=tcp -46:16.414 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:16.414 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-a978aea9-1b3f-5166-f079-48f35b6f8a8d 127.0.0.1 11355 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:a978aea9-1b3f-5166-f079-48f35b6f8a8d port:11357 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11356] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:16.414 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.414 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.414 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.414 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11303 network=udp -46:16.414 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11353 network=tcp -46:16.414 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.414 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.414 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.414 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.915 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.915 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_ExecDisable (0.73s) -=== CONT TestAgent_AddCheck_StartPassing -2022-11-07T22:46:16.135Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.145Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.171 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:263812f5-3da1-ad52-283c-5d16a6c44215 Address:127.0.0.1:11343}]" -46:16.172 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11343 [Follower]" leader= -46:16.172 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-263812f5-3da1-ad52-283c-5d16a6c44215.dc1 127.0.0.1 -46:16.173 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-263812f5-3da1-ad52-283c-5d16a6c44215 127.0.0.1 -2022-11-07T22:46:16.173Z [INFO] agent.router: Initializing LAN area manager -46:16.173 [INFO] TestAgent.server: Adding LAN server: server="Node-263812f5-3da1-ad52-283c-5d16a6c44215 (Addr: tcp/127.0.0.1:11343) (DC: dc1)" -46:16.173 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-263812f5-3da1-ad52-283c-5d16a6c44215.dc1 area=wan -46:16.173 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11359 network=udp -46:16.173 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11359 network=tcp -46:16.174 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11339 network=tcp -46:16.174 [INFO] TestAgent: started state syncer -46:16.174 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11344 network=tcp -46:16.211 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.211 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11343 [Candidate]" term=2 -46:16.227 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.227 [DEBUG] TestAgent.server.raft: vote granted: from=263812f5-3da1-ad52-283c-5d16a6c44215 term=2 tally=1 -46:16.228 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.228 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11343 [Leader]" -46:16.228 [INFO] TestAgent.server: cluster leadership acquired -46:16.228 [INFO] TestAgent.server: New leader elected: payload=Node-263812f5-3da1-ad52-283c-5d16a6c44215 -46:16.235 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11343 -46:16.248 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.269 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.269 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.269 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.269 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.269 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.269 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-263812f5-3da1-ad52-283c-5d16a6c44215 -46:16.269 [INFO] TestAgent.server: member joined, marking health alive: member=Node-263812f5-3da1-ad52-283c-5d16a6c44215 -46:16.276 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.334 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.337 [INFO] TestAgent: Synced node info -46:16.422 [INFO] TestAgent: Requesting shutdown -46:16.422 [INFO] TestAgent.server: shutting down server -46:16.422 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.422 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.422 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.422 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.422 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.422 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.422 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.422 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.431 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.438Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.438Z [INFO] agent.router.manager: shutting down -46:16.438 [INFO] TestAgent: consul server down -46:16.438 [INFO] TestAgent: shutdown complete -46:16.438 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11359 network=tcp -46:16.438 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11359 network=udp -46:16.438 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11339 network=tcp -46:16.938 [INFO] TestAgent: Waiting for endpoints to shut down -46:16.938 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_consulConfig_RaftTrailingLogs (0.81s) -=== CONT TestAgent_ReloadConfigOutgoingRPCConfig +2021-10-05T16:31:53.142Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.231 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b Address:127.0.0.1:30049}]" +31:53.232 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30049 [Follower]" leader= +31:53.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b.dc1 127.0.0.1 +31:53.233 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b 127.0.0.1 +2021-10-05T16:31:53.233Z [INFO] agent.router: Initializing LAN area manager +31:53.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30044 network=udp +31:53.234 [INFO] TestAgent.server: Adding LAN server: server="Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b (Addr: tcp/127.0.0.1:30049) (DC: dc1)" +31:53.234 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b.dc1 area=wan +31:53.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30044 network=tcp +31:53.235 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30045 network=tcp +31:53.235 [INFO] TestAgent: started state syncer +31:53.235 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30015 network=tcp +31:53.286 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:53.286 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30049 [Candidate]" term=2 +31:53.310 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:53.310 [DEBUG] TestAgent.server.raft: vote granted: from=ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b term=2 tally=1 +31:53.310 [INFO] TestAgent.server.raft: election won: tally=1 +31:53.310 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30049 [Leader]" +31:53.312 [INFO] TestAgent.server: cluster leadership acquired +31:53.312 [INFO] TestAgent.server: New leader elected: payload=Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b +31:53.318 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30049 +31:53.324 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.324 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.324 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b +31:53.324 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ca7f8e96-3b7e-e67b-d0a1-9d16d1898d7b +31:53.346 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.356 [INFO] TestAgent: Requesting shutdown +31:53.356 [INFO] TestAgent.server: shutting down server +31:53.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.356 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.356 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.356 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.356 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:53.356 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.365 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.376Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.376Z [INFO] agent.router.manager: shutting down +31:53.376 [INFO] TestAgent: consul server down +31:53.376 [INFO] TestAgent: shutdown complete +31:53.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30044 network=tcp +31:53.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30044 network=udp +31:53.376 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30045 network=tcp +31:53.876 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.876 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectCARoots_empty (0.80s) +=== CONT TestAgent_Monitor_ACLDeny +2021-10-05T16:31:52.891Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.891Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.926Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:52.926Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.020 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6fbdaa14-e7b7-0a41-7f91-df93135da507 Address:127.0.0.1:30056}]" +31:53.020 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30056 [Follower]" leader= +31:53.024 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 127.0.0.1 +31:53.025 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 127.0.0.1 +2021-10-05T16:31:53.025Z [INFO] agent.router: Initializing LAN area manager +31:53.025 [INFO] TestAgent.server: Adding LAN server: server="Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 (Addr: tcp/127.0.0.1:30056) (DC: dc1)" +31:53.025 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30051 network=udp +31:53.025 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 area=wan +31:53.025 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30051 network=tcp +31:53.026 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30052 network=tcp +31:53.026 [INFO] TestAgent: started state syncer +31:53.026 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30057 network=tcp +31:53.062 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:53.062 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30056 [Candidate]" term=2 +31:53.075 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:53.124 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:53.124 [DEBUG] TestAgent.server.raft: vote granted: from=6fbdaa14-e7b7-0a41-7f91-df93135da507 term=2 tally=1 +31:53.124 [INFO] TestAgent.server.raft: election won: tally=1 +31:53.124 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30056 [Leader]" +31:53.128 [INFO] TestAgent.server: cluster leadership acquired +31:53.128 [INFO] TestAgent.server: New leader elected: payload=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.158 [INFO] TestAgent.server: initializing acls +31:53.166 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:53.166 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:53.176 [INFO] TestAgent.server: initializing acls +31:53.176 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:53.206 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:53.208 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:53.216 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:53.216 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:53.216 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:53.216 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:53.216 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.216 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 +31:53.216 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 area=wan +31:53.216 [INFO] TestAgent.server: Updating LAN server: server="Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 (Addr: tcp/127.0.0.1:30056) (DC: dc1)" +31:53.222 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:53.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.223 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 +31:53.223 [INFO] TestAgent.server: Updating LAN server: server="Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 (Addr: tcp/127.0.0.1:30056) (DC: dc1)" +31:53.223 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507.dc1 area=wan +31:53.256 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:53.320 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:53.320 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.320 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.320 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:53.320 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:53.320 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.320 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.338 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.338 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.346 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.364 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6fbdaa14-e7b7-0a41-7f91-df93135da507 +31:53.389 [INFO] TestAgent: Synced node info +31:53.397 [INFO] TestAgent: Synced service: service=test-id +31:53.397 [DEBUG] TestAgent: Check in sync: check=service:test-id +31:53.400 [INFO] TestAgent: Requesting shutdown +31:53.400 [INFO] TestAgent.server: shutting down server +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:53.400 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:53.400 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.400 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.400 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.401 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:53.401 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.401 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.401 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.401 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:53.401 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:53.404 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.410Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.410Z [INFO] agent.router.manager: shutting down +31:53.410 [INFO] TestAgent: consul server down +31:53.410 [INFO] TestAgent: shutdown complete +31:53.410 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30051 network=tcp +31:53.410 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30051 network=udp +31:53.410 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30052 network=tcp +31:53.912 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.912 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectCALeafCert_aclDefaultDeny (1.05s) +=== CONT TestAgent_Monitor +2021-10-05T16:31:52.618Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.664Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.730 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d7cda1b4-bd7c-069b-7f6e-e521f053213b Address:127.0.0.1:29965}]" +31:52.732 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29965 [Follower]" leader= +31:52.733 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 127.0.0.1 +31:52.734 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b 127.0.0.1 +2021-10-05T16:31:52.734Z [INFO] agent.router: Initializing LAN area manager +31:52.735 [INFO] TestAgent.server: Adding LAN server: server="Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b (Addr: tcp/127.0.0.1:29965) (DC: dc1)" +31:52.735 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 area=wan +31:52.735 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29960 network=udp +31:52.735 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29960 network=tcp +31:52.737 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29961 network=tcp +31:52.737 [INFO] TestAgent: started state syncer +31:52.738 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29994 network=tcp +31:52.794 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.794 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29965 [Candidate]" term=2 +31:52.805 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.805 [DEBUG] TestAgent.server.raft: vote granted: from=d7cda1b4-bd7c-069b-7f6e-e521f053213b term=2 tally=1 +31:52.805 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.805 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29965 [Leader]" +31:52.805 [INFO] TestAgent.server: cluster leadership acquired +31:52.806 [INFO] TestAgent.server: New leader elected: payload=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b +31:52.817 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29965 +31:52.843 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.890 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.890 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.890 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.890 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.890 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.890 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b +31:52.890 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b +31:52.904 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:53564 +31:52.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 127.0.0.1 +31:52.908 [INFO] TestAgent.server: Adding LAN server: server="Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 (Addr: tcp/127.0.0.1:29958) (DC: dc1)" +31:52.908 [INFO] TestAgent.server: New leader elected: payload=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.908 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 other=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b +31:52.909 [INFO] TestAgent.server: member joined, marking health alive: member=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.909 [INFO] TestAgent: Requesting shutdown +31:52.909 [INFO] TestAgent.server: shutting down server +31:52.909 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:52.909 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:52.909 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:52.909 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:52.909 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:52.910 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:52.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:52.910 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:52.910 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:52.916 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:37910 +31:52.916 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 127.0.0.1:29957 +31:52.916 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 127.0.0.1 +31:52.917 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 area=wan +31:52.917 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 address=127.0.0.1:29957 +31:52.918 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:52.923 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:31:52.932Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:52.932Z [INFO] agent.router.manager: shutting down +31:52.932 [INFO] TestAgent: consul server down +31:52.932 [INFO] TestAgent: shutdown complete +31:52.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29960 network=tcp +31:52.932 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29960 network=udp +31:52.932 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29961 network=tcp +31:53.432 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.432 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:52.211Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.223Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.281 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:789ce253-6eeb-34cf-ec33-df7b2222dd81 Address:127.0.0.1:29958}]" +31:52.282 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 127.0.0.1 +31:52.283 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 127.0.0.1 +2021-10-05T16:31:52.283Z [INFO] agent.router: Initializing LAN area manager +31:52.283 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30016 network=udp +31:52.283 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29958 [Follower]" leader= +31:52.283 [INFO] TestAgent.server: Adding LAN server: server="Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 (Addr: tcp/127.0.0.1:29958) (DC: dc1)" +31:52.283 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81.dc1 area=wan +31:52.283 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30016 network=tcp +31:52.284 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29954 network=tcp +31:52.284 [INFO] TestAgent: started state syncer +31:52.284 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30008 network=tcp +31:52.327 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.327 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29958 [Candidate]" term=2 +31:52.344 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:52.344 [DEBUG] TestAgent.server.raft: vote granted: from=789ce253-6eeb-34cf-ec33-df7b2222dd81 term=2 tally=1 +31:52.344 [INFO] TestAgent.server.raft: election won: tally=1 +31:52.344 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29958 [Leader]" +31:52.345 [INFO] TestAgent.server: cluster leadership acquired +31:52.345 [INFO] TestAgent.server: New leader elected: payload=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.355 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29958 +31:52.370 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.398 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.398 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:52.398 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:52.398 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:52.398 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:52.398 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.398 [INFO] TestAgent.server: member joined, marking health alive: member=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.436 [INFO] TestAgent.server: federation state anti-entropy synced +31:52.604 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:52.607 [INFO] TestAgent: Synced node info +31:52.607 [DEBUG] TestAgent: Node info in sync +31:52.903 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:29963] +31:52.904 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:29963 +31:52.909 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b 127.0.0.1 +31:52.909 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:52.909 [DEBUG] TestAgent: systemd notify failed: error="No socket" +31:52.909 [INFO] TestAgent.server: Adding LAN server: server="Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b (Addr: tcp/127.0.0.1:29965) (DC: dc1)" +31:52.909 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b other=Node-789ce253-6eeb-34cf-ec33-df7b2222dd81 +31:52.909 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b +31:52.916 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 127.0.0.1:29964 +31:52.916 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:54382 +31:52.917 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 127.0.0.1 +31:52.917 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 area=wan +31:52.917 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b.dc1 address=127.0.0.1:29964 +31:53.432 [INFO] TestAgent: Requesting shutdown +31:53.432 [INFO] TestAgent.server: shutting down server +31:53.432 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.432 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.432 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.432 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.433 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.433 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.433 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.433 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.433 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.433 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.439 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.445Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.445Z [INFO] agent.router.manager: shutting down +31:53.445 [INFO] TestAgent: consul server down +31:53.445 [INFO] TestAgent: shutdown complete +31:53.445 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30016 network=tcp +31:53.445 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30016 network=udp +31:53.445 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29954 network=tcp +31:53.788 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-d7cda1b4-bd7c-069b-7f6e-e521f053213b (timeout reached) +31:53.948 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.948 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Join (1.75s) +=== CONT TestAgent_RegisterCheck_Service +2021-10-05T16:31:52.994Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.009Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.087 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d2664cf1-33f1-9a7f-7739-dd43884819eb Address:127.0.0.1:30042}]" +31:53.088 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30042 [Follower]" leader= +31:53.088 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d2664cf1-33f1-9a7f-7739-dd43884819eb.dc1 127.0.0.1 +31:53.088 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d2664cf1-33f1-9a7f-7739-dd43884819eb 127.0.0.1 +2021-10-05T16:31:53.088Z [INFO] agent.router: Initializing LAN area manager +31:53.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30037 network=udp +31:53.089 [INFO] TestAgent.server: Adding LAN server: server="Node-d2664cf1-33f1-9a7f-7739-dd43884819eb (Addr: tcp/127.0.0.1:30042) (DC: dc1)" +31:53.089 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d2664cf1-33f1-9a7f-7739-dd43884819eb.dc1 area=wan +31:53.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30037 network=tcp +31:53.090 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30038 network=tcp +31:53.090 [INFO] TestAgent: started state syncer +31:53.090 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30064 network=tcp +31:53.152 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:53.152 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30042 [Candidate]" term=2 +31:53.208 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:53.208 [DEBUG] TestAgent.server.raft: vote granted: from=d2664cf1-33f1-9a7f-7739-dd43884819eb term=2 tally=1 +31:53.208 [INFO] TestAgent.server.raft: election won: tally=1 +31:53.208 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30042 [Leader]" +31:53.208 [INFO] TestAgent.server: cluster leadership acquired +31:53.208 [INFO] TestAgent.server: New leader elected: payload=Node-d2664cf1-33f1-9a7f-7739-dd43884819eb +31:53.233 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30042 +31:53.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:53.342 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:53.342 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:53.342 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:53.342 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:53.342 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:53.342 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d2664cf1-33f1-9a7f-7739-dd43884819eb +31:53.342 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d2664cf1-33f1-9a7f-7739-dd43884819eb +31:53.354 [INFO] TestAgent.server: federation state anti-entropy synced +31:53.383 [DEBUG] connect.ca.consul: consul CA provider configured: id=0a:20:1f:5b:13:04:ac:12:64:a5:0e:ad:7e:a0:58:c0:c0:18:66:16:14:5e:e7:6a:9d:ad:65:c2:ce:15:a4:a7 is_primary=true +31:53.424 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul +31:53.439 [DEBUG] connect.ca.consul: consul CA provider configured: id=bf:97:30:14:95:aa:2b:be:43:eb:9d:c1:5f:54:ca:eb:27:9f:c3:9e:de:58:b4:86:e7:5a:86:f8:59:81:2b:97 is_primary=true +31:53.464 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul +31:53.465 [INFO] TestAgent: Requesting shutdown +31:53.465 [INFO] TestAgent.server: shutting down server +31:53.465 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:53.465 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:53.465 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:53.465 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:53.465 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:53.465 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:53.465 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:53.465 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:53.465 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:53.465 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:53.465 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:53.471 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:53.478Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:53.478Z [INFO] agent.router.manager: shutting down +31:53.478 [INFO] TestAgent: consul server down +31:53.478 [INFO] TestAgent: shutdown complete +31:53.478 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30037 network=tcp +31:53.478 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30037 network=udp +31:53.478 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30038 network=tcp +31:53.978 [INFO] TestAgent: Waiting for endpoints to shut down +31:53.978 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentConnectCARoots_list (1.00s) +=== CONT TestAgent_NodeMaintenance_ACLDeny +[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_Monitor/unknown_log_level +=== RUN TestAgent_Monitor/stream_unstructured_logs +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestAgent_Token/bad_token_name === RUN TestAgent_Token/bad_JSON === RUN TestAgent_Token/set_user_legacy @@ -42485,676 +45456,113 @@ === RUN TestAgent_Token/clear_repl_legacy === RUN TestAgent_Token/clear_repl === RUN TestAgent_Token/permission_denied -2022-11-07T22:46:16.258Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.258Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.273Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.273Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.309 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ecf42d35-6ef3-3ffa-8377-36d0a49620aa Address:127.0.0.1:11392}]" -46:16.309 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11392 [Follower]" leader= -46:16.310 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa.dc1 127.0.0.1 -46:16.311 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa 127.0.0.1 -2022-11-07T22:46:16.311Z [INFO] agent.router: Initializing LAN area manager -46:16.311 [INFO] TestAgent.server: Adding LAN server: server="Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa (Addr: tcp/127.0.0.1:11392) (DC: dc1)" -46:16.311 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa.dc1 area=wan -46:16.311 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11387 network=tcp -46:16.311 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11387 network=udp -46:16.312 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11388 network=tcp -46:16.312 [INFO] TestAgent: started state syncer -46:16.312 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11323 network=tcp -46:16.358 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.358 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11392 [Candidate]" term=2 -46:16.361 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:16.368 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.368 [DEBUG] TestAgent.server.raft: vote granted: from=ecf42d35-6ef3-3ffa-8377-36d0a49620aa term=2 tally=1 -46:16.368 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.368 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11392 [Leader]" -46:16.368 [INFO] TestAgent.server: cluster leadership acquired -46:16.368 [INFO] TestAgent.server: New leader elected: payload=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.380 [INFO] TestAgent.server: initializing acls -46:16.386 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:16.386 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:16.399 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:16.416 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:16.416 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:16.416 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:16.416 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa.dc1 -46:16.416 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa.dc1 area=wan -46:16.416 [INFO] TestAgent.server: Updating LAN server: server="Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa (Addr: tcp/127.0.0.1:11392) (DC: dc1)" -46:16.431 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.462 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.462 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.462 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.462 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.462 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.462 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.462 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.481 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.482 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.484 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ecf42d35-6ef3-3ffa-8377-36d0a49620aa -46:16.507 [INFO] TestAgent: Synced node info -46:16.511 [INFO] TestAgent: Synced service: service=test-id -46:16.511 [DEBUG] TestAgent: Check in sync: check=service:test-id -46:16.512 [INFO] TestAgent: Requesting shutdown -46:16.512 [INFO] TestAgent.server: shutting down server -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.512 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.512 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.512 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.517 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.522Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.522Z [INFO] agent.router.manager: shutting down -46:16.522 [INFO] TestAgent: consul server down -46:16.522 [INFO] TestAgent: shutdown complete -46:16.522 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11387 network=tcp -46:16.522 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11387 network=udp -46:16.522 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11388 network=tcp -46:17.022 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.022 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectCALeafCert_aclDefaultDeny (0.78s) -=== CONT TestAgent_AddCheck -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:16.583Z [WARN] TestAgent_ReloadConfigTLSConfigFailure: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.433Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.443Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.488 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:61b05e79-3703-e452-2d2e-ad61bc6b21d7 Address:127.0.0.1:12027}]" -46:16.488 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12027 [Follower]" leader= -46:16.489 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7.dc1 127.0.0.1 -46:16.489 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7 127.0.0.1 -2022-11-07T22:46:16.489Z [INFO] agent.router: Initializing LAN area manager -46:16.489 [INFO] TestAgent.server: Adding LAN server: server="Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7 (Addr: tcp/127.0.0.1:12027) (DC: dc1)" -46:16.489 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7.dc1 area=wan -46:16.489 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12022 network=udp -46:16.489 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12022 network=tcp -46:16.490 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12023 network=tcp -46:16.490 [INFO] TestAgent: started state syncer -46:16.490 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11379 network=tcp -46:16.533 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.533 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12027 [Candidate]" term=2 -46:16.543 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.543 [DEBUG] TestAgent.server.raft: vote granted: from=61b05e79-3703-e452-2d2e-ad61bc6b21d7 term=2 tally=1 -46:16.543 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.543 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12027 [Leader]" -46:16.543 [INFO] TestAgent.server: cluster leadership acquired -46:16.543 [INFO] TestAgent.server: New leader elected: payload=Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7 -46:16.548 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12027 -46:16.560 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.583 [INFO] TestAgent: Requesting shutdown -46:16.583 [INFO] TestAgent.server: shutting down server -46:16.583 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.583 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.589 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:16.596 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.596 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.596 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.596 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.596 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.596 [INFO] TestAgent.server: member joined, marking health alive: member=Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7 -2022-11-07T22:46:16.599Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.599Z [INFO] agent.router.manager: shutting down -46:16.604 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-61b05e79-3703-e452-2d2e-ad61bc6b21d7 127.0.0.1 12025 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:61b05e79-3703-e452-2d2e-ad61bc6b21d7 port:12027 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:12026] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:16.605 [INFO] TestAgent: consul server down -46:16.605 [INFO] TestAgent: shutdown complete -46:16.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12022 network=tcp -46:16.605 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12022 network=udp -46:16.605 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12023 network=tcp -46:17.105 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.105 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReloadConfigTLSConfigFailure (0.68s) -=== CONT TestAgent_Monitor_ACLDeny -2022-11-07T22:46:16.315Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.324Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.363 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38d2c533-7e9b-2e51-a285-b08484c8e6b4 Address:127.0.0.1:11371}]" -46:16.363 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11371 [Follower]" leader= -46:16.364 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4.dc1 127.0.0.1 -46:16.365 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4 127.0.0.1 -2022-11-07T22:46:16.365Z [INFO] agent.router: Initializing LAN area manager -46:16.365 [INFO] TestAgent.server: Adding LAN server: server="Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4 (Addr: tcp/127.0.0.1:11371) (DC: dc1)" -46:16.365 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4.dc1 area=wan -46:16.365 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11422 network=udp -46:16.365 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11422 network=tcp -46:16.365 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11367 network=tcp -46:16.365 [INFO] TestAgent: started state syncer -46:16.366 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11372 network=tcp -46:16.419 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.419 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11371 [Candidate]" term=2 -46:16.432 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.432 [DEBUG] TestAgent.server.raft: vote granted: from=38d2c533-7e9b-2e51-a285-b08484c8e6b4 term=2 tally=1 -46:16.432 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.432 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11371 [Leader]" -46:16.433 [INFO] TestAgent.server: cluster leadership acquired -46:16.433 [INFO] TestAgent.server: New leader elected: payload=Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4 -46:16.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11371 -46:16.464 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.502 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.503 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.503 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4 -46:16.503 [INFO] TestAgent.server: member joined, marking health alive: member=Node-38d2c533-7e9b-2e51-a285-b08484c8e6b4 -46:16.515 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.605 [INFO] TestAgent.server.connect: CA provider config updated -46:16.605 [INFO] TestAgent: Requesting shutdown -46:16.605 [INFO] TestAgent.server: shutting down server -46:16.605 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.605 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.605 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.605 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.605 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.605 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.605 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.605 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.605 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.605 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.617 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.626Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.626Z [INFO] agent.router.manager: shutting down -46:16.626 [INFO] TestAgent: consul server down -46:16.626 [INFO] TestAgent: shutdown complete -46:16.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11422 network=tcp -46:16.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11422 network=udp -46:16.626 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11367 network=tcp -46:17.127 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.127 [INFO] TestAgent: Endpoints down -=== RUN TestConnectCAConfig/setting_state_fails -2022-11-07T22:46:16.350Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.362Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.401 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d23aed1c-6a5a-5a95-fbac-18628889e4eb Address:127.0.0.1:11413}]" -46:16.401 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11413 [Follower]" leader= -46:16.402 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb.dc1 127.0.0.1 -46:16.402 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb 127.0.0.1 -2022-11-07T22:46:16.402Z [INFO] agent.router: Initializing LAN area manager -46:16.402 [INFO] TestAgent.server: Adding LAN server: server="Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb (Addr: tcp/127.0.0.1:11413) (DC: dc1)" -46:16.402 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb.dc1 area=wan -46:16.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11373 network=udp -46:16.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11373 network=tcp -46:16.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11409 network=tcp -46:16.403 [INFO] TestAgent: started state syncer -46:16.403 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12021 network=tcp -46:16.462 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.462 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11413 [Candidate]" term=2 -46:16.472 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.472 [DEBUG] TestAgent.server.raft: vote granted: from=d23aed1c-6a5a-5a95-fbac-18628889e4eb term=2 tally=1 -46:16.472 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.472 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11413 [Leader]" -46:16.472 [INFO] TestAgent.server: cluster leadership acquired -46:16.472 [INFO] TestAgent.server: New leader elected: payload=Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb -46:16.480 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11413 -46:16.486 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.486 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.486 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb -46:16.486 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d23aed1c-6a5a-5a95-fbac-18628889e4eb -46:16.493 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.612 [INFO] TestAgent: Requesting shutdown -46:16.612 [INFO] TestAgent.server: shutting down server -46:16.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.612 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.612 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.612 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.622 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.630Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.630Z [INFO] agent.router.manager: shutting down -46:16.630 [INFO] TestAgent: consul server down -46:16.630 [INFO] TestAgent: shutdown complete -46:16.630 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11373 network=tcp -46:16.630 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11373 network=udp -46:16.630 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11409 network=tcp -46:17.131 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.131 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectCARoots_empty (0.80s) -=== CONT TestAgent_SecurityChecks -2022-11-07T22:46:16.501Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.501Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:16.510Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.510Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:16.536 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dfd5aaf6-2529-30a4-bb35-16d41c5ff810 Address:127.0.0.1:11329}]" -46:16.536 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11329 [Follower]" leader= -46:16.536 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dfd5aaf6-2529-30a4-bb35-16d41c5ff810.dc1 127.0.0.1 -46:16.537 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dfd5aaf6-2529-30a4-bb35-16d41c5ff810 127.0.0.1 -2022-11-07T22:46:16.537Z [INFO] agent.router: Initializing LAN area manager -46:16.537 [INFO] TestAgent.server: Adding LAN server: server="Node-dfd5aaf6-2529-30a4-bb35-16d41c5ff810 (Addr: tcp/127.0.0.1:11329) (DC: dc1)" -46:16.537 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dfd5aaf6-2529-30a4-bb35-16d41c5ff810.dc1 area=wan -46:16.537 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11380 network=udp -46:16.537 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11380 network=tcp -46:16.537 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11325 network=tcp -46:16.537 [INFO] TestAgent: started state syncer -46:16.537 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11449 network=tcp -46:16.605 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.605 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11329 [Candidate]" term=2 -46:16.622 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.622 [DEBUG] TestAgent.server.raft: vote granted: from=dfd5aaf6-2529-30a4-bb35-16d41c5ff810 term=2 tally=1 -46:16.622 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.622 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11329 [Leader]" -46:16.622 [INFO] TestAgent.server: cluster leadership acquired -46:16.623 [INFO] TestAgent.server: New leader elected: payload=Node-dfd5aaf6-2529-30a4-bb35-16d41c5ff810 -46:16.631 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11329 -46:16.635 [INFO] TestAgent: Requesting shutdown -46:16.635 [INFO] TestAgent.server: shutting down server -46:16.635 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.635 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.639 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:16.644 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:16.645Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.645Z [INFO] agent.router.manager: shutting down -46:16.656 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:16.656 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:16.656 [INFO] TestAgent: consul server down -46:16.656 [INFO] TestAgent: shutdown complete -46:16.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11380 network=tcp -46:16.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11380 network=udp -46:16.656 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11325 network=tcp -46:17.157 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.157 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_MissingService (0.67s) -=== CONT TestAgent_IndexChurn -=== RUN TestAgent_IndexChurn/no_tags -2022-11-07T22:46:16.173Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.225 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f50b860-0538-8667-37a9-d3af46f0ea94 Address:127.0.0.1:11322}]" -46:16.225 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11322 [Follower]" leader= -46:16.226 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f50b860-0538-8667-37a9-d3af46f0ea94.dc1 127.0.0.1 -46:16.226 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f50b860-0538-8667-37a9-d3af46f0ea94 127.0.0.1 -2022-11-07T22:46:16.226Z [INFO] agent.router: Initializing LAN area manager -46:16.226 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f50b860-0538-8667-37a9-d3af46f0ea94.dc1 area=wan -46:16.226 [INFO] TestAgent.server: Adding LAN server: server="Node-8f50b860-0538-8667-37a9-d3af46f0ea94 (Addr: tcp/127.0.0.1:11322) (DC: dc1)" -46:16.226 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11345 network=tcp -46:16.226 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11345 network=udp -46:16.227 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11318 network=tcp -46:16.227 [INFO] TestAgent: started state syncer -46:16.227 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11302 network=tcp -46:16.279 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.279 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11322 [Candidate]" term=2 -46:16.288 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.288 [DEBUG] TestAgent.server.raft: vote granted: from=8f50b860-0538-8667-37a9-d3af46f0ea94 term=2 tally=1 -46:16.288 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.288 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11322 [Leader]" -46:16.288 [INFO] TestAgent.server: cluster leadership acquired -46:16.288 [INFO] TestAgent.server: New leader elected: payload=Node-8f50b860-0538-8667-37a9-d3af46f0ea94 -46:16.299 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11322 -46:16.325 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.352 [INFO] TestAgent: Synced node info -46:16.352 [DEBUG] TestAgent: Node info in sync -46:16.361 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.361 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.361 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.361 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.361 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.361 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f50b860-0538-8667-37a9-d3af46f0ea94 -46:16.361 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f50b860-0538-8667-37a9-d3af46f0ea94 -46:16.378 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.654 [INFO] TestAgent: Requesting shutdown -46:16.654 [INFO] TestAgent.server: shutting down server -46:16.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.654 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.654 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.654 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.654 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.654 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.655 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.661 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.668Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.668Z [INFO] agent.router.manager: shutting down -46:16.668 [INFO] TestAgent: consul server down -46:16.668 [INFO] TestAgent: shutdown complete -46:16.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11345 network=tcp -46:16.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11345 network=udp -46:16.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11318 network=tcp -46:17.168 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.168 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:16.284Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.293Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.342 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15250657-1950-42af-18da-b9520e369182 Address:127.0.0.1:11406}]" -46:16.342 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11406 [Follower]" leader= -46:16.342 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15250657-1950-42af-18da-b9520e369182.dc1 127.0.0.1 -46:16.343 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15250657-1950-42af-18da-b9520e369182 127.0.0.1 -2022-11-07T22:46:16.343Z [INFO] agent.router: Initializing LAN area manager -46:16.343 [INFO] TestAgent.server: Adding LAN server: server="Node-15250657-1950-42af-18da-b9520e369182 (Addr: tcp/127.0.0.1:11406) (DC: dc1)" -46:16.343 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15250657-1950-42af-18da-b9520e369182.dc1 area=wan -46:16.343 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11408 network=udp -46:16.343 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11408 network=tcp -46:16.344 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11402 network=tcp -46:16.344 [INFO] TestAgent: started state syncer -46:16.344 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11351 network=tcp -46:16.397 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.397 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11406 [Candidate]" term=2 -46:16.416 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.416 [DEBUG] TestAgent.server.raft: vote granted: from=15250657-1950-42af-18da-b9520e369182 term=2 tally=1 -46:16.416 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.416 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11406 [Leader]" -46:16.416 [INFO] TestAgent.server: cluster leadership acquired -46:16.416 [INFO] TestAgent.server: New leader elected: payload=Node-15250657-1950-42af-18da-b9520e369182 -46:16.425 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11406 -46:16.449 [INFO] TestAgent: Synced node info -46:16.450 [DEBUG] TestAgent: Node info in sync -46:16.450 [DEBUG] TestAgent: Node info in sync -46:16.467 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.490 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.490 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.490 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.490 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.490 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.490 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15250657-1950-42af-18da-b9520e369182 -46:16.490 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15250657-1950-42af-18da-b9520e369182 -46:16.500 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.582 [DEBUG] connect.ca.consul: consul CA provider configured: id=c5:fe:60:fa:3a:5a:b6:91:47:3b:cc:ec:96:d4:48:38:c1:e4:6c:73:f5:38:df:3f:70:2a:fb:0e:34:22:d9:97 is_primary=true -46:16.624 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul -46:16.635 [DEBUG] connect.ca.consul: consul CA provider configured: id=ee:ca:47:6a:4d:37:7f:b2:77:4e:d5:54:eb:e9:82:98:3d:c8:d1:88:8a:9b:19:1a:fa:c8:0f:5a:0d:92:88:ce is_primary=true -46:16.661 [INFO] TestAgent.server.connect: CA rotated to new root under provider: provider=consul -46:16.661 [INFO] TestAgent: Requesting shutdown -46:16.661 [INFO] TestAgent.server: shutting down server -46:16.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.661 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.661 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.661 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.661 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.661 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.661 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.664 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.668Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.668Z [INFO] agent.router.manager: shutting down -46:16.668 [INFO] TestAgent: consul server down -46:16.668 [INFO] TestAgent: shutdown complete -46:16.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11408 network=tcp -46:16.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11408 network=udp -46:16.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11402 network=tcp -46:17.168 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.168 [INFO] TestAgent: Endpoints down ---- PASS: TestConfig_Apply (1.01s) -=== CONT TestAgent_Monitor ---- PASS: TestAgentConnectCARoots_list (0.89s) -=== CONT TestAgent_loadTokens -2022-11-07T22:46:13.411Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:13.420Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:13.464 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7bce3351-25ef-7e2c-c154-4dd66475368a Address:127.0.0.1:12010}]" -46:13.464 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12010 [Follower]" leader= -46:13.465 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7bce3351-25ef-7e2c-c154-4dd66475368a.dc1 127.0.0.1 -46:13.465 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7bce3351-25ef-7e2c-c154-4dd66475368a 127.0.0.1 -2022-11-07T22:46:13.465Z [INFO] agent.router: Initializing LAN area manager -46:13.465 [INFO] TestAgent.server: Adding LAN server: server="Node-7bce3351-25ef-7e2c-c154-4dd66475368a (Addr: tcp/127.0.0.1:12010) (DC: dc1)" -46:13.465 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7bce3351-25ef-7e2c-c154-4dd66475368a.dc1 area=wan -46:13.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11956 network=udp -46:13.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11956 network=tcp -46:13.466 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11957 network=tcp -46:13.466 [INFO] TestAgent: started state syncer -46:13.466 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12011 network=tcp -46:13.508 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:13.508 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12010 [Candidate]" term=2 -46:13.521 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:13.521 [DEBUG] TestAgent.server.raft: vote granted: from=7bce3351-25ef-7e2c-c154-4dd66475368a term=2 tally=1 -46:13.521 [INFO] TestAgent.server.raft: election won: tally=1 -46:13.521 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12010 [Leader]" -46:13.521 [INFO] TestAgent.server: cluster leadership acquired -46:13.521 [INFO] TestAgent.server: New leader elected: payload=Node-7bce3351-25ef-7e2c-c154-4dd66475368a -46:13.529 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12010 -46:13.549 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:13.584 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:13.584 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:13.584 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:13.584 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:13.584 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:13.584 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7bce3351-25ef-7e2c-c154-4dd66475368a -46:13.584 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7bce3351-25ef-7e2c-c154-4dd66475368a -46:13.592 [INFO] TestAgent: Synced node info -46:13.603 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.554 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.557 [INFO] TestAgent: Synced node info -46:16.557 [DEBUG] TestAgent: Node info in sync -46:16.669 [INFO] TestAgent: Requesting shutdown -46:16.669 [INFO] TestAgent.server: shutting down server -46:16.669 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.669 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.669 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.669 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.669 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.669 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.669 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.669 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.669 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.669 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.672 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.676Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.676Z [INFO] agent.router.manager: shutting down -46:16.676 [INFO] TestAgent: consul server down -46:16.676 [INFO] TestAgent: shutdown complete -46:16.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11956 network=tcp -46:16.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11956 network=udp -46:16.676 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11957 network=tcp -46:17.176 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.176 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogNodes_Blocking (3.78s) -=== CONT TestAgent_ReconnectConfigWanDisabled -2022-11-07T22:46:16.295Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.304Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.356 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a507ffe8-e976-90e5-c912-293ae761219b Address:127.0.0.1:11385}]" -46:16.356 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11385 [Follower]" leader= -46:16.357 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a507ffe8-e976-90e5-c912-293ae761219b.dc1 127.0.0.1 -46:16.357 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a507ffe8-e976-90e5-c912-293ae761219b 127.0.0.1 -2022-11-07T22:46:16.357Z [INFO] agent.router: Initializing LAN area manager -46:16.357 [INFO] TestAgent.server: Adding LAN server: server="Node-a507ffe8-e976-90e5-c912-293ae761219b (Addr: tcp/127.0.0.1:11385) (DC: dc1)" -46:16.357 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a507ffe8-e976-90e5-c912-293ae761219b.dc1 area=wan -46:16.357 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11352 network=udp -46:16.357 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11352 network=tcp -46:16.358 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11381 network=tcp -46:16.358 [INFO] TestAgent: started state syncer -46:16.358 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11421 network=tcp -46:16.422 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.422 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11385 [Candidate]" term=2 -46:16.445 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.445 [DEBUG] TestAgent.server.raft: vote granted: from=a507ffe8-e976-90e5-c912-293ae761219b term=2 tally=1 -46:16.445 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.445 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11385 [Leader]" -46:16.445 [INFO] TestAgent.server: cluster leadership acquired -46:16.445 [INFO] TestAgent.server: New leader elected: payload=Node-a507ffe8-e976-90e5-c912-293ae761219b -46:16.454 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11385 -46:16.471 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.503 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.503 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.503 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a507ffe8-e976-90e5-c912-293ae761219b -46:16.503 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a507ffe8-e976-90e5-c912-293ae761219b -46:16.519 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.671 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.674 [INFO] TestAgent: Synced node info -46:16.681 [INFO] TestAgent: Requesting shutdown -46:16.681 [INFO] TestAgent.server: shutting down server -46:16.681 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.681 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.681 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.681 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.681 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.681 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.684 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.688Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.688Z [INFO] agent.router.manager: shutting down -46:16.688 [INFO] TestAgent: consul server down -46:16.688 [INFO] TestAgent: shutdown complete -46:16.688 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11352 network=tcp -46:16.688 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11352 network=udp -46:16.688 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11381 network=tcp -46:17.188 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.188 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_RestoreState (0.90s) -=== CONT TestAgent_ReconnectConfigSettings -2022-11-07T22:46:16.270Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.279Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:231e3884-277b-62b1-cd26-8bd1fcf0774e Address:127.0.0.1:11350}]" -46:16.324 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11350 [Follower]" leader= -46:16.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-231e3884-277b-62b1-cd26-8bd1fcf0774e.dc1 127.0.0.1 -46:16.325 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-231e3884-277b-62b1-cd26-8bd1fcf0774e 127.0.0.1 -2022-11-07T22:46:16.325Z [INFO] agent.router: Initializing LAN area manager -46:16.325 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11324 network=udp -46:16.325 [INFO] TestAgent.server: Adding LAN server: server="Node-231e3884-277b-62b1-cd26-8bd1fcf0774e (Addr: tcp/127.0.0.1:11350) (DC: dc1)" -46:16.326 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-231e3884-277b-62b1-cd26-8bd1fcf0774e.dc1 area=wan -46:16.326 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11324 network=tcp -46:16.326 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11346 network=tcp -46:16.326 [INFO] TestAgent: started state syncer -46:16.326 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11407 network=tcp -46:16.378 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.378 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11350 [Candidate]" term=2 -46:16.394 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.394 [DEBUG] TestAgent.server.raft: vote granted: from=231e3884-277b-62b1-cd26-8bd1fcf0774e term=2 tally=1 -46:16.394 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.394 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11350 [Leader]" -46:16.394 [INFO] TestAgent.server: cluster leadership acquired -46:16.394 [INFO] TestAgent.server: New leader elected: payload=Node-231e3884-277b-62b1-cd26-8bd1fcf0774e -46:16.401 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11350 -46:16.416 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.453 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.453 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.453 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.453 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.453 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.453 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-231e3884-277b-62b1-cd26-8bd1fcf0774e -46:16.453 [INFO] TestAgent.server: member joined, marking health alive: member=Node-231e3884-277b-62b1-cd26-8bd1fcf0774e -46:16.470 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.610 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:16.616 [INFO] TestAgent: Synced node info -46:16.748 [INFO] TestAgent: Requesting shutdown -46:16.748 [INFO] TestAgent.server: shutting down server -46:16.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.748 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.748 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.748 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.748 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.748 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.748 [INFO] TestAgent.server: shutting down Connect CA roots monitor -46:16.748 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.748 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.748 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.751 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.757Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.757Z [INFO] agent.router.manager: shutting down -46:16.757 [INFO] TestAgent: consul server down -46:16.757 [INFO] TestAgent: shutdown complete -46:16.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11324 network=tcp -46:16.757 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11324 network=udp -46:16.757 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11346 network=tcp -46:17.257 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.257 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_consulConfig_AutoEncryptAllowTLS (1.00s) -=== CONT TestAgent_RegisterCheck_Service -[WARN] freeport: 10 out of 73 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:16.686Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.686Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.696Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.696Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.719 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4bc128c5-4420-978a-ae4c-65730014955c Address:127.0.0.1:11497}]" -46:16.719 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11497 [Follower]" leader= -46:16.722 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4bc128c5-4420-978a-ae4c-65730014955c.dc1 127.0.0.1 -46:16.723 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4bc128c5-4420-978a-ae4c-65730014955c 127.0.0.1 -2022-11-07T22:46:16.723Z [INFO] agent.router: Initializing LAN area manager -46:16.723 [INFO] TestAgent.server: Adding LAN server: server="Node-4bc128c5-4420-978a-ae4c-65730014955c (Addr: tcp/127.0.0.1:11497) (DC: dc1)" -46:16.723 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4bc128c5-4420-978a-ae4c-65730014955c.dc1 area=wan -46:16.723 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11450 network=udp -46:16.723 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11450 network=tcp -46:16.723 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11493 network=tcp -46:16.723 [INFO] TestAgent: started state syncer -46:16.723 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11330 network=tcp -46:16.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:16.786 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.786 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11497 [Candidate]" term=2 -46:16.797 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.797 [DEBUG] TestAgent.server.raft: vote granted: from=4bc128c5-4420-978a-ae4c-65730014955c term=2 tally=1 -46:16.797 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.797 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11497 [Leader]" -46:16.797 [INFO] TestAgent.server: cluster leadership acquired -46:16.797 [INFO] TestAgent.server: New leader elected: payload=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.807 [INFO] TestAgent.server: initializing acls -46:16.809 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:16.809 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:16.814 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:16.821 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:16.821 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:16.821 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:16.822 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.822 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4bc128c5-4420-978a-ae4c-65730014955c.dc1 -46:16.822 [INFO] TestAgent.server: Updating LAN server: server="Node-4bc128c5-4420-978a-ae4c-65730014955c (Addr: tcp/127.0.0.1:11497) (DC: dc1)" -46:16.822 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4bc128c5-4420-978a-ae4c-65730014955c.dc1 area=wan -46:16.831 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.856 [INFO] TestAgent.server: federation state anti-entropy synced -46:16.856 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.955 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.955 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4bc128c5-4420-978a-ae4c-65730014955c -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=acl_token -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=default -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=agent -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_master_token -46:16.956 [INFO] TestAgent: Updated agent's ACL token: token=agent_master -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=acl_replication_token -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=replication -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=acl_token -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=default -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=agent -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_master_token -46:16.957 [INFO] TestAgent: Updated agent's ACL token: token=agent_master -46:16.958 [INFO] TestAgent: Updated agent's ACL token: token=acl_replication_token -46:16.958 [INFO] TestAgent: Updated agent's ACL token: token=replication -46:16.958 [INFO] TestAgent: Requesting shutdown -46:16.958 [INFO] TestAgent.server: shutting down server -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:16.958 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:16.958 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:16.958 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:16.958 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:16.958 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:16.963 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:16.970Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:16.970Z [INFO] agent.router.manager: shutting down -46:16.970 [INFO] TestAgent: consul server down -46:16.970 [INFO] TestAgent: shutdown complete -46:16.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11450 network=tcp -46:16.971 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11450 network=udp -46:16.971 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11493 network=tcp -46:17.471 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.471 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Token (0.80s) +=== RUN TestAgent_Monitor/stream_JSON_logs +=== RUN TestAgent_NodeMaintenance_ACLDeny/no_token +=== RUN TestAgent_NodeMaintenance_ACLDeny/root_token +=== RUN TestAgent_Monitor/serf_shutdown_logging +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:53.786Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.786Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.830Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.830Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.942 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8db302a7-a9f4-4748-6339-a3969d057942 Address:127.0.0.1:30112}]" +31:53.943 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 127.0.0.1 +31:53.943 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8db302a7-a9f4-4748-6339-a3969d057942 127.0.0.1 +2021-10-05T16:31:53.943Z [INFO] agent.router: Initializing LAN area manager +31:53.943 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30030 network=udp +31:53.943 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30112 [Follower]" leader= +31:53.944 [INFO] TestAgent.server: Adding LAN server: server="Node-8db302a7-a9f4-4748-6339-a3969d057942 (Addr: tcp/127.0.0.1:30112) (DC: dc1)" +31:53.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30030 network=tcp +31:53.944 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 area=wan +31:53.945 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30108 network=tcp +31:53.945 [INFO] TestAgent: started state syncer +31:53.945 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30036 network=tcp +31:53.994 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:54.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30112 [Candidate]" term=2 +31:54.028 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.028 [DEBUG] TestAgent.server.raft: vote granted: from=8db302a7-a9f4-4748-6339-a3969d057942 term=2 tally=1 +31:54.028 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.028 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30112 [Leader]" +31:54.029 [INFO] TestAgent.server: cluster leadership acquired +31:54.029 [INFO] TestAgent.server: New leader elected: payload=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.043 [INFO] TestAgent.server: initializing acls +31:54.053 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:54.053 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.067 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:54.071 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:54.094 [INFO] TestAgent.server: initializing acls +31:54.094 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.098 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.098 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:54.098 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:54.098 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.098 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 +31:54.099 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 area=wan +31:54.099 [INFO] TestAgent.server: Updating LAN server: server="Node-8db302a7-a9f4-4748-6339-a3969d057942 (Addr: tcp/127.0.0.1:30112) (DC: dc1)" +31:54.106 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.106 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:54.107 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.107 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 +31:54.107 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8db302a7-a9f4-4748-6339-a3969d057942.dc1 area=wan +31:54.107 [INFO] TestAgent.server: Updating LAN server: server="Node-8db302a7-a9f4-4748-6339-a3969d057942 (Addr: tcp/127.0.0.1:30112) (DC: dc1)" +31:54.130 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.167 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.167 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.167 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.167 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.167 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.167 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.167 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.189 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.189 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.201 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.350 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.350 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8db302a7-a9f4-4748-6339-a3969d057942 +31:54.353 [INFO] TestAgent: Updated agent's ACL token: token=acl_token +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=default +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=agent +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_master_token +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=agent_master +31:54.354 [INFO] TestAgent: Updated agent's ACL token: token=acl_replication_token +31:54.355 [INFO] TestAgent: Updated agent's ACL token: token=replication +31:54.355 [INFO] TestAgent: Updated agent's ACL token: token=acl_token +31:54.355 [INFO] TestAgent: Updated agent's ACL token: token=default +31:54.355 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token +31:54.357 [INFO] TestAgent: Updated agent's ACL token: token=agent +31:54.357 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_master_token +31:54.358 [INFO] TestAgent: Updated agent's ACL token: token=agent_master +31:54.358 [INFO] TestAgent: Updated agent's ACL token: token=acl_replication_token +31:54.358 [INFO] TestAgent: Updated agent's ACL token: token=replication +31:54.358 [INFO] TestAgent: Requesting shutdown +31:54.358 [INFO] TestAgent.server: shutting down server +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.358 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.358 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.359 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.359 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.359 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:54.359 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.359 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.360 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:54.360 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.373Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.373Z [INFO] agent.router.manager: shutting down +31:54.373 [INFO] TestAgent: consul server down +31:54.373 [INFO] TestAgent: shutdown complete +31:54.373 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30030 network=tcp +31:54.373 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30030 network=udp +31:54.373 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30108 network=tcp +31:54.876 [INFO] TestAgent: Waiting for endpoints to shut down +31:54.876 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Token (1.14s) --- PASS: TestAgent_Token/bad_token_name (0.00s) --- PASS: TestAgent_Token/bad_JSON (0.00s) --- PASS: TestAgent_Token/set_user_legacy (0.00s) @@ -43174,5845 +45582,4307 @@ --- PASS: TestAgent_Token/clear_repl_legacy (0.00s) --- PASS: TestAgent_Token/clear_repl (0.00s) --- PASS: TestAgent_Token/permission_denied (0.00s) -=== CONT TestAgent_reloadWatchesHTTPS -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:16.874Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.874Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:16.884Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.884Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:16.917 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9773cf85-da20-c361-28e7-88a230f02b45 Address:127.0.0.1:11427}]" -46:16.917 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11427 [Follower]" leader= -46:16.918 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9773cf85-da20-c361-28e7-88a230f02b45.dc1 127.0.0.1 -46:16.918 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9773cf85-da20-c361-28e7-88a230f02b45 127.0.0.1 -2022-11-07T22:46:16.918Z [INFO] agent.router: Initializing LAN area manager -46:16.918 [INFO] TestAgent.server: Adding LAN server: server="Node-9773cf85-da20-c361-28e7-88a230f02b45 (Addr: tcp/127.0.0.1:11427) (DC: dc1)" -46:16.918 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9773cf85-da20-c361-28e7-88a230f02b45.dc1 area=wan -46:16.918 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11471 network=udp -46:16.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11471 network=tcp -46:16.919 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11423 network=tcp -46:16.919 [INFO] TestAgent: started state syncer -46:16.919 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11456 network=tcp -46:16.963 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.963 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11427 [Candidate]" term=2 -46:16.975 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.975 [DEBUG] TestAgent.server.raft: vote granted: from=9773cf85-da20-c361-28e7-88a230f02b45 term=2 tally=1 -46:16.975 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.975 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11427 [Leader]" -46:16.975 [INFO] TestAgent.server: cluster leadership acquired -46:16.975 [INFO] TestAgent.server: New leader elected: payload=Node-9773cf85-da20-c361-28e7-88a230f02b45 -46:16.981 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11427 -46:16.996 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.020 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.020 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.020 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.020 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.020 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.020 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9773cf85-da20-c361-28e7-88a230f02b45 -46:17.020 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9773cf85-da20-c361-28e7-88a230f02b45 -46:17.031 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.094 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.097 [INFO] TestAgent: Synced node info -46:17.112 [WARN] TestAgent: check has interval below minimum: check=mem minimum_interval=1s -46:17.112 [INFO] TestAgent: Requesting shutdown -46:17.112 [INFO] TestAgent.server: shutting down server -46:17.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.112 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.112 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.112 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.112 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.112 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.112 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.119 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.124Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.124Z [INFO] agent.router.manager: shutting down -46:17.124 [INFO] TestAgent: consul server down -46:17.124 [INFO] TestAgent: shutdown complete -46:17.124 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11471 network=tcp -46:17.124 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11471 network=udp -46:17.124 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11423 network=tcp -46:17.624 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.624 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_MinInterval (0.76s) -=== CONT TestAgent_TokenStore -=== RUN TestAgent_loadTokens/original-configuration -=== RUN TestAgent_loadTokens/updated-configuration -=== RUN TestAgent_loadTokens/persisted-tokens -=== RUN TestAgent_loadTokens/persisted-tokens-override -=== RUN TestAgent_loadTokens/partial-persisted -=== RUN TestAgent_loadTokens/persistence-error-not-json -=== RUN TestAgent_loadTokens/persistence-error-wrong-top-level -=== RUN TestAgent_Monitor/unknown_log_level -=== RUN TestAgent_Monitor/stream_unstructured_logs -2022-11-07T22:46:16.931Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.931Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:16.946Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.946Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:16.975 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7e5e9336-3370-f5d7-d32f-f305b743c9ea Address:127.0.0.1:11511}]" -46:16.975 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11511 [Follower]" leader= -46:16.975 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea.dc1 127.0.0.1 -46:16.976 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea 127.0.0.1 -2022-11-07T22:46:16.976Z [INFO] agent.router: Initializing LAN area manager -46:16.977 [INFO] TestAgent.server: Adding LAN server: server="Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea (Addr: tcp/127.0.0.1:11511) (DC: dc1)" -46:16.977 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea.dc1 area=wan -46:16.977 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11478 network=udp -46:16.977 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11478 network=tcp -46:16.977 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11507 network=tcp -46:16.977 [INFO] TestAgent: started state syncer -46:16.977 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11435 network=tcp -46:17.036 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.036 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11511 [Candidate]" term=2 -46:17.051 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.051 [DEBUG] TestAgent.server.raft: vote granted: from=7e5e9336-3370-f5d7-d32f-f305b743c9ea term=2 tally=1 -46:17.051 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.051 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11511 [Leader]" -46:17.051 [INFO] TestAgent.server: cluster leadership acquired -46:17.051 [INFO] TestAgent.server: New leader elected: payload=Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea -46:17.059 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11511 -46:17.079 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.105 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.105 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.105 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.105 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.105 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.105 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea -46:17.105 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7e5e9336-3370-f5d7-d32f-f305b743c9ea -46:17.118 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.161 [INFO] TestAgent: Requesting shutdown -46:17.161 [INFO] TestAgent.server: shutting down server -46:17.161 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.161 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.161 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.161 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.161 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.161 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:17.161 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.161 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.161 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.161 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.161 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.168 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.175Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.175Z [INFO] agent.router.manager: shutting down -46:17.175 [INFO] TestAgent: consul server down -46:17.175 [INFO] TestAgent: shutdown complete -46:17.175 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11478 network=tcp -46:17.175 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11478 network=udp -46:17.175 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11507 network=tcp -46:17.676 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.676 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_StartPassing (0.76s) -=== CONT TestAgentConnectAuthorize_allowTrustDomain -2022-11-07T22:46:17.235Z [WARN] TestAgent_ReloadConfigOutgoingRPCConfig: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.949Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.959Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.991 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f Address:127.0.0.1:11455}]" -46:16.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11455 [Follower]" leader= -46:16.992 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f.dc1 127.0.0.1 -46:16.993 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f 127.0.0.1 -2022-11-07T22:46:16.993Z [INFO] agent.router: Initializing LAN area manager -46:16.993 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f.dc1 area=wan -46:16.993 [INFO] TestAgent.server: Adding LAN server: server="Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f (Addr: tcp/127.0.0.1:11455) (DC: dc1)" -46:16.993 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11436 network=udp -46:16.993 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11436 network=tcp -46:16.993 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11451 network=tcp -46:16.993 [INFO] TestAgent: started state syncer -46:16.993 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11484 network=tcp -46:17.035 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.035 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11455 [Candidate]" term=2 -46:17.051 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.051 [DEBUG] TestAgent.server.raft: vote granted: from=a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f term=2 tally=1 -46:17.051 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.051 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11455 [Leader]" -46:17.051 [INFO] TestAgent.server: cluster leadership acquired -46:17.051 [INFO] TestAgent.server: New leader elected: payload=Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f -46:17.061 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11455 -46:17.096 [INFO] TestAgent: Synced node info -46:17.096 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.112 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.112 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.112 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.112 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.112 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.112 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f -46:17.112 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a3ae71f8-88e8-bd23-7d8d-2abef1fbec0f -46:17.120 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.236 [INFO] TestAgent: Requesting shutdown -46:17.236 [INFO] TestAgent.server: shutting down server -46:17.236 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.236 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.241 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.246Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.246Z [INFO] agent.router.manager: shutting down -46:17.246 [INFO] TestAgent: consul server down -46:17.246 [INFO] TestAgent: shutdown complete -46:17.246 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11436 network=tcp -46:17.246 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11436 network=udp -46:17.246 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11451 network=tcp -46:17.747 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.747 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReloadConfigOutgoingRPCConfig (0.81s) -=== CONT TestAgent_NodeMaintenance_ACLDeny -2022-11-07T22:46:17.241Z [WARN] TestAgent_ReloadConfigIncomingRPCConfig: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.766Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.776Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.805 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f5e0820b-8287-c790-7c09-30d5d026f213 Address:127.0.0.1:11469}]" -46:16.805 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11469 [Follower]" leader= -46:16.805 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f5e0820b-8287-c790-7c09-30d5d026f213.dc1 127.0.0.1 -46:16.806 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f5e0820b-8287-c790-7c09-30d5d026f213 127.0.0.1 -2022-11-07T22:46:16.806Z [INFO] agent.router: Initializing LAN area manager -46:16.806 [INFO] TestAgent.server: Adding LAN server: server="Node-f5e0820b-8287-c790-7c09-30d5d026f213 (Addr: tcp/127.0.0.1:11469) (DC: dc1)" -46:16.806 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f5e0820b-8287-c790-7c09-30d5d026f213.dc1 area=wan -46:16.806 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11331 network=udp -46:16.806 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11331 network=tcp -46:16.807 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11465 network=tcp -46:16.807 [INFO] TestAgent: started state syncer -46:16.807 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11470 network=tcp -46:16.864 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:16.864 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11469 [Candidate]" term=2 -46:16.873 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:16.873 [DEBUG] TestAgent.server.raft: vote granted: from=f5e0820b-8287-c790-7c09-30d5d026f213 term=2 tally=1 -46:16.873 [INFO] TestAgent.server.raft: election won: tally=1 -46:16.873 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11469 [Leader]" -46:16.873 [INFO] TestAgent.server: cluster leadership acquired -46:16.873 [INFO] TestAgent.server: New leader elected: payload=Node-f5e0820b-8287-c790-7c09-30d5d026f213 -46:16.879 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11469 -46:16.891 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:16.918 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:16.918 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:16.918 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:16.918 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:16.918 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:16.918 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f5e0820b-8287-c790-7c09-30d5d026f213 -46:16.918 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f5e0820b-8287-c790-7c09-30d5d026f213 -46:16.929 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.095 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.098 [INFO] TestAgent: Synced node info -46:17.098 [DEBUG] TestAgent: Node info in sync -46:17.242 [INFO] TestAgent: Requesting shutdown -46:17.242 [INFO] TestAgent.server: shutting down server -46:17.242 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.242 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.250 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.256Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.256Z [INFO] agent.router.manager: shutting down -46:17.257 [INFO] TestAgent: consul server down -46:17.257 [INFO] TestAgent: shutdown complete -46:17.257 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11331 network=tcp -46:17.257 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11331 network=udp -46:17.257 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11465 network=tcp -46:17.757 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.757 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReloadConfigIncomingRPCConfig (1.00s) -=== CONT TestAgent_reloadWatches -2022-11-07T22:46:17.033Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.033Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:17.046Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.046Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:17.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8bb9ab4-b15e-549e-b673-8f406e074456 Address:127.0.0.1:11476}]" -46:17.080 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11476 [Follower]" leader= -46:17.080 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8bb9ab4-b15e-549e-b673-8f406e074456.dc1 127.0.0.1 -46:17.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8bb9ab4-b15e-549e-b673-8f406e074456 127.0.0.1 -2022-11-07T22:46:17.081Z [INFO] agent.router: Initializing LAN area manager -46:17.081 [INFO] TestAgent.server: Adding LAN server: server="Node-b8bb9ab4-b15e-549e-b673-8f406e074456 (Addr: tcp/127.0.0.1:11476) (DC: dc1)" -46:17.081 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8bb9ab4-b15e-549e-b673-8f406e074456.dc1 area=wan -46:17.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11485 network=udp -46:17.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11485 network=tcp -46:17.081 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11472 network=tcp -46:17.081 [INFO] TestAgent: started state syncer -46:17.081 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11505 network=tcp -46:17.146 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.146 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11476 [Candidate]" term=2 -46:17.158 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.158 [DEBUG] TestAgent.server.raft: vote granted: from=b8bb9ab4-b15e-549e-b673-8f406e074456 term=2 tally=1 -46:17.158 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.158 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11476 [Leader]" -46:17.158 [INFO] TestAgent.server: cluster leadership acquired -46:17.158 [INFO] TestAgent.server: New leader elected: payload=Node-b8bb9ab4-b15e-549e-b673-8f406e074456 -46:17.165 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11476 -46:17.182 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.215 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.215 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.215 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.215 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.215 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.215 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8bb9ab4-b15e-549e-b673-8f406e074456 -46:17.215 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8bb9ab4-b15e-549e-b673-8f406e074456 -46:17.227 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.245 [INFO] TestAgent: Synced node info -46:17.251 [INFO] TestAgent: Requesting shutdown -46:17.251 [INFO] TestAgent.server: shutting down server -46:17.251 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.251 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.251 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.251 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.251 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.251 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.251 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.251 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.251 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.251 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.256 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.264Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.264Z [INFO] agent.router.manager: shutting down -46:17.264 [INFO] TestAgent: consul server down -46:17.264 [INFO] TestAgent: shutdown complete -46:17.264 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11485 network=tcp -46:17.264 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11485 network=udp -46:17.264 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11472 network=tcp -46:17.765 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.765 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck (0.74s) -=== CONT TestAgentConnectAuthorize_denyWildcard -=== RUN TestAgent_Monitor/stream_JSON_logs -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:17.119Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:17.119Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.128Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:17.128Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.159 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6a9d6b48-b264-4d2e-4db4-8dc567d1e596 Address:127.0.0.1:11441}]" -46:17.159 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11441 [Follower]" leader= -46:17.160 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596.dc1 127.0.0.1 -46:17.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 127.0.0.1 -2022-11-07T22:46:17.160Z [INFO] agent.router: Initializing LAN area manager -46:17.160 [INFO] TestAgent.server: Adding LAN server: server="Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 (Addr: tcp/127.0.0.1:11441) (DC: dc1)" -46:17.160 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596.dc1 area=wan -46:17.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11506 network=udp -46:17.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11506 network=tcp -46:17.161 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11437 network=tcp -46:17.161 [INFO] TestAgent: started state syncer -46:17.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11463 network=tcp -46:17.210 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:17.213 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.213 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11441 [Candidate]" term=2 -46:17.232 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.232 [DEBUG] TestAgent.server.raft: vote granted: from=6a9d6b48-b264-4d2e-4db4-8dc567d1e596 term=2 tally=1 -46:17.232 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.232 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11441 [Leader]" -46:17.232 [INFO] TestAgent.server: cluster leadership acquired -46:17.232 [INFO] TestAgent.server: New leader elected: payload=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.241 [INFO] TestAgent.server: initializing acls -46:17.247 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:17.247 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:17.257 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:17.273 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:17.273 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:17.273 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:17.273 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.273 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596.dc1 -46:17.273 [INFO] TestAgent.server: Updating LAN server: server="Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 (Addr: tcp/127.0.0.1:11441) (DC: dc1)" -46:17.273 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596.dc1 area=wan -46:17.301 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.332 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.332 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.332 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.333 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.333 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.333 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.333 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.345 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.368 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.368 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6a9d6b48-b264-4d2e-4db4-8dc567d1e596 -46:17.368 [INFO] TestAgent: Requesting shutdown -46:17.368 [INFO] TestAgent.server: shutting down server -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:17.368 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:17.368 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:17.368 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.368 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.368 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:17.373 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.381Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.381Z [INFO] agent.router.manager: shutting down -46:17.381 [INFO] TestAgent: consul server down -46:17.381 [INFO] TestAgent: shutdown complete -46:17.381 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11506 network=tcp -46:17.381 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11506 network=udp -46:17.381 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11437 network=tcp -46:17.882 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.882 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Monitor_ACLDeny (0.78s) -=== CONT TestAgentConnectAuthorize_deny ---- PASS: TestAgent_SecurityChecks (0.75s) === CONT TestAgent_NodeMaintenance_Disable -=== RUN TestAgent_Monitor/serf_shutdown_logging -2022-11-07T22:46:17.136Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.154Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.185 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b53e3e33-ccd7-9e76-88be-3a3403b1d245 Address:127.0.0.1:11504}]" -46:17.185 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11504 [Follower]" leader= -46:17.185 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245.dc1 127.0.0.1 -46:17.186 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245 127.0.0.1 -2022-11-07T22:46:17.186Z [INFO] agent.router: Initializing LAN area manager -46:17.186 [INFO] TestAgent.server: Adding LAN server: server="Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245 (Addr: tcp/127.0.0.1:11504) (DC: dc1)" -46:17.186 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245.dc1 area=wan -46:17.186 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11464 network=udp -46:17.186 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11464 network=tcp -46:17.187 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11500 network=tcp -46:17.187 [INFO] TestAgent: started state syncer -46:17.187 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11498 network=tcp -46:17.253 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.253 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11504 [Candidate]" term=2 -46:17.267 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.267 [DEBUG] TestAgent.server.raft: vote granted: from=b53e3e33-ccd7-9e76-88be-3a3403b1d245 term=2 tally=1 -46:17.267 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.267 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11504 [Leader]" -46:17.268 [INFO] TestAgent.server: cluster leadership acquired -46:17.268 [INFO] TestAgent.server: New leader elected: payload=Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245 -46:17.282 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11504 -46:17.308 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.345 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.345 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.345 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.345 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.345 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245 -46:17.345 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b53e3e33-ccd7-9e76-88be-3a3403b1d245 -46:17.360 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.434 [INFO] TestAgent: Requesting shutdown -46:17.435 [INFO] TestAgent.server: shutting down server -46:17.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.435 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.435 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.435 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.435 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.435 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:17.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.435 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.435 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.439 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.446Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.446Z [INFO] agent.router.manager: shutting down -46:17.446 [INFO] TestAgent: consul server down -46:17.446 [INFO] TestAgent: shutdown complete -46:17.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11464 network=tcp -46:17.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11464 network=udp -46:17.446 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11500 network=tcp -46:17.946 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.946 [INFO] TestAgent: Endpoints down -=== RUN TestConnectCAConfig/updating_config_with_same_state -2022-11-07T22:46:17.191Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.201Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.245 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8aa1b5ca-2867-2a34-0382-9592eb340d62 Address:127.0.0.1:11378}]" -46:17.245 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11378 [Follower]" leader= -46:17.245 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8aa1b5ca-2867-2a34-0382-9592eb340d62 127.0.0.1 -2022-11-07T22:46:17.245Z [INFO] agent.router: Initializing LAN area manager -46:17.245 [INFO] TestAgent.server: Adding LAN server: server="Node-8aa1b5ca-2867-2a34-0382-9592eb340d62 (Addr: tcp/127.0.0.1:11378) (DC: dc1)" -46:17.245 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11282 network=tcp -46:17.245 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11282 network=udp -46:17.246 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11374 network=tcp -46:17.246 [INFO] TestAgent: started state syncer -46:17.246 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11442 network=tcp -46:17.310 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.310 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11378 [Candidate]" term=2 -46:17.328 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.328 [DEBUG] TestAgent.server.raft: vote granted: from=8aa1b5ca-2867-2a34-0382-9592eb340d62 term=2 tally=1 -46:17.328 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.328 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11378 [Leader]" -46:17.328 [INFO] TestAgent.server: cluster leadership acquired -46:17.328 [INFO] TestAgent.server: New leader elected: payload=Node-8aa1b5ca-2867-2a34-0382-9592eb340d62 -46:17.337 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11378 -46:17.360 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.383 [INFO] TestAgent: Synced node info -46:17.383 [DEBUG] TestAgent: Node info in sync -46:17.391 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.391 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.391 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.391 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.391 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.391 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8aa1b5ca-2867-2a34-0382-9592eb340d62 -46:17.391 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8aa1b5ca-2867-2a34-0382-9592eb340d62 -46:17.404 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.444 [INFO] TestAgent: Requesting shutdown -46:17.444 [INFO] TestAgent.server: shutting down server -46:17.444 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.444 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.444 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.444 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.444 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.444 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.444 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.444 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.444 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -2022-11-07T22:46:17.449Z [INFO] agent.router.manager: shutting down -46:17.449 [INFO] TestAgent: consul server down -46:17.449 [INFO] TestAgent: shutdown complete -46:17.449 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11282 network=tcp -46:17.449 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11282 network=udp -46:17.449 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11374 network=tcp -46:17.949 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.949 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReconnectConfigWanDisabled (0.77s) -=== CONT TestAgent_GetCoordinate -=== RUN TestAgent_NodeMaintenance_ACLDeny/no_token -=== RUN TestAgent_NodeMaintenance_ACLDeny/root_token -2022-11-07T22:46:17.268Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.277Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.332 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:253b5d18-b308-6393-99cf-9cdb51a2693c Address:127.0.0.1:11273}]" -46:17.332 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11273 [Follower]" leader= -46:17.332 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-253b5d18-b308-6393-99cf-9cdb51a2693c.dc1 127.0.0.1 -46:17.333 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-253b5d18-b308-6393-99cf-9cdb51a2693c 127.0.0.1 -2022-11-07T22:46:17.333Z [INFO] agent.router: Initializing LAN area manager -46:17.333 [INFO] TestAgent.server: Adding LAN server: server="Node-253b5d18-b308-6393-99cf-9cdb51a2693c (Addr: tcp/127.0.0.1:11273) (DC: dc1)" -46:17.333 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-253b5d18-b308-6393-99cf-9cdb51a2693c.dc1 area=wan -46:17.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11394 network=udp -46:17.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11394 network=tcp -46:17.334 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11269 network=tcp -46:17.334 [INFO] TestAgent: started state syncer -46:17.334 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11414 network=tcp -46:17.390 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.390 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11273 [Candidate]" term=2 -46:17.401 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.401 [DEBUG] TestAgent.server.raft: vote granted: from=253b5d18-b308-6393-99cf-9cdb51a2693c term=2 tally=1 -46:17.401 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.401 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11273 [Leader]" -46:17.402 [INFO] TestAgent.server: cluster leadership acquired -46:17.402 [INFO] TestAgent.server: New leader elected: payload=Node-253b5d18-b308-6393-99cf-9cdb51a2693c -46:17.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11273 -46:17.422 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.444 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.444 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.444 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.444 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.444 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.444 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-253b5d18-b308-6393-99cf-9cdb51a2693c -46:17.444 [INFO] TestAgent.server: member joined, marking health alive: member=Node-253b5d18-b308-6393-99cf-9cdb51a2693c -46:17.458 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.465 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.469 [INFO] TestAgent: Synced node info -46:17.476 [INFO] TestAgent: Synced service: service=memcache -46:17.476 [DEBUG] TestAgent: Check in sync: check=service:memcache -46:17.486 [DEBUG] TestAgent: Node info in sync -46:17.486 [DEBUG] TestAgent: Service in sync: service=memcache -46:17.486 [DEBUG] TestAgent: Check in sync: check=service:memcache -46:17.492 [DEBUG] TestAgent: Node info in sync -46:17.492 [DEBUG] TestAgent: Service in sync: service=memcache -46:17.492 [DEBUG] TestAgent: Check in sync: check=service:memcache -46:17.495 [INFO] TestAgent: Synced check: check=memcache_check2 -46:17.495 [INFO] TestAgent: Requesting shutdown -46:17.495 [INFO] TestAgent.server: shutting down server -46:17.495 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.495 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.495 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.495 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.495 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.495 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.495 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.495 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.495 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.495 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.499 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.505Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.505Z [INFO] agent.router.manager: shutting down -46:17.506 [INFO] TestAgent: consul server down -46:17.506 [INFO] TestAgent: shutdown complete -46:17.506 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11394 network=tcp -46:17.506 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11394 network=udp -46:17.506 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11269 network=tcp -46:18.007 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.007 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck_Service (0.75s) -=== CONT TestAgent_RPCPing -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:17.490Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.500Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.530 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cbb5b631-d58b-c672-bb5c-cf3a32bff93b Address:127.0.0.1:11483}]" -46:17.531 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11483 [Follower]" leader= -46:17.531 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cbb5b631-d58b-c672-bb5c-cf3a32bff93b.dc1 127.0.0.1 -46:17.532 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cbb5b631-d58b-c672-bb5c-cf3a32bff93b 127.0.0.1 -2022-11-07T22:46:17.532Z [INFO] agent.router: Initializing LAN area manager -46:17.532 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11519 network=udp -46:17.532 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cbb5b631-d58b-c672-bb5c-cf3a32bff93b.dc1 area=wan -46:17.532 [INFO] TestAgent.server: Adding LAN server: server="Node-cbb5b631-d58b-c672-bb5c-cf3a32bff93b (Addr: tcp/127.0.0.1:11483) (DC: dc1)" -46:17.532 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11519 network=tcp -46:17.533 [INFO] TestAgent: Started HTTPS server: address=127.0.0.1:11480 network=tcp -46:17.533 [INFO] TestAgent: started state syncer -46:17.533 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11491 network=tcp -46:17.589 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.590 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11483 [Candidate]" term=2 -46:17.599 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.599 [DEBUG] TestAgent.server.raft: vote granted: from=cbb5b631-d58b-c672-bb5c-cf3a32bff93b term=2 tally=1 -46:17.599 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.599 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11483 [Leader]" -46:17.599 [INFO] TestAgent.server: cluster leadership acquired -46:17.599 [INFO] TestAgent.server: New leader elected: payload=Node-cbb5b631-d58b-c672-bb5c-cf3a32bff93b -46:17.601 [INFO] TestAgent: Requesting shutdown -46:17.601 [INFO] TestAgent.server: shutting down server -46:17.601 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.602 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:17.605 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11483 -46:17.607 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.612Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.612Z [INFO] agent.router.manager: shutting down -46:17.615 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" -46:17.615 [INFO] TestAgent: consul server down -46:17.615 [INFO] TestAgent: shutdown complete -46:17.615 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11519 network=tcp -46:17.615 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11519 network=udp -46:17.615 [INFO] TestAgent: Stopping server: protocol=HTTPS address=127.0.0.1:11480 network=tcp -46:18.116 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.116 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_reloadWatchesHTTPS (0.64s) -=== CONT TestAgent_purgeCheckState -2022-11-07T22:46:17.179Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.189Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.233 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4905d663-6273-2822-71a6-4af1a369141f Address:127.0.0.1:11448}]" -46:17.233 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11448 [Follower]" leader= -46:17.234 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4905d663-6273-2822-71a6-4af1a369141f.dc1 127.0.0.1 -46:17.234 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4905d663-6273-2822-71a6-4af1a369141f 127.0.0.1 -2022-11-07T22:46:17.234Z [INFO] agent.router: Initializing LAN area manager -46:17.234 [INFO] TestAgent.server: Adding LAN server: server="Node-4905d663-6273-2822-71a6-4af1a369141f (Addr: tcp/127.0.0.1:11448) (DC: dc1)" -46:17.234 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4905d663-6273-2822-71a6-4af1a369141f.dc1 area=wan -46:17.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11526 network=udp -46:17.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11526 network=tcp -46:17.235 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11444 network=tcp -46:17.235 [INFO] TestAgent: started state syncer -46:17.235 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11281 network=tcp -46:17.276 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.276 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11448 [Candidate]" term=2 -46:17.284 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:17.310 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.310 [DEBUG] TestAgent.server.raft: vote granted: from=4905d663-6273-2822-71a6-4af1a369141f term=2 tally=1 -46:17.310 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.310 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11448 [Leader]" -46:17.310 [INFO] TestAgent.server: cluster leadership acquired -46:17.310 [INFO] TestAgent.server: New leader elected: payload=Node-4905d663-6273-2822-71a6-4af1a369141f -46:17.323 [INFO] TestAgent.server: initializing acls -46:17.327 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:17.339 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:17.339 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:17.339 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:17.339 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4905d663-6273-2822-71a6-4af1a369141f -46:17.339 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4905d663-6273-2822-71a6-4af1a369141f.dc1 -46:17.339 [INFO] TestAgent.server: Updating LAN server: server="Node-4905d663-6273-2822-71a6-4af1a369141f (Addr: tcp/127.0.0.1:11448) (DC: dc1)" -46:17.339 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4905d663-6273-2822-71a6-4af1a369141f.dc1 area=wan -46:17.366 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.385 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:17.388 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.388 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.388 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.388 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.388 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.388 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4905d663-6273-2822-71a6-4af1a369141f -46:17.388 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4905d663-6273-2822-71a6-4af1a369141f -46:17.394 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.401 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4905d663-6273-2822-71a6-4af1a369141f -46:17.631 [WARN] TestAgent: "default" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "replication" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "default" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token -46:17.631 [WARN] TestAgent: "replication" token present in both the configuration and persisted token store, using the persisted token -46:17.632 [WARN] TestAgent: "agent" token present in both the configuration and persisted token store, using the persisted token -46:17.632 [WARN] TestAgent: "agent_master" token present in both the configuration and persisted token store, using the persisted token -46:17.633 [WARN] TestAgent: unable to load persisted tokens: error="failed to decode tokens file "/tmp/consul-test/TestAgent_loadTokens-TestAgent-agent460646371/acl-tokens.json": invalid character '\x01' looking for beginning of value" -46:17.633 [WARN] TestAgent: unable to load persisted tokens: error="failed to decode tokens file "/tmp/consul-test/TestAgent_loadTokens-TestAgent-agent460646371/acl-tokens.json": json: cannot unmarshal array into Go value of type agent.persistedTokens" -46:17.633 [INFO] TestAgent: Requesting shutdown -46:17.633 [INFO] TestAgent.server: shutting down server -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.633 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.633 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:17.633 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.633 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:17.637 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.641Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.641Z [INFO] agent.router.manager: shutting down -46:17.641 [INFO] TestAgent: consul server down -46:17.641 [INFO] TestAgent: shutdown complete -46:17.641 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11526 network=tcp -46:17.641 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11526 network=udp -46:17.641 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11444 network=tcp -46:18.142 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.142 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_loadTokens (0.97s) - --- PASS: TestAgent_loadTokens/original-configuration (0.00s) - --- PASS: TestAgent_loadTokens/updated-configuration (0.00s) - --- PASS: TestAgent_loadTokens/persisted-tokens (0.00s) - --- PASS: TestAgent_loadTokens/persisted-tokens-override (0.00s) - --- PASS: TestAgent_loadTokens/partial-persisted (0.00s) - --- PASS: TestAgent_loadTokens/persistence-error-not-json (0.00s) - --- PASS: TestAgent_loadTokens/persistence-error-wrong-top-level (0.00s) -=== CONT TestAgentConnectAuthorize_allow -2022-11-07T22:46:16.895Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.895Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:16.905Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:16.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:16.936 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 Address:127.0.0.1:11420}]" -46:16.936 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11420 [Follower]" leader= -46:16.937 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48.dc1 127.0.0.1 -46:16.937 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 127.0.0.1 -2022-11-07T22:46:16.937Z [INFO] agent.router: Initializing LAN area manager -46:16.937 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11457 network=udp -46:16.937 [INFO] TestAgent.server: Adding LAN server: server="Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 (Addr: tcp/127.0.0.1:11420) (DC: dc1)" -46:16.937 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48.dc1 area=wan -46:16.937 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11457 network=tcp -46:16.938 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11416 network=tcp -46:16.938 [INFO] TestAgent: started state syncer -46:16.938 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11477 network=tcp -46:16.987 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:17.001 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.001 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11420 [Candidate]" term=2 -46:17.012 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.012 [DEBUG] TestAgent.server.raft: vote granted: from=7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 term=2 tally=1 -46:17.012 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.012 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11420 [Leader]" -46:17.013 [INFO] TestAgent.server: cluster leadership acquired -46:17.013 [INFO] TestAgent.server: New leader elected: payload=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.019 [INFO] TestAgent.server: initializing acls -46:17.022 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:17.022 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:17.029 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:17.037 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:17.037 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:17.037 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:17.037 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.037 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48.dc1 -46:17.037 [INFO] TestAgent.server: Updating LAN server: server="Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 (Addr: tcp/127.0.0.1:11420) (DC: dc1)" -46:17.037 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48.dc1 area=wan -46:17.053 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.081 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.081 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.081 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.081 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.081 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.081 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.081 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.093 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.094 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.327 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -46:17.327 [WARN] TestAgent: Node info update blocked by ACLs: node=7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 accessorID=00000000-0000-0000-0000-000000000002 -46:17.327 [DEBUG] TestAgent: Node info in sync -46:17.340 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.340 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7070cd49-f4a8-1c30-e56b-2b6b1dc97f48 -46:17.352 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token -46:17.652 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service="{consul {}}" -46:17.652 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.656 [INFO] TestAgent: Synced node info -46:17.661 [INFO] TestAgent: Requesting shutdown -46:17.661 [INFO] TestAgent.server: shutting down server -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.661 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.661 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.661 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.661 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.665 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.673Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.673Z [INFO] agent.router.manager: shutting down -46:17.673 [INFO] TestAgent: consul server down -46:17.673 [INFO] TestAgent: shutdown complete -46:17.673 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11457 network=tcp -46:17.673 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11457 network=udp -46:17.673 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11416 network=tcp -46:18.173 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.173 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_TokenTriggersFullSync/agent +2021-10-05T16:31:53.857Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.857Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.905Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.982 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f491f49-49cb-5b2e-a251-1e913962dcf7 Address:127.0.0.1:30063}]" +31:53.982 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30063 [Follower]" leader= +31:53.983 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f491f49-49cb-5b2e-a251-1e913962dcf7.dc1 127.0.0.1 +31:53.983 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 127.0.0.1 +2021-10-05T16:31:53.983Z [INFO] agent.router: Initializing LAN area manager +31:53.984 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7.dc1 area=wan +31:53.984 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30023 network=udp +31:53.984 [INFO] TestAgent.server: Adding LAN server: server="Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 (Addr: tcp/127.0.0.1:30063) (DC: dc1)" +31:53.984 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30023 network=tcp +31:53.984 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30059 network=tcp +31:53.984 [INFO] TestAgent: started state syncer +31:53.984 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30106 network=tcp +31:54.028 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.029 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30063 [Candidate]" term=2 +31:54.034 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:54.058 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.058 [DEBUG] TestAgent.server.raft: vote granted: from=8f491f49-49cb-5b2e-a251-1e913962dcf7 term=2 tally=1 +31:54.058 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.058 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30063 [Leader]" +31:54.058 [INFO] TestAgent.server: cluster leadership acquired +31:54.058 [INFO] TestAgent.server: New leader elected: payload=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.077 [INFO] TestAgent.server: initializing acls +31:54.093 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:54.093 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.105 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:54.124 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.124 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:54.124 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:54.124 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.124 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8f491f49-49cb-5b2e-a251-1e913962dcf7.dc1 +31:54.124 [INFO] TestAgent.server: Updating LAN server: server="Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 (Addr: tcp/127.0.0.1:30063) (DC: dc1)" +31:54.125 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7.dc1 area=wan +31:54.144 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.202 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.202 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.202 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.202 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.202 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.202 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.202 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.216 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.222 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.368 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8f491f49-49cb-5b2e-a251-1e913962dcf7 +31:54.368 [INFO] TestAgent.server.fsm: snapshot created: duration=54.406µs +31:54.368 [INFO] TestAgent.server.raft: starting snapshot up to: index=14 +31:54.368 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_Options_GET#02-TestAgent-agent138019399/raft/snapshots/2-14-1633451514368.tmp +31:54.448 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:54.496 [INFO] TestAgent: Synced node info +31:54.496 [DEBUG] TestAgent: Node info in sync +31:54.508 [INFO] TestAgent.server.raft: snapshot complete up to: index=14 +31:54.540 [INFO] TestAgent: Requesting shutdown +31:54.541 [INFO] TestAgent.server: shutting down server +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.541 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:54.541 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.541 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.541 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:54.546 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.547Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.547Z [INFO] agent.router.manager: shutting down +31:54.547 [INFO] TestAgent: consul server down +31:54.547 [INFO] TestAgent: shutdown complete +31:54.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30023 network=tcp +31:54.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30023 network=udp +31:54.547 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30059 network=tcp +31:55.048 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.048 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:53.911Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.911Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.954Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.954Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:54.051 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:376a3752-3096-e38e-f647-25c6a9ada849 Address:127.0.0.1:30098}]" +31:54.052 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30098 [Follower]" leader= +31:54.052 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-376a3752-3096-e38e-f647-25c6a9ada849.dc1 127.0.0.1 +31:54.052 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-376a3752-3096-e38e-f647-25c6a9ada849 127.0.0.1 +2021-10-05T16:31:54.053Z [INFO] agent.router: Initializing LAN area manager +31:54.053 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30114 network=udp +31:54.053 [INFO] TestAgent.server: Adding LAN server: server="Node-376a3752-3096-e38e-f647-25c6a9ada849 (Addr: tcp/127.0.0.1:30098) (DC: dc1)" +31:54.053 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-376a3752-3096-e38e-f647-25c6a9ada849.dc1 area=wan +31:54.053 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30114 network=tcp +31:54.053 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30094 network=tcp +31:54.054 [INFO] TestAgent: started state syncer +31:54.054 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30099 network=tcp +31:54.088 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.088 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30098 [Candidate]" term=2 +31:54.103 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:54.114 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.114 [DEBUG] TestAgent.server.raft: vote granted: from=376a3752-3096-e38e-f647-25c6a9ada849 term=2 tally=1 +31:54.115 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.115 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30098 [Leader]" +31:54.115 [INFO] TestAgent.server: cluster leadership acquired +31:54.115 [INFO] TestAgent.server: New leader elected: payload=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.132 [INFO] TestAgent.server: initializing acls +31:54.137 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:54.137 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.158 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:54.180 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.180 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:54.180 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:54.180 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.180 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-376a3752-3096-e38e-f647-25c6a9ada849.dc1 +31:54.181 [INFO] TestAgent.server: Updating LAN server: server="Node-376a3752-3096-e38e-f647-25c6a9ada849 (Addr: tcp/127.0.0.1:30098) (DC: dc1)" +31:54.181 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-376a3752-3096-e38e-f647-25c6a9ada849.dc1 area=wan +31:54.247 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.247 [INFO] TestAgent: Synced node info +31:54.247 [DEBUG] TestAgent: Node info in sync +31:54.284 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.284 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.284 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.284 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.284 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.284 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.284 [INFO] TestAgent.server: member joined, marking health alive: member=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.316 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.316 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.508 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.508 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-376a3752-3096-e38e-f647-25c6a9ada849 +31:54.508 [INFO] TestAgent: Requesting shutdown +31:54.508 [INFO] TestAgent.server: shutting down server +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.508 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:54.508 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.508 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:54.508 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.540 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.544Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.544Z [INFO] agent.router.manager: shutting down +31:54.544 [INFO] TestAgent: consul server down +31:54.544 [INFO] TestAgent: shutdown complete +31:54.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30114 network=tcp +31:54.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30114 network=udp +31:54.544 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30094 network=tcp +31:55.048 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.048 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Monitor_ACLDeny (1.17s) +=== CONT TestAgent_NodeMaintenance_Enable +=== RUN TestSnapshot_Options/PUT [WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:17.178Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.187Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.232 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab8bc7e1-4024-0da7-f822-8dde1ac68abb Address:127.0.0.1:11531}]" -46:17.232 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11531 [Follower]" leader= -46:17.232 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb.dc1 127.0.0.1 -46:17.233 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb 127.0.0.1 -2022-11-07T22:46:17.233Z [INFO] agent.router: Initializing LAN area manager -46:17.233 [INFO] TestAgent.server: Adding LAN server: server="Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb (Addr: tcp/127.0.0.1:11531) (DC: dc1)" -46:17.233 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb.dc1 area=wan -46:17.233 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11533 network=udp -46:17.233 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11533 network=tcp -46:17.234 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11527 network=tcp -46:17.234 [INFO] TestAgent: started state syncer -46:17.234 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11514 network=tcp -46:17.276 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.276 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11531 [Candidate]" term=2 -46:17.295 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.295 [DEBUG] TestAgent.server.raft: vote granted: from=ab8bc7e1-4024-0da7-f822-8dde1ac68abb term=2 tally=1 -46:17.295 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11531 [Leader]" -46:17.295 [INFO] TestAgent.server: cluster leadership acquired -46:17.295 [INFO] TestAgent.server: New leader elected: payload=Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb -46:17.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11531 -46:17.342 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.371 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.371 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.371 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.372 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.372 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.372 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb -46:17.372 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab8bc7e1-4024-0da7-f822-8dde1ac68abb -46:17.379 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.657 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.661 [INFO] TestAgent: Synced node info -46:17.661 [DEBUG] TestAgent: Node info in sync -46:17.677 [DEBUG] TestAgent: Node info in sync -46:17.684 [INFO] TestAgent: Synced service: service=monitor -46:17.684 [DEBUG] TestAgent: Check in sync: check=service:monitor -46:17.804 [DEBUG] TestAgent: Node info in sync -46:17.810 [INFO] TestAgent: Synced service: service=monitor -46:17.810 [DEBUG] TestAgent: Check in sync: check=service:monitor -46:17.911 [INFO] TestAgent: Requesting shutdown -46:17.911 [INFO] TestAgent.server: shutting down server -46:17.911 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:17.911 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:17.911 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:17.911 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:17.911 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.911 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.911 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:17.911 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:17.911 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:17.911 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:17.917 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.924Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.924Z [INFO] agent.router.manager: shutting down -46:17.924 [INFO] TestAgent: consul server down -46:17.924 [INFO] TestAgent: shutdown complete -46:17.924 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11533 network=tcp -46:17.924 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11533 network=udp -46:17.924 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11527 network=tcp -46:18.424 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.424 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Monitor (1.36s) +2021-10-05T16:31:53.866Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.866Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.905Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:53.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:53.990 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:36ee82cc-a1ce-c78b-0d38-918d729fc18a Address:127.0.0.1:30070}]" +31:53.991 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 127.0.0.1 +31:53.991 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a 127.0.0.1 +2021-10-05T16:31:53.991Z [INFO] agent.router: Initializing LAN area manager +31:53.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30070 [Follower]" leader= +31:53.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30093 network=udp +31:53.992 [INFO] TestAgent.server: Adding LAN server: server="Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a (Addr: tcp/127.0.0.1:30070) (DC: dc1)" +31:53.992 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 area=wan +31:53.992 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30093 network=tcp +31:53.992 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30066 network=tcp +31:53.993 [INFO] TestAgent: started state syncer +31:53.993 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30113 network=tcp +31:54.034 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.034 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30070 [Candidate]" term=2 +31:54.042 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:54.063 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.063 [DEBUG] TestAgent.server.raft: vote granted: from=36ee82cc-a1ce-c78b-0d38-918d729fc18a term=2 tally=1 +31:54.063 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.063 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30070 [Leader]" +31:54.064 [INFO] TestAgent.server: cluster leadership acquired +31:54.064 [INFO] TestAgent.server: New leader elected: payload=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.088 [INFO] TestAgent.server: initializing acls +31:54.095 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:54.095 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.120 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:54.143 [INFO] TestAgent.server: initializing acls +31:54.143 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.152 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.152 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:54.152 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:54.152 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.152 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 +31:54.152 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 area=wan +31:54.153 [INFO] TestAgent.server: Updating LAN server: server="Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a (Addr: tcp/127.0.0.1:30070) (DC: dc1)" +31:54.185 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.185 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:54.185 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.185 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 +31:54.185 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a.dc1 area=wan +31:54.185 [INFO] TestAgent.server: Updating LAN server: server="Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a (Addr: tcp/127.0.0.1:30070) (DC: dc1)" +31:54.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.284 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.284 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.284 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.284 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.284 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.284 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.284 [INFO] TestAgent.server: member joined, marking health alive: member=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.301 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.301 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.316 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.321 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-36ee82cc-a1ce-c78b-0d38-918d729fc18a +31:54.335 [INFO] TestAgent: Updated agent's ACL token: token=acl_agent_token +31:54.491 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +31:54.525 [INFO] TestAgent: Synced node info +31:54.545 [INFO] TestAgent: Requesting shutdown +31:54.545 [INFO] TestAgent.server: shutting down server +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:54.545 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.545 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.545 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:54.545 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:54.568 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.572Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.572Z [INFO] agent.router.manager: shutting down +31:54.573 [INFO] TestAgent: consul server down +31:54.573 [INFO] TestAgent: shutdown complete +31:54.573 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30093 network=tcp +31:54.573 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30093 network=udp +31:54.573 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30066 network=tcp +31:55.073 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.073 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_TokenTriggersFullSync/agent +2021-10-05T16:31:54.005Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:54.005Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:54.029Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:54.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:54.110 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 Address:127.0.0.1:30035}]" +31:54.111 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30035 [Follower]" leader= +31:54.111 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 127.0.0.1 +31:54.112 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 127.0.0.1 +2021-10-05T16:31:54.112Z [INFO] agent.router: Initializing LAN area manager +31:54.112 [INFO] TestAgent.server: Adding LAN server: server="Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 (Addr: tcp/127.0.0.1:30035) (DC: dc1)" +31:54.112 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 area=wan +31:54.112 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30058 network=tcp +31:54.112 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30058 network=udp +31:54.113 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30031 network=tcp +31:54.113 [INFO] TestAgent: started state syncer +31:54.113 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30137 network=tcp +31:54.163 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:54.171 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.171 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30035 [Candidate]" term=2 +31:54.202 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.202 [DEBUG] TestAgent.server.raft: vote granted: from=3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 term=2 tally=1 +31:54.202 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.202 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30035 [Leader]" +31:54.202 [INFO] TestAgent.server: cluster leadership acquired +31:54.202 [INFO] TestAgent.server: New leader elected: payload=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.228 [INFO] TestAgent.server: initializing acls +31:54.242 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:54.242 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.256 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:54.264 [INFO] TestAgent.server: initializing acls +31:54.264 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:54.282 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.282 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:54.282 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:54.283 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.283 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 +31:54.283 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 area=wan +31:54.283 [INFO] TestAgent.server: Updating LAN server: server="Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 (Addr: tcp/127.0.0.1:30035) (DC: dc1)" +31:54.312 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:54.312 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:54.313 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.313 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 +31:54.313 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1.dc1 area=wan +31:54.313 [INFO] TestAgent.server: Updating LAN server: server="Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 (Addr: tcp/127.0.0.1:30035) (DC: dc1)" +31:54.348 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.380 [INFO] TestAgent: Synced node info +31:54.474 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.475 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.487 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.487 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.487 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.487 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.487 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.487 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.487 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.509 [INFO] TestAgent: Node entered maintenance mode +31:54.509 [DEBUG] TestAgent: Node info in sync +31:54.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e0cab74-004d-c4f1-9e1c-5a162c6dc3c1 +31:54.552 [INFO] TestAgent: Synced check: check=_node_maintenance +31:54.552 [INFO] TestAgent: Requesting shutdown +31:54.552 [INFO] TestAgent.server: shutting down server +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.552 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.552 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.552 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:54.552 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:54.552 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.552 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.552 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.570 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.572 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.576Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.576Z [INFO] agent.router.manager: shutting down +31:54.576 [INFO] TestAgent: consul server down +31:54.576 [INFO] TestAgent: shutdown complete +31:54.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30058 network=tcp +31:54.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30058 network=udp +31:54.576 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30031 network=tcp +31:55.076 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.076 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_NodeMaintenance_ACLDeny (1.11s) + --- PASS: TestAgent_NodeMaintenance_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_NodeMaintenance_ACLDeny/root_token (0.08s) +=== CONT TestAgent_NodeMaintenance_BadRequest +2021-10-05T16:31:53.969Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:54.009Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:54.084 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a336161f-64d3-0ef7-9431-3fd30db145d0 Address:127.0.0.1:30014}]" +31:54.084 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30014 [Follower]" leader= +31:54.085 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a336161f-64d3-0ef7-9431-3fd30db145d0.dc1 127.0.0.1 +31:54.086 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a336161f-64d3-0ef7-9431-3fd30db145d0 127.0.0.1 +2021-10-05T16:31:54.086Z [INFO] agent.router: Initializing LAN area manager +31:54.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30072 network=udp +31:54.087 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a336161f-64d3-0ef7-9431-3fd30db145d0.dc1 area=wan +31:54.087 [INFO] TestAgent.server: Adding LAN server: server="Node-a336161f-64d3-0ef7-9431-3fd30db145d0 (Addr: tcp/127.0.0.1:30014) (DC: dc1)" +31:54.087 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30072 network=tcp +31:54.088 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30010 network=tcp +31:54.088 [INFO] TestAgent: started state syncer +31:54.088 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30043 network=tcp +31:54.153 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.153 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30014 [Candidate]" term=2 +31:54.187 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.187 [DEBUG] TestAgent.server.raft: vote granted: from=a336161f-64d3-0ef7-9431-3fd30db145d0 term=2 tally=1 +31:54.187 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.187 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30014 [Leader]" +31:54.188 [INFO] TestAgent.server: cluster leadership acquired +31:54.188 [INFO] TestAgent.server: New leader elected: payload=Node-a336161f-64d3-0ef7-9431-3fd30db145d0 +31:54.204 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30014 +31:54.241 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.316 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.316 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.316 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.316 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.316 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.316 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a336161f-64d3-0ef7-9431-3fd30db145d0 +31:54.316 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a336161f-64d3-0ef7-9431-3fd30db145d0 +31:54.339 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.472 [INFO] TestAgent: Synced node info +31:54.532 [INFO] TestAgent: Synced service: service=memcache +31:54.532 [DEBUG] TestAgent: Check in sync: check=service:memcache +31:54.532 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:54.541 [INFO] TestAgent: Synced node info +31:54.549 [INFO] TestAgent: Synced service: service=memcache +31:54.549 [DEBUG] TestAgent: Check in sync: check=service:memcache +31:54.549 [DEBUG] TestAgent: Node info in sync +31:54.549 [DEBUG] TestAgent: Service in sync: service=memcache +31:54.549 [DEBUG] TestAgent: Check in sync: check=service:memcache +31:54.616 [DEBUG] TestAgent: Node info in sync +31:54.616 [DEBUG] TestAgent: Service in sync: service=memcache +31:54.616 [DEBUG] TestAgent: Check in sync: check=service:memcache +31:54.653 [INFO] TestAgent: Synced check: check=memcache_check2 +31:54.653 [INFO] TestAgent: Requesting shutdown +31:54.653 [INFO] TestAgent.server: shutting down server +31:54.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.653 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.653 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.654 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.654 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.654 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.654 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.657 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.660Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.660Z [INFO] agent.router.manager: shutting down +31:54.668 [INFO] TestAgent: consul server down +31:54.668 [INFO] TestAgent: shutdown complete +31:54.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30072 network=tcp +31:54.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30072 network=udp +31:54.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30010 network=tcp +31:55.170 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.178 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheck_Service (1.23s) +=== CONT TestAgent_ServiceMaintenance_ACLDeny +2021-10-05T16:31:53.941Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:53.955Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:54.027 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:557d98be-24fe-7243-1db2-8fa8fd1a25fb Address:127.0.0.1:30126}]" +31:54.028 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb.dc1 127.0.0.1 +31:54.028 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30126 [Follower]" leader= +31:54.031 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb 127.0.0.1 +2021-10-05T16:31:54.031Z [INFO] agent.router: Initializing LAN area manager +31:54.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30107 network=udp +31:54.031 [INFO] TestAgent.server: Adding LAN server: server="Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb (Addr: tcp/127.0.0.1:30126) (DC: dc1)" +31:54.031 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb.dc1 area=wan +31:54.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30107 network=tcp +31:54.032 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30122 network=tcp +31:54.032 [INFO] TestAgent: started state syncer +31:54.032 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30071 network=tcp +31:54.076 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:54.076 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30126 [Candidate]" term=2 +31:54.104 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:54.104 [DEBUG] TestAgent.server.raft: vote granted: from=557d98be-24fe-7243-1db2-8fa8fd1a25fb term=2 tally=1 +31:54.104 [INFO] TestAgent.server.raft: election won: tally=1 +31:54.104 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30126 [Leader]" +31:54.105 [INFO] TestAgent.server: cluster leadership acquired +31:54.106 [INFO] TestAgent.server: New leader elected: payload=Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb +31:54.118 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30126 +31:54.144 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:54.204 [INFO] TestAgent: Synced node info +31:54.204 [DEBUG] TestAgent: Node info in sync +31:54.215 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:54.215 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:54.215 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:54.215 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:54.215 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:54.216 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb +31:54.216 [INFO] TestAgent.server: member joined, marking health alive: member=Node-557d98be-24fe-7243-1db2-8fa8fd1a25fb +31:54.232 [INFO] TestAgent.server: federation state anti-entropy synced +31:54.257 [DEBUG] TestAgent: Node info in sync +31:54.264 [INFO] TestAgent: Synced service: service=monitor +31:54.264 [DEBUG] TestAgent: Check in sync: check=service:monitor +31:54.443 [DEBUG] TestAgent: Node info in sync +31:54.469 [INFO] TestAgent: Synced service: service=monitor +31:54.469 [DEBUG] TestAgent: Check in sync: check=service:monitor +31:54.570 [INFO] TestAgent: Requesting shutdown +31:54.570 [INFO] TestAgent.server: shutting down server +31:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.570 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:54.570 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:54.588 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:54.590Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:54.590Z [INFO] agent.router.manager: shutting down +31:54.590 [INFO] TestAgent: consul server down +31:54.590 [INFO] TestAgent: shutdown complete +31:54.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30107 network=tcp +31:54.590 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30107 network=udp +31:54.590 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30122 network=tcp +31:55.092 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.092 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Monitor (1.28s) --- PASS: TestAgent_Monitor/unknown_log_level (0.00s) --- PASS: TestAgent_Monitor/stream_unstructured_logs (0.12s) - --- PASS: TestAgent_Monitor/stream_JSON_logs (0.13s) - --- PASS: TestAgent_Monitor/serf_shutdown_logging (0.61s) -=== CONT TestAgentConnectAuthorize_idNotService -2022-11-07T22:46:17.763Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:17.763Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.778Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:17.778Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.823 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:109cc40b-765c-0731-cfe9-8e669e756d20 Address:127.0.0.1:11566}]" -46:17.823 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11566 [Follower]" leader= -46:17.824 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-109cc40b-765c-0731-cfe9-8e669e756d20.dc1 127.0.0.1 -46:17.824 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-109cc40b-765c-0731-cfe9-8e669e756d20 127.0.0.1 -2022-11-07T22:46:17.824Z [INFO] agent.router: Initializing LAN area manager -46:17.824 [INFO] TestAgent.server: Adding LAN server: server="Node-109cc40b-765c-0731-cfe9-8e669e756d20 (Addr: tcp/127.0.0.1:11566) (DC: dc1)" -46:17.824 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-109cc40b-765c-0731-cfe9-8e669e756d20.dc1 area=wan -46:17.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11589 network=tcp -46:17.825 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11589 network=udp -46:17.825 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11562 network=tcp -46:17.825 [INFO] TestAgent: started state syncer -46:17.825 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11609 network=tcp -46:17.875 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:17.884 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.884 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11566 [Candidate]" term=2 -46:17.896 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.896 [DEBUG] TestAgent.server.raft: vote granted: from=109cc40b-765c-0731-cfe9-8e669e756d20 term=2 tally=1 -46:17.896 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.896 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11566 [Leader]" -46:17.896 [INFO] TestAgent.server: cluster leadership acquired -46:17.896 [INFO] TestAgent.server: New leader elected: payload=Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:17.909 [INFO] TestAgent.server: initializing acls -46:17.914 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:17.914 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:17.923 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:17.935 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:17.935 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:17.935 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:17.935 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:17.935 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-109cc40b-765c-0731-cfe9-8e669e756d20.dc1 -46:17.935 [INFO] TestAgent.server: Updating LAN server: server="Node-109cc40b-765c-0731-cfe9-8e669e756d20 (Addr: tcp/127.0.0.1:11566) (DC: dc1)" -46:17.935 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-109cc40b-765c-0731-cfe9-8e669e756d20.dc1 area=wan -46:17.946 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.969 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.969 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.969 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.969 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.969 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.969 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:17.969 [INFO] TestAgent.server: member joined, marking health alive: member=Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:17.974 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.984 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:18.001 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-109cc40b-765c-0731-cfe9-8e669e756d20 -46:18.005 [INFO] TestAgent: Node entered maintenance mode -46:18.016 [INFO] TestAgent: Synced node info -46:18.020 [INFO] TestAgent: Synced check: check=_node_maintenance -46:18.020 [INFO] TestAgent: Requesting shutdown -46:18.020 [INFO] TestAgent.server: shutting down server -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.020 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.020 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.020 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.020 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.020 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.025 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.036Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.036Z [INFO] agent.router.manager: shutting down -46:18.036 [INFO] TestAgent: consul server down -46:18.036 [INFO] TestAgent: shutdown complete -46:18.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11589 network=tcp -46:18.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11589 network=udp -46:18.036 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11562 network=tcp -46:18.536 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.536 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_NodeMaintenance_ACLDeny (0.79s) - --- PASS: TestAgent_NodeMaintenance_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_NodeMaintenance_ACLDeny/root_token (0.02s) -=== CONT TestAgent_loadCheckState -2022-11-07T22:46:17.167Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.176Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.213 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:324cb218-96af-f308-d032-9c2a05a3abfa Address:127.0.0.1:11490}]" -46:17.213 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11490 [Follower]" leader= -46:17.213 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-324cb218-96af-f308-d032-9c2a05a3abfa.dc1 127.0.0.1 -46:17.214 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-324cb218-96af-f308-d032-9c2a05a3abfa 127.0.0.1 -2022-11-07T22:46:17.214Z [INFO] agent.router: Initializing LAN area manager -46:17.214 [INFO] TestAgent.server: Adding LAN server: server="Node-324cb218-96af-f308-d032-9c2a05a3abfa (Addr: tcp/127.0.0.1:11490) (DC: dc1)" -46:17.214 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-324cb218-96af-f308-d032-9c2a05a3abfa.dc1 area=wan -46:17.214 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11429 network=udp -46:17.214 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11429 network=tcp -46:17.214 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11486 network=tcp -46:17.214 [INFO] TestAgent: started state syncer -46:17.214 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11525 network=tcp -46:17.260 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.260 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11490 [Candidate]" term=2 -46:17.279 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.279 [DEBUG] TestAgent.server.raft: vote granted: from=324cb218-96af-f308-d032-9c2a05a3abfa term=2 tally=1 -46:17.279 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.279 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11490 [Leader]" -46:17.279 [INFO] TestAgent.server: cluster leadership acquired -46:17.279 [INFO] TestAgent.server: New leader elected: payload=Node-324cb218-96af-f308-d032-9c2a05a3abfa -46:17.294 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11490 -46:17.319 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.353 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.353 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.353 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.353 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.353 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.353 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-324cb218-96af-f308-d032-9c2a05a3abfa -46:17.353 [INFO] TestAgent.server: member joined, marking health alive: member=Node-324cb218-96af-f308-d032-9c2a05a3abfa -46:17.371 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.477 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.486 [INFO] TestAgent: Synced node info -46:17.486 [DEBUG] TestAgent: Node info in sync -46:17.509 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.509 [DEBUG] TestAgent: Node info in sync -46:17.515 [INFO] TestAgent: Synced service: service=redis -46:17.521 [INFO] TestAgent: Synced check: check=node-check -46:17.521 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.021 [DEBUG] TestAgent: Registered node: node="Node-level check" -46:18.021 [DEBUG] TestAgent: Registered node: node="Serf Health Status" -46:18.021 [DEBUG] TestAgent: Registered node: node="Service-level check" -46:18.021 [INFO] TestAgent: Sync in progress: iteration=1 -46:18.021 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.021 [DEBUG] TestAgent: Node info in sync -46:18.021 [DEBUG] TestAgent: Service in sync: service=redis -46:18.021 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.021 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.021 [INFO] TestAgent: Sync in progress: iteration=2 -46:18.021 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.021 [DEBUG] TestAgent: Node info in sync -46:18.021 [DEBUG] TestAgent: Service in sync: service=redis -46:18.021 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.021 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.021 [INFO] TestAgent: Sync in progress: iteration=3 -46:18.021 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.021 [DEBUG] TestAgent: Node info in sync -46:18.021 [DEBUG] TestAgent: Service in sync: service=redis -46:18.021 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.021 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.021 [INFO] TestAgent: Sync in progress: iteration=4 -46:18.021 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.021 [DEBUG] TestAgent: Node info in sync -46:18.021 [DEBUG] TestAgent: Service in sync: service=redis -46:18.021 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.021 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.021 [INFO] TestAgent: Sync in progress: iteration=5 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Sync in progress: iteration=6 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Sync in progress: iteration=7 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Sync in progress: iteration=8 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Sync in progress: iteration=9 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Sync in progress: iteration=10 -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.022 [DEBUG] TestAgent: Node info in sync -46:18.022 [DEBUG] TestAgent: Service in sync: service=redis -46:18.022 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.022 [DEBUG] TestAgent: Check in sync: check=node-check -46:18.022 [INFO] TestAgent: Requesting shutdown -46:18.022 [INFO] TestAgent.server: shutting down server -46:18.022 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.022 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.022 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.022 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.022 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.022 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.022 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.022 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.022 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.022 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.030 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.039Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.039Z [INFO] agent.router.manager: shutting down -46:18.039 [INFO] TestAgent: consul server down -46:18.039 [INFO] TestAgent: shutdown complete -46:18.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11429 network=tcp -46:18.039 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11429 network=udp -46:18.039 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11486 network=tcp -46:18.540 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.540 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_IndexChurn/with_tags -2022-11-07T22:46:17.636Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.644Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.668 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:133ca85e-ac4d-e396-ddce-da1a3ba07d34 Address:127.0.0.1:11552}]" -46:17.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11552 [Follower]" leader= -46:17.669 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34.dc1 127.0.0.1 -46:17.669 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34 127.0.0.1 -2022-11-07T22:46:17.669Z [INFO] agent.router: Initializing LAN area manager -46:17.669 [INFO] TestAgent.server: Adding LAN server: server="Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34 (Addr: tcp/127.0.0.1:11552) (DC: dc1)" -46:17.669 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34.dc1 area=wan -46:17.669 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11540 network=udp -46:17.669 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11540 network=tcp -46:17.670 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11548 network=tcp -46:17.670 [INFO] TestAgent: started state syncer -46:17.670 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11400 network=tcp -46:17.724 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.724 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11552 [Candidate]" term=2 -46:17.734 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.734 [DEBUG] TestAgent.server.raft: vote granted: from=133ca85e-ac4d-e396-ddce-da1a3ba07d34 term=2 tally=1 -46:17.734 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.734 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11552 [Leader]" -46:17.734 [INFO] TestAgent.server: cluster leadership acquired -46:17.734 [INFO] TestAgent.server: New leader elected: payload=Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34 -46:17.745 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11552 -46:17.759 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.788 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.788 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.788 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.788 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.788 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.788 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34 -46:17.789 [INFO] TestAgent.server: member joined, marking health alive: member=Node-133ca85e-ac4d-e396-ddce-da1a3ba07d34 -46:17.811 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.051 [INFO] TestAgent: Requesting shutdown -46:18.051 [INFO] TestAgent.server: shutting down server -46:18.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.051 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.051 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.051 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.051 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.051 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.051 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.051 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.051 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.061 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.068Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.068Z [INFO] agent.router.manager: shutting down -46:18.068 [INFO] TestAgent: consul server down -46:18.068 [INFO] TestAgent: shutdown complete -46:18.068 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11540 network=tcp -46:18.068 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11540 network=udp -46:18.068 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11548 network=tcp -46:18.569 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.569 [INFO] TestAgent: Endpoints down -=== CONT TestAgent_StartStop ---- PASS: TestAgent_TokenStore (0.95s) -2022-11-07T22:46:17.868Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.878Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.917 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 Address:127.0.0.1:11643}]" -46:17.917 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11643 [Follower]" leader= -46:17.918 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2.dc1 127.0.0.1 -46:17.918 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 127.0.0.1 -2022-11-07T22:46:17.918Z [INFO] agent.router: Initializing LAN area manager -46:17.918 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11561 network=udp -46:17.918 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2.dc1 area=wan -46:17.918 [INFO] TestAgent.server: Adding LAN server: server="Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 (Addr: tcp/127.0.0.1:11643) (DC: dc1)" -46:17.918 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11561 network=tcp -46:17.919 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11639 network=tcp -46:17.919 [INFO] TestAgent: started state syncer -46:17.919 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11567 network=tcp -46:17.980 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.980 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11643 [Candidate]" term=2 -46:17.990 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.990 [DEBUG] TestAgent.server.raft: vote granted: from=1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 term=2 tally=1 -46:17.990 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.990 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11643 [Leader]" -46:17.990 [INFO] TestAgent.server: cluster leadership acquired -46:17.990 [INFO] TestAgent.server: New leader elected: payload=Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 -46:17.999 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11643 -46:18.020 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.049 [INFO] TestAgent: Requesting shutdown -46:18.049 [INFO] TestAgent.server: shutting down server -46:18.049 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.049 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.057 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:18.060 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.060 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.060 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.060 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.060 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.060 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 -46:18.060 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 -2022-11-07T22:46:18.065Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.065Z [INFO] agent.router.manager: shutting down -46:18.065 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: raft is already shutdown" -46:18.071 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 127.0.0.1 11641 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:1fbe3f2a-9e69-9798-3f4c-a9939a29a5e2 port:11643 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11642] alive 1 5 2 2 5 4}" error="leadership lost while committing log" -46:18.071 [INFO] TestAgent: consul server down -46:18.071 [INFO] TestAgent: shutdown complete -46:18.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.071 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.071 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.071 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11561 network=tcp -46:18.071 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.071 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.071 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11561 network=udp -46:18.071 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11639 network=tcp -46:18.571 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.571 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:17.202Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.214Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.256 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ea2e3954-576e-e664-0522-b2cc048c6d48 Address:127.0.0.1:11434}]" -46:17.256 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11434 [Follower]" leader= -46:17.256 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ea2e3954-576e-e664-0522-b2cc048c6d48.dc1 127.0.0.1 -46:17.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ea2e3954-576e-e664-0522-b2cc048c6d48 127.0.0.1 -2022-11-07T22:46:17.257Z [INFO] agent.router: Initializing LAN area manager -46:17.257 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ea2e3954-576e-e664-0522-b2cc048c6d48.dc1 area=wan -46:17.257 [INFO] TestAgent.server: Adding LAN server: server="Node-ea2e3954-576e-e664-0522-b2cc048c6d48 (Addr: tcp/127.0.0.1:11434) (DC: dc1)" -46:17.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11443 network=udp -46:17.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11443 network=tcp -46:17.257 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11430 network=tcp -46:17.257 [INFO] TestAgent: started state syncer -46:17.257 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11393 network=tcp -46:17.301 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.301 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11434 [Candidate]" term=2 -46:17.318 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.318 [DEBUG] TestAgent.server.raft: vote granted: from=ea2e3954-576e-e664-0522-b2cc048c6d48 term=2 tally=1 -46:17.318 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.318 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11434 [Leader]" -46:17.318 [INFO] TestAgent.server: cluster leadership acquired -46:17.318 [INFO] TestAgent.server: New leader elected: payload=Node-ea2e3954-576e-e664-0522-b2cc048c6d48 -46:17.333 [INFO] TestAgent: Synced node info -46:17.333 [DEBUG] TestAgent: Node info in sync -46:17.333 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11434 -46:17.335 [INFO] TestAgent: Requesting shutdown -46:17.335 [INFO] TestAgent.server: shutting down server -46:17.335 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:17.342 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.350Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.350Z [INFO] agent.router.manager: shutting down -46:17.358 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" -46:17.358 [INFO] TestAgent: consul server down -46:17.358 [INFO] TestAgent: shutdown complete -46:17.358 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11443 network=tcp -46:17.358 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11443 network=udp -46:17.358 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11430 network=tcp -46:17.858 [INFO] TestAgent: Waiting for endpoints to shut down -46:17.858 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ReconnectConfigSettings (1.38s) -=== CONT TestAgentConnectAuthorize_idInvalidFormat + --- PASS: TestAgent_Monitor/stream_JSON_logs (0.20s) + --- PASS: TestAgent_Monitor/serf_shutdown_logging (0.63s) +=== CONT TestAgent_ServiceMaintenance_Disable +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:43.472Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:43.507Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:43.600 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:11cb0771-3341-f728-53b2-2b5965a5ba00 Address:127.0.0.1:29434}]" +31:43.601 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-11cb0771-3341-f728-53b2-2b5965a5ba00.dc1 127.0.0.1 +31:43.601 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29434 [Follower]" leader= +31:43.601 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-11cb0771-3341-f728-53b2-2b5965a5ba00 127.0.0.1 +2021-10-05T16:31:43.601Z [INFO] agent.router: Initializing LAN area manager +31:43.602 [INFO] TestAgent.server: Adding LAN server: server="Node-11cb0771-3341-f728-53b2-2b5965a5ba00 (Addr: tcp/127.0.0.1:29434) (DC: dc1)" +31:43.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29443 network=udp +31:43.602 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-11cb0771-3341-f728-53b2-2b5965a5ba00.dc1 area=wan +31:43.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29443 network=tcp +31:43.602 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29430 network=tcp +31:43.602 [INFO] TestAgent: started state syncer +31:43.602 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29505 network=tcp +31:43.653 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:43.653 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29434 [Candidate]" term=2 +31:43.764 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:43.764 [DEBUG] TestAgent.server.raft: vote granted: from=11cb0771-3341-f728-53b2-2b5965a5ba00 term=2 tally=1 +31:43.764 [INFO] TestAgent.server.raft: election won: tally=1 +31:43.764 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29434 [Leader]" +31:43.767 [INFO] TestAgent.server: cluster leadership acquired +31:43.767 [INFO] TestAgent.server: New leader elected: payload=Node-11cb0771-3341-f728-53b2-2b5965a5ba00 +31:43.815 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29434 +31:43.860 [INFO] TestAgent.server: server starting leave +31:43.861 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-11cb0771-3341-f728-53b2-2b5965a5ba00.dc1 127.0.0.1 +31:43.861 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-11cb0771-3341-f728-53b2-2b5965a5ba00.dc1 area=wan +2021-10-05T16:31:43.861Z [INFO] agent.router.manager: shutting down +31:43.868 [INFO] TestAgent: Synced node info +31:43.868 [DEBUG] TestAgent: Node info in sync +31:43.902 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:43.943 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:43.943 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:43.943 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:43.943 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:43.943 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:43.943 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-11cb0771-3341-f728-53b2-2b5965a5ba00 +31:43.943 [INFO] TestAgent.server: member joined, marking health alive: member=Node-11cb0771-3341-f728-53b2-2b5965a5ba00 +31:43.973 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.166 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.166 [DEBUG] TestAgent: Node info in sync +31:46.861 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-11cb0771-3341-f728-53b2-2b5965a5ba00 127.0.0.1 +31:46.861 [INFO] TestAgent.server: Removing LAN server: server="Node-11cb0771-3341-f728-53b2-2b5965a5ba00 (Addr: tcp/127.0.0.1:29434) (DC: dc1)" +31:46.861 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-11cb0771-3341-f728-53b2-2b5965a5ba00 +31:47.872 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +31:49.862 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s +31:49.871 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +31:51.872 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +31:53.869 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" +31:53.869 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +31:54.864 [INFO] TestAgent: Requesting shutdown +31:54.864 [INFO] TestAgent.server: shutting down server +31:54.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:54.864 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:54.864 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:54.864 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:54.864 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:54.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:54.864 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:54.864 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:54.864 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +2021-10-05T16:31:54.880Z [INFO] agent.router.manager: shutting down +31:54.880 [INFO] TestAgent: consul server down +31:54.880 [INFO] TestAgent: shutdown complete +31:54.880 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29443 network=tcp +31:54.880 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29443 network=udp +31:54.880 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29430 network=tcp +31:55.384 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.384 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_StartStop (11.95s) +=== CONT TestAgent_ServiceMaintenance_Enable +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_ServiceMaintenance_ACLDeny/no_token +=== RUN TestAgent_ServiceMaintenance_ACLDeny/root_token +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:54.901Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:54.921Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:54.955 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c2032ecb-35c5-c54c-88ab-da78f354cbc8 Address:127.0.0.1:30143}]" +31:54.955 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8.dc1 127.0.0.1 +31:54.956 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8 127.0.0.1 +2021-10-05T16:31:54.956Z [INFO] agent.router: Initializing LAN area manager +31:54.956 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30138 network=udp +31:54.956 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30143 [Follower]" leader= +31:54.957 [INFO] TestAgent.server: Adding LAN server: server="Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8 (Addr: tcp/127.0.0.1:30143) (DC: dc1)" +31:54.957 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8.dc1 area=wan +31:54.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30138 network=tcp +31:54.957 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30139 network=tcp +31:54.957 [INFO] TestAgent: started state syncer +31:54.958 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30074 network=tcp +31:55.028 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.028 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30143 [Candidate]" term=2 +31:55.072 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.072 [DEBUG] TestAgent.server.raft: vote granted: from=c2032ecb-35c5-c54c-88ab-da78f354cbc8 term=2 tally=1 +31:55.072 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.072 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30143 [Leader]" +31:55.073 [INFO] TestAgent.server: cluster leadership acquired +31:55.073 [INFO] TestAgent.server: New leader elected: payload=Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8 +31:55.100 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30143 +31:55.168 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.236 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.236 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.236 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.236 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.236 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.236 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8 +31:55.236 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c2032ecb-35c5-c54c-88ab-da78f354cbc8 +31:55.260 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.292 [INFO] TestAgent: Synced node info +31:55.292 [DEBUG] TestAgent: Node info in sync +31:55.324 [INFO] TestAgent: Node entered maintenance mode +31:55.324 [DEBUG] TestAgent: removed check: check=_node_maintenance +31:55.324 [INFO] TestAgent: Node left maintenance mode +31:55.324 [DEBUG] TestAgent: Node info in sync +31:55.340 [INFO] TestAgent: Deregistered check: check=_node_maintenance +31:55.340 [INFO] TestAgent: Requesting shutdown +31:55.340 [INFO] TestAgent.server: shutting down server +31:55.340 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.340 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.340 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.340 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.340 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.340 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.348 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.352Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.352Z [INFO] agent.router.manager: shutting down +31:55.352 [INFO] TestAgent: consul server down +31:55.352 [INFO] TestAgent: shutdown complete +31:55.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30138 network=tcp +31:55.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30138 network=udp +31:55.352 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30139 network=tcp +31:55.853 [INFO] TestAgent: Waiting for endpoints to shut down +31:55.853 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_NodeMaintenance_Disable (0.98s) +=== CONT TestAgent_ServiceMaintenance_BadRequest +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:55.109Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.109Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.199Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.199Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.300 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:587b2b40-4989-5a78-0763-30d873ddfc29 Address:127.0.0.1:30150}]" +31:55.302 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30150 [Follower]" leader= +31:55.304 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 127.0.0.1 +31:55.305 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-587b2b40-4989-5a78-0763-30d873ddfc29 127.0.0.1 +2021-10-05T16:31:55.305Z [INFO] agent.router: Initializing LAN area manager +31:55.305 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30145 network=udp +31:55.305 [INFO] TestAgent.server: Adding LAN server: server="Node-587b2b40-4989-5a78-0763-30d873ddfc29 (Addr: tcp/127.0.0.1:30150) (DC: dc1)" +31:55.306 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 area=wan +31:55.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30145 network=tcp +31:55.306 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30146 network=tcp +31:55.306 [INFO] TestAgent: started state syncer +31:55.306 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30151 network=tcp +31:55.356 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:55.371 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.371 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30150 [Candidate]" term=2 +31:55.400 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.400 [DEBUG] TestAgent.server.raft: vote granted: from=587b2b40-4989-5a78-0763-30d873ddfc29 term=2 tally=1 +31:55.400 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.400 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30150 [Leader]" +31:55.400 [INFO] TestAgent.server: cluster leadership acquired +31:55.400 [INFO] TestAgent.server: New leader elected: payload=Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.429 [INFO] TestAgent.server: initializing acls +31:55.434 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:55.434 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.445 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:55.456 [INFO] TestAgent.server: initializing acls +31:55.456 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.490 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.490 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:55.490 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:55.491 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.491 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 +31:55.491 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 area=wan +31:55.491 [INFO] TestAgent.server: Updating LAN server: server="Node-587b2b40-4989-5a78-0763-30d873ddfc29 (Addr: tcp/127.0.0.1:30150) (DC: dc1)" +31:55.504 [INFO] TestAgent: Synced node info +31:55.504 [DEBUG] TestAgent: Node info in sync +31:55.504 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.504 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:55.504 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.504 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 +31:55.504 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-587b2b40-4989-5a78-0763-30d873ddfc29.dc1 area=wan +31:55.504 [INFO] TestAgent.server: Updating LAN server: server="Node-587b2b40-4989-5a78-0763-30d873ddfc29 (Addr: tcp/127.0.0.1:30150) (DC: dc1)" +31:55.535 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.587 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.587 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.587 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.587 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.587 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.588 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.588 [INFO] TestAgent.server: member joined, marking health alive: member=Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.591 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-587b2b40-4989-5a78-0763-30d873ddfc29 +31:55.591 [INFO] TestAgent: Requesting shutdown +31:55.591 [INFO] TestAgent.server: shutting down server +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.592 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.592 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.592 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.592 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:55.592 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:55.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.592 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.597 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:55.601 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.601 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:31:55.602Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.602Z [INFO] agent.router.manager: shutting down +31:55.603 [INFO] TestAgent: consul server down +31:55.603 [INFO] TestAgent: shutdown complete +31:55.603 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30145 network=tcp +31:55.603 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30145 network=udp +31:55.603 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30146 network=tcp +31:56.103 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.103 [INFO] TestAgent: Endpoints down +=== RUN TestSnapshot_Options/PUT#01 +=== RUN TestAgent_ServiceMaintenance_BadRequest/not_enabled +=== RUN TestAgent_ServiceMaintenance_BadRequest/no_service_id +=== RUN TestAgent_ServiceMaintenance_BadRequest/bad_service_id +2021-10-05T16:31:55.075Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.155Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.316 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c66fff29-44c4-b90d-9ec3-87302adc505a Address:127.0.0.1:30136}]" +31:55.317 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c66fff29-44c4-b90d-9ec3-87302adc505a.dc1 127.0.0.1 +31:55.317 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c66fff29-44c4-b90d-9ec3-87302adc505a 127.0.0.1 +2021-10-05T16:31:55.317Z [INFO] agent.router: Initializing LAN area manager +31:55.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30075 network=udp +31:55.318 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30136 [Follower]" leader= +31:55.318 [INFO] TestAgent.server: Adding LAN server: server="Node-c66fff29-44c4-b90d-9ec3-87302adc505a (Addr: tcp/127.0.0.1:30136) (DC: dc1)" +31:55.318 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c66fff29-44c4-b90d-9ec3-87302adc505a.dc1 area=wan +31:55.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30075 network=tcp +31:55.319 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30076 network=tcp +31:55.319 [INFO] TestAgent: started state syncer +31:55.319 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30144 network=tcp +31:55.360 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.360 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30136 [Candidate]" term=2 +31:55.424 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.424 [DEBUG] TestAgent.server.raft: vote granted: from=c66fff29-44c4-b90d-9ec3-87302adc505a term=2 tally=1 +31:55.424 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.424 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30136 [Leader]" +31:55.424 [INFO] TestAgent.server: cluster leadership acquired +31:55.424 [INFO] TestAgent.server: New leader elected: payload=Node-c66fff29-44c4-b90d-9ec3-87302adc505a +31:55.460 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30136 +31:55.488 [INFO] TestAgent: Synced node info +31:55.532 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.600 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.600 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.600 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.600 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.600 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.600 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c66fff29-44c4-b90d-9ec3-87302adc505a +31:55.600 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c66fff29-44c4-b90d-9ec3-87302adc505a +31:55.636 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.660 [INFO] TestAgent: Node entered maintenance mode +31:55.660 [DEBUG] TestAgent: Node info in sync +31:55.666 [INFO] TestAgent: Synced check: check=_node_maintenance +31:55.666 [INFO] TestAgent: Requesting shutdown +31:55.666 [INFO] TestAgent.server: shutting down server +31:55.666 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.667 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.667 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.667 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.667 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.667 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.667 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.667 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.667 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.667 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.675 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.678Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.678Z [INFO] agent.router.manager: shutting down +31:55.678 [INFO] TestAgent: consul server down +31:55.678 [INFO] TestAgent: shutdown complete +31:55.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30075 network=tcp +31:55.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30075 network=udp +31:55.678 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30076 network=tcp +31:56.179 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.179 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_NodeMaintenance_Enable (1.13s) +=== CONT TestAgent_DeregisterService_ACLDeny +2021-10-05T16:31:55.202Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.253Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.337 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:afab7003-dd57-95a9-e38f-979a8f3e961c Address:127.0.0.1:30105}]" +31:55.338 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-afab7003-dd57-95a9-e38f-979a8f3e961c.dc1 127.0.0.1 +31:55.339 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-afab7003-dd57-95a9-e38f-979a8f3e961c 127.0.0.1 +2021-10-05T16:31:55.339Z [INFO] agent.router: Initializing LAN area manager +31:55.339 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30105 [Follower]" leader= +31:55.339 [INFO] TestAgent.server: Adding LAN server: server="Node-afab7003-dd57-95a9-e38f-979a8f3e961c (Addr: tcp/127.0.0.1:30105) (DC: dc1)" +31:55.339 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-afab7003-dd57-95a9-e38f-979a8f3e961c.dc1 area=wan +31:55.339 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30065 network=tcp +31:55.339 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30065 network=udp +31:55.340 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30101 network=tcp +31:55.341 [INFO] TestAgent: started state syncer +31:55.341 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30158 network=tcp +31:55.408 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.408 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30105 [Candidate]" term=2 +31:55.485 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.485 [DEBUG] TestAgent.server.raft: vote granted: from=afab7003-dd57-95a9-e38f-979a8f3e961c term=2 tally=1 +31:55.485 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.485 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30105 [Leader]" +31:55.485 [INFO] TestAgent.server: cluster leadership acquired +31:55.486 [INFO] TestAgent.server: New leader elected: payload=Node-afab7003-dd57-95a9-e38f-979a8f3e961c +31:55.507 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30105 +31:55.553 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.617 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.617 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.617 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.617 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.617 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.617 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-afab7003-dd57-95a9-e38f-979a8f3e961c +31:55.617 [INFO] TestAgent.server: member joined, marking health alive: member=Node-afab7003-dd57-95a9-e38f-979a8f3e961c +31:55.657 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.709 [INFO] TestAgent: Requesting shutdown +31:55.709 [INFO] TestAgent.server: shutting down server +31:55.709 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.709 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.709 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.709 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.709 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.709 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.709 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.709 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:55.709 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.709 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.709 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.719 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.722Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.722Z [INFO] agent.router.manager: shutting down +31:55.722 [INFO] TestAgent: consul server down +31:55.722 [INFO] TestAgent: shutdown complete +31:55.722 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30065 network=tcp +31:55.722 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30065 network=udp +31:55.723 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30101 network=tcp +31:56.224 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.224 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_NodeMaintenance_BadRequest (1.13s) +=== CONT TestAgent_DeregisterService +2021-10-05T16:31:55.234Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.253Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.348 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f666680-1a45-692c-f3da-57ff1844fe6a Address:127.0.0.1:30157}]" +31:55.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0f666680-1a45-692c-f3da-57ff1844fe6a.dc1 127.0.0.1 +31:55.349 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0f666680-1a45-692c-f3da-57ff1844fe6a 127.0.0.1 +2021-10-05T16:31:55.349Z [INFO] agent.router: Initializing LAN area manager +31:55.349 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30082 network=udp +31:55.349 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30157 [Follower]" leader= +31:55.350 [INFO] TestAgent.server: Adding LAN server: server="Node-0f666680-1a45-692c-f3da-57ff1844fe6a (Addr: tcp/127.0.0.1:30157) (DC: dc1)" +31:55.350 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0f666680-1a45-692c-f3da-57ff1844fe6a.dc1 area=wan +31:55.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30082 network=tcp +31:55.350 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30083 network=tcp +31:55.350 [INFO] TestAgent: started state syncer +31:55.351 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30165 network=tcp +31:55.420 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.420 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30157 [Candidate]" term=2 +31:55.496 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.496 [DEBUG] TestAgent.server.raft: vote granted: from=0f666680-1a45-692c-f3da-57ff1844fe6a term=2 tally=1 +31:55.496 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.496 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30157 [Leader]" +31:55.497 [INFO] TestAgent.server: cluster leadership acquired +31:55.498 [INFO] TestAgent.server: New leader elected: payload=Node-0f666680-1a45-692c-f3da-57ff1844fe6a +31:55.532 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30157 +31:55.562 [INFO] TestAgent: Synced node info +31:55.606 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.665 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.665 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.665 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.665 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.665 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.665 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f666680-1a45-692c-f3da-57ff1844fe6a +31:55.665 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0f666680-1a45-692c-f3da-57ff1844fe6a +31:55.698 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.726 [INFO] TestAgent: Service entered maintenance mode: service=test +31:55.726 [DEBUG] TestAgent: removed check: check=_service_maintenance:test +31:55.726 [INFO] TestAgent: Service left maintenance mode: service=test +31:55.726 [DEBUG] TestAgent: Node info in sync +31:55.737 [INFO] TestAgent: Synced service: service=test +31:55.742 [INFO] TestAgent: Deregistered check: check=_service_maintenance:test +31:55.742 [INFO] TestAgent: Requesting shutdown +31:55.742 [INFO] TestAgent.server: shutting down server +31:55.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.742 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.742 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.742 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.742 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.742 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.742 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.742 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.742 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.742 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.746 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.749Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.749Z [INFO] agent.router.manager: shutting down +31:55.749 [INFO] TestAgent: consul server down +31:55.749 [INFO] TestAgent: shutdown complete +31:55.749 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30082 network=tcp +31:55.749 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30082 network=udp +31:55.749 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30083 network=tcp +31:56.252 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.252 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServiceMaintenance_Disable (1.06s) +=== CONT TestAgent_UpdateCheck_ACLDeny +2021-10-05T16:31:55.230Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.230Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.270Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.270Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.356 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1cc86b31-a8d1-3e00-c528-93b6bd88f1ad Address:127.0.0.1:30164}]" +31:55.357 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 127.0.0.1 +31:55.358 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad 127.0.0.1 +2021-10-05T16:31:55.358Z [INFO] agent.router: Initializing LAN area manager +31:55.358 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30159 network=udp +31:55.358 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30164 [Follower]" leader= +31:55.359 [INFO] TestAgent.server: Adding LAN server: server="Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad (Addr: tcp/127.0.0.1:30164) (DC: dc1)" +31:55.359 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 area=wan +31:55.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30159 network=tcp +31:55.359 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30160 network=tcp +31:55.360 [INFO] TestAgent: started state syncer +31:55.360 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30081 network=tcp +31:55.412 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:55.424 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.424 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30164 [Candidate]" term=2 +31:55.443 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.444 [DEBUG] TestAgent.server.raft: vote granted: from=1cc86b31-a8d1-3e00-c528-93b6bd88f1ad term=2 tally=1 +31:55.444 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.444 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30164 [Leader]" +31:55.444 [INFO] TestAgent.server: cluster leadership acquired +31:55.444 [INFO] TestAgent.server: New leader elected: payload=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.496 [INFO] TestAgent.server: initializing acls +31:55.506 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:55.506 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.513 [INFO] TestAgent.server: initializing acls +31:55.513 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.552 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:55.552 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:55.585 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.585 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:55.585 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:55.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 +31:55.587 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.587 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:55.587 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.587 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 +31:55.587 [INFO] TestAgent.server: Updating LAN server: server="Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad (Addr: tcp/127.0.0.1:30164) (DC: dc1)" +31:55.587 [INFO] TestAgent.server: Updating LAN server: server="Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad (Addr: tcp/127.0.0.1:30164) (DC: dc1)" +31:55.587 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 area=wan +31:55.587 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad.dc1 area=wan +31:55.621 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.687 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.687 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.687 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.687 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.687 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.687 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.687 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.696 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.696 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.712 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.718 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-1cc86b31-a8d1-3e00-c528-93b6bd88f1ad +31:55.726 [INFO] TestAgent: Service entered maintenance mode: service=test +31:55.734 [INFO] TestAgent: Synced node info +31:55.734 [WARN] TestAgent: Service registration blocked by ACLs: service=test accessorID=00000000-0000-0000-0000-000000000002 +31:55.743 [INFO] TestAgent: Synced check: check=_service_maintenance:test +31:55.743 [INFO] TestAgent: Requesting shutdown +31:55.743 [INFO] TestAgent.server: shutting down server +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:55.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.743 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.743 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.743 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:55.743 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:55.746 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.748Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.748Z [INFO] agent.router.manager: shutting down +31:55.748 [INFO] TestAgent: consul server down +31:55.748 [INFO] TestAgent: shutdown complete +31:55.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30159 network=tcp +31:55.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30159 network=udp +31:55.748 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30160 network=tcp +31:56.256 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.256 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServiceMaintenance_ACLDeny (1.08s) + --- PASS: TestAgent_ServiceMaintenance_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_ServiceMaintenance_ACLDeny/root_token (0.02s) +=== CONT TestAgent_FailCheck_ACLDeny +2021-10-05T16:31:55.411Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.441Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.555 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6948ad5a-2409-21f9-af18-0ae59283a2c9 Address:127.0.0.1:30171}]" +31:55.556 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6948ad5a-2409-21f9-af18-0ae59283a2c9.dc1 127.0.0.1 +31:55.556 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30171 [Follower]" leader= +31:55.556 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6948ad5a-2409-21f9-af18-0ae59283a2c9 127.0.0.1 +2021-10-05T16:31:55.556Z [INFO] agent.router: Initializing LAN area manager +31:55.557 [INFO] TestAgent.server: Adding LAN server: server="Node-6948ad5a-2409-21f9-af18-0ae59283a2c9 (Addr: tcp/127.0.0.1:30171) (DC: dc1)" +31:55.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30166 network=udp +31:55.557 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6948ad5a-2409-21f9-af18-0ae59283a2c9.dc1 area=wan +31:55.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30166 network=tcp +31:55.557 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30167 network=tcp +31:55.557 [INFO] TestAgent: started state syncer +31:55.557 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30172 network=tcp +31:55.591 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.591 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30171 [Candidate]" term=2 +31:55.616 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.616 [DEBUG] TestAgent.server.raft: vote granted: from=6948ad5a-2409-21f9-af18-0ae59283a2c9 term=2 tally=1 +31:55.616 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.616 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30171 [Leader]" +31:55.618 [INFO] TestAgent.server: cluster leadership acquired +31:55.618 [INFO] TestAgent.server: New leader elected: payload=Node-6948ad5a-2409-21f9-af18-0ae59283a2c9 +31:55.640 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30171 +31:55.664 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.714 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.714 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.715 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.715 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.715 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.715 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6948ad5a-2409-21f9-af18-0ae59283a2c9 +31:55.715 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6948ad5a-2409-21f9-af18-0ae59283a2c9 +31:55.737 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.750 [INFO] TestAgent: Service entered maintenance mode: service=test +31:55.753 [INFO] TestAgent: Synced node info +31:55.756 [INFO] TestAgent: Synced service: service=test +31:55.762 [INFO] TestAgent: Synced check: check=_service_maintenance:test +31:55.762 [INFO] TestAgent: Requesting shutdown +31:55.762 [INFO] TestAgent.server: shutting down server +31:55.762 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:55.762 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:55.762 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:55.762 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:55.762 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:55.762 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:55.762 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:55.762 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:55.762 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:55.762 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:55.762 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:55.767 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:55.767Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:55.767Z [INFO] agent.router.manager: shutting down +31:55.767 [INFO] TestAgent: consul server down +31:55.767 [INFO] TestAgent: shutdown complete +31:55.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30166 network=tcp +31:55.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30166 network=udp +31:55.767 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30167 network=tcp +31:56.268 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.268 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServiceMaintenance_Enable (0.89s) +=== CONT TestAgent_FailCheck [WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:17.772Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.784Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.827 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:697fba90-3629-5d43-b5a4-1dd20c94a0dd Address:127.0.0.1:11559}]" -46:17.827 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11559 [Follower]" leader= -46:17.829 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd.dc1 127.0.0.1 -46:17.829 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd 127.0.0.1 -2022-11-07T22:46:17.829Z [INFO] agent.router: Initializing LAN area manager -46:17.829 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd.dc1 area=wan -46:17.829 [INFO] TestAgent.server: Adding LAN server: server="Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd (Addr: tcp/127.0.0.1:11559) (DC: dc1)" -46:17.829 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11638 network=udp -46:17.829 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11638 network=tcp -46:17.829 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11555 network=tcp -46:17.830 [INFO] TestAgent: started state syncer -46:17.830 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11595 network=tcp -46:17.879 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.879 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11559 [Candidate]" term=2 -46:17.891 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.891 [DEBUG] TestAgent.server.raft: vote granted: from=697fba90-3629-5d43-b5a4-1dd20c94a0dd term=2 tally=1 -46:17.891 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.891 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11559 [Leader]" -46:17.891 [INFO] TestAgent.server: cluster leadership acquired -46:17.891 [INFO] TestAgent.server: New leader elected: payload=Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd -46:17.899 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11559 -46:17.915 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.943 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.943 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.943 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.943 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.943 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.943 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd -46:17.943 [INFO] TestAgent.server: member joined, marking health alive: member=Node-697fba90-3629-5d43-b5a4-1dd20c94a0dd -46:17.949 [INFO] TestAgent.server: federation state anti-entropy synced -46:17.994 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:17.998 [INFO] TestAgent: Synced node info -46:18.107 [INFO] TestAgent: Requesting shutdown -46:18.107 [INFO] TestAgent.server: shutting down server -46:18.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.107 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.107 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.107 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.107 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.107 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.107 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.113 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.119Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.119Z [INFO] agent.router.manager: shutting down -46:18.120 [INFO] TestAgent: consul server down -46:18.120 [INFO] TestAgent: shutdown complete -46:18.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11638 network=tcp -46:18.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11638 network=udp -46:18.120 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11555 network=tcp -46:18.620 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.620 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:17.686Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.697Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.720 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f06de262-fa67-19a6-1f4d-178f095786e1 Address:127.0.0.1:11538}]" -46:17.720 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11538 [Follower]" leader= -46:17.721 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f06de262-fa67-19a6-1f4d-178f095786e1.dc1 127.0.0.1 -46:17.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f06de262-fa67-19a6-1f4d-178f095786e1 127.0.0.1 -2022-11-07T22:46:17.721Z [INFO] agent.router: Initializing LAN area manager -46:17.722 [INFO] TestAgent.server: Adding LAN server: server="Node-f06de262-fa67-19a6-1f4d-178f095786e1 (Addr: tcp/127.0.0.1:11538) (DC: dc1)" -46:17.722 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f06de262-fa67-19a6-1f4d-178f095786e1.dc1 area=wan -46:17.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11401 network=udp -46:17.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11401 network=tcp -46:17.722 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11534 network=tcp -46:17.722 [INFO] TestAgent: started state syncer -46:17.722 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11602 network=tcp -46:17.783 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.783 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11538 [Candidate]" term=2 -46:17.802 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.802 [DEBUG] TestAgent.server.raft: vote granted: from=f06de262-fa67-19a6-1f4d-178f095786e1 term=2 tally=1 -46:17.802 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.802 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11538 [Leader]" -46:17.802 [INFO] TestAgent.server: cluster leadership acquired -46:17.802 [INFO] TestAgent.server: New leader elected: payload=Node-f06de262-fa67-19a6-1f4d-178f095786e1 -46:17.811 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11538 -46:17.827 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f06de262-fa67-19a6-1f4d-178f095786e1 -46:17.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f06de262-fa67-19a6-1f4d-178f095786e1 -46:17.853 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.052 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.055 [INFO] TestAgent: Synced node info -46:18.055 [DEBUG] TestAgent: Node info in sync -46:18.106 [INFO] TestAgent: Requesting shutdown -46:18.106 [INFO] TestAgent.server: shutting down server -46:18.106 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.106 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.106 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.106 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.106 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.106 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.106 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.106 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.106 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.106 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.113 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.119Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.119Z [INFO] agent.router.manager: shutting down -46:18.120 [INFO] TestAgent: consul server down -46:18.120 [INFO] TestAgent: shutdown complete -46:18.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11401 network=tcp -46:18.120 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11401 network=udp -46:18.120 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11534 network=tcp -46:18.620 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.620 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_reloadWatches (0.86s) -=== CONT TestAgent_persistCheckState ---- PASS: TestAgentConnectAuthorize_allowTrustDomain (0.94s) -=== CONT TestAgentConnectAuthorize_noTarget -2022-11-07T22:46:17.897Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.918Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.951 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf Address:127.0.0.1:11608}]" -46:17.951 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11608 [Follower]" leader= -46:17.952 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf.dc1 127.0.0.1 -46:17.953 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf 127.0.0.1 -2022-11-07T22:46:17.953Z [INFO] agent.router: Initializing LAN area manager -46:17.953 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11568 network=udp -46:17.953 [INFO] TestAgent.server: Adding LAN server: server="Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf (Addr: tcp/127.0.0.1:11608) (DC: dc1)" -46:17.953 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf.dc1 area=wan -46:17.953 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11568 network=tcp -46:17.954 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11604 network=tcp -46:17.954 [INFO] TestAgent: started state syncer -46:17.954 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11574 network=tcp -46:18.017 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.017 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11608 [Candidate]" term=2 -46:18.032 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.032 [DEBUG] TestAgent.server.raft: vote granted: from=e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf term=2 tally=1 -46:18.032 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.033 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11608 [Leader]" -46:18.033 [INFO] TestAgent.server: cluster leadership acquired -46:18.033 [INFO] TestAgent.server: New leader elected: payload=Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf -46:18.043 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11608 -46:18.064 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.091 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.091 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.091 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.091 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.091 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.091 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf -46:18.091 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e6a5e3e0-bcd7-a813-4ecd-af26eaefd7bf -46:18.099 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.126 [INFO] TestAgent: Requesting shutdown -46:18.126 [INFO] TestAgent.server: shutting down server -46:18.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.126 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.126 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.126 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.126 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.126 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.126 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.126 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.130 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.133Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.133Z [INFO] agent.router.manager: shutting down -46:18.134 [INFO] TestAgent: consul server down -46:18.134 [INFO] TestAgent: shutdown complete -46:18.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11568 network=tcp -46:18.134 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11568 network=udp -46:18.134 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11604 network=tcp -46:18.634 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.634 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_deny (0.75s) -=== CONT TestAgent_loadChecks_checkFails -2022-11-07T22:46:17.779Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.788Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.830 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46e6acaa-3c08-179f-8c2d-7be99e3ac052 Address:127.0.0.1:11601}]" -46:17.830 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11601 [Follower]" leader= -46:17.830 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052.dc1 127.0.0.1 -46:17.831 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052 127.0.0.1 -2022-11-07T22:46:17.831Z [INFO] agent.router: Initializing LAN area manager -46:17.831 [INFO] TestAgent.server: Adding LAN server: server="Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052 (Addr: tcp/127.0.0.1:11601) (DC: dc1)" -46:17.831 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11617 network=udp -46:17.831 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052.dc1 area=wan -46:17.831 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11617 network=tcp -46:17.831 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11597 network=tcp -46:17.831 [INFO] TestAgent: started state syncer -46:17.832 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11560 network=tcp -46:17.889 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.890 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11601 [Candidate]" term=2 -46:17.906 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:17.906 [DEBUG] TestAgent.server.raft: vote granted: from=46e6acaa-3c08-179f-8c2d-7be99e3ac052 term=2 tally=1 -46:17.906 [INFO] TestAgent.server.raft: election won: tally=1 -46:17.906 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11601 [Leader]" -46:17.906 [INFO] TestAgent.server: cluster leadership acquired -46:17.906 [INFO] TestAgent.server: New leader elected: payload=Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052 -46:17.914 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11601 -46:17.928 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:17.963 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:17.964 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:17.964 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:17.964 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:17.964 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:17.964 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052 -46:17.964 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46e6acaa-3c08-179f-8c2d-7be99e3ac052 -46:17.980 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.022 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.028 [INFO] TestAgent: Synced node info -46:18.150 [INFO] TestAgent: Requesting shutdown -46:18.150 [INFO] TestAgent.server: shutting down server -46:18.150 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.150 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.150 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.150 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.150 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.150 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.150 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.150 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.150 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.150 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.155 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.162Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.162Z [INFO] agent.router.manager: shutting down -46:18.162 [INFO] TestAgent: consul server down -46:18.162 [INFO] TestAgent: shutdown complete -46:18.162 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11617 network=tcp -46:18.162 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11617 network=udp -46:18.162 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11597 network=tcp -46:18.662 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.662 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_denyWildcard (0.90s) -=== CONT TestAgent_Services_ExposeConfig -2022-11-07T22:46:18.018Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.028Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.066 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 Address:127.0.0.1:11622}]" -46:18.066 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11622 [Follower]" leader= -46:18.066 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40.dc1 127.0.0.1 -46:18.066 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 127.0.0.1 -2022-11-07T22:46:18.067Z [INFO] agent.router: Initializing LAN area manager -46:18.067 [INFO] TestAgent.server: Adding LAN server: server="Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 (Addr: tcp/127.0.0.1:11622) (DC: dc1)" -46:18.067 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40.dc1 area=wan -46:18.067 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11624 network=tcp -46:18.067 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11624 network=udp -46:18.067 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11618 network=tcp -46:18.067 [INFO] TestAgent: started state syncer -46:18.067 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11651 network=tcp -46:18.117 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.117 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11622 [Candidate]" term=2 -46:18.130 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.130 [DEBUG] TestAgent.server.raft: vote granted: from=87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 term=2 tally=1 -46:18.130 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.130 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11622 [Leader]" -46:18.130 [INFO] TestAgent.server: cluster leadership acquired -46:18.130 [INFO] TestAgent.server: New leader elected: payload=Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 -46:18.139 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11622 -46:18.157 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.185 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.185 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.185 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.185 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.185 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.185 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 -46:18.185 [INFO] TestAgent.server: member joined, marking health alive: member=Node-87fe49ef-6dfd-37aa-c5d2-7c33c1b16c40 -46:18.198 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.259 [INFO] TestAgent: Requesting shutdown -46:18.259 [INFO] TestAgent.server: shutting down server -46:18.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.259 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.259 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.259 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.259 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.259 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.259 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.259 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.259 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.259 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.264 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.267Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.267Z [INFO] agent.router.manager: shutting down -46:18.267 [INFO] TestAgent: consul server down -46:18.267 [INFO] TestAgent: shutdown complete -46:18.267 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11624 network=tcp -46:18.267 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11624 network=udp -46:18.267 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11618 network=tcp -46:18.767 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.767 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RPCPing (0.76s) -=== CONT TestAgent_HostBadACL +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_UpdateCheck_ACLDeny/no_token +=== RUN TestAgent_UpdateCheck_ACLDeny/root_token +=== RUN TestAgent_FailCheck_ACLDeny/no_token +=== RUN TestAgent_FailCheck_ACLDeny/root_token +2021-10-05T16:31:55.876Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.901Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.978 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1445d07f-355e-3404-c6ad-98a2bef12a19 Address:127.0.0.1:30178}]" +31:55.978 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1445d07f-355e-3404-c6ad-98a2bef12a19.dc1 127.0.0.1 +31:55.979 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1445d07f-355e-3404-c6ad-98a2bef12a19 127.0.0.1 +2021-10-05T16:31:55.979Z [INFO] agent.router: Initializing LAN area manager +31:55.979 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30173 network=udp +31:55.979 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30178 [Follower]" leader= +31:55.980 [INFO] TestAgent.server: Adding LAN server: server="Node-1445d07f-355e-3404-c6ad-98a2bef12a19 (Addr: tcp/127.0.0.1:30178) (DC: dc1)" +31:55.980 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1445d07f-355e-3404-c6ad-98a2bef12a19.dc1 area=wan +31:55.980 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30173 network=tcp +31:55.981 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30174 network=tcp +31:55.981 [INFO] TestAgent: started state syncer +31:55.981 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30186 network=tcp +31:56.049 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.049 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30178 [Candidate]" term=2 +31:56.062 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.062 [DEBUG] TestAgent.server.raft: vote granted: from=1445d07f-355e-3404-c6ad-98a2bef12a19 term=2 tally=1 +31:56.062 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.062 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30178 [Leader]" +31:56.062 [INFO] TestAgent.server: cluster leadership acquired +31:56.062 [INFO] TestAgent.server: New leader elected: payload=Node-1445d07f-355e-3404-c6ad-98a2bef12a19 +31:56.069 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30178 +31:56.092 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.136 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.136 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.136 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.136 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.136 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.136 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1445d07f-355e-3404-c6ad-98a2bef12a19 +31:56.136 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1445d07f-355e-3404-c6ad-98a2bef12a19 +31:56.151 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.171 [INFO] TestAgent: Requesting shutdown +31:56.171 [INFO] TestAgent.server: shutting down server +31:56.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.171 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.171 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.171 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.171 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:56.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.171 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:56.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.171 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.171 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.179 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.179Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.179Z [INFO] agent.router.manager: shutting down +31:56.179 [INFO] TestAgent: consul server down +31:56.179 [INFO] TestAgent: shutdown complete +31:56.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30173 network=tcp +31:56.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30173 network=udp +31:56.179 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30174 network=tcp +31:56.680 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.680 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServiceMaintenance_BadRequest (0.83s) + --- PASS: TestAgent_ServiceMaintenance_BadRequest/not_enabled (0.00s) + --- PASS: TestAgent_ServiceMaintenance_BadRequest/no_service_id (0.00s) + --- PASS: TestAgent_ServiceMaintenance_BadRequest/bad_service_id (0.00s) +=== CONT TestAgent_WarnCheck_ACLDeny +=== RUN TestAgent_DeregisterService_ACLDeny/no_token +=== RUN TestAgent_DeregisterService_ACLDeny/root_token +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:56.122Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.122Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.134Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.134Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.171 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 Address:127.0.0.1:30216}]" +31:56.173 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 127.0.0.1 +31:56.173 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 127.0.0.1 +2021-10-05T16:31:56.173Z [INFO] agent.router: Initializing LAN area manager +31:56.174 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30187 network=udp +31:56.174 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30216 [Follower]" leader= +31:56.174 [INFO] TestAgent.server: Adding LAN server: server="Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 (Addr: tcp/127.0.0.1:30216) (DC: dc1)" +31:56.174 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 area=wan +31:56.174 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30187 network=tcp +31:56.174 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30188 network=tcp +31:56.175 [INFO] TestAgent: started state syncer +31:56.175 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30179 network=tcp +31:56.224 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:56.244 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.244 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30216 [Candidate]" term=2 +31:56.280 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.280 [DEBUG] TestAgent.server.raft: vote granted: from=8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 term=2 tally=1 +31:56.280 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.280 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30216 [Leader]" +31:56.280 [INFO] TestAgent.server: cluster leadership acquired +31:56.280 [INFO] TestAgent.server: New leader elected: payload=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.295 [INFO] TestAgent.server: initializing acls +31:56.304 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:56.304 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.311 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.324 [INFO] TestAgent.server: initializing acls +31:56.325 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.333 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.333 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:56.333 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:56.334 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.334 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 +31:56.334 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 area=wan +31:56.334 [INFO] TestAgent.server: Updating LAN server: server="Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 (Addr: tcp/127.0.0.1:30216) (DC: dc1)" +31:56.339 [INFO] TestAgent: Synced node info +31:56.339 [DEBUG] TestAgent: Node info in sync +31:56.339 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.339 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:56.339 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.339 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 +31:56.339 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4.dc1 area=wan +31:56.339 [INFO] TestAgent.server: Updating LAN server: server="Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 (Addr: tcp/127.0.0.1:30216) (DC: dc1)" +31:56.354 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.394 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.394 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.394 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.394 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.394 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.394 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.394 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.416 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8ed1f317-9e1c-ee1b-9dd0-4e9e9d76a3b4 +31:56.416 [INFO] TestAgent: Requesting shutdown +31:56.416 [INFO] TestAgent.server: shutting down server +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.416 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:56.416 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:56.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.423 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +31:56.428 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.428 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:31:56.432Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.432Z [INFO] agent.router.manager: shutting down +31:56.432 [INFO] TestAgent: consul server down +31:56.432 [INFO] TestAgent: shutdown complete +31:56.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30187 network=tcp +31:56.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30187 network=udp +31:56.432 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30188 network=tcp +31:56.935 [INFO] TestAgent: Waiting for endpoints to shut down +31:56.935 [INFO] TestAgent: Endpoints down +=== RUN TestSnapshot_Options/PUT#02 +=== RUN TestAgent_WarnCheck_ACLDeny/no_token +=== RUN TestAgent_WarnCheck_ACLDeny/root_token +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:56.267Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.267Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.280Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.280Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.333 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:92a004a5-5374-472e-802b-47cff2c6a2bc Address:127.0.0.1:30252}]" +31:56.334 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 127.0.0.1 +31:56.334 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30252 [Follower]" leader= +31:56.334 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-92a004a5-5374-472e-802b-47cff2c6a2bc 127.0.0.1 +2021-10-05T16:31:56.334Z [INFO] agent.router: Initializing LAN area manager +31:56.334 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30240 network=udp +31:56.335 [INFO] TestAgent.server: Adding LAN server: server="Node-92a004a5-5374-472e-802b-47cff2c6a2bc (Addr: tcp/127.0.0.1:30252) (DC: dc1)" +31:56.335 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 area=wan +31:56.335 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30240 network=tcp +31:56.335 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30241 network=tcp +31:56.335 [INFO] TestAgent: started state syncer +31:56.335 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30225 network=tcp +31:56.385 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:56.400 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.400 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30252 [Candidate]" term=2 +31:56.441 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.441 [DEBUG] TestAgent.server.raft: vote granted: from=92a004a5-5374-472e-802b-47cff2c6a2bc term=2 tally=1 +31:56.441 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.441 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30252 [Leader]" +31:56.441 [INFO] TestAgent.server: cluster leadership acquired +31:56.441 [INFO] TestAgent.server: New leader elected: payload=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.463 [INFO] TestAgent.server: initializing acls +31:56.484 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:56.484 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.485 [INFO] TestAgent.server: initializing acls +31:56.485 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.493 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.518 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.518 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:56.518 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:56.518 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.518 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 +31:56.518 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.519 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:56.519 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.519 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 +31:56.519 [INFO] TestAgent.server: Updating LAN server: server="Node-92a004a5-5374-472e-802b-47cff2c6a2bc (Addr: tcp/127.0.0.1:30252) (DC: dc1)" +31:56.519 [INFO] TestAgent.server: Updating LAN server: server="Node-92a004a5-5374-472e-802b-47cff2c6a2bc (Addr: tcp/127.0.0.1:30252) (DC: dc1)" +31:56.519 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 area=wan +31:56.519 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-92a004a5-5374-472e-802b-47cff2c6a2bc.dc1 area=wan +31:56.541 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.572 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.572 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.572 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.572 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.572 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.572 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.572 [INFO] TestAgent.server: member joined, marking health alive: member=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.581 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.581 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.583 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.583 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-92a004a5-5374-472e-802b-47cff2c6a2bc +31:56.585 [DEBUG] TestAgent: Check status updated: check=test status=passing +31:56.620 [INFO] TestAgent: Synced node info +31:56.621 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 +31:56.621 [INFO] TestAgent: Requesting shutdown +31:56.621 [INFO] TestAgent.server: shutting down server +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.621 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.621 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.621 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.621 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.621 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:56.621 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.621 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:56.622 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:56.622 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.622 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.633 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.640Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.640Z [INFO] agent.router.manager: shutting down +31:56.644 [INFO] TestAgent: consul server down +31:56.644 [INFO] TestAgent: shutdown complete +31:56.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30240 network=tcp +31:56.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30240 network=udp +31:56.644 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30241 network=tcp +31:56.644 [WARN] TestAgent: Syncing node info failed.: error="leadership lost while committing log" +31:56.644 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="leadership lost while committing log" +31:57.144 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.144 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_UpdateCheck_ACLDeny (0.89s) + --- PASS: TestAgent_UpdateCheck_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_UpdateCheck_ACLDeny/root_token (0.04s) +=== CONT TestAgent_WarnCheck +2021-10-05T16:31:56.241Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.260Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.325 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d2897bdd-5998-a856-d877-d7615ba14481 Address:127.0.0.1:30210}]" +31:56.325 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30210 [Follower]" leader= +31:56.326 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d2897bdd-5998-a856-d877-d7615ba14481.dc1 127.0.0.1 +31:56.326 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d2897bdd-5998-a856-d877-d7615ba14481 127.0.0.1 +2021-10-05T16:31:56.326Z [INFO] agent.router: Initializing LAN area manager +31:56.327 [INFO] TestAgent.server: Adding LAN server: server="Node-d2897bdd-5998-a856-d877-d7615ba14481 (Addr: tcp/127.0.0.1:30210) (DC: dc1)" +31:56.327 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d2897bdd-5998-a856-d877-d7615ba14481.dc1 area=wan +31:56.327 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30205 network=udp +31:56.327 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30205 network=tcp +31:56.327 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30206 network=tcp +31:56.327 [INFO] TestAgent: started state syncer +31:56.327 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30239 network=tcp +31:56.363 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.363 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30210 [Candidate]" term=2 +31:56.416 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.416 [DEBUG] TestAgent.server.raft: vote granted: from=d2897bdd-5998-a856-d877-d7615ba14481 term=2 tally=1 +31:56.416 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.416 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30210 [Leader]" +31:56.416 [INFO] TestAgent.server: cluster leadership acquired +31:56.416 [INFO] TestAgent.server: New leader elected: payload=Node-d2897bdd-5998-a856-d877-d7615ba14481 +31:56.452 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30210 +31:56.508 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.573 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.573 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.573 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.573 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.573 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.573 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d2897bdd-5998-a856-d877-d7615ba14481 +31:56.574 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d2897bdd-5998-a856-d877-d7615ba14481 +31:56.605 [DEBUG] TestAgent: removed service: service=test +31:56.619 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.648 [INFO] TestAgent: Synced node info +31:56.653 [INFO] TestAgent: Deregistered service: service=test +31:56.653 [INFO] TestAgent: Requesting shutdown +31:56.653 [INFO] TestAgent.server: shutting down server +31:56.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.653 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.653 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.653 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.653 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:56.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.653 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:56.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.653 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.653 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.659 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.664Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.664Z [INFO] agent.router.manager: shutting down +31:56.665 [INFO] TestAgent: consul server down +31:56.665 [INFO] TestAgent: shutdown complete +31:56.665 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30205 network=tcp +31:56.665 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30205 network=udp +31:56.665 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30206 network=tcp +31:57.165 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.165 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_DeregisterService (0.94s) +=== CONT TestAgent_PassCheck_ACLDeny +2021-10-05T16:31:56.281Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.281Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.297Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.297Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.352 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f588c486-2cee-856e-b8bf-cb411a65d95f Address:127.0.0.1:30238}]" +31:56.352 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30238 [Follower]" leader= +31:56.353 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 127.0.0.1 +31:56.353 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f588c486-2cee-856e-b8bf-cb411a65d95f 127.0.0.1 +2021-10-05T16:31:56.353Z [INFO] agent.router: Initializing LAN area manager +31:56.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30233 network=udp +31:56.354 [INFO] TestAgent.server: Adding LAN server: server="Node-f588c486-2cee-856e-b8bf-cb411a65d95f (Addr: tcp/127.0.0.1:30238) (DC: dc1)" +31:56.354 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 area=wan +31:56.354 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30233 network=tcp +31:56.354 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30234 network=tcp +31:56.354 [INFO] TestAgent: started state syncer +31:56.354 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30253 network=tcp +31:56.404 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:56.416 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.416 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30238 [Candidate]" term=2 +31:56.454 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.454 [DEBUG] TestAgent.server.raft: vote granted: from=f588c486-2cee-856e-b8bf-cb411a65d95f term=2 tally=1 +31:56.454 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.454 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30238 [Leader]" +31:56.454 [INFO] TestAgent.server: cluster leadership acquired +31:56.454 [INFO] TestAgent.server: New leader elected: payload=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.480 [INFO] TestAgent.server: initializing acls +31:56.487 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:56.487 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.499 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.508 [INFO] TestAgent.server: initializing acls +31:56.508 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.527 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.527 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:56.527 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:56.527 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:56.527 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.527 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 +31:56.527 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 area=wan +31:56.527 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.527 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.528 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 +31:56.528 [INFO] TestAgent.server: Updating LAN server: server="Node-f588c486-2cee-856e-b8bf-cb411a65d95f (Addr: tcp/127.0.0.1:30238) (DC: dc1)" +31:56.528 [INFO] TestAgent.server: Updating LAN server: server="Node-f588c486-2cee-856e-b8bf-cb411a65d95f (Addr: tcp/127.0.0.1:30238) (DC: dc1)" +31:56.528 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f588c486-2cee-856e-b8bf-cb411a65d95f.dc1 area=wan +31:56.547 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.576 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.576 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.576 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.576 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.576 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.577 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.577 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.613 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.613 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.613 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.619 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-f588c486-2cee-856e-b8bf-cb411a65d95f +31:56.620 [DEBUG] TestAgent: Check status updated: check=test status=critical +31:56.644 [INFO] TestAgent: Synced node info +31:56.644 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 +31:56.644 [INFO] TestAgent: Requesting shutdown +31:56.644 [INFO] TestAgent.server: shutting down server +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.644 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.644 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.644 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:56.644 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:56.645 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.645 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.645 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.646 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:56.646 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:56.660 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.672Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.672Z [INFO] agent.router.manager: shutting down +31:56.672 [INFO] TestAgent: consul server down +31:56.672 [INFO] TestAgent: shutdown complete +31:56.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30233 network=tcp +31:56.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30233 network=udp +31:56.672 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30234 network=tcp +31:57.173 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.173 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_FailCheck_ACLDeny (0.92s) + --- PASS: TestAgent_FailCheck_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_FailCheck_ACLDeny/root_token (0.02s) +=== CONT TestAgent_PassCheck +2021-10-05T16:31:56.308Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.324Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.400 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:86fbd723-536e-4ad0-b4b8-c9cbca1b6384 Address:127.0.0.1:30245}]" +31:56.401 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384.dc1 127.0.0.1 +31:56.401 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384 127.0.0.1 +2021-10-05T16:31:56.401Z [INFO] agent.router: Initializing LAN area manager +31:56.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30254 network=udp +31:56.402 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30245 [Follower]" leader= +31:56.402 [INFO] TestAgent.server: Adding LAN server: server="Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384 (Addr: tcp/127.0.0.1:30245) (DC: dc1)" +31:56.402 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384.dc1 area=wan +31:56.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30254 network=tcp +31:56.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30255 network=tcp +31:56.403 [INFO] TestAgent: started state syncer +31:56.403 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30246 network=tcp +31:56.444 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.444 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30245 [Candidate]" term=2 +31:56.484 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.484 [DEBUG] TestAgent.server.raft: vote granted: from=86fbd723-536e-4ad0-b4b8-c9cbca1b6384 term=2 tally=1 +31:56.484 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.484 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30245 [Leader]" +31:56.484 [INFO] TestAgent.server: cluster leadership acquired +31:56.484 [INFO] TestAgent.server: New leader elected: payload=Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384 +31:56.501 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30245 +31:56.528 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.566 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.566 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.566 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.566 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.566 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.566 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384 +31:56.566 [INFO] TestAgent.server: member joined, marking health alive: member=Node-86fbd723-536e-4ad0-b4b8-c9cbca1b6384 +31:56.594 [DEBUG] TestAgent: Check status updated: check=test status=critical +31:56.618 [INFO] TestAgent: Synced node info +31:56.619 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.652 [INFO] TestAgent: Synced check: check=test +31:56.652 [INFO] TestAgent: Requesting shutdown +31:56.652 [INFO] TestAgent.server: shutting down server +31:56.652 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.652 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.652 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.652 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.652 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.652 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:56.652 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.652 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:56.652 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.652 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.652 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.664 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.676Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.676Z [INFO] agent.router.manager: shutting down +31:56.676 [INFO] TestAgent: consul server down +31:56.676 [INFO] TestAgent: shutdown complete +31:56.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30254 network=tcp +31:56.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30254 network=udp +31:56.676 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30255 network=tcp +31:57.176 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.176 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_FailCheck (0.90s) +=== CONT TestAgent_DeregisterCheckACLDeny +2021-10-05T16:31:56.221Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.221Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.232Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.232Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.302 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:20354e63-4208-41bd-9d91-d3b51af394ea Address:127.0.0.1:30185}]" +31:56.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 127.0.0.1 +31:56.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-20354e63-4208-41bd-9d91-d3b51af394ea 127.0.0.1 +2021-10-05T16:31:56.303Z [INFO] agent.router: Initializing LAN area manager +31:56.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30180 network=udp +31:56.303 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30185 [Follower]" leader= +31:56.304 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 area=wan +31:56.304 [INFO] TestAgent.server: Adding LAN server: server="Node-20354e63-4208-41bd-9d91-d3b51af394ea (Addr: tcp/127.0.0.1:30185) (DC: dc1)" +31:56.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30180 network=tcp +31:56.304 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30181 network=tcp +31:56.304 [INFO] TestAgent: started state syncer +31:56.305 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30217 network=tcp +31:56.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:56.360 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.360 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30185 [Candidate]" term=2 +31:56.380 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.380 [DEBUG] TestAgent.server.raft: vote granted: from=20354e63-4208-41bd-9d91-d3b51af394ea term=2 tally=1 +31:56.380 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.380 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30185 [Leader]" +31:56.381 [INFO] TestAgent.server: cluster leadership acquired +31:56.381 [INFO] TestAgent.server: New leader elected: payload=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.406 [INFO] TestAgent.server: initializing acls +31:56.420 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:56.420 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.440 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.454 [INFO] TestAgent.server: initializing acls +31:56.454 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.480 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.480 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:56.480 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:56.480 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:56.480 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.480 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 +31:56.480 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 area=wan +31:56.480 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.480 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.481 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 +31:56.481 [INFO] TestAgent.server: Updating LAN server: server="Node-20354e63-4208-41bd-9d91-d3b51af394ea (Addr: tcp/127.0.0.1:30185) (DC: dc1)" +31:56.481 [INFO] TestAgent.server: Updating LAN server: server="Node-20354e63-4208-41bd-9d91-d3b51af394ea (Addr: tcp/127.0.0.1:30185) (DC: dc1)" +31:56.481 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-20354e63-4208-41bd-9d91-d3b51af394ea.dc1 area=wan +31:56.513 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:56.553 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:56.553 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:56.553 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:56.553 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:56.553 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:56.554 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.554 [INFO] TestAgent.server: member joined, marking health alive: member=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.560 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.560 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.569 [INFO] TestAgent.server: federation state anti-entropy synced +31:56.645 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:56.662 [INFO] TestAgent: Synced node info +31:56.662 [DEBUG] TestAgent: Node info in sync +31:56.683 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.683 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-20354e63-4208-41bd-9d91-d3b51af394ea +31:56.683 [DEBUG] TestAgent: removed service: service=test +31:56.683 [DEBUG] TestAgent: Node info in sync +31:56.683 [INFO] TestAgent: Deregistered service: service=test +31:56.683 [INFO] TestAgent: Requesting shutdown +31:56.684 [INFO] TestAgent.server: shutting down server +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.684 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.684 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:56.684 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.684 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.692 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.700Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.700Z [INFO] agent.router.manager: shutting down +31:56.700 [INFO] TestAgent: consul server down +31:56.700 [INFO] TestAgent: shutdown complete +31:56.700 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30180 network=tcp +31:56.700 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30180 network=udp +31:56.700 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30181 network=tcp +31:57.200 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.200 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_DeregisterService_ACLDeny (1.03s) + --- PASS: TestAgent_DeregisterService_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_DeregisterService_ACLDeny/root_token (0.00s) +=== CONT TestAgent_DeregisterCheck [WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:17.901Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.919Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.962 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:27daabfb-a8d2-026c-386e-c861dc125086 Address:127.0.0.1:11615}]" -46:17.962 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11615 [Follower]" leader= -46:17.963 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-27daabfb-a8d2-026c-386e-c861dc125086.dc1 127.0.0.1 -46:17.963 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-27daabfb-a8d2-026c-386e-c861dc125086 127.0.0.1 -2022-11-07T22:46:17.963Z [INFO] agent.router: Initializing LAN area manager -46:17.963 [INFO] TestAgent.server: Adding LAN server: server="Node-27daabfb-a8d2-026c-386e-c861dc125086 (Addr: tcp/127.0.0.1:11615) (DC: dc1)" -46:17.964 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-27daabfb-a8d2-026c-386e-c861dc125086.dc1 area=wan -46:17.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11575 network=udp -46:17.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11575 network=tcp -46:17.964 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11611 network=tcp -46:17.964 [INFO] TestAgent: started state syncer -46:17.964 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11623 network=tcp -46:18.016 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.016 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11615 [Candidate]" term=2 -46:18.031 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.031 [DEBUG] TestAgent.server.raft: vote granted: from=27daabfb-a8d2-026c-386e-c861dc125086 term=2 tally=1 -46:18.031 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.031 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11615 [Leader]" -46:18.031 [INFO] TestAgent.server: cluster leadership acquired -46:18.031 [INFO] TestAgent.server: New leader elected: payload=Node-27daabfb-a8d2-026c-386e-c861dc125086 -46:18.043 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11615 -46:18.063 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.095 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.095 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.095 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.095 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.095 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.095 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-27daabfb-a8d2-026c-386e-c861dc125086 -46:18.095 [INFO] TestAgent.server: member joined, marking health alive: member=Node-27daabfb-a8d2-026c-386e-c861dc125086 -46:18.113 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.279 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.284 [INFO] TestAgent: Synced node info -46:18.284 [DEBUG] TestAgent: Node info in sync -46:18.327 [INFO] TestAgent: Node entered maintenance mode -46:18.327 [DEBUG] TestAgent: removed check: check=_node_maintenance -46:18.327 [INFO] TestAgent: Node left maintenance mode -46:18.327 [DEBUG] TestAgent: Node info in sync -46:18.330 [INFO] TestAgent: Deregistered check: check=_node_maintenance -46:18.330 [INFO] TestAgent: Requesting shutdown -46:18.330 [INFO] TestAgent.server: shutting down server -46:18.330 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.330 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.330 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.330 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.330 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.330 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.330 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.330 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.330 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.330 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.338 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.341Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.341Z [INFO] agent.router.manager: shutting down -46:18.341 [INFO] TestAgent: consul server down -46:18.341 [INFO] TestAgent: shutdown complete -46:18.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11575 network=tcp -46:18.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11575 network=udp -46:18.341 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11611 network=tcp -46:18.842 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.842 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_NodeMaintenance_Disable (0.95s) -=== CONT TestAgentConnectAuthorize_badBody -2022-11-07T22:46:17.962Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.973Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.011 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e Address:127.0.0.1:11657}]" -46:18.011 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11657 [Follower]" leader= -46:18.012 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e.dc1 127.0.0.1 -46:18.012 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e 127.0.0.1 -2022-11-07T22:46:18.012Z [INFO] agent.router: Initializing LAN area manager -46:18.012 [INFO] TestAgent.server: Adding LAN server: server="Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e (Addr: tcp/127.0.0.1:11657) (DC: dc1)" -46:18.012 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e.dc1 area=wan -46:18.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11610 network=udp -46:18.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11610 network=tcp -46:18.013 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11653 network=tcp -46:18.013 [INFO] TestAgent: started state syncer -46:18.013 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11581 network=tcp -46:18.068 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.068 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11657 [Candidate]" term=2 -46:18.079 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.079 [DEBUG] TestAgent.server.raft: vote granted: from=cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e term=2 tally=1 -46:18.079 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.079 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11657 [Leader]" -46:18.079 [INFO] TestAgent.server: cluster leadership acquired -46:18.079 [INFO] TestAgent.server: New leader elected: payload=Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e -46:18.089 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11657 -46:18.110 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.148 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.148 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.148 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.148 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.148 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.148 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e -46:18.148 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd8aab47-3eba-0fb8-1a14-2c9b4c55ed2e -46:18.169 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.197 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.201 [INFO] TestAgent: Synced node info -46:18.393 [INFO] TestAgent.server.connect: CA provider config updated -46:18.393 [INFO] TestAgent: Requesting shutdown -46:18.393 [INFO] TestAgent.server: shutting down server -46:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.393 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.393 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.393 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.393 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.396 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.402Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.402Z [INFO] agent.router.manager: shutting down -46:18.403 [INFO] TestAgent: consul server down -46:18.403 [INFO] TestAgent: shutdown complete -46:18.403 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11610 network=tcp -46:18.403 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11610 network=udp -46:18.403 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11653 network=tcp -46:18.903 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.903 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCAConfig (5.03s) - --- PASS: TestConnectCAConfig/basic (0.85s) - --- PASS: TestConnectCAConfig/basic_with_IntermediateCertTTL (0.74s) - --- PASS: TestConnectCAConfig/force_without_cross_sign_CamelCase (0.84s) - --- PASS: TestConnectCAConfig/force_without_cross_sign_snake_case (0.82s) - --- PASS: TestConnectCAConfig/setting_state_fails (0.82s) - --- PASS: TestConnectCAConfig/updating_config_with_same_state (0.96s) -=== CONT TestAgent_Host -2022-11-07T22:46:18.126Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.136Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.171 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7a80d35b-8b80-275b-ab69-83a8fdd4747c Address:127.0.0.1:11629}]" -46:18.171 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11629 [Follower]" leader= -46:18.172 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c.dc1 127.0.0.1 -46:18.172 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c 127.0.0.1 -2022-11-07T22:46:18.172Z [INFO] agent.router: Initializing LAN area manager -46:18.172 [INFO] TestAgent.server: Adding LAN server: server="Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c (Addr: tcp/127.0.0.1:11629) (DC: dc1)" -46:18.172 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c.dc1 area=wan -46:18.172 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11673 network=udp -46:18.172 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11673 network=tcp -46:18.173 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11625 network=tcp -46:18.173 [INFO] TestAgent: started state syncer -46:18.173 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11658 network=tcp -46:18.223 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.223 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11629 [Candidate]" term=2 -46:18.231 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.231 [DEBUG] TestAgent.server.raft: vote granted: from=7a80d35b-8b80-275b-ab69-83a8fdd4747c term=2 tally=1 -46:18.231 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.231 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11629 [Leader]" -46:18.231 [INFO] TestAgent.server: cluster leadership acquired -46:18.231 [INFO] TestAgent.server: New leader elected: payload=Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c -46:18.237 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11629 -46:18.251 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.272 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.272 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.272 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.272 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.272 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.272 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c -46:18.272 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7a80d35b-8b80-275b-ab69-83a8fdd4747c -46:18.287 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.406 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.409 [INFO] TestAgent: Synced node info -46:18.409 [DEBUG] TestAgent: Node info in sync -46:18.426 [INFO] TestAgent: Requesting shutdown -46:18.426 [INFO] TestAgent.server: shutting down server -46:18.426 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.426 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.426 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.426 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.426 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.426 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.426 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.426 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.426 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.426 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.429 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.434Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.434Z [INFO] agent.router.manager: shutting down -46:18.434 [INFO] TestAgent: consul server down -46:18.434 [INFO] TestAgent: shutdown complete -46:18.434 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11673 network=tcp -46:18.434 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11673 network=udp -46:18.434 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11625 network=tcp -46:18.934 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.934 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_purgeCheckState (0.82s) -=== CONT TestAgent_checkStateSnapshot -2022-11-07T22:46:18.156Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.166Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.198 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:78969ca0-96e2-7188-4ef5-231c21309229 Address:127.0.0.1:11580}]" -46:18.198 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11580 [Follower]" leader= -46:18.198 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-78969ca0-96e2-7188-4ef5-231c21309229.dc1 127.0.0.1 -46:18.198 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-78969ca0-96e2-7188-4ef5-231c21309229 127.0.0.1 -2022-11-07T22:46:18.198Z [INFO] agent.router: Initializing LAN area manager -46:18.199 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-78969ca0-96e2-7188-4ef5-231c21309229.dc1 area=wan -46:18.199 [INFO] TestAgent.server: Adding LAN server: server="Node-78969ca0-96e2-7188-4ef5-231c21309229 (Addr: tcp/127.0.0.1:11580) (DC: dc1)" -46:18.199 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11645 network=udp -46:18.199 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11645 network=tcp -46:18.199 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11576 network=tcp -46:18.199 [INFO] TestAgent: started state syncer -46:18.199 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11665 network=tcp -46:18.260 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.260 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11580 [Candidate]" term=2 -46:18.271 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.271 [DEBUG] TestAgent.server.raft: vote granted: from=78969ca0-96e2-7188-4ef5-231c21309229 term=2 tally=1 -46:18.271 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.271 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11580 [Leader]" -46:18.271 [INFO] TestAgent.server: cluster leadership acquired -46:18.271 [INFO] TestAgent.server: New leader elected: payload=Node-78969ca0-96e2-7188-4ef5-231c21309229 -46:18.279 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11580 -46:18.295 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.318 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.318 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.318 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.318 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.318 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.318 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78969ca0-96e2-7188-4ef5-231c21309229 -46:18.318 [INFO] TestAgent.server: member joined, marking health alive: member=Node-78969ca0-96e2-7188-4ef5-231c21309229 -46:18.332 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.547 [INFO] TestAgent: Requesting shutdown -46:18.547 [INFO] TestAgent.server: shutting down server -46:18.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.548 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.548 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.548 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.548 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.548 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.548 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.548 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.548 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.548 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.561 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.566Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.566Z [INFO] agent.router.manager: shutting down -46:18.567 [INFO] TestAgent: consul server down -46:18.567 [INFO] TestAgent: shutdown complete -46:18.567 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11645 network=tcp -46:18.567 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11645 network=udp -46:18.567 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11576 network=tcp -46:19.067 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.067 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_allow (0.93s) -=== CONT TestAgentConnectAuthorize_defaultAllow -2022-11-07T22:46:18.184Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.184Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.193Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.193Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.220 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76349493-84d4-641a-5f5f-cd34b79d1847 Address:127.0.0.1:11678}]" -46:18.220 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11678 [Follower]" leader= -46:18.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76349493-84d4-641a-5f5f-cd34b79d1847.dc1 127.0.0.1 -46:18.221 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76349493-84d4-641a-5f5f-cd34b79d1847 127.0.0.1 -2022-11-07T22:46:18.221Z [INFO] agent.router: Initializing LAN area manager -46:18.221 [INFO] TestAgent.server: Adding LAN server: server="Node-76349493-84d4-641a-5f5f-cd34b79d1847 (Addr: tcp/127.0.0.1:11678) (DC: dc1)" -46:18.221 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76349493-84d4-641a-5f5f-cd34b79d1847.dc1 area=wan -46:18.221 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11680 network=udp -46:18.221 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11680 network=tcp -46:18.222 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11674 network=tcp -46:18.222 [INFO] TestAgent: started state syncer -46:18.222 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11644 network=tcp -46:18.272 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:18.283 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.283 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11678 [Candidate]" term=2 -46:18.295 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.295 [DEBUG] TestAgent.server.raft: vote granted: from=76349493-84d4-641a-5f5f-cd34b79d1847 term=2 tally=1 -46:18.295 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11678 [Leader]" -46:18.295 [INFO] TestAgent.server: cluster leadership acquired -46:18.295 [INFO] TestAgent.server: New leader elected: payload=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.302 [INFO] TestAgent.server: initializing acls -46:18.307 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:18.307 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:18.312 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:18.320 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:18.320 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:18.320 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:18.320 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.320 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-76349493-84d4-641a-5f5f-cd34b79d1847.dc1 -46:18.321 [INFO] TestAgent.server: Updating LAN server: server="Node-76349493-84d4-641a-5f5f-cd34b79d1847 (Addr: tcp/127.0.0.1:11678) (DC: dc1)" -46:18.321 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-76349493-84d4-641a-5f5f-cd34b79d1847.dc1 area=wan -46:18.331 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.350 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.350 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.350 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.350 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.350 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.351 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.351 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.355 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.355 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.509 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.509 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-76349493-84d4-641a-5f5f-cd34b79d1847 -46:18.518 [INFO] TestAgent: Updated agent's ACL token: token=agent -46:18.559 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -46:18.570 [INFO] TestAgent: Synced node info -46:18.570 [DEBUG] TestAgent: Node info in sync -46:18.576 [INFO] TestAgent: Requesting shutdown -46:18.576 [INFO] TestAgent.server: shutting down server -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.576 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.576 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.576 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.576 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.582 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.586Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.586Z [INFO] agent.router.manager: shutting down -46:18.586 [INFO] TestAgent: consul server down -46:18.586 [INFO] TestAgent: shutdown complete -46:18.586 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11680 network=tcp -46:18.586 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11680 network=udp -46:18.586 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11674 network=tcp -46:19.086 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.086 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:55.169Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.169Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:55.226Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:55.226Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:55.328 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 Address:127.0.0.1:30080}]" +31:55.329 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 127.0.0.1 +31:55.329 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 127.0.0.1 +2021-10-05T16:31:55.329Z [INFO] agent.router: Initializing LAN area manager +31:55.329 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30152 network=udp +31:55.329 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30080 [Follower]" leader= +31:55.330 [INFO] TestAgent.server: Adding LAN server: server="Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 (Addr: tcp/127.0.0.1:30080) (DC: dc1)" +31:55.330 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 area=wan +31:55.330 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30152 network=tcp +31:55.330 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30153 network=tcp +31:55.330 [INFO] TestAgent: started state syncer +31:55.330 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30050 network=tcp +31:55.380 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:55.388 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:55.388 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30080 [Candidate]" term=2 +31:55.457 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:55.457 [DEBUG] TestAgent.server.raft: vote granted: from=a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 term=2 tally=1 +31:55.457 [INFO] TestAgent.server.raft: election won: tally=1 +31:55.457 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30080 [Leader]" +31:55.458 [INFO] TestAgent.server: cluster leadership acquired +31:55.458 [INFO] TestAgent.server: New leader elected: payload=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.480 [INFO] TestAgent.server: initializing acls +31:55.490 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:55.490 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.490 [INFO] TestAgent.server: initializing acls +31:55.490 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:55.507 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:55.534 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:55.552 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.552 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:55.552 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:55.552 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:55.552 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.552 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 +31:55.552 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 area=wan +31:55.552 [INFO] TestAgent.server: Updating LAN server: server="Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 (Addr: tcp/127.0.0.1:30080) (DC: dc1)" +31:55.568 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:55.568 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.568 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 +31:55.572 [INFO] TestAgent.server: Updating LAN server: server="Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 (Addr: tcp/127.0.0.1:30080) (DC: dc1)" +31:55.572 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7.dc1 area=wan +31:55.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:55.650 [WARN] TestAgent: Node info update blocked by ACLs: node=a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 accessorID=00000000-0000-0000-0000-000000000002 +31:55.658 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:55.658 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:55.658 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:55.658 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:55.658 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:55.658 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.658 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.666 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.666 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.678 [INFO] TestAgent.server: federation state anti-entropy synced +31:55.689 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a0b35e84-c5e9-ef76-efe2-27bb7699e8d7 +31:55.709 [INFO] TestAgent: Updated agent's ACL token: token=agent +31:56.860 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service="{consul {}}" +31:56.860 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:56.880 [INFO] TestAgent: Synced node info +31:56.880 [DEBUG] TestAgent: Node info in sync +31:56.896 [INFO] TestAgent: Requesting shutdown +31:56.896 [INFO] TestAgent.server: shutting down server +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:56.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:56.896 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:56.896 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:56.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:56.902 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:56.903Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:56.903Z [INFO] agent.router.manager: shutting down +31:56.903 [INFO] TestAgent: consul server down +31:56.903 [INFO] TestAgent: shutdown complete +31:56.903 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30152 network=tcp +31:56.903 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30152 network=udp +31:56.903 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30153 network=tcp +31:57.404 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.404 [INFO] TestAgent: Endpoints down === RUN TestAgent_TokenTriggersFullSync/acl_token -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:18.565Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.581Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.614 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b085094-e66b-ab9e-e44c-56248c331d9c Address:127.0.0.1:11671}]" -46:18.614 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11671 [Follower]" leader= -46:18.614 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b085094-e66b-ab9e-e44c-56248c331d9c.dc1 127.0.0.1 -46:18.615 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b085094-e66b-ab9e-e44c-56248c331d9c 127.0.0.1 -2022-11-07T22:46:18.615Z [INFO] agent.router: Initializing LAN area manager -46:18.615 [INFO] TestAgent.server: Adding LAN server: server="Node-2b085094-e66b-ab9e-e44c-56248c331d9c (Addr: tcp/127.0.0.1:11671) (DC: dc1)" -46:18.615 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b085094-e66b-ab9e-e44c-56248c331d9c.dc1 area=wan -46:18.615 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11666 network=udp -46:18.615 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11666 network=tcp -46:18.615 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11667 network=tcp -46:18.615 [INFO] TestAgent: started state syncer -46:18.615 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11588 network=tcp -46:18.665 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.665 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11671 [Candidate]" term=2 -46:18.675 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.675 [DEBUG] TestAgent.server.raft: vote granted: from=2b085094-e66b-ab9e-e44c-56248c331d9c term=2 tally=1 -46:18.675 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.675 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11671 [Leader]" -46:18.675 [INFO] TestAgent.server: cluster leadership acquired -46:18.676 [INFO] TestAgent.server: New leader elected: payload=Node-2b085094-e66b-ab9e-e44c-56248c331d9c -46:18.684 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11671 -46:18.706 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.730 [INFO] TestAgent: Synced node info -46:18.730 [DEBUG] TestAgent: Node info in sync -46:18.750 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.750 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.750 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.750 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.750 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.750 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b085094-e66b-ab9e-e44c-56248c331d9c -46:18.750 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b085094-e66b-ab9e-e44c-56248c331d9c -46:18.760 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.784 [DEBUG] TestAgent: check state expired, not restoring: check=check1 -46:18.784 [INFO] TestAgent: Requesting shutdown -46:18.784 [INFO] TestAgent.server: shutting down server -46:18.784 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.784 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.784 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.784 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.784 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.790 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.798Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.798Z [INFO] agent.router.manager: shutting down -46:18.798 [INFO] TestAgent: consul server down -46:18.798 [INFO] TestAgent: shutdown complete -46:18.798 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11666 network=tcp -46:18.798 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11666 network=udp -46:18.798 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11667 network=tcp -46:19.298 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.298 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_loadCheckState (0.76s) -=== CONT TestAgentConnectCALeafCert_secondaryDC_good +=== RUN TestAgent_PassCheck_ACLDeny/no_token +=== RUN TestAgent_PassCheck_ACLDeny/root_token [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:18.535Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.543Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.574 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6dcc890b-c57b-ac73-04ff-65aef9e2ec49 Address:127.0.0.1:11573}]" -46:18.574 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49.dc1 127.0.0.1 -46:18.575 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49 127.0.0.1 -2022-11-07T22:46:18.575Z [INFO] agent.router: Initializing LAN area manager -46:18.575 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11631 network=udp -46:18.575 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11573 [Follower]" leader= -46:18.575 [INFO] TestAgent.server: Adding LAN server: server="Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49 (Addr: tcp/127.0.0.1:11573) (DC: dc1)" -46:18.576 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49.dc1 area=wan -46:18.576 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11631 network=tcp -46:18.576 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11569 network=tcp -46:18.576 [INFO] TestAgent: started state syncer -46:18.576 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11686 network=tcp -46:18.621 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.621 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11573 [Candidate]" term=2 -46:18.643 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.643 [DEBUG] TestAgent.server.raft: vote granted: from=6dcc890b-c57b-ac73-04ff-65aef9e2ec49 term=2 tally=1 -46:18.643 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.643 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11573 [Leader]" -46:18.643 [INFO] TestAgent.server: cluster leadership acquired -46:18.643 [INFO] TestAgent.server: New leader elected: payload=Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49 -46:18.651 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11573 -46:18.667 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.707 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.707 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.707 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.707 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.707 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.707 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49 -46:18.707 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6dcc890b-c57b-ac73-04ff-65aef9e2ec49 -46:18.728 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.791 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.797 [INFO] TestAgent: Synced node info -46:18.829 [INFO] TestAgent: Requesting shutdown -46:18.829 [INFO] TestAgent.server: shutting down server -46:18.829 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.829 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.829 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.829 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.829 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.829 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.829 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.829 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.829 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.829 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.835 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.842Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.842Z [INFO] agent.router.manager: shutting down -46:18.842 [INFO] TestAgent: consul server down -46:18.842 [INFO] TestAgent: shutdown complete -46:18.842 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11631 network=tcp -46:18.842 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11631 network=udp -46:18.843 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11569 network=tcp -46:19.343 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.343 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_idNotService (0.82s) -=== CONT TestAgentConnectAuthorize_defaultDeny -2022-11-07T22:46:18.630Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.638Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.675 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae1775f3-182b-a28a-534b-48def7124f13 Address:127.0.0.1:11699}]" -46:18.675 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11699 [Follower]" leader= -46:18.675 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ae1775f3-182b-a28a-534b-48def7124f13.dc1 127.0.0.1 -46:18.676 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ae1775f3-182b-a28a-534b-48def7124f13 127.0.0.1 -2022-11-07T22:46:18.676Z [INFO] agent.router: Initializing LAN area manager -46:18.676 [INFO] TestAgent.server: Adding LAN server: server="Node-ae1775f3-182b-a28a-534b-48def7124f13 (Addr: tcp/127.0.0.1:11699) (DC: dc1)" -46:18.676 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ae1775f3-182b-a28a-534b-48def7124f13.dc1 area=wan -46:18.676 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11694 network=udp -46:18.676 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11694 network=tcp -46:18.677 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11695 network=tcp -46:18.677 [INFO] TestAgent: started state syncer -46:18.677 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11679 network=tcp -46:18.729 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.729 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11699 [Candidate]" term=2 -46:18.741 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.741 [DEBUG] TestAgent.server.raft: vote granted: from=ae1775f3-182b-a28a-534b-48def7124f13 term=2 tally=1 -46:18.741 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.741 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11699 [Leader]" -46:18.741 [INFO] TestAgent.server: cluster leadership acquired -46:18.741 [INFO] TestAgent.server: New leader elected: payload=Node-ae1775f3-182b-a28a-534b-48def7124f13 -46:18.754 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11699 -46:18.777 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.813 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.813 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.813 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.813 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.813 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.813 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae1775f3-182b-a28a-534b-48def7124f13 -46:18.813 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ae1775f3-182b-a28a-534b-48def7124f13 -46:18.832 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.872 [INFO] TestAgent: Requesting shutdown -46:18.872 [INFO] TestAgent.server: shutting down server -46:18.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.872 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.872 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.872 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.872 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.872 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.872 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.872 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.872 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.878 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.885Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.885Z [INFO] agent.router.manager: shutting down -46:18.885 [INFO] TestAgent: consul server down -46:18.885 [INFO] TestAgent: shutdown complete -46:18.885 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11694 network=tcp -46:18.885 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11694 network=udp -46:18.885 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11695 network=tcp -46:19.386 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.386 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:18.673Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.684Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.724 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 Address:127.0.0.1:11692}]" -46:18.724 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11692 [Follower]" leader= -46:18.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2.dc1 127.0.0.1 -46:18.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 127.0.0.1 -2022-11-07T22:46:18.725Z [INFO] agent.router: Initializing LAN area manager -46:18.725 [INFO] TestAgent.server: Adding LAN server: server="Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 (Addr: tcp/127.0.0.1:11692) (DC: dc1)" -46:18.725 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2.dc1 area=wan -46:18.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11715 network=udp -46:18.725 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11715 network=tcp -46:18.726 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11688 network=tcp -46:18.726 [INFO] TestAgent: started state syncer -46:18.726 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11700 network=tcp -46:18.773 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.773 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11692 [Candidate]" term=2 -46:18.786 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.786 [DEBUG] TestAgent.server.raft: vote granted: from=bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 term=2 tally=1 -46:18.786 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.786 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11692 [Leader]" -46:18.786 [INFO] TestAgent.server: cluster leadership acquired -46:18.786 [INFO] TestAgent.server: New leader elected: payload=Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 -46:18.795 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11692 -46:18.815 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.847 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.847 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.847 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.847 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.847 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.847 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 -46:18.847 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc0290b2-fc48-69dd-6d8f-cfe2efa34cf2 -46:18.855 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.872 [INFO] TestAgent: Requesting shutdown -46:18.873 [INFO] TestAgent.server: shutting down server -46:18.873 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.873 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.873 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.873 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.873 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.873 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.873 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.873 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.873 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.873 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.873 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.878 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.885Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.885Z [INFO] agent.router.manager: shutting down -46:18.885 [INFO] TestAgent: consul server down -46:18.885 [INFO] TestAgent: shutdown complete -46:18.885 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11715 network=tcp -46:18.886 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11715 network=udp -46:18.886 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11688 network=tcp -46:19.387 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.387 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_ExposeConfig (0.72s) -=== CONT TestAgent_NodeMaintenanceMode ---- PASS: TestAgent_persistCheckState (0.77s) -=== CONT TestAgentConnectAuthorize_serviceWrite -2022-11-07T22:46:18.630Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.640Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.683 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f55272c-4b6f-7bfb-a77b-60d281491ff2 Address:127.0.0.1:11685}]" -46:18.683 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11685 [Follower]" leader= -46:18.685 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2.dc1 127.0.0.1 -46:18.685 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2 127.0.0.1 -2022-11-07T22:46:18.685Z [INFO] agent.router: Initializing LAN area manager -46:18.685 [INFO] TestAgent.server: Adding LAN server: server="Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2 (Addr: tcp/127.0.0.1:11685) (DC: dc1)" -46:18.685 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2.dc1 area=wan -46:18.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11659 network=udp -46:18.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11659 network=tcp -46:18.686 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11681 network=tcp -46:18.686 [INFO] TestAgent: started state syncer -46:18.686 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11693 network=tcp -46:18.745 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.745 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11685 [Candidate]" term=2 -46:18.762 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.762 [DEBUG] TestAgent.server.raft: vote granted: from=0f55272c-4b6f-7bfb-a77b-60d281491ff2 term=2 tally=1 -46:18.762 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.762 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11685 [Leader]" -46:18.762 [INFO] TestAgent.server: cluster leadership acquired -46:18.762 [INFO] TestAgent.server: New leader elected: payload=Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2 -46:18.770 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11685 -46:18.788 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.822 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.822 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.822 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.822 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.822 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.822 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2 -46:18.822 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0f55272c-4b6f-7bfb-a77b-60d281491ff2 -46:18.840 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.867 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.870 [INFO] TestAgent: Synced node info -46:18.913 [INFO] TestAgent: Requesting shutdown -46:18.913 [INFO] TestAgent.server: shutting down server -46:18.913 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.913 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.913 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.913 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.913 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.913 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.913 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.913 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.913 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.913 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.921 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.928Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.928Z [INFO] agent.router.manager: shutting down -46:18.928 [INFO] TestAgent: consul server down -46:18.928 [INFO] TestAgent: shutdown complete -46:18.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11659 network=tcp -46:18.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11659 network=udp -46:18.928 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11681 network=tcp -46:19.428 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.428 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_noTarget (0.81s) -=== CONT TestAgent_PurgeCheckOnDuplicate -2022-11-07T22:46:18.587Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.601Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.642 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bba01198-b550-6f89-7e4c-f31fee271168 Address:127.0.0.1:11664}]" -46:18.642 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11664 [Follower]" leader= -46:18.643 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bba01198-b550-6f89-7e4c-f31fee271168.dc1 127.0.0.1 -46:18.643 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bba01198-b550-6f89-7e4c-f31fee271168 127.0.0.1 -2022-11-07T22:46:18.643Z [INFO] agent.router: Initializing LAN area manager -46:18.643 [INFO] TestAgent.server: Adding LAN server: server="Node-bba01198-b550-6f89-7e4c-f31fee271168 (Addr: tcp/127.0.0.1:11664) (DC: dc1)" -46:18.643 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bba01198-b550-6f89-7e4c-f31fee271168.dc1 area=wan -46:18.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11652 network=udp -46:18.643 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11652 network=tcp -46:18.644 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11660 network=tcp -46:18.644 [INFO] TestAgent: started state syncer -46:18.644 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11672 network=tcp -46:18.692 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.692 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11664 [Candidate]" term=2 -46:18.706 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.706 [DEBUG] TestAgent.server.raft: vote granted: from=bba01198-b550-6f89-7e4c-f31fee271168 term=2 tally=1 -46:18.706 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.706 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11664 [Leader]" -46:18.706 [INFO] TestAgent.server: cluster leadership acquired -46:18.706 [INFO] TestAgent.server: New leader elected: payload=Node-bba01198-b550-6f89-7e4c-f31fee271168 -46:18.713 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11664 -46:18.732 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.766 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.766 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.766 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.766 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.766 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.766 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bba01198-b550-6f89-7e4c-f31fee271168 -46:18.766 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bba01198-b550-6f89-7e4c-f31fee271168 -46:18.776 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.954 [INFO] TestAgent: Requesting shutdown -46:18.955 [INFO] TestAgent.server: shutting down server -46:18.955 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.955 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.955 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.955 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.955 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.955 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.955 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.955 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.956 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.956 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.956 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.970 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.975Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.975Z [INFO] agent.router.manager: shutting down -46:18.975 [INFO] TestAgent: consul server down -46:18.975 [INFO] TestAgent: shutdown complete -46:18.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11652 network=tcp -46:18.975 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11652 network=udp -46:18.976 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11660 network=tcp -46:19.476 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.476 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_idInvalidFormat (0.90s) -=== CONT TestAgent_AddCheck_restoresSnapshot -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:18.649Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.711 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 Address:127.0.0.1:11720}]" -46:18.711 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11720 [Follower]" leader= -46:18.711 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6.dc1 127.0.0.1 -46:18.712 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 127.0.0.1 -2022-11-07T22:46:18.712Z [INFO] agent.router: Initializing LAN area manager -46:18.712 [INFO] TestAgent.server: Adding LAN server: server="Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 (Addr: tcp/127.0.0.1:11720) (DC: dc1)" -46:18.712 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6.dc1 area=wan -46:18.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11701 network=udp -46:18.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11701 network=tcp -46:18.712 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11716 network=tcp -46:18.712 [INFO] TestAgent: started state syncer -46:18.712 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11728 network=tcp -46:18.772 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.772 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11720 [Candidate]" term=2 -46:18.790 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.790 [DEBUG] TestAgent.server.raft: vote granted: from=ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 term=2 tally=1 -46:18.790 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.790 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11720 [Leader]" -46:18.790 [INFO] TestAgent.server: cluster leadership acquired -46:18.790 [INFO] TestAgent.server: New leader elected: payload=Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 -46:18.799 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11720 -46:18.825 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.852 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.852 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.852 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.852 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.852 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.852 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 -46:18.852 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ae7951a0-ba56-c4ab-f2f4-fde8cddb0cf6 -46:18.868 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.945 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.951 [INFO] TestAgent: Synced node info -46:18.951 [DEBUG] TestAgent: Node info in sync -46:19.083 [WARN] TestAgent: Failed to restore check: check=service:redis error="ServiceID "nope" does not exist" -46:19.083 [DEBUG] TestAgent: restored health check from file: check=service:redis file=/tmp/consul-test/TestAgent_loadChecks_checkFails-TestAgent-agent798167647/checks/60a2ef12de014a05ecdc850d9aab46da -46:19.083 [INFO] TestAgent: Requesting shutdown -46:19.083 [INFO] TestAgent.server: shutting down server -46:19.083 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.083 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.083 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.083 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.083 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.084 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.084 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.084 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.084 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.089 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.096Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.096Z [INFO] agent.router.manager: shutting down -46:19.096 [INFO] TestAgent: consul server down -46:19.096 [INFO] TestAgent: shutdown complete -46:19.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11701 network=tcp -46:19.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11701 network=udp -46:19.096 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11716 network=tcp -46:19.596 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.597 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_loadChecks_checkFails (0.96s) -=== CONT TestAgent_Service_MaintenanceMode -2022-11-07T22:46:18.847Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.861Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.886 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:032b3029-8764-bb01-547e-dcbc001a8061 Address:127.0.0.1:11797}]" -46:18.886 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11797 [Follower]" leader= -46:18.887 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-032b3029-8764-bb01-547e-dcbc001a8061.dc1 127.0.0.1 -46:18.887 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-032b3029-8764-bb01-547e-dcbc001a8061 127.0.0.1 -2022-11-07T22:46:18.887Z [INFO] agent.router: Initializing LAN area manager -46:18.887 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11750 network=udp -46:18.887 [INFO] TestAgent.server: Adding LAN server: server="Node-032b3029-8764-bb01-547e-dcbc001a8061 (Addr: tcp/127.0.0.1:11797) (DC: dc1)" -46:18.887 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-032b3029-8764-bb01-547e-dcbc001a8061.dc1 area=wan -46:18.887 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11750 network=tcp -46:18.888 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11793 network=tcp -46:18.888 [INFO] TestAgent: started state syncer -46:18.888 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11742 network=tcp -46:18.928 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.928 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11797 [Candidate]" term=2 -46:18.939 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.939 [DEBUG] TestAgent.server.raft: vote granted: from=032b3029-8764-bb01-547e-dcbc001a8061 term=2 tally=1 -46:18.939 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.939 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11797 [Leader]" -46:18.940 [INFO] TestAgent.server: cluster leadership acquired -46:18.940 [INFO] TestAgent.server: New leader elected: payload=Node-032b3029-8764-bb01-547e-dcbc001a8061 -46:18.950 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11797 -46:18.977 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.004 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.004 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.004 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.004 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.004 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.004 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-032b3029-8764-bb01-547e-dcbc001a8061 -46:19.004 [INFO] TestAgent.server: member joined, marking health alive: member=Node-032b3029-8764-bb01-547e-dcbc001a8061 -46:19.019 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.171 [INFO] TestAgent: Requesting shutdown -46:19.171 [INFO] TestAgent.server: shutting down server -46:19.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.171 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.171 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.171 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.171 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.171 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.171 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.171 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.177 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.184Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.184Z [INFO] agent.router.manager: shutting down -46:19.184 [INFO] TestAgent: consul server down -46:19.184 [INFO] TestAgent: shutdown complete -46:19.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11750 network=tcp -46:19.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11750 network=udp -46:19.184 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11793 network=tcp -46:19.685 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.685 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:17.966Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.980Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.016 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ccfe2f26-d660-f782-b39f-1af83f47c567 Address:127.0.0.1:11587}]" -46:18.016 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11587 [Follower]" leader= -46:18.017 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ccfe2f26-d660-f782-b39f-1af83f47c567.dc1 127.0.0.1 -46:18.017 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ccfe2f26-d660-f782-b39f-1af83f47c567 127.0.0.1 -2022-11-07T22:46:18.017Z [INFO] agent.router: Initializing LAN area manager -46:18.018 [INFO] TestAgent.server: Adding LAN server: server="Node-ccfe2f26-d660-f782-b39f-1af83f47c567 (Addr: tcp/127.0.0.1:11587) (DC: dc1)" -46:18.018 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ccfe2f26-d660-f782-b39f-1af83f47c567.dc1 area=wan -46:18.018 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11582 network=udp -46:18.018 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11582 network=tcp -46:18.018 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11583 network=tcp -46:18.018 [INFO] TestAgent: started state syncer -46:18.018 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11616 network=tcp -46:18.053 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.053 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11587 [Candidate]" term=2 -46:18.065 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.065 [DEBUG] TestAgent.server.raft: vote granted: from=ccfe2f26-d660-f782-b39f-1af83f47c567 term=2 tally=1 -46:18.065 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.065 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11587 [Leader]" -46:18.066 [INFO] TestAgent.server: cluster leadership acquired -46:18.066 [INFO] TestAgent.server: New leader elected: payload=Node-ccfe2f26-d660-f782-b39f-1af83f47c567 -46:18.073 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11587 -46:18.097 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.126 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.126 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.126 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.126 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.126 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.126 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ccfe2f26-d660-f782-b39f-1af83f47c567 -46:18.126 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ccfe2f26-d660-f782-b39f-1af83f47c567 -46:18.144 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.321 [INFO] TestAgent: Requesting shutdown -46:18.321 [INFO] TestAgent.server: shutting down server -46:18.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:18.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:18.321 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:18.321 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:18.321 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:18.321 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:18.321 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:18.321 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:18.321 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:18.321 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:18.321 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:18.327 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:18.333Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:18.333Z [INFO] agent.router.manager: shutting down -46:18.333 [INFO] TestAgent: consul server down -46:18.333 [INFO] TestAgent: shutdown complete -46:18.333 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11582 network=tcp -46:18.333 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11582 network=udp -46:18.333 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11583 network=tcp -46:18.834 [INFO] TestAgent: Waiting for endpoints to shut down -46:18.834 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_GetCoordinate (1.74s) -=== CONT TestAgent_PurgeCheck -2022-11-07T22:46:18.782Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.782Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.791Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.791Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.836 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:70dfe23e-f037-fc2e-fb8c-45cb49b29067 Address:127.0.0.1:11727}]" -46:18.836 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067.dc1 127.0.0.1 -46:18.837 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11727 [Follower]" leader= -46:18.837 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 127.0.0.1 -2022-11-07T22:46:18.837Z [INFO] agent.router: Initializing LAN area manager -46:18.837 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11687 network=udp -46:18.838 [INFO] TestAgent.server: Adding LAN server: server="Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 (Addr: tcp/127.0.0.1:11727) (DC: dc1)" -46:18.838 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067.dc1 area=wan -46:18.838 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11687 network=tcp -46:18.838 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11723 network=tcp -46:18.838 [INFO] TestAgent: started state syncer -46:18.838 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11735 network=tcp -46:18.888 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:18.890 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.890 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11727 [Candidate]" term=2 -46:18.902 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.902 [DEBUG] TestAgent.server.raft: vote granted: from=70dfe23e-f037-fc2e-fb8c-45cb49b29067 term=2 tally=1 -46:18.902 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.902 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11727 [Leader]" -46:18.902 [INFO] TestAgent.server: cluster leadership acquired -46:18.902 [INFO] TestAgent.server: New leader elected: payload=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:18.908 [INFO] TestAgent.server: initializing acls -46:18.911 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:18.911 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:18.919 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:18.934 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:18.934 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:18.934 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:18.934 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:18.934 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067.dc1 -46:18.934 [INFO] TestAgent.server: Updating LAN server: server="Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 (Addr: tcp/127.0.0.1:11727) (DC: dc1)" -46:18.934 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067.dc1 area=wan -46:18.950 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.980 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.980 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.980 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.980 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.980 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.980 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:18.980 [INFO] TestAgent.server: member joined, marking health alive: member=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:18.988 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:18.992 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.011 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:19.174 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:19.175 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-70dfe23e-f037-fc2e-fb8c-45cb49b29067 -46:19.175 [INFO] TestAgent: Requesting shutdown -46:19.175 [INFO] TestAgent.server: shutting down server -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.175 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.175 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:19.175 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.175 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.180 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.189Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.189Z [INFO] agent.router.manager: shutting down -46:19.189 [INFO] TestAgent: consul server down -46:19.189 [INFO] TestAgent: shutdown complete -46:19.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11687 network=tcp -46:19.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11687 network=udp -46:19.189 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11723 network=tcp -46:19.689 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.689 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_HostBadACL (0.92s) -=== CONT TestAgent_PersistCheck -2022-11-07T22:46:18.913Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.913Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.923Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:18.923Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.962 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 Address:127.0.0.1:11734}]" -46:18.962 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11734 [Follower]" leader= -46:18.963 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03.dc1 127.0.0.1 -46:18.964 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 127.0.0.1 -2022-11-07T22:46:18.964Z [INFO] agent.router: Initializing LAN area manager -46:18.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11708 network=udp -46:18.964 [INFO] TestAgent.server: Adding LAN server: server="Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 (Addr: tcp/127.0.0.1:11734) (DC: dc1)" -46:18.964 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03.dc1 area=wan -46:18.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11708 network=tcp -46:18.965 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11730 network=tcp -46:18.965 [INFO] TestAgent: started state syncer -46:18.965 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11756 network=tcp -46:19.015 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:19.028 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.028 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11734 [Candidate]" term=2 -46:19.037 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.037 [DEBUG] TestAgent.server.raft: vote granted: from=e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 term=2 tally=1 -46:19.037 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.037 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11734 [Leader]" -46:19.037 [INFO] TestAgent.server: cluster leadership acquired -46:19.038 [INFO] TestAgent.server: New leader elected: payload=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 -46:19.044 [INFO] TestAgent.server: initializing acls -46:19.047 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:19.047 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:19.057 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:19.070 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:19.070 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:19.070 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:19.070 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 -46:19.070 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03.dc1 -46:19.070 [INFO] TestAgent.server: Updating LAN server: server="Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 (Addr: tcp/127.0.0.1:11734) (DC: dc1)" -46:19.070 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03.dc1 area=wan -46:19.083 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.102 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.102 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.102 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.102 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.102 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.102 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 -46:19.102 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 -46:19.111 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e7e6dbc9-2b36-1eb2-7d65-0e0ea454bf03 -46:19.114 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:19.119 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.208 [INFO] TestAgent: Requesting shutdown -46:19.208 [INFO] TestAgent.server: shutting down server -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.208 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.208 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.208 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.208 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.215 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.219Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.219Z [INFO] agent.router.manager: shutting down -46:19.219 [INFO] TestAgent: consul server down -46:19.219 [INFO] TestAgent: shutdown complete -46:19.219 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11708 network=tcp -46:19.219 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11708 network=udp -46:19.219 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11730 network=tcp -46:19.719 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.719 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Host (0.82s) -=== CONT TestAgent_unloadChecks -2022-11-07T22:46:19.077Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.077Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.090Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.090Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.121 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd315a77-7bf0-038c-5de2-3e6b0f54252d Address:127.0.0.1:11776}]" -46:19.121 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11776 [Follower]" leader= -46:19.122 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d.dc1 127.0.0.1 -46:19.122 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d 127.0.0.1 -2022-11-07T22:46:19.122Z [INFO] agent.router: Initializing LAN area manager -46:19.122 [INFO] TestAgent.server: Adding LAN server: server="Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d (Addr: tcp/127.0.0.1:11776) (DC: dc1)" -46:19.122 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d.dc1 area=wan -46:19.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11799 network=udp -46:19.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11799 network=tcp -46:19.123 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11772 network=tcp -46:19.123 [INFO] TestAgent: started state syncer -46:19.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11553 network=tcp -46:19.173 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:19.185 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.185 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11776 [Candidate]" term=2 -46:19.198 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.198 [DEBUG] TestAgent.server.raft: vote granted: from=dd315a77-7bf0-038c-5de2-3e6b0f54252d term=2 tally=1 -46:19.198 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.198 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11776 [Leader]" -46:19.198 [INFO] TestAgent.server: cluster leadership acquired -46:19.198 [INFO] TestAgent.server: New leader elected: payload=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d -46:19.205 [INFO] TestAgent.server: initializing acls -46:19.209 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:19.209 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:19.216 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:19.228 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:19.228 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:19.228 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:19.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d -46:19.229 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d.dc1 -46:19.229 [INFO] TestAgent.server: Updating LAN server: server="Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d (Addr: tcp/127.0.0.1:11776) (DC: dc1)" -46:19.229 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d.dc1 area=wan -46:19.239 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.256 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.256 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.256 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.256 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.256 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.256 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d -46:19.256 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d -46:19.259 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.266 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd315a77-7bf0-038c-5de2-3e6b0f54252d -46:19.274 [INFO] TestAgent: Requesting shutdown -46:19.274 [INFO] TestAgent.server: shutting down server -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.274 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.274 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.274 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.279 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.285Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.285Z [INFO] agent.router.manager: shutting down -46:19.285 [INFO] TestAgent: consul server down -46:19.285 [INFO] TestAgent: shutdown complete -46:19.285 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11799 network=tcp -46:19.285 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11799 network=udp -46:19.285 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11772 network=tcp -46:19.786 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.786 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_defaultAllow (0.72s) -=== CONT TestAgent_updateTTLCheck -2022-11-07T22:46:18.857Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.868Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.899 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 Address:127.0.0.1:11769}]" -46:18.899 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11769 [Follower]" leader= -46:18.900 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9.dc1 127.0.0.1 -46:18.900 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 127.0.0.1 -2022-11-07T22:46:18.900Z [INFO] agent.router: Initializing LAN area manager -46:18.900 [INFO] TestAgent.server: Adding LAN server: server="Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 (Addr: tcp/127.0.0.1:11769) (DC: dc1)" -46:18.900 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9.dc1 area=wan -46:18.900 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11729 network=udp -46:18.900 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11729 network=tcp -46:18.901 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11765 network=tcp -46:18.901 [INFO] TestAgent: started state syncer -46:18.901 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11707 network=tcp -46:18.941 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.941 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11769 [Candidate]" term=2 -46:18.970 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.970 [DEBUG] TestAgent.server.raft: vote granted: from=c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 term=2 tally=1 -46:18.970 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.970 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11769 [Leader]" -46:18.970 [INFO] TestAgent.server: cluster leadership acquired -46:18.970 [INFO] TestAgent.server: New leader elected: payload=Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 -46:18.977 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11769 -46:18.990 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.005 [INFO] TestAgent: Synced node info -46:19.016 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.016 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.016 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.016 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.016 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.016 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 -46:19.016 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c64a88a5-4a49-97ba-677b-3a6f54a8cfe9 -46:19.026 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.278 [INFO] TestAgent: Requesting shutdown -46:19.278 [INFO] TestAgent.server: shutting down server -46:19.278 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.278 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.278 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.278 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.278 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.278 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.278 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.278 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.278 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.278 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.282 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.287Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.287Z [INFO] agent.router.manager: shutting down -46:19.287 [INFO] TestAgent: consul server down -46:19.287 [INFO] TestAgent: shutdown complete -46:19.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11729 network=tcp -46:19.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11729 network=udp -46:19.288 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11765 network=tcp -46:19.788 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.788 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_badBody (0.95s) -=== CONT TestAgent_loadChecks_token -2022-11-07T22:46:18.562Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.581Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.612 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:beae1e2f-6726-a5ce-a7fd-013604b45670 Address:127.0.0.1:11650}]" -46:18.612 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11650 [Follower]" leader= -46:18.612 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-beae1e2f-6726-a5ce-a7fd-013604b45670.dc1 127.0.0.1 -46:18.613 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-beae1e2f-6726-a5ce-a7fd-013604b45670 127.0.0.1 -2022-11-07T22:46:18.613Z [INFO] agent.router: Initializing LAN area manager -46:18.613 [INFO] TestAgent.server: Adding LAN server: server="Node-beae1e2f-6726-a5ce-a7fd-013604b45670 (Addr: tcp/127.0.0.1:11650) (DC: dc1)" -46:18.613 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-beae1e2f-6726-a5ce-a7fd-013604b45670.dc1 area=wan -46:18.613 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11596 network=udp -46:18.613 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11596 network=tcp -46:18.613 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11646 network=tcp -46:18.613 [INFO] TestAgent: started state syncer -46:18.613 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11630 network=tcp -46:18.679 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.679 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11650 [Candidate]" term=2 -46:18.698 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.698 [DEBUG] TestAgent.server.raft: vote granted: from=beae1e2f-6726-a5ce-a7fd-013604b45670 term=2 tally=1 -46:18.698 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.698 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11650 [Leader]" -46:18.698 [INFO] TestAgent.server: cluster leadership acquired -46:18.698 [INFO] TestAgent.server: New leader elected: payload=Node-beae1e2f-6726-a5ce-a7fd-013604b45670 -46:18.711 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11650 -46:18.722 [INFO] TestAgent: Synced node info -46:18.722 [DEBUG] TestAgent: Node info in sync -46:18.745 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.758 [DEBUG] TestAgent: Node info in sync -46:18.767 [INFO] TestAgent: Synced service: service=redis -46:18.783 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.783 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.783 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.783 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.783 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.783 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-beae1e2f-6726-a5ce-a7fd-013604b45670 -46:18.783 [INFO] TestAgent.server: member joined, marking health alive: member=Node-beae1e2f-6726-a5ce-a7fd-013604b45670 -46:18.783 [INFO] TestAgent: Synced check: check=node-check -46:18.783 [DEBUG] TestAgent: Check in sync: check=redis-check -46:18.797 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.284 [DEBUG] TestAgent: Registered node: node="Node-level check" -46:19.284 [DEBUG] TestAgent: Registered node: node="Serf Health Status" -46:19.284 [DEBUG] TestAgent: Registered node: node="Service-level check" -46:19.284 [INFO] TestAgent: Sync in progress: iteration=1 -46:19.284 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.284 [DEBUG] TestAgent: Node info in sync -46:19.284 [DEBUG] TestAgent: Service in sync: service=redis -46:19.284 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.284 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.284 [INFO] TestAgent: Sync in progress: iteration=2 -46:19.284 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.284 [DEBUG] TestAgent: Node info in sync -46:19.284 [DEBUG] TestAgent: Service in sync: service=redis -46:19.284 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.284 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.284 [INFO] TestAgent: Sync in progress: iteration=3 -46:19.284 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.284 [DEBUG] TestAgent: Node info in sync -46:19.284 [DEBUG] TestAgent: Service in sync: service=redis -46:19.284 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.284 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.284 [INFO] TestAgent: Sync in progress: iteration=4 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=5 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=6 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=7 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=8 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=9 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Sync in progress: iteration=10 -46:19.285 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.285 [DEBUG] TestAgent: Node info in sync -46:19.285 [DEBUG] TestAgent: Service in sync: service=redis -46:19.285 [DEBUG] TestAgent: Check in sync: check=redis-check -46:19.285 [DEBUG] TestAgent: Check in sync: check=node-check -46:19.285 [INFO] TestAgent: Requesting shutdown -46:19.285 [INFO] TestAgent.server: shutting down server -46:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.285 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.285 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.285 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.285 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.290 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.293Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.293Z [INFO] agent.router.manager: shutting down -46:19.293 [INFO] TestAgent: consul server down -46:19.293 [INFO] TestAgent: shutdown complete -46:19.293 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11596 network=tcp -46:19.293 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11596 network=udp -46:19.293 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11646 network=tcp -46:19.794 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.794 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_IndexChurn (2.64s) - --- PASS: TestAgent_IndexChurn/no_tags (1.38s) - --- PASS: TestAgent_IndexChurn/with_tags (1.25s) -=== CONT TestAgent_HTTPCheck_TLSSkipVerify -[WARN] freeport: 10 out of 66 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:18.946Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.973Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.006 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a860cba3-7313-ba64-7098-2becb07782ed Address:127.0.0.1:11741}]" -46:19.006 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11741 [Follower]" leader= -46:19.006 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a860cba3-7313-ba64-7098-2becb07782ed.dc1 127.0.0.1 -46:19.007 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a860cba3-7313-ba64-7098-2becb07782ed 127.0.0.1 -2022-11-07T22:46:19.007Z [INFO] agent.router: Initializing LAN area manager -46:19.007 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11743 network=udp -46:19.007 [INFO] TestAgent.server: Adding LAN server: server="Node-a860cba3-7313-ba64-7098-2becb07782ed (Addr: tcp/127.0.0.1:11741) (DC: dc1)" -46:19.007 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a860cba3-7313-ba64-7098-2becb07782ed.dc1 area=wan -46:19.007 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11743 network=tcp -46:19.008 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11737 network=tcp -46:19.008 [INFO] TestAgent: started state syncer -46:19.008 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11812 network=tcp -46:19.056 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.056 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11741 [Candidate]" term=2 -46:19.070 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.070 [DEBUG] TestAgent.server.raft: vote granted: from=a860cba3-7313-ba64-7098-2becb07782ed term=2 tally=1 -46:19.070 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.070 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11741 [Leader]" -46:19.070 [INFO] TestAgent.server: cluster leadership acquired -46:19.070 [INFO] TestAgent.server: New leader elected: payload=Node-a860cba3-7313-ba64-7098-2becb07782ed -46:19.079 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11741 -46:19.095 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.120 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.120 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.120 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.120 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.120 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.120 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a860cba3-7313-ba64-7098-2becb07782ed -46:19.120 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a860cba3-7313-ba64-7098-2becb07782ed -46:19.135 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.146 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.150 [INFO] TestAgent: Synced node info -46:19.425 [DEBUG] TestAgent: removed check: check=service:redis -46:19.425 [DEBUG] TestAgent: restored health check from file: check=service:redis file=/tmp/consul-test/TestAgent_checkStateSnapshot-TestAgent-agent545180973/checks/60a2ef12de014a05ecdc850d9aab46da -46:19.425 [INFO] TestAgent: Requesting shutdown -46:19.425 [INFO] TestAgent.server: shutting down server -46:19.425 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.425 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.425 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.425 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.425 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.425 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.425 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.425 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.425 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.425 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.431 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.439Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.439Z [INFO] agent.router.manager: shutting down -46:19.439 [INFO] TestAgent: consul server down -46:19.439 [INFO] TestAgent: shutdown complete -46:19.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11743 network=tcp -46:19.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11743 network=udp -46:19.439 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11737 network=tcp -46:19.939 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.939 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_checkStateSnapshot (1.03s) -=== CONT TestAgent_DeregisterPersistedSidecarAfterRestart +2021-10-05T16:31:56.698Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.698Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:56.723Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.723Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:56.801 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38d335a2-ac1e-e9f3-8e74-a168b18d575d Address:127.0.0.1:30231}]" +31:56.802 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d.dc1 127.0.0.1 +31:56.804 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30231 [Follower]" leader= +31:56.804 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d 127.0.0.1 +2021-10-05T16:31:56.804Z [INFO] agent.router: Initializing LAN area manager +31:56.804 [INFO] TestAgent.server: Adding LAN server: server="Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d (Addr: tcp/127.0.0.1:30231) (DC: dc1)" +31:56.804 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d.dc1 area=wan +31:56.804 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30247 network=udp +31:56.804 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30247 network=tcp +31:56.805 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30248 network=tcp +31:56.805 [INFO] TestAgent: started state syncer +31:56.805 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30211 network=tcp +31:56.856 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:56.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:56.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30231 [Candidate]" term=2 +31:56.884 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:56.884 [DEBUG] TestAgent.server.raft: vote granted: from=38d335a2-ac1e-e9f3-8e74-a168b18d575d term=2 tally=1 +31:56.884 [INFO] TestAgent.server.raft: election won: tally=1 +31:56.884 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30231 [Leader]" +31:56.884 [INFO] TestAgent.server: cluster leadership acquired +31:56.884 [INFO] TestAgent.server: New leader elected: payload=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:56.892 [INFO] TestAgent.server: initializing acls +31:56.901 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:56.901 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:56.916 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:56.940 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:56.940 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:56.940 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:56.940 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:56.940 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d.dc1 +31:56.941 [INFO] TestAgent.server: Updating LAN server: server="Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d (Addr: tcp/127.0.0.1:30231) (DC: dc1)" +31:56.941 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d.dc1 area=wan +31:56.976 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.017 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.017 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.017 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.017 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.017 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.017 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:57.017 [INFO] TestAgent.server: member joined, marking health alive: member=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:57.032 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.032 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:57.056 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:57.058 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:57.058 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-38d335a2-ac1e-e9f3-8e74-a168b18d575d +31:57.064 [INFO] TestAgent: Synced node info +31:57.064 [DEBUG] TestAgent: Check status updated: check=test status=warning +31:57.064 [DEBUG] TestAgent: Node info in sync +31:57.064 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 +31:57.064 [INFO] TestAgent: Requesting shutdown +31:57.064 [INFO] TestAgent.server: shutting down server +31:57.064 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.064 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.064 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.064 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.064 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:57.065 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:57.065 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.065 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:57.065 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:57.072 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.092Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.092Z [INFO] agent.router.manager: shutting down +31:57.092 [INFO] TestAgent: consul server down +31:57.092 [INFO] TestAgent: shutdown complete +31:57.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30247 network=tcp +31:57.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30247 network=udp +31:57.092 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30248 network=tcp +31:57.592 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.592 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_WarnCheck_ACLDeny (0.91s) + --- PASS: TestAgent_WarnCheck_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_WarnCheck_ACLDeny/root_token (0.00s) +=== CONT TestAgent_RegisterCheck_ACLDeny +=== RUN TestAgent_DeregisterCheckACLDeny/no_token +=== RUN TestAgent_DeregisterCheckACLDeny/root_token [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:19.489Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.498Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.534 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d0572bf6-56ca-c509-dcd8-eab6abffcb69 Address:127.0.0.1:11280}]" -46:19.534 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11280 [Follower]" leader= -46:19.534 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d0572bf6-56ca-c509-dcd8-eab6abffcb69.dc1 127.0.0.1 -46:19.535 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d0572bf6-56ca-c509-dcd8-eab6abffcb69 127.0.0.1 -2022-11-07T22:46:19.535Z [INFO] agent.router: Initializing LAN area manager -46:19.535 [INFO] TestAgent.server: Adding LAN server: server="Node-d0572bf6-56ca-c509-dcd8-eab6abffcb69 (Addr: tcp/127.0.0.1:11280) (DC: dc1)" -46:19.535 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d0572bf6-56ca-c509-dcd8-eab6abffcb69.dc1 area=wan -46:19.535 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11205 network=tcp -46:19.535 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11205 network=udp -46:19.535 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11276 network=tcp -46:19.535 [INFO] TestAgent: started state syncer -46:19.535 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11777 network=tcp -46:19.599 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.599 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11280 [Candidate]" term=2 -46:19.614 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.614 [DEBUG] TestAgent.server.raft: vote granted: from=d0572bf6-56ca-c509-dcd8-eab6abffcb69 term=2 tally=1 -46:19.614 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.614 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11280 [Leader]" -46:19.614 [INFO] TestAgent.server: cluster leadership acquired -46:19.614 [INFO] TestAgent.server: New leader elected: payload=Node-d0572bf6-56ca-c509-dcd8-eab6abffcb69 -46:19.624 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11280 -46:19.631 [INFO] TestAgent: Requesting shutdown -46:19.632 [INFO] TestAgent.server: shutting down server -46:19.632 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.632 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.636 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:19.637 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:19.642Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.642Z [INFO] agent.router.manager: shutting down -46:19.649 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:19.649 [INFO] TestAgent: consul server down -46:19.649 [INFO] TestAgent: shutdown complete -46:19.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11205 network=tcp -46:19.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11205 network=udp -46:19.649 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11276 network=tcp -46:20.149 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.149 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_AddCheck_restoresSnapshot (0.67s) -=== CONT TestAgent_RemoveCheck -2022-11-07T22:46:19.399Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.399Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.409Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.409Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.445 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e2446597-a322-0b37-337f-5a16e08b3a60 Address:127.0.0.1:11811}]" -46:19.445 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11811 [Follower]" leader= -46:19.445 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e2446597-a322-0b37-337f-5a16e08b3a60.dc1 127.0.0.1 -46:19.446 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e2446597-a322-0b37-337f-5a16e08b3a60 127.0.0.1 -2022-11-07T22:46:19.446Z [INFO] agent.router: Initializing LAN area manager -46:19.446 [INFO] TestAgent.server: Adding LAN server: server="Node-e2446597-a322-0b37-337f-5a16e08b3a60 (Addr: tcp/127.0.0.1:11811) (DC: dc1)" -46:19.446 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11785 network=udp -46:19.446 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11785 network=tcp -46:19.446 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e2446597-a322-0b37-337f-5a16e08b3a60.dc1 area=wan -46:19.446 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11807 network=tcp -46:19.446 [INFO] TestAgent: started state syncer -46:19.446 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11851 network=tcp -46:19.494 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.494 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11811 [Candidate]" term=2 -46:19.496 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:19.505 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.505 [DEBUG] TestAgent.server.raft: vote granted: from=e2446597-a322-0b37-337f-5a16e08b3a60 term=2 tally=1 -46:19.505 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.505 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11811 [Leader]" -46:19.505 [INFO] TestAgent.server: cluster leadership acquired -46:19.505 [INFO] TestAgent.server: New leader elected: payload=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.516 [INFO] TestAgent.server: initializing acls -46:19.524 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:19.524 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:19.533 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:19.544 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:19.544 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:19.544 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:19.544 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.544 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e2446597-a322-0b37-337f-5a16e08b3a60.dc1 -46:19.544 [INFO] TestAgent.server: Updating LAN server: server="Node-e2446597-a322-0b37-337f-5a16e08b3a60 (Addr: tcp/127.0.0.1:11811) (DC: dc1)" -46:19.544 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e2446597-a322-0b37-337f-5a16e08b3a60.dc1 area=wan -46:19.554 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.570 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.570 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.570 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.570 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.570 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.570 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.570 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.574 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.579 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.707 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.707 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e2446597-a322-0b37-337f-5a16e08b3a60 -46:19.719 [INFO] TestAgent: Requesting shutdown -46:19.719 [INFO] TestAgent.server: shutting down server -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.719 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.719 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:19.719 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.719 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.719 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.733 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.744Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.744Z [INFO] agent.router.manager: shutting down -46:19.744 [INFO] TestAgent: consul server down -46:19.744 [INFO] TestAgent: shutdown complete -46:19.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11785 network=tcp -46:19.744 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11785 network=udp -46:19.744 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11807 network=tcp -46:20.245 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.245 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_serviceWrite (0.86s) +2021-10-05T16:31:56.974Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:56.974Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.002Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.002Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.034 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:28607d04-31bd-602c-fc15-477c852ba160 Address:127.0.0.1:30192}]" +31:57.035 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-28607d04-31bd-602c-fc15-477c852ba160.dc1 127.0.0.1 +31:57.035 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-28607d04-31bd-602c-fc15-477c852ba160 127.0.0.1 +2021-10-05T16:31:57.035Z [INFO] agent.router: Initializing LAN area manager +31:57.035 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30226 network=udp +31:57.035 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30192 [Follower]" leader= +31:57.036 [INFO] TestAgent.server: Adding LAN server: server="Node-28607d04-31bd-602c-fc15-477c852ba160 (Addr: tcp/127.0.0.1:30192) (DC: dc1)" +31:57.036 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-28607d04-31bd-602c-fc15-477c852ba160.dc1 area=wan +31:57.036 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30226 network=tcp +31:57.036 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30227 network=tcp +31:57.037 [INFO] TestAgent: started state syncer +31:57.037 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30232 network=tcp +31:57.084 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.084 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30192 [Candidate]" term=2 +31:57.088 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:57.119 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.119 [DEBUG] TestAgent.server.raft: vote granted: from=28607d04-31bd-602c-fc15-477c852ba160 term=2 tally=1 +31:57.119 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.119 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30192 [Leader]" +31:57.119 [INFO] TestAgent.server: cluster leadership acquired +31:57.119 [INFO] TestAgent.server: New leader elected: payload=Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.126 [INFO] TestAgent.server: initializing acls +31:57.131 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:57.131 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:57.141 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:57.158 [INFO] TestAgent: Synced node info +31:57.158 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:57.158 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:57.158 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:57.158 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.158 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-28607d04-31bd-602c-fc15-477c852ba160.dc1 +31:57.159 [INFO] TestAgent.server: Updating LAN server: server="Node-28607d04-31bd-602c-fc15-477c852ba160 (Addr: tcp/127.0.0.1:30192) (DC: dc1)" +31:57.159 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-28607d04-31bd-602c-fc15-477c852ba160.dc1 area=wan +31:57.187 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.221 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.221 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.221 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.221 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.221 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.221 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.221 [INFO] TestAgent.server: member joined, marking health alive: member=Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.229 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.237 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.303 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-28607d04-31bd-602c-fc15-477c852ba160 +31:57.303 [INFO] TestAgent: Requesting shutdown +31:57.303 [INFO] TestAgent.server: shutting down server +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.303 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.303 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.303 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:57.303 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.312 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.324Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.324Z [INFO] agent.router.manager: shutting down +31:57.325 [INFO] TestAgent: consul server down +31:57.325 [INFO] TestAgent: shutdown complete +31:57.325 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30226 network=tcp +31:57.325 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30226 network=udp +31:57.325 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30227 network=tcp +31:57.825 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.825 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshot_Options (5.88s) + --- PASS: TestSnapshot_Options/GET (0.83s) + --- PASS: TestSnapshot_Options/GET#01 (1.04s) + --- PASS: TestSnapshot_Options/GET#02 (1.23s) + --- PASS: TestSnapshot_Options/PUT (1.05s) + --- PASS: TestSnapshot_Options/PUT#01 (0.83s) + --- PASS: TestSnapshot_Options/PUT#02 (0.89s) +=== CONT TestAgent_RegisterCheck_BadStatus +2021-10-05T16:31:57.187Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.204Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.239 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97b791b9-fe6f-1c82-765c-2bb2321f36c3 Address:127.0.0.1:30259}]" +31:57.239 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30259 [Follower]" leader= +31:57.240 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3.dc1 127.0.0.1 +31:57.241 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3 127.0.0.1 +2021-10-05T16:31:57.241Z [INFO] agent.router: Initializing LAN area manager +31:57.241 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30275 network=udp +31:57.241 [INFO] TestAgent.server: Adding LAN server: server="Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3 (Addr: tcp/127.0.0.1:30259) (DC: dc1)" +31:57.241 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3.dc1 area=wan +31:57.241 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30275 network=tcp +31:57.242 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30276 network=tcp +31:57.242 [INFO] TestAgent: started state syncer +31:57.242 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30193 network=tcp +31:57.280 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.280 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30259 [Candidate]" term=2 +31:57.311 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.311 [DEBUG] TestAgent.server.raft: vote granted: from=97b791b9-fe6f-1c82-765c-2bb2321f36c3 term=2 tally=1 +31:57.311 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.311 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30259 [Leader]" +31:57.312 [INFO] TestAgent.server: cluster leadership acquired +31:57.312 [INFO] TestAgent.server: New leader elected: payload=Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3 +31:57.324 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30259 +31:57.344 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.398 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.398 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.398 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.398 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.398 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.398 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3 +31:57.398 [INFO] TestAgent.server: member joined, marking health alive: member=Node-97b791b9-fe6f-1c82-765c-2bb2321f36c3 +31:57.425 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.425 [DEBUG] TestAgent: Check status updated: check=test status=passing +31:57.433 [INFO] TestAgent: Synced node info +31:57.445 [INFO] TestAgent: Synced check: check=test +31:57.445 [INFO] TestAgent: Requesting shutdown +31:57.445 [INFO] TestAgent.server: shutting down server +31:57.445 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.445 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.445 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.445 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.445 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.445 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.445 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.445 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:57.445 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.445 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.445 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.454 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.459Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.460Z [INFO] agent.router.manager: shutting down +31:57.460 [INFO] TestAgent: consul server down +31:57.460 [INFO] TestAgent: shutdown complete +31:57.460 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30275 network=tcp +31:57.460 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30275 network=udp +31:57.460 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30276 network=tcp +31:57.961 [INFO] TestAgent: Waiting for endpoints to shut down +31:57.961 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_PassCheck (0.79s) +=== CONT TestAgent_RegisterCheck_Passing +2021-10-05T16:31:57.227Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.244Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.316 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:733e3b1b-33ff-94e2-2859-c143ee57b607 Address:127.0.0.1:30115}]" +31:57.316 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30115 [Follower]" leader= +31:57.317 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-733e3b1b-33ff-94e2-2859-c143ee57b607.dc1 127.0.0.1 +31:57.317 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-733e3b1b-33ff-94e2-2859-c143ee57b607 127.0.0.1 +2021-10-05T16:31:57.317Z [INFO] agent.router: Initializing LAN area manager +31:57.317 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30261 network=udp +31:57.317 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-733e3b1b-33ff-94e2-2859-c143ee57b607.dc1 area=wan +31:57.317 [INFO] TestAgent.server: Adding LAN server: server="Node-733e3b1b-33ff-94e2-2859-c143ee57b607 (Addr: tcp/127.0.0.1:30115) (DC: dc1)" +31:57.318 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30261 network=tcp +31:57.318 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30262 network=tcp +31:57.318 [INFO] TestAgent: started state syncer +31:57.318 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30116 network=tcp +31:57.356 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.356 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30115 [Candidate]" term=2 +31:57.390 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.390 [DEBUG] TestAgent.server.raft: vote granted: from=733e3b1b-33ff-94e2-2859-c143ee57b607 term=2 tally=1 +31:57.390 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.390 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30115 [Leader]" +31:57.392 [INFO] TestAgent.server: cluster leadership acquired +31:57.392 [INFO] TestAgent.server: New leader elected: payload=Node-733e3b1b-33ff-94e2-2859-c143ee57b607 +31:57.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30115 +31:57.437 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.476 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.476 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.476 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.476 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.476 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.476 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-733e3b1b-33ff-94e2-2859-c143ee57b607 +31:57.476 [INFO] TestAgent.server: member joined, marking health alive: member=Node-733e3b1b-33ff-94e2-2859-c143ee57b607 +31:57.486 [INFO] TestAgent: Synced node info +31:57.486 [DEBUG] TestAgent: Node info in sync +31:57.507 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.520 [DEBUG] TestAgent: removed check: check=test +31:57.520 [DEBUG] TestAgent: Node info in sync +31:57.524 [INFO] TestAgent: Deregistered check: check=test +31:57.525 [INFO] TestAgent: Requesting shutdown +31:57.525 [INFO] TestAgent.server: shutting down server +31:57.525 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.525 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.525 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.525 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.525 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.525 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.525 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.525 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.525 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.525 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.536 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.547Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.547Z [INFO] agent.router.manager: shutting down +31:57.547 [INFO] TestAgent: consul server down +31:57.547 [INFO] TestAgent: shutdown complete +31:57.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30261 network=tcp +31:57.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30261 network=udp +31:57.547 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30262 network=tcp +31:58.048 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.048 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_DeregisterCheck (0.84s) +=== CONT TestAgent_RegisterCheckScriptsExecRemoteDisable +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_RegisterCheck_ACLDeny/no_token_-_node_check +=== RUN TestAgent_RegisterCheck_ACLDeny/svc_token_-_node_check +=== RUN TestAgent_RegisterCheck_ACLDeny/node_token_-_node_check +2021-10-05T16:31:57.194Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.194Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.225Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.225Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.328 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c44cc17f-67a3-339f-64b7-cf75057356a0 Address:127.0.0.1:30273}]" +31:57.328 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c44cc17f-67a3-339f-64b7-cf75057356a0.dc1 127.0.0.1 +31:57.329 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c44cc17f-67a3-339f-64b7-cf75057356a0 127.0.0.1 +2021-10-05T16:31:57.329Z [INFO] agent.router: Initializing LAN area manager +31:57.329 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30268 network=udp +31:57.329 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30273 [Follower]" leader= +31:57.330 [INFO] TestAgent.server: Adding LAN server: server="Node-c44cc17f-67a3-339f-64b7-cf75057356a0 (Addr: tcp/127.0.0.1:30273) (DC: dc1)" +31:57.330 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c44cc17f-67a3-339f-64b7-cf75057356a0.dc1 area=wan +31:57.330 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30268 network=tcp +31:57.330 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30269 network=tcp +31:57.331 [INFO] TestAgent: started state syncer +31:57.331 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30274 network=tcp +31:57.384 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:57.392 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.392 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30273 [Candidate]" term=2 +31:57.418 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.418 [DEBUG] TestAgent.server.raft: vote granted: from=c44cc17f-67a3-339f-64b7-cf75057356a0 term=2 tally=1 +31:57.418 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.418 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30273 [Leader]" +31:57.418 [INFO] TestAgent.server: cluster leadership acquired +31:57.418 [INFO] TestAgent.server: New leader elected: payload=Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.430 [INFO] TestAgent.server: initializing acls +31:57.443 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:57.443 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:57.455 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:57.473 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:57.473 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:57.473 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:57.473 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.474 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c44cc17f-67a3-339f-64b7-cf75057356a0.dc1 +31:57.474 [INFO] TestAgent.server: Updating LAN server: server="Node-c44cc17f-67a3-339f-64b7-cf75057356a0 (Addr: tcp/127.0.0.1:30273) (DC: dc1)" +31:57.474 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c44cc17f-67a3-339f-64b7-cf75057356a0.dc1 area=wan +31:57.491 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.534 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.535 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.535 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.535 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.535 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.535 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.535 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.544 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.551 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.566 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c44cc17f-67a3-339f-64b7-cf75057356a0 +31:57.567 [DEBUG] TestAgent: Check status updated: check=test status=passing +31:57.588 [INFO] TestAgent: Synced node info +31:57.588 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 +31:57.588 [INFO] TestAgent: Requesting shutdown +31:57.588 [INFO] TestAgent.server: shutting down server +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:57.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.588 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.588 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.588 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:57.588 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:57.600 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.607Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.607Z [INFO] agent.router.manager: shutting down +31:57.607 [INFO] TestAgent: consul server down +31:57.607 [INFO] TestAgent: shutdown complete +31:57.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30268 network=tcp +31:57.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30268 network=udp +31:57.607 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30269 network=tcp +31:58.108 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.108 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:57.159Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.173Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.221 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e7a4f633-2caf-5992-4c05-6b141ea303a8 Address:127.0.0.1:30199}]" +31:57.222 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e7a4f633-2caf-5992-4c05-6b141ea303a8.dc1 127.0.0.1 +31:57.222 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30199 [Follower]" leader= +31:57.223 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e7a4f633-2caf-5992-4c05-6b141ea303a8 127.0.0.1 +2021-10-05T16:31:57.223Z [INFO] agent.router: Initializing LAN area manager +31:57.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30219 network=udp +31:57.223 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e7a4f633-2caf-5992-4c05-6b141ea303a8.dc1 area=wan +31:57.223 [INFO] TestAgent.server: Adding LAN server: server="Node-e7a4f633-2caf-5992-4c05-6b141ea303a8 (Addr: tcp/127.0.0.1:30199) (DC: dc1)" +31:57.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30219 network=tcp +31:57.224 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30220 network=tcp +31:57.224 [INFO] TestAgent: started state syncer +31:57.224 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30267 network=tcp +31:57.280 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.280 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30199 [Candidate]" term=2 +31:57.316 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.316 [DEBUG] TestAgent.server.raft: vote granted: from=e7a4f633-2caf-5992-4c05-6b141ea303a8 term=2 tally=1 +31:57.316 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.316 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30199 [Leader]" +31:57.316 [INFO] TestAgent.server: cluster leadership acquired +31:57.317 [INFO] TestAgent.server: New leader elected: payload=Node-e7a4f633-2caf-5992-4c05-6b141ea303a8 +31:57.334 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30199 +31:57.353 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.386 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.386 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.386 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.386 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.386 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.386 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e7a4f633-2caf-5992-4c05-6b141ea303a8 +31:57.386 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e7a4f633-2caf-5992-4c05-6b141ea303a8 +31:57.407 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.412 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:57.421 [INFO] TestAgent: Synced node info +31:57.572 [DEBUG] TestAgent: Check status updated: check=test status=warning +31:57.576 [DEBUG] TestAgent: Node info in sync +31:57.592 [INFO] TestAgent: Synced check: check=test +31:57.592 [INFO] TestAgent: Requesting shutdown +31:57.592 [INFO] TestAgent.server: shutting down server +31:57.592 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.592 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.592 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.592 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.592 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.592 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.592 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.592 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.599 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.607Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.607Z [INFO] agent.router.manager: shutting down +31:57.607 [INFO] TestAgent: consul server down +31:57.607 [INFO] TestAgent: shutdown complete +31:57.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30219 network=tcp +31:57.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30219 network=udp +31:57.607 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30220 network=tcp +31:58.108 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.108 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_PassCheck_ACLDeny (0.94s) + --- PASS: TestAgent_PassCheck_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_PassCheck_ACLDeny/root_token (0.02s) +=== CONT TestAgent_RegisterCheckScriptsExecDisable +--- PASS: TestAgent_WarnCheck (0.96s) +=== CONT TestAgent_RegisterCheck_Scripts +=== RUN TestAgent_RegisterCheck_ACLDeny/no_token_-_svc_check +=== RUN TestAgent_RegisterCheck_ACLDeny/node_token_-_svc_check +=== RUN TestAgent_RegisterCheck_ACLDeny/svc_token_-_svc_check +2021-10-05T16:31:57.208Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.208Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.231Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.231Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef5969e6-afe5-6821-3a55-cff78b3b99f1 Address:127.0.0.1:30224}]" +31:57.320 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1.dc1 127.0.0.1 +31:57.321 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 127.0.0.1 +2021-10-05T16:31:57.321Z [INFO] agent.router: Initializing LAN area manager +31:57.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30194 network=udp +31:57.321 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30224 [Follower]" leader= +31:57.322 [INFO] TestAgent.server: Adding LAN server: server="Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 (Addr: tcp/127.0.0.1:30224) (DC: dc1)" +31:57.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1.dc1 area=wan +31:57.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30194 network=tcp +31:57.322 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30195 network=tcp +31:57.322 [INFO] TestAgent: started state syncer +31:57.322 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30260 network=tcp +31:57.370 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.370 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30224 [Candidate]" term=2 +31:57.372 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:57.389 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.389 [DEBUG] TestAgent.server.raft: vote granted: from=ef5969e6-afe5-6821-3a55-cff78b3b99f1 term=2 tally=1 +31:57.389 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.389 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30224 [Leader]" +31:57.389 [INFO] TestAgent.server: cluster leadership acquired +31:57.389 [INFO] TestAgent.server: New leader elected: payload=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.399 [INFO] TestAgent.server: initializing acls +31:57.405 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:57.405 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:57.417 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:57.436 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:57.436 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:57.436 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:57.436 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.436 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1.dc1 +31:57.437 [INFO] TestAgent.server: Updating LAN server: server="Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 (Addr: tcp/127.0.0.1:30224) (DC: dc1)" +31:57.437 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1.dc1 area=wan +31:57.461 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.487 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.487 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.487 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.487 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.487 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.487 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.487 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.497 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.505 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.739 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:57.750 [INFO] TestAgent: Synced node info +31:57.751 [DEBUG] TestAgent: Node info in sync +31:57.759 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ef5969e6-afe5-6821-3a55-cff78b3b99f1 +31:57.759 [DEBUG] TestAgent: removed check: check=test +31:57.759 [DEBUG] TestAgent: Node info in sync +31:57.759 [INFO] TestAgent: Deregistered check: check=test +31:57.759 [INFO] TestAgent: Requesting shutdown +31:57.759 [INFO] TestAgent.server: shutting down server +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.759 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:57.759 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:57.760 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:57.760 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.769 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.773Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.773Z [INFO] agent.router.manager: shutting down +31:57.774 [INFO] TestAgent: consul server down +31:57.774 [INFO] TestAgent: shutdown complete +31:57.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30194 network=tcp +31:57.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30194 network=udp +31:57.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30195 network=tcp +31:58.274 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.274 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_DeregisterCheckACLDeny (1.10s) + --- PASS: TestAgent_DeregisterCheckACLDeny/no_token (0.00s) + --- PASS: TestAgent_DeregisterCheckACLDeny/root_token (0.00s) +=== CONT TestAgent_ForceLeavePrune +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_node_check +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_top-level_service_check +=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_slice-based_service_check +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_node_check +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_top-level_service_check +2021-10-05T16:31:57.836Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.846Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.897 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3a25b85f-988c-f598-1b2f-dd1f828888b5 Address:127.0.0.1:30294}]" +31:57.898 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3a25b85f-988c-f598-1b2f-dd1f828888b5.dc1 127.0.0.1 +31:57.899 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3a25b85f-988c-f598-1b2f-dd1f828888b5 127.0.0.1 +2021-10-05T16:31:57.899Z [INFO] agent.router: Initializing LAN area manager +31:57.899 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30289 network=udp +31:57.899 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30294 [Follower]" leader= +31:57.899 [INFO] TestAgent.server: Adding LAN server: server="Node-3a25b85f-988c-f598-1b2f-dd1f828888b5 (Addr: tcp/127.0.0.1:30294) (DC: dc1)" +31:57.899 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3a25b85f-988c-f598-1b2f-dd1f828888b5.dc1 area=wan +31:57.900 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30289 network=tcp +31:57.903 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30290 network=tcp +31:57.903 [INFO] TestAgent: started state syncer +31:57.903 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30316 network=tcp +31:57.959 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.960 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30294 [Candidate]" term=2 +31:58.004 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.004 [DEBUG] TestAgent.server.raft: vote granted: from=3a25b85f-988c-f598-1b2f-dd1f828888b5 term=2 tally=1 +31:58.004 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.004 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30294 [Leader]" +31:58.007 [INFO] TestAgent.server: cluster leadership acquired +31:58.007 [INFO] TestAgent.server: New leader elected: payload=Node-3a25b85f-988c-f598-1b2f-dd1f828888b5 +31:58.014 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30294 +31:58.036 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.086 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.086 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.086 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.086 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.086 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.086 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3a25b85f-988c-f598-1b2f-dd1f828888b5 +31:58.087 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3a25b85f-988c-f598-1b2f-dd1f828888b5 +31:58.128 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.129 [INFO] TestAgent: Requesting shutdown +31:58.129 [INFO] TestAgent.server: shutting down server +31:58.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.129 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.129 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.129 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.129 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.129 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.129 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:58.129 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.129 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.129 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.144 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.164Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.164Z [INFO] agent.router.manager: shutting down +31:58.166 [INFO] TestAgent: consul server down +31:58.166 [INFO] TestAgent: shutdown complete +31:58.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30289 network=tcp +31:58.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30289 network=udp +31:58.166 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30290 network=tcp +31:58.667 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.667 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheck_BadStatus (0.84s) === CONT TestAgent_ForceLeave_ACLDeny -2022-11-07T22:46:19.397Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.407Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.441 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c3c0276-b644-4177-a226-d446882d9a87 Address:127.0.0.1:11818}]" -46:19.441 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11818 [Follower]" leader= -46:19.442 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6c3c0276-b644-4177-a226-d446882d9a87.dc1 127.0.0.1 -46:19.442 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6c3c0276-b644-4177-a226-d446882d9a87 127.0.0.1 -2022-11-07T22:46:19.442Z [INFO] agent.router: Initializing LAN area manager -46:19.442 [INFO] TestAgent.server: Adding LAN server: server="Node-6c3c0276-b644-4177-a226-d446882d9a87 (Addr: tcp/127.0.0.1:11818) (DC: dc1)" -46:19.442 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6c3c0276-b644-4177-a226-d446882d9a87.dc1 area=wan -46:19.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11852 network=udp -46:19.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11852 network=tcp -46:19.443 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11814 network=tcp -46:19.443 [INFO] TestAgent: started state syncer -46:19.443 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11246 network=tcp -46:19.501 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.501 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11818 [Candidate]" term=2 -46:19.517 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.517 [DEBUG] TestAgent.server.raft: vote granted: from=6c3c0276-b644-4177-a226-d446882d9a87 term=2 tally=1 -46:19.517 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.517 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11818 [Leader]" -46:19.517 [INFO] TestAgent.server: cluster leadership acquired -46:19.517 [INFO] TestAgent.server: New leader elected: payload=Node-6c3c0276-b644-4177-a226-d446882d9a87 -46:19.530 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11818 -46:19.545 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.569 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.569 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.569 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.569 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.569 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.570 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c3c0276-b644-4177-a226-d446882d9a87 -46:19.570 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6c3c0276-b644-4177-a226-d446882d9a87 -46:19.582 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.752 [INFO] TestAgent: Node entered maintenance mode -46:19.752 [DEBUG] TestAgent: removed check: check=_node_maintenance -46:19.752 [INFO] TestAgent: Node left maintenance mode -46:19.759 [INFO] TestAgent: Node entered maintenance mode -46:19.759 [INFO] TestAgent: Requesting shutdown -46:19.759 [INFO] TestAgent.server: shutting down server -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.759 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.760 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.760 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.760 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.760 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.760 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.760 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.765 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.774Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.774Z [INFO] agent.router.manager: shutting down -46:19.774 [INFO] TestAgent: consul server down -46:19.774 [INFO] TestAgent: shutdown complete -46:19.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11852 network=tcp -46:19.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11852 network=udp -46:19.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11814 network=tcp -46:20.274 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.274 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:19.356Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.356Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.366Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.366Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.389 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:44c00329-69b6-cc38-981d-9310051126ef Address:127.0.0.1:11748}]" -46:19.389 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11748 [Follower]" leader= -46:19.390 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-44c00329-69b6-cc38-981d-9310051126ef.dc1 127.0.0.1 -46:19.390 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-44c00329-69b6-cc38-981d-9310051126ef 127.0.0.1 -2022-11-07T22:46:19.390Z [INFO] agent.router: Initializing LAN area manager -46:19.390 [INFO] TestAgent.server: Adding LAN server: server="Node-44c00329-69b6-cc38-981d-9310051126ef (Addr: tcp/127.0.0.1:11748) (DC: dc1)" -46:19.390 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-44c00329-69b6-cc38-981d-9310051126ef.dc1 area=wan -46:19.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11771 network=udp -46:19.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11771 network=tcp -46:19.391 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11744 network=tcp -46:19.391 [INFO] TestAgent: started state syncer -46:19.391 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11798 network=tcp -46:19.441 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:19.446 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.446 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11748 [Candidate]" term=2 -46:19.457 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.457 [DEBUG] TestAgent.server.raft: vote granted: from=44c00329-69b6-cc38-981d-9310051126ef term=2 tally=1 -46:19.457 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.457 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11748 [Leader]" -46:19.457 [INFO] TestAgent.server: cluster leadership acquired -46:19.457 [INFO] TestAgent.server: New leader elected: payload=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.464 [INFO] TestAgent.server: initializing acls -46:19.468 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:19.468 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:19.475 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:19.486 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:19.486 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:19.486 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:19.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.486 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-44c00329-69b6-cc38-981d-9310051126ef.dc1 -46:19.486 [INFO] TestAgent.server: Updating LAN server: server="Node-44c00329-69b6-cc38-981d-9310051126ef (Addr: tcp/127.0.0.1:11748) (DC: dc1)" -46:19.486 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-44c00329-69b6-cc38-981d-9310051126ef.dc1 area=wan -46:19.496 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.523 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.523 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.523 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.523 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.523 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.523 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.523 [INFO] TestAgent.server: member joined, marking health alive: member=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.542 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.542 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.707 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:19.719 [INFO] TestAgent: Synced node info -46:19.758 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.758 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-44c00329-69b6-cc38-981d-9310051126ef -46:19.759 [INFO] TestAgent: Requesting shutdown -46:19.759 [INFO] TestAgent.server: shutting down server -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.759 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.759 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.759 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:19.759 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.765 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.774Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.774Z [INFO] agent.router.manager: shutting down -46:19.774 [INFO] TestAgent: consul server down -46:19.774 [INFO] TestAgent: shutdown complete -46:19.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11771 network=tcp -46:19.774 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11771 network=udp -46:19.774 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11744 network=tcp -46:20.274 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.274 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentConnectAuthorize_defaultDeny (0.93s) -=== CONT TestAgent_Join ---- PASS: TestAgent_NodeMaintenanceMode (0.89s) +2021-10-05T16:31:57.616Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.630Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.690 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a71a3f31-e554-22ce-9c9d-a12f2e114b80 Address:127.0.0.1:30301}]" +31:57.690 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30301 [Follower]" leader= +31:57.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80.dc1 127.0.0.1 +31:57.691 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 127.0.0.1 +2021-10-05T16:31:57.691Z [INFO] agent.router: Initializing LAN area manager +31:57.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30296 network=udp +31:57.692 [INFO] TestAgent.server: Adding LAN server: server="Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 (Addr: tcp/127.0.0.1:30301) (DC: dc1)" +31:57.692 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30296 network=tcp +31:57.692 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80.dc1 area=wan +31:57.692 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30297 network=tcp +31:57.692 [INFO] TestAgent: started state syncer +31:57.692 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30288 network=tcp +31:57.744 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:57.751 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.751 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30301 [Candidate]" term=2 +31:57.765 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.765 [DEBUG] TestAgent.server.raft: vote granted: from=a71a3f31-e554-22ce-9c9d-a12f2e114b80 term=2 tally=1 +31:57.765 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.765 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30301 [Leader]" +31:57.766 [INFO] TestAgent.server: cluster leadership acquired +31:57.766 [INFO] TestAgent.server: New leader elected: payload=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:57.773 [INFO] TestAgent.server: initializing acls +31:57.786 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:57.786 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:57.795 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:57.811 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:57.811 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:57.811 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:57.811 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:57.811 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80.dc1 +31:57.811 [INFO] TestAgent.server: Updating LAN server: server="Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 (Addr: tcp/127.0.0.1:30301) (DC: dc1)" +31:57.811 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80.dc1 area=wan +31:57.833 [INFO] TestAgent: Synced node info +31:57.833 [DEBUG] TestAgent: Node info in sync +31:57.864 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.932 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.932 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.932 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.932 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.932 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.932 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:57.932 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:57.954 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.954 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:58.017 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:58.018 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a71a3f31-e554-22ce-9c9d-a12f2e114b80 +31:58.028 [DEBUG] TestAgent: Node info in sync +31:58.038 [INFO] TestAgent: Synced service: service=foo:1234 +31:58.086 [DEBUG] TestAgent: Node info in sync +31:58.086 [DEBUG] TestAgent: Service in sync: service=foo:1234 +31:58.110 [INFO] TestAgent: Synced check: check=test +31:58.128 [DEBUG] TestAgent: Node info in sync +31:58.128 [DEBUG] TestAgent: Service in sync: service=foo:1234 +31:58.128 [DEBUG] TestAgent: Check in sync: check=test +31:58.148 [INFO] TestAgent: Synced check: check=test2 +31:58.148 [INFO] TestAgent: Requesting shutdown +31:58.148 [INFO] TestAgent.server: shutting down server +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.148 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.148 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.148 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.148 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:58.148 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:58.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.149 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.149 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.163 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.166Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.166Z [INFO] agent.router.manager: shutting down +31:58.166 [INFO] TestAgent: consul server down +31:58.166 [INFO] TestAgent: shutdown complete +31:58.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30296 network=tcp +31:58.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30296 network=udp +31:58.166 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30297 network=tcp +31:58.669 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.669 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_slice-based_service_check +--- PASS: TestAgent_RegisterCheck_ACLDeny (1.09s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/no_token_-_node_check (0.00s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/svc_token_-_node_check (0.00s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/node_token_-_node_check (0.03s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/no_token_-_svc_check (0.00s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/node_token_-_svc_check (0.00s) + --- PASS: TestAgent_RegisterCheck_ACLDeny/svc_token_-_svc_check (0.04s) === CONT TestOpenMetricsMimeTypeHeaders --- PASS: TestOpenMetricsMimeTypeHeaders (0.00s) === CONT TestAgent_ForceLeave -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:19.985Z [WARN] agent: Node name "Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:19.985Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:19.994Z [WARN] agent.auto_config: Node name "Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:19.994Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:19.995 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5 127.0.0.1 -2022-11-07T22:46:19.995Z [INFO] agent.router: Initializing LAN area manager -46:19.995 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node -46:19.995 [DEBUG] Agent2: check exists, not restoring from file: check=mem file=/tmp/consul-test/TestAgent_PurgeCheckOnDuplicate-TestAgent-agent000932846/checks/afc4fc7e48a0710a1dc94ef3e8bc5764 -46:19.995 [INFO] Agent2: Started DNS server: address=127.0.0.1:11873 network=udp -46:19.995 [INFO] Agent2: Started DNS server: address=127.0.0.1:11873 network=tcp -46:19.996 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11902 network=tcp -46:19.996 [INFO] Agent2: started state syncer -46:19.996 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11805 network=tcp -2022-11-07T22:46:19.996Z [WARN] agent.router.manager: No servers available -46:19.996 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:19.996 [INFO] Agent2: Requesting shutdown -46:19.996 [INFO] Agent2.client: shutting down client -46:19.996 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave -46:20.001 [INFO] Agent2: consul client down -46:20.001 [INFO] Agent2: shutdown complete -46:20.001 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11873 network=tcp -46:20.001 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11873 network=udp -46:20.001 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11902 network=tcp -46:20.501 [INFO] Agent2: Waiting for endpoints to shut down -46:20.501 [INFO] Agent2: Endpoints down -2022-11-07T22:46:19.438Z [WARN] agent: Node name "Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:19.438Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:19.448Z [WARN] agent.auto_config: Node name "Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:19.448Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:19.448 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node 4b9fd705-df0f-b3aa-c5b5-addf5756f9b5 127.0.0.1 -2022-11-07T22:46:19.448Z [INFO] agent.router: Initializing LAN area manager -46:19.448 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11247 network=udp -46:19.448 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11247 network=tcp -46:19.448 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11283 network=tcp -46:19.449 [INFO] TestAgent: started state syncer -46:19.449 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11204 network=tcp -2022-11-07T22:46:19.449Z [WARN] agent.router.manager: No servers available -46:19.449 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:19.453 [INFO] TestAgent: Requesting shutdown -46:19.453 [INFO] TestAgent.client: shutting down client -46:19.453 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:19.457 [INFO] TestAgent: consul client down -46:19.457 [INFO] TestAgent: shutdown complete -46:19.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11247 network=tcp -46:19.457 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11247 network=udp -46:19.457 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11283 network=tcp -46:19.957 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.957 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_PurgeCheckOnDuplicate (1.07s) -=== CONT TestAgent_Services_TerminatingGateway -2022-11-07T22:46:19.606Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.615Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.652 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bdeaa823-f5c7-e24b-43d5-7715cfcabf2d Address:127.0.0.1:11713}]" -46:19.652 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11713 [Follower]" leader= -46:19.653 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d.dc1 127.0.0.1 -46:19.653 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d 127.0.0.1 -2022-11-07T22:46:19.653Z [INFO] agent.router: Initializing LAN area manager -46:19.653 [INFO] TestAgent.server: Adding LAN server: server="Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d (Addr: tcp/127.0.0.1:11713) (DC: dc1)" -46:19.653 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d.dc1 area=wan -46:19.653 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11778 network=udp -46:19.654 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11778 network=tcp -46:19.654 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11709 network=tcp -46:19.654 [INFO] TestAgent: started state syncer -46:19.654 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11714 network=tcp -46:19.715 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.715 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11713 [Candidate]" term=2 -46:19.737 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.737 [DEBUG] TestAgent.server.raft: vote granted: from=bdeaa823-f5c7-e24b-43d5-7715cfcabf2d term=2 tally=1 -46:19.737 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.737 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11713 [Leader]" -46:19.737 [INFO] TestAgent.server: cluster leadership acquired -46:19.737 [INFO] TestAgent.server: New leader elected: payload=Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d -46:19.748 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11713 -46:19.763 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.791 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.791 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.791 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.791 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.791 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.791 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d -46:19.791 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bdeaa823-f5c7-e24b-43d5-7715cfcabf2d -46:19.802 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.018 [INFO] TestAgent: Service entered maintenance mode: service=redis -46:20.018 [DEBUG] TestAgent: removed check: check=_service_maintenance:redis -46:20.018 [INFO] TestAgent: Service left maintenance mode: service=redis -46:20.021 [INFO] TestAgent: Service entered maintenance mode: service=redis -46:20.021 [INFO] TestAgent: Requesting shutdown -46:20.021 [INFO] TestAgent.server: shutting down server -46:20.021 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.021 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.021 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.022 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.022 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.022 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:20.022 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.022 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.022 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.026 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.030Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.030Z [INFO] agent.router.manager: shutting down -46:20.030 [INFO] TestAgent: consul server down -46:20.030 [INFO] TestAgent: shutdown complete -46:20.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11778 network=tcp -46:20.030 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11778 network=udp -46:20.030 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11709 network=tcp -46:20.530 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.530 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Service_MaintenanceMode (0.93s) -=== CONT TestAgent_Members_ACLFilter -2022-11-07T22:46:19.703Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.715Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.764 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c935d096-0c99-658e-bf47-e3898b17790c Address:127.0.0.1:11829}]" -46:19.764 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11829 [Follower]" leader= -46:19.765 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c935d096-0c99-658e-bf47-e3898b17790c.dc1 127.0.0.1 -46:19.765 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c935d096-0c99-658e-bf47-e3898b17790c 127.0.0.1 -2022-11-07T22:46:19.765Z [INFO] agent.router: Initializing LAN area manager -46:19.765 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c935d096-0c99-658e-bf47-e3898b17790c.dc1 area=wan -46:19.765 [INFO] TestAgent.server: Adding LAN server: server="Node-c935d096-0c99-658e-bf47-e3898b17790c (Addr: tcp/127.0.0.1:11829) (DC: dc1)" -46:19.765 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11722 network=udp -46:19.765 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11722 network=tcp -46:19.766 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11838 network=tcp -46:19.766 [INFO] TestAgent: started state syncer -46:19.766 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11749 network=tcp -46:19.807 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.807 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11829 [Candidate]" term=2 -46:19.816 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.816 [DEBUG] TestAgent.server.raft: vote granted: from=c935d096-0c99-658e-bf47-e3898b17790c term=2 tally=1 -46:19.816 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.816 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11829 [Leader]" -46:19.817 [INFO] TestAgent.server: cluster leadership acquired -46:19.817 [INFO] TestAgent.server: New leader elected: payload=Node-c935d096-0c99-658e-bf47-e3898b17790c -46:19.824 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11829 -46:19.847 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.874 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.874 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.874 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.874 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.874 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.874 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c935d096-0c99-658e-bf47-e3898b17790c -46:19.874 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c935d096-0c99-658e-bf47-e3898b17790c -46:19.884 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.041 [DEBUG] TestAgent: removed check: check=mem -46:20.042 [DEBUG] TestAgent: removed check: check=mem -46:20.042 [INFO] TestAgent: Requesting shutdown -46:20.042 [INFO] TestAgent.server: shutting down server -46:20.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.042 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.042 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.042 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.042 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:20.042 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.042 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.042 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.050Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.050Z [INFO] agent.router.manager: shutting down -46:20.050 [INFO] TestAgent: consul server down -46:20.050 [INFO] TestAgent: shutdown complete -46:20.050 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11722 network=tcp -46:20.050 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11722 network=udp -46:20.050 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11838 network=tcp -46:20.551 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.551 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_PurgeCheck (0.87s) +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_node_check +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_top-level_service_check +=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_slice-based_service_check +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:58.177Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.197Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 Address:127.0.0.1:30315}]" +31:58.252 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30315 [Follower]" leader= +31:58.253 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463.dc1 127.0.0.1 +31:58.253 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 127.0.0.1 +2021-10-05T16:31:58.253Z [INFO] agent.router: Initializing LAN area manager +31:58.253 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30310 network=udp +31:58.254 [INFO] TestAgent.server: Adding LAN server: server="Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 (Addr: tcp/127.0.0.1:30315) (DC: dc1)" +31:58.254 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463.dc1 area=wan +31:58.254 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30310 network=tcp +31:58.254 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30311 network=tcp +31:58.254 [INFO] TestAgent: started state syncer +31:58.254 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30337 network=tcp +31:58.296 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.296 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30315 [Candidate]" term=2 +31:58.339 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.339 [DEBUG] TestAgent.server.raft: vote granted: from=4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 term=2 tally=1 +31:58.339 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.339 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30315 [Leader]" +31:58.340 [INFO] TestAgent.server: cluster leadership acquired +31:58.340 [INFO] TestAgent.server: New leader elected: payload=Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 +31:58.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30315 +31:58.388 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.431 [INFO] TestAgent: Synced node info +31:58.441 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.441 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.441 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.441 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.441 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.441 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 +31:58.441 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b9aa5a8-5458-03e9-2bbf-bf5cf4906463 +31:58.461 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.479 [INFO] TestAgent: Requesting shutdown +31:58.479 [INFO] TestAgent.server: shutting down server +31:58.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.479 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.479 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.479 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.479 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.479 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.479 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.485 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.508Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.508Z [INFO] agent.router.manager: shutting down +31:58.508 [INFO] TestAgent: consul server down +31:58.508 [INFO] TestAgent: shutdown complete +31:58.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30310 network=tcp +31:58.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30310 network=udp +31:58.508 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30311 network=tcp +31:59.012 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.012 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheckScriptsExecDisable (0.90s) === CONT TestAgent_Leave_ACLDeny -2022-11-07T22:46:19.728Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.737Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.776 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c29e7755-1900-8a5c-ac9d-a1af627515f7 Address:127.0.0.1:11825}]" -46:19.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11825 [Follower]" leader= -46:19.776 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c29e7755-1900-8a5c-ac9d-a1af627515f7.dc1 127.0.0.1 -46:19.777 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c29e7755-1900-8a5c-ac9d-a1af627515f7 127.0.0.1 -2022-11-07T22:46:19.777Z [INFO] agent.router: Initializing LAN area manager -46:19.777 [INFO] TestAgent.server: Adding LAN server: server="Node-c29e7755-1900-8a5c-ac9d-a1af627515f7 (Addr: tcp/127.0.0.1:11825) (DC: dc1)" -46:19.777 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c29e7755-1900-8a5c-ac9d-a1af627515f7.dc1 area=wan -46:19.777 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11831 network=udp -46:19.777 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11831 network=tcp -46:19.778 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11821 network=tcp -46:19.778 [INFO] TestAgent: started state syncer -46:19.778 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11819 network=tcp -46:19.823 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.823 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11825 [Candidate]" term=2 -46:19.836 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.836 [DEBUG] TestAgent.server.raft: vote granted: from=c29e7755-1900-8a5c-ac9d-a1af627515f7 term=2 tally=1 -46:19.836 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.836 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11825 [Leader]" -46:19.836 [INFO] TestAgent.server: cluster leadership acquired -46:19.836 [INFO] TestAgent.server: New leader elected: payload=Node-c29e7755-1900-8a5c-ac9d-a1af627515f7 -46:19.843 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11825 -46:19.860 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.879 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.880 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.880 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.880 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.880 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.880 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c29e7755-1900-8a5c-ac9d-a1af627515f7 -46:19.880 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c29e7755-1900-8a5c-ac9d-a1af627515f7 -46:19.892 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.012 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.015 [INFO] TestAgent: Synced node info -46:20.015 [DEBUG] TestAgent: Node info in sync -46:20.046 [DEBUG] TestAgent: removed check: check=service:redis -46:20.046 [INFO] TestAgent: Requesting shutdown -46:20.047 [INFO] TestAgent.server: shutting down server -46:20.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.047 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.047 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.047 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.047 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.047 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.047 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.047 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.047 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.053 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.056Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.056Z [INFO] agent.router.manager: shutting down -46:20.056 [INFO] TestAgent: consul server down -46:20.056 [INFO] TestAgent: shutdown complete -46:20.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11831 network=tcp -46:20.056 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11831 network=udp -46:20.056 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11821 network=tcp -46:20.557 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.557 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_unloadChecks (0.84s) -=== CONT TestAgent_Members_WAN -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:58.003Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.037Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.156 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1177de88-73b4-8f74-a31e-02cf03a12fd2 Address:127.0.0.1:30322}]" +31:58.157 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30322 [Follower]" leader= +31:58.157 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1177de88-73b4-8f74-a31e-02cf03a12fd2.dc1 127.0.0.1 +31:58.159 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1177de88-73b4-8f74-a31e-02cf03a12fd2 127.0.0.1 +2021-10-05T16:31:58.159Z [INFO] agent.router: Initializing LAN area manager +31:58.159 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30317 network=udp +31:58.159 [INFO] TestAgent.server: Adding LAN server: server="Node-1177de88-73b4-8f74-a31e-02cf03a12fd2 (Addr: tcp/127.0.0.1:30322) (DC: dc1)" +31:58.159 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1177de88-73b4-8f74-a31e-02cf03a12fd2.dc1 area=wan +31:58.159 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30317 network=tcp +31:58.160 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30318 network=tcp +31:58.160 [INFO] TestAgent: started state syncer +31:58.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30358 network=tcp +31:58.202 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.202 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30322 [Candidate]" term=2 +31:58.220 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.220 [DEBUG] TestAgent.server.raft: vote granted: from=1177de88-73b4-8f74-a31e-02cf03a12fd2 term=2 tally=1 +31:58.220 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.220 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30322 [Leader]" +31:58.221 [INFO] TestAgent.server: cluster leadership acquired +31:58.221 [INFO] TestAgent.server: New leader elected: payload=Node-1177de88-73b4-8f74-a31e-02cf03a12fd2 +31:58.231 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30322 +31:58.255 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.296 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.296 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.296 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.296 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.296 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.296 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1177de88-73b4-8f74-a31e-02cf03a12fd2 +31:58.296 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1177de88-73b4-8f74-a31e-02cf03a12fd2 +31:58.324 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.371 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:58.376 [INFO] TestAgent: Synced node info +31:58.376 [DEBUG] TestAgent: Node info in sync +31:58.542 [DEBUG] TestAgent: Node info in sync +31:58.546 [INFO] TestAgent: Synced check: check=test +31:58.546 [INFO] TestAgent: Requesting shutdown +31:58.546 [INFO] TestAgent.server: shutting down server +31:58.546 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.546 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.546 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.546 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.546 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.546 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.546 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.546 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.546 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.546 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.556 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.564Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.564Z [INFO] agent.router.manager: shutting down +31:58.564 [INFO] TestAgent: consul server down +31:58.564 [INFO] TestAgent: shutdown complete +31:58.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30317 network=tcp +31:58.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30317 network=udp +31:58.564 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30318 network=tcp +31:59.065 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.065 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheck_Passing (1.10s) +=== CONT TestAgent_Leave +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestAgent_ForceLeave_ACLDeny/no_token === RUN TestAgent_ForceLeave_ACLDeny/agent_master_token === RUN TestAgent_ForceLeave_ACLDeny/read-only_token -=== RUN TestAgent_ForceLeave_ACLDeny/operator_write_token -2022-11-07T22:46:19.799Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:edabc424-71de-6618-1eb8-8e0786706e45 Address:127.0.0.1:11755}]" -46:19.859 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11755 [Follower]" leader= -46:19.860 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-edabc424-71de-6618-1eb8-8e0786706e45.dc1 127.0.0.1 -46:19.860 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-edabc424-71de-6618-1eb8-8e0786706e45 127.0.0.1 -2022-11-07T22:46:19.860Z [INFO] agent.router: Initializing LAN area manager -46:19.860 [INFO] TestAgent.server: Adding LAN server: server="Node-edabc424-71de-6618-1eb8-8e0786706e45 (Addr: tcp/127.0.0.1:11755) (DC: dc1)" -46:19.860 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-edabc424-71de-6618-1eb8-8e0786706e45.dc1 area=wan -46:19.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11736 network=udp -46:19.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11736 network=tcp -46:19.861 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11751 network=tcp -46:19.861 [INFO] TestAgent: started state syncer -46:19.861 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11826 network=tcp -46:19.914 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.914 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11755 [Candidate]" term=2 -46:19.926 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.926 [DEBUG] TestAgent.server.raft: vote granted: from=edabc424-71de-6618-1eb8-8e0786706e45 term=2 tally=1 -46:19.926 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.926 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11755 [Leader]" -46:19.926 [INFO] TestAgent.server: cluster leadership acquired -46:19.926 [INFO] TestAgent.server: New leader elected: payload=Node-edabc424-71de-6618-1eb8-8e0786706e45 -46:19.938 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11755 -46:19.951 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.973 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.973 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.973 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.973 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.973 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.973 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-edabc424-71de-6618-1eb8-8e0786706e45 -46:19.973 [INFO] TestAgent.server: member joined, marking health alive: member=Node-edabc424-71de-6618-1eb8-8e0786706e45 -46:19.986 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.172 [INFO] TestAgent: Requesting shutdown -46:20.172 [INFO] TestAgent.server: shutting down server -46:20.172 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.172 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.172 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.172 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.172 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.172 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.172 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.172 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:20.172 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.172 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.172 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.179 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.187Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.187Z [INFO] agent.router.manager: shutting down -46:20.187 [INFO] TestAgent: consul server down -46:20.187 [INFO] TestAgent: shutdown complete -46:20.187 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11736 network=tcp -46:20.187 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11736 network=udp -46:20.187 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11751 network=tcp -46:20.687 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.687 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_loadChecks_token (0.90s) -=== CONT TestAgent_Members -2022-11-07T22:46:19.796Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.849 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:873bbb23-2f5d-29a2-640e-91cff7ec2f34 Address:127.0.0.1:11762}]" -46:19.849 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11762 [Follower]" leader= -46:19.850 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34.dc1 127.0.0.1 -46:19.850 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34 127.0.0.1 -2022-11-07T22:46:19.850Z [INFO] agent.router: Initializing LAN area manager -46:19.850 [INFO] TestAgent.server: Adding LAN server: server="Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34 (Addr: tcp/127.0.0.1:11762) (DC: dc1)" -46:19.851 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34.dc1 area=wan -46:19.851 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11757 network=tcp -46:19.851 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11757 network=udp -46:19.851 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11758 network=tcp -46:19.851 [INFO] TestAgent: started state syncer -46:19.851 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11721 network=tcp -46:19.900 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.900 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11762 [Candidate]" term=2 -46:19.909 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.909 [DEBUG] TestAgent.server.raft: vote granted: from=873bbb23-2f5d-29a2-640e-91cff7ec2f34 term=2 tally=1 -46:19.909 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.909 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11762 [Leader]" -46:19.909 [INFO] TestAgent.server: cluster leadership acquired -46:19.909 [INFO] TestAgent.server: New leader elected: payload=Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34 -46:19.915 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11762 -46:19.932 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.958 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.958 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.958 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.958 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.958 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.958 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34 -46:19.958 [INFO] TestAgent.server: member joined, marking health alive: member=Node-873bbb23-2f5d-29a2-640e-91cff7ec2f34 -46:19.972 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.222 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.225 [INFO] TestAgent: Synced node info -46:20.225 [DEBUG] TestAgent: Node info in sync -46:20.225 [DEBUG] TestAgent: Check status updated: check=mem status=passing -46:20.225 [DEBUG] TestAgent: Check status updated: check=mem status=critical -46:20.225 [INFO] TestAgent: Requesting shutdown -46:20.225 [INFO] TestAgent.server: shutting down server -46:20.225 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.225 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.225 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.225 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.225 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.225 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.225 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.225 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.225 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.225 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.229 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.233Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.233Z [INFO] agent.router.manager: shutting down -46:20.233 [INFO] TestAgent: consul server down -46:20.233 [INFO] TestAgent: shutdown complete -46:20.233 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11757 network=tcp -46:20.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11757 network=udp -46:20.234 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11758 network=tcp -46:20.734 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.734 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_updateTTLCheck (0.95s) -=== CONT TestAgent_Leave -=== RUN TestAgent_Members_ACLFilter/no_token -=== RUN TestAgent_Members_ACLFilter/root_token -2022-11-07T22:46:20.262Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:20.277Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:20.278 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node-9c4314fa-bbdc-c9d4-b4a0-08dd1d62ab6f 127.0.0.1 -2022-11-07T22:46:20.278Z [INFO] agent.router: Initializing LAN area manager -46:20.278 [INFO] Agent2.client.serf.lan: serf: Attempting re-join to previously known node: Node-7944a9ca-22ff-8135-ed03-a91ea587c6b4: 127.0.0.1:11704 -46:20.278 [DEBUG] Agent2: restored health check from file: check=mem file=/tmp/consul-test/TestAgent_PersistCheck-TestAgent-agent016792834/checks/afc4fc7e48a0710a1dc94ef3e8bc5764 -46:20.278 [DEBUG] Agent2.client.memberlist.lan: memberlist: Failed to join 127.0.0.1: dial tcp 127.0.0.1:11704: connect: connection refused -46:20.278 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node -46:20.278 [INFO] Agent2: Started DNS server: address=127.0.0.1:11894 network=udp -46:20.278 [INFO] Agent2: Started DNS server: address=127.0.0.1:11894 network=tcp -46:20.279 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11930 network=tcp -46:20.279 [INFO] Agent2: started state syncer -46:20.279 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11784 network=tcp -2022-11-07T22:46:20.279Z [WARN] agent.router.manager: No servers available -46:20.279 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:20.279 [INFO] Agent2: Requesting shutdown -46:20.279 [INFO] Agent2.client: shutting down client -46:20.279 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave -46:20.286 [INFO] Agent2: consul client down -46:20.286 [INFO] Agent2: shutdown complete -46:20.286 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11894 network=tcp -46:20.286 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11894 network=udp -46:20.286 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11930 network=tcp -46:20.786 [INFO] Agent2: Waiting for endpoints to shut down -46:20.786 [INFO] Agent2: Endpoints down -2022-11-07T22:46:19.711Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:19.721Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:19.721 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-7944a9ca-22ff-8135-ed03-a91ea587c6b4 127.0.0.1 -2022-11-07T22:46:19.721Z [INFO] agent.router: Initializing LAN area manager -46:19.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11764 network=tcp -46:19.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11764 network=udp -46:19.722 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11702 network=tcp -46:19.722 [INFO] TestAgent: started state syncer -46:19.722 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11830 network=tcp -2022-11-07T22:46:19.722Z [WARN] agent.router.manager: No servers available -46:19.722 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:19.745 [INFO] TestAgent: Requesting shutdown -46:19.745 [INFO] TestAgent.client: shutting down client -46:19.745 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:19.751 [INFO] TestAgent: consul client down -46:19.751 [INFO] TestAgent: shutdown complete -46:19.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11764 network=tcp -46:19.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11764 network=udp -46:19.751 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11702 network=tcp -46:20.252 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.252 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_PersistCheck (1.09s) +2021-10-05T16:31:58.080Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.141Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.222 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c233b5c-cfad-0920-e31f-b597c946c0a3 Address:127.0.0.1:30364}]" +31:58.222 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c233b5c-cfad-0920-e31f-b597c946c0a3.dc1 127.0.0.1 +31:58.223 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c233b5c-cfad-0920-e31f-b597c946c0a3 127.0.0.1 +2021-10-05T16:31:58.223Z [INFO] agent.router: Initializing LAN area manager +31:58.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30359 network=udp +31:58.223 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30364 [Follower]" leader= +31:58.223 [INFO] TestAgent.server: Adding LAN server: server="Node-0c233b5c-cfad-0920-e31f-b597c946c0a3 (Addr: tcp/127.0.0.1:30364) (DC: dc1)" +31:58.223 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c233b5c-cfad-0920-e31f-b597c946c0a3.dc1 area=wan +31:58.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30359 network=tcp +31:58.224 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30360 network=tcp +31:58.224 [INFO] TestAgent: started state syncer +31:58.224 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30309 network=tcp +31:58.272 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.272 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30364 [Candidate]" term=2 +31:58.286 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.286 [DEBUG] TestAgent.server.raft: vote granted: from=0c233b5c-cfad-0920-e31f-b597c946c0a3 term=2 tally=1 +31:58.286 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.286 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30364 [Leader]" +31:58.287 [INFO] TestAgent.server: cluster leadership acquired +31:58.287 [INFO] TestAgent.server: New leader elected: payload=Node-0c233b5c-cfad-0920-e31f-b597c946c0a3 +31:58.297 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30364 +31:58.338 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.380 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.380 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.380 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.380 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.380 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.380 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c233b5c-cfad-0920-e31f-b597c946c0a3 +31:58.380 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c233b5c-cfad-0920-e31f-b597c946c0a3 +31:58.413 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.436 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:58.442 [INFO] TestAgent: Synced node info +31:58.585 [INFO] TestAgent: Requesting shutdown +31:58.585 [INFO] TestAgent.server: shutting down server +31:58.585 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.585 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.585 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.585 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.585 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.585 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.585 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.585 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.585 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.585 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.591 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.596Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.596Z [INFO] agent.router.manager: shutting down +31:58.596 [INFO] TestAgent: consul server down +31:58.596 [INFO] TestAgent: shutdown complete +31:58.596 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30359 network=tcp +31:58.596 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30359 network=udp +31:58.596 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30360 network=tcp +31:59.097 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.097 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheckScriptsExecRemoteDisable (1.05s) === CONT TestAgent_JoinLANNotify +=== RUN TestAgent_ForceLeave_ACLDeny/operator_write_token +2021-10-05T16:31:58.185Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.185Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +2021-10-05T16:31:58.201Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.201Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ +31:58.249 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:545fb5b4-7c96-12b7-cb3b-5f2928b237ac Address:127.0.0.1:30350}]" +31:58.250 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac.dc1 127.0.0.1 +31:58.250 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30350 [Follower]" leader= +31:58.251 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac 127.0.0.1 +2021-10-05T16:31:58.251Z [INFO] agent.router: Initializing LAN area manager +31:58.251 [INFO] TestAgent.server: Adding LAN server: server="Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac (Addr: tcp/127.0.0.1:30350) (DC: dc1)" +31:58.251 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac.dc1 area=wan +31:58.251 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30338 network=udp +31:58.251 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30338 network=tcp +31:58.252 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30339 network=tcp +31:58.252 [INFO] TestAgent: started state syncer +31:58.252 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30302 network=tcp +31:58.296 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.296 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30350 [Candidate]" term=2 +31:58.324 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.324 [DEBUG] TestAgent.server.raft: vote granted: from=545fb5b4-7c96-12b7-cb3b-5f2928b237ac term=2 tally=1 +31:58.324 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.324 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30350 [Leader]" +31:58.324 [INFO] TestAgent.server: cluster leadership acquired +31:58.324 [INFO] TestAgent.server: New leader elected: payload=Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac +31:58.339 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30350 +31:58.371 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.438 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.438 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.438 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.438 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.438 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.438 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac +31:58.438 [INFO] TestAgent.server: member joined, marking health alive: member=Node-545fb5b4-7c96-12b7-cb3b-5f2928b237ac +31:58.458 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.477 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:58.483 [INFO] TestAgent: Synced node info +31:58.572 [DEBUG] TestAgent: Node info in sync +31:58.584 [INFO] TestAgent: Synced check: check=test +31:58.598 [DEBUG] TestAgent: Node info in sync +31:58.610 [INFO] TestAgent: Synced service: service=a +31:58.610 [DEBUG] TestAgent: Check in sync: check=test +31:58.610 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.632 [DEBUG] TestAgent: Node info in sync +31:58.637 [INFO] TestAgent: Synced service: service=a +31:58.637 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.637 [DEBUG] TestAgent: Check in sync: check=test +31:58.648 [DEBUG] TestAgent: Node info in sync +31:58.648 [DEBUG] TestAgent: Service in sync: service=a +31:58.658 [INFO] TestAgent: Synced check: check=test +31:58.658 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.667 [DEBUG] TestAgent: Node info in sync +31:58.674 [INFO] TestAgent: Synced service: service=a +31:58.674 [DEBUG] TestAgent: Check in sync: check=test +31:58.674 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.690 [DEBUG] TestAgent: Node info in sync +31:58.708 [INFO] TestAgent: Synced service: service=a +31:58.708 [DEBUG] TestAgent: Check in sync: check=test +31:58.708 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.717 [DEBUG] TestAgent: Node info in sync +31:58.717 [DEBUG] TestAgent: Service in sync: service=a +31:58.724 [INFO] TestAgent: Synced check: check=test +31:58.724 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.743 [DEBUG] TestAgent: Node info in sync +31:58.748 [INFO] TestAgent: Synced service: service=a +31:58.748 [DEBUG] TestAgent: Check in sync: check=test +31:58.748 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.769 [DEBUG] TestAgent: Node info in sync +31:58.779 [INFO] TestAgent: Synced service: service=a +31:58.779 [DEBUG] TestAgent: Check in sync: check=test +31:58.779 [DEBUG] TestAgent: Check in sync: check=service:a +31:58.779 [INFO] TestAgent: Requesting shutdown +31:58.779 [INFO] TestAgent.server: shutting down server +31:58.779 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.779 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.779 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.779 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.779 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.779 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.779 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.779 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.779 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.779 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.786 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.791Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.791Z [INFO] agent.router.manager: shutting down +31:58.791 [INFO] TestAgent: consul server down +31:58.791 [INFO] TestAgent: shutdown complete +31:58.791 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30338 network=tcp +31:58.791 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30338 network=udp +31:58.791 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30339 network=tcp +31:59.292 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.292 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_RegisterCheck_Scripts (1.19s) + --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_node_check (0.02s) + --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_top-level_service_check (0.03s) + --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_slice-based_service_check (0.03s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_node_check (0.02s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_top-level_service_check (0.02s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_slice-based_service_check (0.03s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_node_check (0.02s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_top-level_service_check (0.02s) + --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_slice-based_service_check (0.03s) +=== CONT TestAgent_Join_ACLDeny +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:57.451Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.451Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.483Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:57.483Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.523 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 Address:127.0.0.1:30280}]" +31:57.524 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30280 [Follower]" leader= +31:57.524 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9.dc1 127.0.0.1 +31:57.525 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 127.0.0.1 +2021-10-05T16:31:57.525Z [INFO] agent.router: Initializing LAN area manager +31:57.525 [INFO] TestAgent.server: Adding LAN server: server="Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 (Addr: tcp/127.0.0.1:30280) (DC: dc1)" +31:57.525 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9.dc1 area=wan +31:57.525 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30117 network=udp +31:57.526 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30117 network=tcp +31:57.526 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30118 network=tcp +31:57.526 [INFO] TestAgent: started state syncer +31:57.526 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30295 network=tcp +31:57.576 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:57.586 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.586 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30280 [Candidate]" term=2 +31:57.627 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.627 [DEBUG] TestAgent.server.raft: vote granted: from=e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 term=2 tally=1 +31:57.627 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.627 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30280 [Leader]" +31:57.627 [INFO] TestAgent.server: cluster leadership acquired +31:57.627 [INFO] TestAgent.server: New leader elected: payload=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.638 [INFO] TestAgent.server: initializing acls +31:57.645 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:57.645 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:57.652 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:57.661 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:57.664 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:57.664 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:57.664 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:57.664 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.664 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9.dc1 +31:57.664 [INFO] TestAgent.server: Updating LAN server: server="Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 (Addr: tcp/127.0.0.1:30280) (DC: dc1)" +31:57.665 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9.dc1 area=wan +31:57.687 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.718 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.718 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.718 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.718 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.718 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.718 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.718 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.732 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.741 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.777 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +31:57.777 [WARN] TestAgent: Node info update blocked by ACLs: node=e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 accessorID=00000000-0000-0000-0000-000000000002 +31:57.777 [DEBUG] TestAgent: Node info in sync +31:57.786 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.786 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e3e6be5f-77af-ef9a-03fc-b5c37d9e6ee9 +31:57.810 [INFO] TestAgent: Updated agent's ACL token: token=acl_token +31:58.888 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service="{consul {}}" +31:58.888 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:58.900 [INFO] TestAgent: Synced node info +31:58.909 [INFO] TestAgent: Requesting shutdown +31:58.909 [INFO] TestAgent.server: shutting down server +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:58.909 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:58.910 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:58.910 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.910 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:58.916 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.920Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.920Z [INFO] agent.router.manager: shutting down +31:58.920 [INFO] TestAgent: consul server down +31:58.920 [INFO] TestAgent: shutdown complete +31:58.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30117 network=tcp +31:58.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30117 network=udp +31:58.920 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30118 network=tcp +31:59.424 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.424 [INFO] TestAgent: Endpoints down +=== RUN TestAgent_TokenTriggersFullSync/default === RUN TestAgent_Leave_ACLDeny/no_token === RUN TestAgent_Leave_ACLDeny/read-only_token === RUN TestAgent_Leave_ACLDeny/agent_master_token -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:20.162Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.162Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:20.172Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.172Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:20.205 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e Address:127.0.0.1:11783}]" -46:20.205 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11783 [Follower]" leader= -46:20.205 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e.dc1 127.0.0.1 -46:20.206 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e 127.0.0.1 -2022-11-07T22:46:20.206Z [INFO] agent.router: Initializing LAN area manager -46:20.206 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e.dc1 area=wan -46:20.206 [INFO] TestAgent.server: Adding LAN server: server="Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e (Addr: tcp/127.0.0.1:11783) (DC: dc1)" -46:20.206 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11806 network=udp -46:20.206 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11806 network=tcp -46:20.206 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11779 network=tcp -46:20.206 [INFO] TestAgent: started state syncer -46:20.206 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11844 network=tcp -46:20.261 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.261 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11783 [Candidate]" term=2 -46:20.274 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.274 [DEBUG] TestAgent.server.raft: vote granted: from=2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e term=2 tally=1 -46:20.274 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.274 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11783 [Leader]" -46:20.274 [INFO] TestAgent.server: cluster leadership acquired -46:20.274 [INFO] TestAgent.server: New leader elected: payload=Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e -46:20.282 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11783 -46:20.316 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.343 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.343 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.343 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.343 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.343 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.343 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e -46:20.343 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2ed5ef0f-ff68-c3f0-fd7c-bc90f0b3755e -46:20.350 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.356 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.359 [INFO] TestAgent: Synced node info -46:20.379 [DEBUG] TestAgent: removed check: check=mem -46:20.379 [DEBUG] TestAgent: removed check: check=mem -46:20.379 [INFO] TestAgent: Requesting shutdown -46:20.379 [INFO] TestAgent.server: shutting down server -46:20.379 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.379 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.379 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.379 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.379 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.379 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.379 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.379 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.379 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.383 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.393Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.393Z [INFO] agent.router.manager: shutting down -46:20.393 [INFO] TestAgent: consul server down -46:20.393 [INFO] TestAgent: shutdown complete -46:20.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11806 network=tcp -46:20.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11806 network=udp -46:20.393 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11779 network=tcp -46:20.893 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.893 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RemoveCheck (0.74s) -=== CONT TestAgent_Reload_ACLDeny -2022-11-07T22:46:20.517Z [WARN] agent: Node name "Node 305c7ff8-d207-947c-8b43-026386536eb2" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:20.527Z [WARN] agent.auto_config: Node name "Node 305c7ff8-d207-947c-8b43-026386536eb2" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -46:20.527 [INFO] Agent2.client.serf.lan: serf: EventMemberJoin: Node 305c7ff8-d207-947c-8b43-026386536eb2 127.0.0.1 -2022-11-07T22:46:20.527Z [INFO] agent.router: Initializing LAN area manager -46:20.527 [WARN] Agent2.client.serf.lan: serf: Failed to re-join any previously known node -46:20.528 [DEBUG] Agent2: restored service definition from file: service=svc-sidecar-proxy file=/tmp/consul-test/TestAgent_DeregisterPersistedSidecarAfterRestart-TestAgent-agent769706744/services/18ed17ea501b78b6980ecfa920fcf57a -46:20.528 [DEBUG] Agent2: restored service definition from file: service=svc file=/tmp/consul-test/TestAgent_DeregisterPersistedSidecarAfterRestart-TestAgent-agent769706744/services/961e38b5146c0b07d078f53dec9b699a -46:20.528 [INFO] Agent2: Started DNS server: address=127.0.0.1:11915 network=udp -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [INFO] Agent2: Started DNS server: address=127.0.0.1:11915 network=tcp -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:20.528Z [WARN] agent.router.manager: No servers available -46:20.528 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:20.529 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11881 network=tcp -46:20.529 [INFO] Agent2: started state syncer -46:20.529 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11935 network=tcp -2022-11-07T22:46:20.529Z [WARN] agent.router.manager: No servers available -46:20.529 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:20.529 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" -46:20.529 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" -46:20.529 [DEBUG] Agent2: removed service: service=svc -46:20.529 [ERROR] Agent2.proxycfg: watch error: service_id=svc-sidecar-proxy id=service-http-checks:svc error="error filling agent cache: Internal cache failure: service 'svc' not in agent state" -46:20.529 [DEBUG] Agent2: removed service: service=svc-sidecar-proxy -46:20.529 [INFO] Agent2: Requesting shutdown -46:20.529 [INFO] Agent2.client: shutting down client -46:20.529 [WARN] Agent2.client.serf.lan: serf: Shutdown without a Leave -46:20.533 [INFO] Agent2: consul client down -46:20.533 [INFO] Agent2: shutdown complete -46:20.533 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11915 network=tcp -46:20.533 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11915 network=udp -46:20.533 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11881 network=tcp -46:21.034 [INFO] Agent2: Waiting for endpoints to shut down -46:21.034 [INFO] Agent2: Endpoints down -2022-11-07T22:46:19.980Z [WARN] agent: Node name "Node 305c7ff8-d207-947c-8b43-026386536eb2" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:19.990Z [WARN] agent.auto_config: Node name "Node 305c7ff8-d207-947c-8b43-026386536eb2" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -46:19.990 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node 305c7ff8-d207-947c-8b43-026386536eb2 127.0.0.1 -2022-11-07T22:46:19.990Z [INFO] agent.router: Initializing LAN area manager -46:19.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11820 network=udp -46:19.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11820 network=tcp -46:19.991 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11786 network=tcp -46:19.991 [INFO] TestAgent: started state syncer -46:19.991 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11791 network=tcp -2022-11-07T22:46:19.991Z [WARN] agent.router.manager: No servers available -46:19.991 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=intentions error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -46:19.998 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.998Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -46:19.999 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=roots error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -46:19.999 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -46:19.999 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -2022-11-07T22:46:19.999Z [WARN] agent.router.manager: No servers available -46:19.999 [ERROR] TestAgent.proxycfg: watch error: service_id=svc-sidecar-proxy id=leaf error="error filling agent cache: No known Consul servers" -46:20.003 [INFO] TestAgent: Requesting shutdown -46:20.003 [INFO] TestAgent.client: shutting down client -46:20.003 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:20.007 [INFO] TestAgent: consul client down -46:20.007 [INFO] TestAgent: shutdown complete -46:20.007 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11820 network=tcp -46:20.007 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11820 network=udp -46:20.007 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11786 network=tcp -46:20.507 [INFO] TestAgent: Waiting for endpoints to shut down -46:20.507 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_DeregisterPersistedSidecarAfterRestart (1.06s) -=== CONT TestAgent_Join_ACLDeny -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:19.804Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.813Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:85549767-305c-99b5-ab37-d62e160c0eda Address:127.0.0.1:11850}]" -46:19.859 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11850 [Follower]" leader= -46:19.859 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-85549767-305c-99b5-ab37-d62e160c0eda.dc1 127.0.0.1 -46:19.860 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-85549767-305c-99b5-ab37-d62e160c0eda 127.0.0.1 -2022-11-07T22:46:19.860Z [INFO] agent.router: Initializing LAN area manager -46:19.860 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-85549767-305c-99b5-ab37-d62e160c0eda.dc1 area=wan -46:19.860 [INFO] TestAgent.server: Adding LAN server: server="Node-85549767-305c-99b5-ab37-d62e160c0eda (Addr: tcp/127.0.0.1:11850) (DC: dc1)" -46:19.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11813 network=udp -46:19.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11813 network=tcp -46:19.861 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11832 network=tcp -46:19.861 [INFO] TestAgent: started state syncer -46:19.861 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11872 network=tcp -46:19.917 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.917 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11850 [Candidate]" term=2 -46:19.931 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.931 [DEBUG] TestAgent.server.raft: vote granted: from=85549767-305c-99b5-ab37-d62e160c0eda term=2 tally=1 -46:19.931 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.931 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11850 [Leader]" -46:19.931 [INFO] TestAgent.server: cluster leadership acquired -46:19.931 [INFO] TestAgent.server: New leader elected: payload=Node-85549767-305c-99b5-ab37-d62e160c0eda -46:19.941 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11850 -46:19.955 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.980 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.980 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.980 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.980 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.980 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.980 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-85549767-305c-99b5-ab37-d62e160c0eda -46:19.980 [INFO] TestAgent.server: member joined, marking health alive: member=Node-85549767-305c-99b5-ab37-d62e160c0eda -46:19.996 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.999 [INFO] TestAgent: Synced node info -46:20.021 [WARN] TestAgent: check has interval below minimum: check=tls minimum_interval=1s -46:20.248 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.248 [DEBUG] TestAgent: Node info in sync -46:20.253 [INFO] TestAgent: Synced check: check=tls -46:20.253 [DEBUG] TestAgent: Node info in sync -46:20.253 [DEBUG] TestAgent: Check in sync: check=tls -46:20.588 [DEBUG] TestAgent: Check status updated: check=tls status=passing -46:20.588 [DEBUG] TestAgent: Node info in sync -46:20.600 [INFO] TestAgent: Synced check: check=tls -46:20.609 [INFO] TestAgent: Requesting shutdown -46:20.609 [INFO] TestAgent.server: shutting down server -46:20.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.609 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.609 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.609 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.609 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.609 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.609 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.609 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.614 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.617Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.617Z [INFO] agent.router.manager: shutting down -46:20.617 [INFO] TestAgent: consul server down -46:20.617 [INFO] TestAgent: shutdown complete -46:20.617 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11813 network=tcp -46:20.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11813 network=udp -46:20.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11832 network=tcp -46:21.118 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.118 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_HTTPCheck_TLSSkipVerify (1.32s) -=== CONT TestAgent_Reload -2022-11-07T22:46:20.256Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.256Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.267Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.267Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.314 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:90bdd7c1-6294-16d7-dca4-6524cefeeaba Address:127.0.0.1:11804}]" -46:20.314 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11804 [Follower]" leader= -46:20.314 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba.dc1 127.0.0.1 -46:20.315 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba 127.0.0.1 -2022-11-07T22:46:20.315Z [INFO] agent.router: Initializing LAN area manager -46:20.315 [INFO] TestAgent.server: Adding LAN server: server="Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba (Addr: tcp/127.0.0.1:11804) (DC: dc1)" -46:20.315 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba.dc1 area=wan -46:20.315 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11845 network=udp -46:20.315 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11845 network=tcp -46:20.315 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11800 network=tcp -46:20.315 [INFO] TestAgent: started state syncer -46:20.315 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11893 network=tcp -46:20.366 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:20.378 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.378 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11804 [Candidate]" term=2 -46:20.387 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.387 [DEBUG] TestAgent.server.raft: vote granted: from=90bdd7c1-6294-16d7-dca4-6524cefeeaba term=2 tally=1 -46:20.387 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.387 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11804 [Leader]" -46:20.387 [INFO] TestAgent.server: cluster leadership acquired -46:20.387 [INFO] TestAgent.server: New leader elected: payload=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.396 [INFO] TestAgent.server: initializing acls -46:20.400 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:20.400 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:20.407 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:20.422 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:20.422 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:20.422 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:20.422 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.422 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba.dc1 -46:20.422 [INFO] TestAgent.server: Updating LAN server: server="Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba (Addr: tcp/127.0.0.1:11804) (DC: dc1)" -46:20.422 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba.dc1 area=wan -46:20.441 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.465 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.465 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.465 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.465 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.465 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.465 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.465 [INFO] TestAgent.server: member joined, marking health alive: member=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.469 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.476 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.481 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.488 [INFO] TestAgent: Synced node info -46:20.611 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.611 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.623 [INFO] TestAgent: Force leaving node: node=Node-90bdd7c1-6294-16d7-dca4-6524cefeeaba -46:20.623 [DEBUG] TestAgent.server.serf.lan: serf: Refuting an older leave intent -46:20.623 [DEBUG] TestAgent.server.serf.wan: serf: Refuting an older leave intent -46:20.623 [INFO] TestAgent: Requesting shutdown -46:20.623 [INFO] TestAgent.server: shutting down server -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:20.623 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:20.623 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.623 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:20.623 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:20.628 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.633Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.633Z [INFO] agent.router.manager: shutting down -46:20.633 [INFO] TestAgent: consul server down -46:20.633 [INFO] TestAgent: shutdown complete -46:20.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11845 network=tcp -46:20.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11845 network=udp -46:20.633 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11800 network=tcp -46:21.133 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.133 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ForceLeave_ACLDeny (0.89s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:58.682Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:58.682Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.695Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:58.695Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.748 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 Address:127.0.0.1:30336}]" +31:58.748 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30336 [Follower]" leader= +31:58.748 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2.dc1 127.0.0.1 +31:58.749 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 127.0.0.1 +2021-10-05T16:31:58.749Z [INFO] agent.router: Initializing LAN area manager +31:58.749 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2.dc1 area=wan +31:58.749 [INFO] TestAgent.server: Adding LAN server: server="Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 (Addr: tcp/127.0.0.1:30336) (DC: dc1)" +31:58.749 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30345 network=tcp +31:58.749 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30345 network=udp +31:58.750 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30346 network=tcp +31:58.750 [INFO] TestAgent: started state syncer +31:58.750 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30365 network=tcp +31:58.799 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.799 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30336 [Candidate]" term=2 +31:58.799 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:58.832 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.832 [DEBUG] TestAgent.server.raft: vote granted: from=b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 term=2 tally=1 +31:58.832 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.832 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30336 [Leader]" +31:58.832 [INFO] TestAgent.server: cluster leadership acquired +31:58.832 [INFO] TestAgent.server: New leader elected: payload=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:58.848 [INFO] TestAgent.server: initializing acls +31:58.859 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:58.859 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:58.866 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:58.879 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:58.879 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:58.879 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:58.879 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:58.879 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2.dc1 +31:58.879 [INFO] TestAgent.server: Updating LAN server: server="Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 (Addr: tcp/127.0.0.1:30336) (DC: dc1)" +31:58.879 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2.dc1 area=wan +31:58.896 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.948 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.948 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.948 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.948 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.948 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.948 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:58.948 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:58.970 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:58.988 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.068 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.086 [INFO] TestAgent: Synced node info +31:59.092 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:59.092 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:59.137 [INFO] TestAgent: Force leaving node: node=Node-b43d3f2e-a9b9-cafc-e773-fc053cdddbc2 +31:59.137 [DEBUG] TestAgent.server.serf.lan: serf: Refuting an older leave intent +31:59.137 [DEBUG] TestAgent.server.serf.wan: serf: Refuting an older leave intent +31:59.137 [INFO] TestAgent: Requesting shutdown +31:59.137 [INFO] TestAgent.server: shutting down server +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:59.137 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:59.137 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:59.138 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:59.138 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:59.149 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:59.155Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:59.155Z [INFO] agent.router.manager: shutting down +31:59.155 [INFO] TestAgent: consul server down +31:59.155 [INFO] TestAgent: shutdown complete +31:59.155 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30345 network=tcp +31:59.155 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30345 network=udp +31:59.156 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30346 network=tcp +31:59.656 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.656 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ForceLeave_ACLDeny (0.99s) --- PASS: TestAgent_ForceLeave_ACLDeny/no_token (0.00s) --- PASS: TestAgent_ForceLeave_ACLDeny/agent_master_token (0.00s) - --- PASS: TestAgent_ForceLeave_ACLDeny/read-only_token (0.00s) - --- PASS: TestAgent_ForceLeave_ACLDeny/operator_write_token (0.01s) + --- PASS: TestAgent_ForceLeave_ACLDeny/read-only_token (0.01s) + --- PASS: TestAgent_ForceLeave_ACLDeny/operator_write_token (0.04s) === CONT TestAgent_Join_WAN -2022-11-07T22:46:19.098Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.098Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.111Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:19.112Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.150 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9deaef2-c62c-8621-b0fe-79aa498fa5b0 Address:127.0.0.1:11545}]" -46:19.150 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11545 [Follower]" leader= -46:19.151 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0.dc1 127.0.0.1 -46:19.152 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 127.0.0.1 -2022-11-07T22:46:19.152Z [INFO] agent.router: Initializing LAN area manager -46:19.152 [INFO] TestAgent.server: Adding LAN server: server="Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 (Addr: tcp/127.0.0.1:11545) (DC: dc1)" -46:19.152 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0.dc1 area=wan -46:19.152 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11554 network=udp -46:19.152 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11554 network=tcp -46:19.153 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11541 network=tcp -46:19.153 [INFO] TestAgent: started state syncer -46:19.153 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11546 network=tcp -46:19.190 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.190 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11545 [Candidate]" term=2 -46:19.203 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:19.205 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.205 [DEBUG] TestAgent.server.raft: vote granted: from=c9deaef2-c62c-8621-b0fe-79aa498fa5b0 term=2 tally=1 -46:19.205 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.205 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11545 [Leader]" -46:19.205 [INFO] TestAgent.server: cluster leadership acquired -46:19.205 [INFO] TestAgent.server: New leader elected: payload=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.212 [INFO] TestAgent.server: initializing acls -46:19.216 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:19.216 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:19.223 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:19.232 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:19.232 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:19.232 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:19.232 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.232 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0.dc1 -46:19.232 [INFO] TestAgent.server: Updating LAN server: server="Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 (Addr: tcp/127.0.0.1:11545) (DC: dc1)" -46:19.232 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0.dc1 area=wan -46:19.243 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.261 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.261 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.261 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.261 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.261 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.261 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.261 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.264 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.271 [WARN] TestAgent: Node info update blocked by ACLs: node=c9deaef2-c62c-8621-b0fe-79aa498fa5b0 accessorID=00000000-0000-0000-0000-000000000002 -46:19.272 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.508 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.508 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c9deaef2-c62c-8621-b0fe-79aa498fa5b0 -46:19.524 [INFO] TestAgent: Updated agent's ACL token: token=acl_token -46:20.701 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service="{consul {}}" -46:20.701 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.707 [INFO] TestAgent: Synced node info -46:20.707 [DEBUG] TestAgent: Node info in sync -46:20.719 [INFO] TestAgent: Requesting shutdown -46:20.719 [INFO] TestAgent.server: shutting down server -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:20.719 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.719 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.719 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:20.719 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:20.728 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.735Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.735Z [INFO] agent.router.manager: shutting down -46:20.735 [INFO] TestAgent: consul server down -46:20.735 [INFO] TestAgent: shutdown complete -46:20.735 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11554 network=tcp -46:20.735 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11554 network=udp -46:20.735 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11541 network=tcp -46:21.235 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.235 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_TokenTriggersFullSync/default -2022-11-07T22:46:20.515Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.529Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.561 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:44a93758-adee-abe7-3bc5-07010da2edce Address:127.0.0.1:11899}]" -46:20.561 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11899 [Follower]" leader= -46:20.562 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-44a93758-adee-abe7-3bc5-07010da2edce.dc1 127.0.0.1 -46:20.562 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-44a93758-adee-abe7-3bc5-07010da2edce 127.0.0.1 -2022-11-07T22:46:20.562Z [INFO] agent.router: Initializing LAN area manager -46:20.562 [INFO] TestAgent.server: Adding LAN server: server="Node-44a93758-adee-abe7-3bc5-07010da2edce (Addr: tcp/127.0.0.1:11899) (DC: dc1)" -46:20.563 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-44a93758-adee-abe7-3bc5-07010da2edce.dc1 area=wan -46:20.563 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11887 network=tcp -46:20.563 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11887 network=udp -46:20.563 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11895 network=tcp -46:20.563 [INFO] TestAgent: started state syncer -46:20.563 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11879 network=tcp -46:20.625 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.625 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11899 [Candidate]" term=2 -46:20.635 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.635 [DEBUG] TestAgent.server.raft: vote granted: from=44a93758-adee-abe7-3bc5-07010da2edce term=2 tally=1 -46:20.635 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.635 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11899 [Leader]" -46:20.635 [INFO] TestAgent.server: cluster leadership acquired -46:20.635 [INFO] TestAgent.server: New leader elected: payload=Node-44a93758-adee-abe7-3bc5-07010da2edce -46:20.642 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11899 -46:20.660 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.706 [INFO] TestAgent: Synced node info -46:20.706 [DEBUG] TestAgent: Node info in sync -46:20.706 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.706 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.706 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.706 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.706 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.706 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44a93758-adee-abe7-3bc5-07010da2edce -46:20.707 [INFO] TestAgent.server: member joined, marking health alive: member=Node-44a93758-adee-abe7-3bc5-07010da2edce -46:20.724 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.728 [INFO] TestAgent: Requesting shutdown -46:20.728 [INFO] TestAgent.server: shutting down server -46:20.728 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.728 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.728 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.728 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.728 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.728 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.728 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.728 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.728 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.728 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.735 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.747Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.747Z [INFO] agent.router.manager: shutting down -46:20.747 [INFO] TestAgent: consul server down -46:20.747 [INFO] TestAgent: shutdown complete -46:20.747 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11887 network=tcp -46:20.747 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11887 network=udp -46:20.747 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11895 network=tcp -46:21.248 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.248 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_TerminatingGateway (0.75s) -=== CONT TestAgent_RegisterCheck_Scripts -2022-11-07T22:46:20.543Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.543Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.553Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.553Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.578 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5001c141-f2e0-3aff-9526-9397cc8aec00 Address:127.0.0.1:11878}]" -46:20.578 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11878 [Follower]" leader= -46:20.578 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5001c141-f2e0-3aff-9526-9397cc8aec00.dc1 127.0.0.1 -46:20.579 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5001c141-f2e0-3aff-9526-9397cc8aec00 127.0.0.1 -2022-11-07T22:46:20.579Z [INFO] agent.router: Initializing LAN area manager -46:20.579 [INFO] TestAgent.server: Adding LAN server: server="Node-5001c141-f2e0-3aff-9526-9397cc8aec00 (Addr: tcp/127.0.0.1:11878) (DC: dc1)" -46:20.579 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5001c141-f2e0-3aff-9526-9397cc8aec00.dc1 area=wan -46:20.579 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11936 network=udp -46:20.579 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11936 network=tcp -46:20.580 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11874 network=tcp -46:20.580 [INFO] TestAgent: started state syncer -46:20.580 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11942 network=tcp -46:20.629 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:20.640 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.640 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11878 [Candidate]" term=2 -46:20.656 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.656 [DEBUG] TestAgent.server.raft: vote granted: from=5001c141-f2e0-3aff-9526-9397cc8aec00 term=2 tally=1 -46:20.656 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.656 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11878 [Leader]" -46:20.656 [INFO] TestAgent.server: cluster leadership acquired -46:20.656 [INFO] TestAgent.server: New leader elected: payload=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.663 [INFO] TestAgent.server: initializing acls -46:20.669 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:20.669 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:20.680 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:20.701 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:20.701 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:20.701 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:20.701 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.701 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5001c141-f2e0-3aff-9526-9397cc8aec00.dc1 -46:20.701 [INFO] TestAgent.server: Updating LAN server: server="Node-5001c141-f2e0-3aff-9526-9397cc8aec00 (Addr: tcp/127.0.0.1:11878) (DC: dc1)" -46:20.701 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5001c141-f2e0-3aff-9526-9397cc8aec00.dc1 area=wan -46:20.718 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.749 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.749 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.749 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.749 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.749 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.750 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.750 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.763 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.767 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.783 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 -46:20.784 [DEBUG] TestAgent: dropping node from result due to ACLs: node=Node-5001c141-f2e0-3aff-9526-9397cc8aec00 accessorID=00000000-0000-0000-0000-000000000002 -46:20.784 [INFO] TestAgent: Requesting shutdown -46:20.784 [INFO] TestAgent.server: shutting down server -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:20.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.784 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.784 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.784 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:20.784 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:20.791 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.796Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.796Z [INFO] agent.router.manager: shutting down -46:20.796 [INFO] TestAgent: consul server down -46:20.796 [INFO] TestAgent: shutdown complete -46:20.796 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11936 network=tcp -46:20.796 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11936 network=udp -46:20.796 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11874 network=tcp -46:21.296 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.296 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Members_ACLFilter (0.77s) - --- PASS: TestAgent_Members_ACLFilter/no_token (0.00s) - --- PASS: TestAgent_Members_ACLFilter/root_token (0.00s) -=== CONT TestAgent_RegisterCheck_Passing -=== RUN TestAgent_Reload_ACLDeny/no_token -=== RUN TestAgent_Reload_ACLDeny/read-only_token -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:20.567Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.577Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.613 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a18adbf1-00ff-0789-ada7-077e0b73d5be Address:127.0.0.1:11920}]" -46:20.613 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11920 [Follower]" leader= -46:20.614 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a18adbf1-00ff-0789-ada7-077e0b73d5be.dc1 127.0.0.1 -46:20.615 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a18adbf1-00ff-0789-ada7-077e0b73d5be 127.0.0.1 -2022-11-07T22:46:20.615Z [INFO] agent.router: Initializing LAN area manager -46:20.615 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a18adbf1-00ff-0789-ada7-077e0b73d5be.dc1 area=wan -46:20.615 [INFO] TestAgent.server: Adding LAN server: server="Node-a18adbf1-00ff-0789-ada7-077e0b73d5be (Addr: tcp/127.0.0.1:11920) (DC: dc1)" -46:20.615 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11901 network=udp -46:20.615 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11901 network=tcp -46:20.615 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11916 network=tcp -46:20.615 [INFO] TestAgent: started state syncer -46:20.615 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11970 network=tcp -46:20.667 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.667 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11920 [Candidate]" term=2 -46:20.688 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.688 [DEBUG] TestAgent.server.raft: vote granted: from=a18adbf1-00ff-0789-ada7-077e0b73d5be term=2 tally=1 -46:20.688 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.688 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11920 [Leader]" -46:20.688 [INFO] TestAgent.server: cluster leadership acquired -46:20.689 [INFO] TestAgent.server: New leader elected: payload=Node-a18adbf1-00ff-0789-ada7-077e0b73d5be -46:20.696 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11920 -46:20.715 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.760 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.760 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.760 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.760 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.760 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.760 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a18adbf1-00ff-0789-ada7-077e0b73d5be -46:20.760 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a18adbf1-00ff-0789-ada7-077e0b73d5be -46:20.767 [INFO] TestAgent: Synced node info -46:20.779 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.781 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.786 [INFO] TestAgent: Synced node info -46:20.786 [DEBUG] TestAgent: Node info in sync -46:20.834 [INFO] TestAgent: Requesting shutdown -46:20.835 [INFO] TestAgent.server: shutting down server -46:20.835 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.835 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.835 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.835 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.835 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.835 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.835 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.835 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.835 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.835 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.841 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.847Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.847Z [INFO] agent.router.manager: shutting down -46:20.847 [INFO] TestAgent: consul server down -46:20.847 [INFO] TestAgent: shutdown complete -46:20.847 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11901 network=tcp -46:20.847 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11901 network=udp -46:20.847 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11916 network=tcp -46:21.347 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.347 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Members_WAN (0.79s) -=== CONT TestAgent_RegisterCheckScriptsExecRemoteDisable -2022-11-07T22:46:20.698Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.707Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.754 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22f1ccf8-1322-3838-6620-ec34b6492ffc Address:127.0.0.1:11990}]" -46:20.754 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11990 [Follower]" leader= -46:20.754 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-22f1ccf8-1322-3838-6620-ec34b6492ffc.dc1 127.0.0.1 -46:20.755 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-22f1ccf8-1322-3838-6620-ec34b6492ffc 127.0.0.1 -2022-11-07T22:46:20.755Z [INFO] agent.router: Initializing LAN area manager -46:20.755 [INFO] TestAgent.server: Adding LAN server: server="Node-22f1ccf8-1322-3838-6620-ec34b6492ffc (Addr: tcp/127.0.0.1:11990) (DC: dc1)" -46:20.755 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-22f1ccf8-1322-3838-6620-ec34b6492ffc.dc1 area=wan -46:20.755 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11978 network=udp -46:20.755 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11978 network=tcp -46:20.756 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11986 network=tcp -46:20.756 [INFO] TestAgent: started state syncer -46:20.756 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11928 network=tcp -46:20.823 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.823 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11990 [Candidate]" term=2 -46:20.842 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.842 [DEBUG] TestAgent.server.raft: vote granted: from=22f1ccf8-1322-3838-6620-ec34b6492ffc term=2 tally=1 -46:20.842 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.842 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11990 [Leader]" -46:20.842 [INFO] TestAgent.server: cluster leadership acquired -46:20.842 [INFO] TestAgent.server: New leader elected: payload=Node-22f1ccf8-1322-3838-6620-ec34b6492ffc -46:20.853 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11990 -46:20.870 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.899 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.899 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.899 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.899 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.899 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.899 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-22f1ccf8-1322-3838-6620-ec34b6492ffc -46:20.899 [INFO] TestAgent.server: member joined, marking health alive: member=Node-22f1ccf8-1322-3838-6620-ec34b6492ffc -46:20.911 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.933 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.936 [INFO] TestAgent: Requesting shutdown -46:20.936 [INFO] TestAgent.server: shutting down server -46:20.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.936 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.936 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.936 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.936 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.936 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.936 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.938 [INFO] TestAgent: Synced node info -46:20.942 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.953Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.953Z [INFO] agent.router.manager: shutting down -46:20.953 [INFO] TestAgent: consul server down -46:20.953 [INFO] TestAgent: shutdown complete -46:20.953 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11978 network=tcp -46:20.953 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11978 network=udp -46:20.953 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11986 network=tcp -46:21.453 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.453 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Members (0.77s) -=== CONT TestAgent_RegisterCheck -=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_node_check -=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_top-level_service_check +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestAgent_Join_ACLDeny/no_token === RUN TestAgent_Join_ACLDeny/agent_master_token === RUN TestAgent_Join_ACLDeny/read-only_token -=== RUN TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_slice-based_service_check -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_node_check -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_top-level_service_check -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_slice-based_service_check -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_node_check -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_top-level_service_check -=== RUN TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_slice-based_service_check -2022-11-07T22:46:20.904Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.904Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.913Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.913Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.957 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:35ec3c29-a27f-c18c-08b0-9169ff09b92f Address:127.0.0.1:11927}]" -46:20.957 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11927 [Follower]" leader= -46:20.958 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f.dc1 127.0.0.1 -46:20.958 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f 127.0.0.1 -2022-11-07T22:46:20.958Z [INFO] agent.router: Initializing LAN area manager -46:20.958 [INFO] TestAgent.server: Adding LAN server: server="Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f (Addr: tcp/127.0.0.1:11927) (DC: dc1)" -46:20.958 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11964 network=udp -46:20.958 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f.dc1 area=wan -46:20.958 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11964 network=tcp -46:20.959 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11923 network=tcp -46:20.959 [INFO] TestAgent: started state syncer -46:20.959 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11984 network=tcp -46:21.009 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:21.011 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.011 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11927 [Candidate]" term=2 -46:21.024 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.024 [DEBUG] TestAgent.server.raft: vote granted: from=35ec3c29-a27f-c18c-08b0-9169ff09b92f term=2 tally=1 -46:21.024 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.024 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11927 [Leader]" -46:21.024 [INFO] TestAgent.server: cluster leadership acquired -46:21.024 [INFO] TestAgent.server: New leader elected: payload=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.031 [INFO] TestAgent.server: initializing acls -46:21.034 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:21.034 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:21.041 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:21.052 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:21.052 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:21.052 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:21.052 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.052 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f.dc1 -46:21.052 [INFO] TestAgent.server: Updating LAN server: server="Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f (Addr: tcp/127.0.0.1:11927) (DC: dc1)" -46:21.052 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f.dc1 area=wan -46:21.066 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.089 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.089 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.089 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.089 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.089 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.089 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.089 [INFO] TestAgent.server: member joined, marking health alive: member=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.094 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.100 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.108 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.115 [INFO] TestAgent: Synced node info -46:21.115 [DEBUG] TestAgent: Node info in sync -46:21.298 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.298 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-35ec3c29-a27f-c18c-08b0-9169ff09b92f -46:21.308 [INFO] TestAgent: Requesting shutdown -46:21.308 [INFO] TestAgent.server: shutting down server -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.308 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:21.308 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.308 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.308 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:21.316 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.323Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.323Z [INFO] agent.router.manager: shutting down -46:21.323 [INFO] TestAgent: consul server down -46:21.323 [INFO] TestAgent: shutdown complete -46:21.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11964 network=tcp -46:21.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11964 network=udp -46:21.323 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11923 network=tcp -46:21.824 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.824 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Reload_ACLDeny (0.93s) - --- PASS: TestAgent_Reload_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_Reload_ACLDeny/read-only_token (0.01s) -=== CONT TestAgent_ForceLeavePrune +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:59.445Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.445Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.536Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.536Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.624 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:60967cc9-5c22-e184-e54e-8fce3542003c Address:127.0.0.1:30406}]" +31:59.625 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30406 [Follower]" leader= +31:59.625 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-60967cc9-5c22-e184-e54e-8fce3542003c.dc1 127.0.0.1 +31:59.626 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-60967cc9-5c22-e184-e54e-8fce3542003c 127.0.0.1 +2021-10-05T16:31:59.626Z [INFO] agent.router: Initializing LAN area manager +31:59.626 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30394 network=udp +31:59.626 [INFO] TestAgent.server: Adding LAN server: server="Node-60967cc9-5c22-e184-e54e-8fce3542003c (Addr: tcp/127.0.0.1:30406) (DC: dc1)" +31:59.626 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-60967cc9-5c22-e184-e54e-8fce3542003c.dc1 area=wan +31:59.626 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30394 network=tcp +31:59.627 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30395 network=tcp +31:59.627 [INFO] TestAgent: started state syncer +31:59.627 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29478 network=tcp +31:59.675 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.675 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30406 [Candidate]" term=2 +31:59.676 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:59.696 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.696 [DEBUG] TestAgent.server.raft: vote granted: from=60967cc9-5c22-e184-e54e-8fce3542003c term=2 tally=1 +31:59.696 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30406 [Leader]" +31:59.696 [INFO] TestAgent.server: cluster leadership acquired +31:59.696 [INFO] TestAgent.server: New leader elected: payload=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.705 [INFO] TestAgent.server: initializing acls +31:59.709 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:59.709 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:59.720 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:59.740 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:59.740 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:59.740 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:59.740 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.740 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-60967cc9-5c22-e184-e54e-8fce3542003c.dc1 +31:59.741 [INFO] TestAgent.server: Updating LAN server: server="Node-60967cc9-5c22-e184-e54e-8fce3542003c (Addr: tcp/127.0.0.1:30406) (DC: dc1)" +31:59.741 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-60967cc9-5c22-e184-e54e-8fce3542003c.dc1 area=wan +31:59.756 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.781 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.781 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.781 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.781 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.781 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.781 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.781 [INFO] TestAgent.server: member joined, marking health alive: member=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.789 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.797 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.815 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.815 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-60967cc9-5c22-e184-e54e-8fce3542003c +31:59.838 [INFO] TestAgent: Updated agent's ACL token: token=default +31:59.930 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +31:59.934 [INFO] TestAgent: Synced node info +31:59.940 [INFO] TestAgent: Requesting shutdown +31:59.940 [INFO] TestAgent.server: shutting down server +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:59.940 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:59.940 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:59.940 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:59.940 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:59.948 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:59.958Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:59.958Z [INFO] agent.router.manager: shutting down +31:59.958 [INFO] TestAgent: consul server down +31:59.958 [INFO] TestAgent: shutdown complete +31:59.958 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30394 network=tcp +31:59.958 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30394 network=udp +31:59.958 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30395 network=tcp +32:00.458 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.458 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_TokenTriggersFullSync (6.63s) + --- PASS: TestAgent_TokenTriggersFullSync/acl_agent_token (1.24s) + --- PASS: TestAgent_TokenTriggersFullSync/agent (2.34s) + --- PASS: TestAgent_TokenTriggersFullSync/acl_token (2.01s) + --- PASS: TestAgent_TokenTriggersFullSync/default (1.03s) +=== CONT TestParseWait_InvalidIndex +--- PASS: TestParseWait_InvalidIndex (0.00s) +=== CONT TestParseConsistency [WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:20.570Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.578Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.610 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f4c34940-c304-6ce4-ed02-4966edc24e29 Address:127.0.0.1:11941}]" -46:20.610 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11941 [Follower]" leader= -46:20.611 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4c34940-c304-6ce4-ed02-4966edc24e29.dc1 127.0.0.1 -46:20.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4c34940-c304-6ce4-ed02-4966edc24e29 127.0.0.1 -2022-11-07T22:46:20.612Z [INFO] agent.router: Initializing LAN area manager -46:20.612 [INFO] TestAgent.server: Adding LAN server: server="Node-f4c34940-c304-6ce4-ed02-4966edc24e29 (Addr: tcp/127.0.0.1:11941) (DC: dc1)" -46:20.612 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4c34940-c304-6ce4-ed02-4966edc24e29.dc1 area=wan -46:20.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11971 network=udp -46:20.612 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11971 network=tcp -46:20.612 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11937 network=tcp -46:20.612 [INFO] TestAgent: started state syncer -46:20.612 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11977 network=tcp -46:20.648 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.648 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11941 [Candidate]" term=2 -46:20.662 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.662 [DEBUG] TestAgent.server.raft: vote granted: from=f4c34940-c304-6ce4-ed02-4966edc24e29 term=2 tally=1 -46:20.662 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.662 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11941 [Leader]" -46:20.662 [INFO] TestAgent.server: cluster leadership acquired -46:20.662 [INFO] TestAgent.server: New leader elected: payload=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 -46:20.672 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11941 -46:20.695 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.733 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.733 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.733 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.733 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.733 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.733 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 -46:20.733 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 -46:20.753 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.842 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.848 [INFO] TestAgent: Synced node info -46:20.848 [DEBUG] TestAgent: Node info in sync -46:20.959 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:48970 -46:20.959 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b7e12046-c634-538b-1de0-f4a9e3825f1d 127.0.0.1 -46:20.959 [INFO] TestAgent.server: Adding LAN server: server="Node-b7e12046-c634-538b-1de0-f4a9e3825f1d (Addr: tcp/127.0.0.1:11871) (DC: dc1)" -46:20.959 [INFO] TestAgent.server: New leader elected: payload=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.959 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d other=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 -46:20.959 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.960 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 127.0.0.1:11870 -46:20.960 [INFO] TestAgent: Requesting shutdown -46:20.960 [INFO] TestAgent.server: shutting down server -46:20.960 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.960 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.960 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.960 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.960 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.960 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.960 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.960 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.960 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.960 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.960 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 127.0.0.1 -46:20.960 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 address=127.0.0.1:11870 -46:20.960 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 area=wan -46:20.966 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.969Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.969Z [INFO] agent.router.manager: shutting down -46:20.969 [INFO] TestAgent: consul server down -46:20.969 [INFO] TestAgent: shutdown complete -46:20.969 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11971 network=tcp -46:20.969 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11971 network=udp -46:20.969 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11937 network=tcp -46:21.470 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.470 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.294Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.305Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.349 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b7e12046-c634-538b-1de0-f4a9e3825f1d Address:127.0.0.1:11871}]" -46:20.350 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11871 [Follower]" leader= -46:20.350 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 127.0.0.1 -46:20.350 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b7e12046-c634-538b-1de0-f4a9e3825f1d 127.0.0.1 -2022-11-07T22:46:20.350Z [INFO] agent.router: Initializing LAN area manager -46:20.351 [INFO] TestAgent.server: Adding LAN server: server="Node-b7e12046-c634-538b-1de0-f4a9e3825f1d (Addr: tcp/127.0.0.1:11871) (DC: dc1)" -46:20.351 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d.dc1 area=wan -46:20.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11792 network=tcp -46:20.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11792 network=udp -46:20.351 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11867 network=tcp -46:20.351 [INFO] TestAgent: started state syncer -46:20.351 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11907 network=tcp -46:20.408 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.408 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11871 [Candidate]" term=2 -46:20.423 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.423 [DEBUG] TestAgent.server.raft: vote granted: from=b7e12046-c634-538b-1de0-f4a9e3825f1d term=2 tally=1 -46:20.423 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.423 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11871 [Leader]" -46:20.424 [INFO] TestAgent.server: cluster leadership acquired -46:20.424 [INFO] TestAgent.server: New leader elected: payload=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.434 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11871 -46:20.452 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.475 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.475 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.475 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.475 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.475 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.475 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.475 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.483 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.675 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.682 [INFO] TestAgent: Synced node info -46:20.682 [DEBUG] TestAgent: Node info in sync -46:20.959 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11939] -46:20.959 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11939 -46:20.960 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:55956 -46:20.960 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4c34940-c304-6ce4-ed02-4966edc24e29 127.0.0.1 -46:20.960 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:20.960 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:20.960 [INFO] TestAgent.server: Adding LAN server: server="Node-f4c34940-c304-6ce4-ed02-4966edc24e29 (Addr: tcp/127.0.0.1:11941) (DC: dc1)" -46:20.960 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 other=Node-b7e12046-c634-538b-1de0-f4a9e3825f1d -46:20.960 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4c34940-c304-6ce4-ed02-4966edc24e29 -46:20.960 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4c34940-c304-6ce4-ed02-4966edc24e29.dc1 127.0.0.1 -46:20.960 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4c34940-c304-6ce4-ed02-4966edc24e29.dc1 area=wan -46:21.470 [INFO] TestAgent: Requesting shutdown -46:21.470 [INFO] TestAgent.server: shutting down server -46:21.470 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.470 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.470 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.470 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.470 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.470 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.470 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.470 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.470 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.470 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.477 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.483Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.483Z [INFO] agent.router.manager: shutting down -46:21.483 [INFO] TestAgent: consul server down -46:21.483 [INFO] TestAgent: shutdown complete -46:21.483 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11792 network=tcp -46:21.483 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11792 network=udp -46:21.483 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11867 network=tcp -46:21.858 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-f4c34940-c304-6ce4-ed02-4966edc24e29 (timeout reached) -46:21.983 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.983 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Join (1.71s) -=== CONT TestAgent_RegisterCheckScriptsExecDisable +31:59.582 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-92023ef8-08cc-0a76-fce7-139ab9eb93ef 127.0.0.1 +2021-10-05T16:31:59.582Z [INFO] agent.router: Initializing LAN area manager +31:59.582 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29444 network=udp +31:59.583 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29444 network=tcp +31:59.583 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29445 network=tcp +31:59.583 [INFO] TestAgent: started state syncer +31:59.585 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Stream connection from=127.0.0.1:35654 +31:59.586 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 127.0.0.1 +31:59.586 [INFO] TestAgent.client: adding server: server="Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 (Addr: tcp/127.0.0.1:30399) (DC: dc1)" +31:59.586 [INFO] TestAgent.client: New leader elected: payload=Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 +31:59.586 [INFO] TestAgent: Requesting shutdown +31:59.586 [INFO] TestAgent.client: shutting down client +31:59.586 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave +31:59.588 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30351 network=tcp +31:59.588 [ERROR] TestAgent: gRPC server failed: error="grpc: the server has been stopped" +31:59.604 [INFO] TestAgent: consul client down +31:59.604 [INFO] TestAgent: shutdown complete +31:59.604 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29444 network=tcp +31:59.604 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29444 network=udp +31:59.604 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29445 network=tcp +31:59.604 [ERROR] TestAgent.client: RPC failed to server: method=Catalog.Register server=127.0.0.1:30399 error="rpc error making call: EOF" +31:59.604 [WARN] TestAgent: Syncing node info failed.: error="rpc error making call: EOF" +31:59.604 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="rpc error making call: EOF" +32:00.104 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.104 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:59.113Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.127Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.183 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:19e3f2f3-509a-82cb-4954-dc738ccf7e93 Address:127.0.0.1:30399}]" +31:59.184 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30399 [Follower]" leader= +31:59.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93.dc1 127.0.0.1 +31:59.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 127.0.0.1 +2021-10-05T16:31:59.184Z [INFO] agent.router: Initializing LAN area manager +31:59.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30380 network=udp +31:59.185 [INFO] TestAgent.server: Adding LAN server: server="Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 (Addr: tcp/127.0.0.1:30399) (DC: dc1)" +31:59.185 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93.dc1 area=wan +31:59.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30380 network=tcp +31:59.185 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30381 network=tcp +31:59.186 [INFO] TestAgent: started state syncer +31:59.186 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30414 network=tcp +31:59.231 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.231 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30399 [Candidate]" term=2 +31:59.257 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.257 [DEBUG] TestAgent.server.raft: vote granted: from=19e3f2f3-509a-82cb-4954-dc738ccf7e93 term=2 tally=1 +31:59.257 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.257 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30399 [Leader]" +31:59.257 [INFO] TestAgent.server: cluster leadership acquired +31:59.257 [INFO] TestAgent.server: New leader elected: payload=Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 +31:59.278 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30399 +31:59.309 [INFO] TestAgent: Synced node info +31:59.309 [DEBUG] TestAgent: Node info in sync +31:59.334 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.370 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.370 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.370 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.370 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.370 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.370 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 +31:59.370 [INFO] TestAgent.server: member joined, marking health alive: member=Node-19e3f2f3-509a-82cb-4954-dc738ccf7e93 +31:59.393 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.584 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:29447] +31:59.585 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:29447 +31:59.586 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-92023ef8-08cc-0a76-fce7-139ab9eb93ef 127.0.0.1 +31:59.586 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:59.587 [INFO] TestAgent.server: member joined, marking health alive: member=Node-92023ef8-08cc-0a76-fce7-139ab9eb93ef +32:00.104 [INFO] TestAgent: Requesting shutdown +32:00.104 [INFO] TestAgent.server: shutting down server +32:00.104 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.104 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.104 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.104 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.104 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.104 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.104 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.104 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.104 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.105 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.119 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.125Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.125Z [INFO] agent.router.manager: shutting down +32:00.125 [INFO] TestAgent: consul server down +32:00.125 [INFO] TestAgent: shutdown complete +32:00.125 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30380 network=tcp +32:00.125 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30380 network=udp +32:00.125 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30381 network=tcp +32:00.626 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.626 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_JoinLANNotify (1.53s) +=== CONT TestOperator_RaftPeer +=== RUN TestOperator_RaftPeer/#00 +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:59.646Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.673Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.749 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8426086-03f6-4686-c3c1-8f34bb410155 Address:127.0.0.1:30357}]" +31:59.749 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8426086-03f6-4686-c3c1-8f34bb410155.dc1 127.0.0.1 +31:59.750 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30357 [Follower]" leader= +31:59.750 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8426086-03f6-4686-c3c1-8f34bb410155 127.0.0.1 +2021-10-05T16:31:59.750Z [INFO] agent.router: Initializing LAN area manager +31:59.750 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30352 network=udp +31:59.750 [INFO] TestAgent.server: Adding LAN server: server="Node-f8426086-03f6-4686-c3c1-8f34bb410155 (Addr: tcp/127.0.0.1:30357) (DC: dc1)" +31:59.750 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8426086-03f6-4686-c3c1-8f34bb410155.dc1 area=wan +31:59.750 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30352 network=tcp +31:59.751 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30353 network=tcp +31:59.751 [INFO] TestAgent: started state syncer +31:59.751 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29408 network=tcp +31:59.816 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.816 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30357 [Candidate]" term=2 +31:59.841 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.841 [DEBUG] TestAgent.server.raft: vote granted: from=f8426086-03f6-4686-c3c1-8f34bb410155 term=2 tally=1 +31:59.841 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.841 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30357 [Leader]" +31:59.841 [INFO] TestAgent.server: cluster leadership acquired +31:59.841 [INFO] TestAgent.server: New leader elected: payload=Node-f8426086-03f6-4686-c3c1-8f34bb410155 +31:59.849 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30357 +31:59.870 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.898 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.898 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.898 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.898 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.898 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.898 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8426086-03f6-4686-c3c1-8f34bb410155 +31:59.898 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8426086-03f6-4686-c3c1-8f34bb410155 +31:59.922 [INFO] TestAgent.server: federation state anti-entropy synced +32:00.023 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:00.024 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:44992 +32:00.025 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70f24462-3485-00ce-04b3-1596890ad559 127.0.0.1 +32:00.025 [INFO] TestAgent.server: Adding LAN server: server="Node-70f24462-3485-00ce-04b3-1596890ad559 (Addr: tcp/127.0.0.1:29400) (DC: dc1)" +32:00.025 [INFO] TestAgent.server: New leader elected: payload=Node-70f24462-3485-00ce-04b3-1596890ad559 +32:00.025 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-70f24462-3485-00ce-04b3-1596890ad559 other=Node-f8426086-03f6-4686-c3c1-8f34bb410155 +32:00.025 [INFO] TestAgent.server: member joined, marking health alive: member=Node-70f24462-3485-00ce-04b3-1596890ad559 +32:00.026 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 127.0.0.1:30412 +32:00.027 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 127.0.0.1 +32:00.027 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 address=127.0.0.1:30412 +32:00.027 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 area=wan +32:00.028 [INFO] TestAgent: Synced node info +32:00.028 [DEBUG] TestAgent: Node info in sync +32:00.046 [INFO] TestAgent: Requesting shutdown +32:00.046 [INFO] TestAgent.server: shutting down server +32:00.046 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.046 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.046 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.046 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.046 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.046 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.046 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.046 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.046 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.046 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.052 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.057Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.058Z [INFO] agent.router.manager: shutting down +32:00.058 [INFO] TestAgent: consul server down +32:00.058 [INFO] TestAgent: shutdown complete +32:00.058 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30352 network=tcp +32:00.058 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30352 network=udp +32:00.058 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30353 network=tcp +32:00.558 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.558 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:59.323Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.323Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.343Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.343Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.409 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:70f24462-3485-00ce-04b3-1596890ad559 Address:127.0.0.1:29400}]" +31:59.410 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 127.0.0.1 +31:59.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70f24462-3485-00ce-04b3-1596890ad559 127.0.0.1 +2021-10-05T16:31:59.411Z [INFO] agent.router: Initializing LAN area manager +31:59.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30408 network=udp +31:59.411 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29400 [Follower]" leader= +31:59.412 [INFO] TestAgent.server: Adding LAN server: server="Node-70f24462-3485-00ce-04b3-1596890ad559 (Addr: tcp/127.0.0.1:29400) (DC: dc1)" +31:59.412 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 area=wan +31:59.412 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30408 network=tcp +31:59.413 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30409 network=tcp +31:59.413 [INFO] TestAgent: started state syncer +31:59.413 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29401 network=tcp +31:59.463 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:59.484 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.484 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29400 [Candidate]" term=2 +31:59.517 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.517 [DEBUG] TestAgent.server.raft: vote granted: from=70f24462-3485-00ce-04b3-1596890ad559 term=2 tally=1 +31:59.517 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.517 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29400 [Leader]" +31:59.522 [INFO] TestAgent.server: cluster leadership acquired +31:59.527 [INFO] TestAgent.server: initializing acls +31:59.528 [INFO] TestAgent.server: New leader elected: payload=Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.532 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:59.532 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:59.548 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:59.564 [INFO] TestAgent.server: initializing acls +31:59.564 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:59.587 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:59.587 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:59.587 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:59.587 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:59.587 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.588 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 +31:59.588 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 area=wan +31:59.588 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:59.588 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.588 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 +31:59.589 [INFO] TestAgent.server: Updating LAN server: server="Node-70f24462-3485-00ce-04b3-1596890ad559 (Addr: tcp/127.0.0.1:29400) (DC: dc1)" +31:59.589 [INFO] TestAgent.server: Updating LAN server: server="Node-70f24462-3485-00ce-04b3-1596890ad559 (Addr: tcp/127.0.0.1:29400) (DC: dc1)" +31:59.590 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-70f24462-3485-00ce-04b3-1596890ad559.dc1 area=wan +31:59.627 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.653 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.653 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.653 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.653 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.653 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.653 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.653 [INFO] TestAgent.server: member joined, marking health alive: member=Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.658 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.668 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.668 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70f24462-3485-00ce-04b3-1596890ad559 +31:59.713 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.734 [INFO] TestAgent: Synced node info +31:59.734 [DEBUG] TestAgent: Node info in sync +32:00.024 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-70f24462-3485-00ce-04b3-1596890ad559 +32:00.024 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:30355] +32:00.024 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30355 +32:00.025 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8426086-03f6-4686-c3c1-8f34bb410155 127.0.0.1 +32:00.026 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +32:00.026 [DEBUG] TestAgent: systemd notify failed: error="No socket" +32:00.026 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:48810 +32:00.026 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8426086-03f6-4686-c3c1-8f34bb410155.dc1 127.0.0.1 +32:00.027 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8426086-03f6-4686-c3c1-8f34bb410155.dc1 area=wan +32:00.027 [INFO] TestAgent.server: Adding LAN server: server="Node-f8426086-03f6-4686-c3c1-8f34bb410155 (Addr: tcp/127.0.0.1:30357) (DC: dc1)" +32:00.027 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-f8426086-03f6-4686-c3c1-8f34bb410155 other=Node-70f24462-3485-00ce-04b3-1596890ad559 +32:00.027 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8426086-03f6-4686-c3c1-8f34bb410155 +32:00.558 [INFO] TestAgent: Requesting shutdown +32:00.558 [INFO] TestAgent.server: shutting down server +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.558 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.558 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.558 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.558 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.562 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.566Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.566Z [INFO] agent.router.manager: shutting down +32:00.566 [INFO] TestAgent: consul server down +32:00.566 [INFO] TestAgent: shutdown complete +32:00.566 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30408 network=tcp +32:00.566 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30408 network=udp +32:00.566 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30409 network=tcp +32:00.913 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-f8426086-03f6-4686-c3c1-8f34bb410155 (timeout reached) +32:01.068 [INFO] TestAgent: Waiting for endpoints to shut down +32:01.068 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Join_ACLDeny (1.77s) + --- PASS: TestAgent_Join_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_Join_ACLDeny/agent_master_token (0.00s) + --- PASS: TestAgent_Join_ACLDeny/read-only_token (0.02s) +=== CONT TestOperator_KeyringInstall [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:21.599Z [WARN] TestAgent_Reload: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.127Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.137Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.177 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ff00f524-8f3b-0854-d47c-e7f30b2f3daa Address:127.0.0.1:11976}]" -46:21.177 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11976 [Follower]" leader= -46:21.177 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa.dc1 127.0.0.1 -46:21.177 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa 127.0.0.1 -2022-11-07T22:46:21.177Z [INFO] agent.router: Initializing LAN area manager -46:21.178 [INFO] TestAgent.server: Adding LAN server: server="Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa (Addr: tcp/127.0.0.1:11976) (DC: dc1)" -46:21.178 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa.dc1 area=wan -46:21.178 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12013 network=udp -46:21.178 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12013 network=tcp -46:21.178 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11972 network=tcp -46:21.178 [WARN] TestAgent: The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html -46:21.178 [INFO] TestAgent: started state syncer -46:21.178 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11858 network=tcp -46:21.230 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.230 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11976 [Candidate]" term=2 -46:21.249 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.249 [DEBUG] TestAgent.server.raft: vote granted: from=ff00f524-8f3b-0854-d47c-e7f30b2f3daa term=2 tally=1 -46:21.249 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.249 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11976 [Leader]" -46:21.249 [INFO] TestAgent.server: cluster leadership acquired -46:21.249 [INFO] TestAgent.server: New leader elected: payload=Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa -46:21.264 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11976 -46:21.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.321 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.321 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.321 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.321 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.321 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.321 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa -46:21.321 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ff00f524-8f3b-0854-d47c-e7f30b2f3daa -46:21.342 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.391 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.398 [INFO] TestAgent: Synced node info -46:21.407 [INFO] TestAgent: Synced service: service=redis -46:21.407 [DEBUG] TestAgent: Node info in sync -46:21.407 [DEBUG] TestAgent: Service in sync: service=redis -46:21.453 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test?dc=dc1 from=127.0.0.1:52806 latency=274.103892ms -46:21.458 [DEBUG] TestAgent: watch handler output: watch_handler=true output= -46:21.599 [INFO] TestAgent: Requesting shutdown -46:21.599 [INFO] TestAgent.server: shutting down server -46:21.599 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.599 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.605 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.611Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.611Z [INFO] agent.router.manager: shutting down -46:21.611 [INFO] TestAgent: consul server down -46:21.611 [INFO] TestAgent: shutdown complete -46:21.611 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12013 network=tcp -46:21.611 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12013 network=udp -46:21.611 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11972 network=tcp -46:22.112 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.112 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Reload (0.99s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:00.049Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:00.072Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:00.118 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:710b27fc-6301-ef80-4683-c86632fa3f53 Address:127.0.0.1:29435}]" +32:00.118 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-710b27fc-6301-ef80-4683-c86632fa3f53.dc1 127.0.0.1 +32:00.119 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-710b27fc-6301-ef80-4683-c86632fa3f53 127.0.0.1 +2021-10-05T16:32:00.119Z [INFO] agent.router: Initializing LAN area manager +32:00.119 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29486 network=udp +32:00.119 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29435 [Follower]" leader= +32:00.119 [INFO] TestAgent.server: Adding LAN server: server="Node-710b27fc-6301-ef80-4683-c86632fa3f53 (Addr: tcp/127.0.0.1:29435) (DC: dc1)" +32:00.120 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-710b27fc-6301-ef80-4683-c86632fa3f53.dc1 area=wan +32:00.120 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29486 network=tcp +32:00.120 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29487 network=tcp +32:00.120 [INFO] TestAgent: started state syncer +32:00.120 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29471 network=tcp +32:00.156 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:00.156 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29435 [Candidate]" term=2 +32:00.180 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:00.180 [DEBUG] TestAgent.server.raft: vote granted: from=710b27fc-6301-ef80-4683-c86632fa3f53 term=2 tally=1 +32:00.180 [INFO] TestAgent.server.raft: election won: tally=1 +32:00.180 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29435 [Leader]" +32:00.180 [INFO] TestAgent.server: cluster leadership acquired +32:00.180 [INFO] TestAgent.server: New leader elected: payload=Node-710b27fc-6301-ef80-4683-c86632fa3f53 +32:00.196 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29435 +32:00.223 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:00.267 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:00.268 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:00.268 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:00.268 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:00.268 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:00.268 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-710b27fc-6301-ef80-4683-c86632fa3f53 +32:00.268 [INFO] TestAgent.server: member joined, marking health alive: member=Node-710b27fc-6301-ef80-4683-c86632fa3f53 +32:00.284 [INFO] TestAgent.server: federation state anti-entropy synced +32:00.304 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:60606 +32:00.304 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d1403890-9142-3ca6-025d-6510d2141537.dc1 127.0.0.1 +32:00.304 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d1403890-9142-3ca6-025d-6510d2141537.dc1 area=wan +32:00.305 [INFO] TestAgent: Requesting shutdown +32:00.305 [INFO] TestAgent.server: shutting down server +32:00.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.305 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.305 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.305 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.305 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.305 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.305 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:00.305 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.305 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.305 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.313 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.323Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.323Z [INFO] agent.router.manager: shutting down +32:00.323 [INFO] TestAgent: consul server down +32:00.323 [INFO] TestAgent: shutdown complete +32:00.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29486 network=tcp +32:00.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29486 network=udp +32:00.323 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29487 network=tcp +32:00.823 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.823 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:59.673Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.702Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.788 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d1403890-9142-3ca6-025d-6510d2141537 Address:127.0.0.1:29414}]" +31:59.788 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29414 [Follower]" leader= +31:59.789 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d1403890-9142-3ca6-025d-6510d2141537.dc1 127.0.0.1 +31:59.789 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d1403890-9142-3ca6-025d-6510d2141537 127.0.0.1 +2021-10-05T16:31:59.789Z [INFO] agent.router: Initializing LAN area manager +31:59.789 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29409 network=udp +31:59.789 [INFO] TestAgent.server: Adding LAN server: server="Node-d1403890-9142-3ca6-025d-6510d2141537 (Addr: tcp/127.0.0.1:29414) (DC: dc1)" +31:59.789 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d1403890-9142-3ca6-025d-6510d2141537.dc1 area=wan +31:59.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29409 network=tcp +31:59.790 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29410 network=tcp +31:59.790 [INFO] TestAgent: started state syncer +31:59.790 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29415 network=tcp +31:59.856 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.856 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29414 [Candidate]" term=2 +31:59.894 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.894 [DEBUG] TestAgent.server.raft: vote granted: from=d1403890-9142-3ca6-025d-6510d2141537 term=2 tally=1 +31:59.894 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.894 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29414 [Leader]" +31:59.896 [INFO] TestAgent.server: cluster leadership acquired +31:59.896 [INFO] TestAgent.server: New leader elected: payload=Node-d1403890-9142-3ca6-025d-6510d2141537 +31:59.905 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29414 +31:59.948 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.986 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.986 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.986 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.986 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.986 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.986 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d1403890-9142-3ca6-025d-6510d2141537 +31:59.986 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d1403890-9142-3ca6-025d-6510d2141537 +32:00.000 [INFO] TestAgent.server: federation state anti-entropy synced +32:00.204 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:00.209 [INFO] TestAgent: Synced node info +32:00.209 [DEBUG] TestAgent: Node info in sync +32:00.303 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29490] +32:00.303 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29490 +32:00.305 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-710b27fc-6301-ef80-4683-c86632fa3f53.dc1 127.0.0.1 +32:00.305 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +32:00.305 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-710b27fc-6301-ef80-4683-c86632fa3f53.dc1 area=wan +32:00.823 [INFO] TestAgent: Requesting shutdown +32:00.823 [INFO] TestAgent.server: shutting down server +32:00.823 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.823 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.823 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.823 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.823 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.823 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.823 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.824 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.824 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.824 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.826 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.831Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.831Z [INFO] agent.router.manager: shutting down +32:00.831 [INFO] TestAgent: consul server down +32:00.831 [INFO] TestAgent: shutdown complete +32:00.831 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29409 network=tcp +32:00.831 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29409 network=udp +32:00.831 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29410 network=tcp +32:01.331 [INFO] TestAgent: Waiting for endpoints to shut down +32:01.331 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Join_WAN (1.68s) +=== CONT TestParseWait_InvalidTime +--- PASS: TestParseWait_InvalidTime (0.00s) +=== CONT TestACL_filterChecks +2021-10-05T16:32:01.343Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:01.343Z [WARN] agent: bootstrap = true: do not enable unless necessary +32:01.343 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-node +32:01.343 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-other +32:01.343 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-service +32:01.343 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-other +--- PASS: TestACL_filterChecks (0.01s) +=== CONT TestAgent_Members_ACLFilter +2021-10-05T16:32:00.474Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:00.495Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:00.554 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5136b235-46ca-75b6-5a4d-8af0e81182e4 Address:127.0.0.1:29484}]" +32:00.554 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29484 [Follower]" leader= +32:00.555 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5136b235-46ca-75b6-5a4d-8af0e81182e4.dc1 127.0.0.1 +32:00.556 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5136b235-46ca-75b6-5a4d-8af0e81182e4 127.0.0.1 +2021-10-05T16:32:00.556Z [INFO] agent.router: Initializing LAN area manager +32:00.556 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29423 network=udp +32:00.556 [INFO] TestAgent.server: Adding LAN server: server="Node-5136b235-46ca-75b6-5a4d-8af0e81182e4 (Addr: tcp/127.0.0.1:29484) (DC: dc1)" +32:00.556 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5136b235-46ca-75b6-5a4d-8af0e81182e4.dc1 area=wan +32:00.556 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29423 network=tcp +32:00.557 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29424 network=tcp +32:00.557 [INFO] TestAgent: started state syncer +32:00.557 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29436 network=tcp +32:00.624 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:00.624 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29484 [Candidate]" term=2 +32:00.660 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:00.660 [DEBUG] TestAgent.server.raft: vote granted: from=5136b235-46ca-75b6-5a4d-8af0e81182e4 term=2 tally=1 +32:00.660 [INFO] TestAgent.server.raft: election won: tally=1 +32:00.660 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29484 [Leader]" +32:00.660 [INFO] TestAgent.server: cluster leadership acquired +32:00.660 [INFO] TestAgent.server: New leader elected: payload=Node-5136b235-46ca-75b6-5a4d-8af0e81182e4 +32:00.688 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29484 +32:00.726 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:00.750 [INFO] TestAgent: Synced node info +32:00.750 [DEBUG] TestAgent: Node info in sync +32:00.762 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:00.762 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:00.762 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:00.762 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:00.762 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:00.762 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5136b235-46ca-75b6-5a4d-8af0e81182e4 +32:00.762 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5136b235-46ca-75b6-5a4d-8af0e81182e4 +32:00.782 [INFO] TestAgent.server: federation state anti-entropy synced +32:00.844 [INFO] TestAgent: Requesting shutdown +32:00.844 [INFO] TestAgent.server: shutting down server +32:00.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.844 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.844 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.844 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.844 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.844 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.844 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.849 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.861Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.861Z [INFO] agent.router.manager: shutting down +32:00.861 [INFO] TestAgent: consul server down +32:00.861 [INFO] TestAgent: shutdown complete +32:00.861 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29423 network=tcp +32:00.861 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29423 network=udp +32:00.861 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29424 network=tcp +32:01.364 [INFO] TestAgent: Waiting for endpoints to shut down +32:01.364 [INFO] TestAgent: Endpoints down +--- PASS: TestParseConsistency (0.91s) +=== CONT TestAgent_Members_WAN +2021-10-05T16:31:52.725Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.749Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.832 [INFO] dc2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d7aa34bd-d7b6-593e-4946-2b559f4b7f99 Address:127.0.0.1:30028}]" +31:52.836 [INFO] dc2.server.raft: entering follower state: follower="Node at 127.0.0.1:30028 [Follower]" leader= +31:52.836 [INFO] dc2.server.serf.wan: serf: EventMemberJoin: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 127.0.0.1 +31:52.837 [INFO] dc2.server.serf.lan: serf: EventMemberJoin: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99 127.0.0.1 +2021-10-05T16:31:52.837Z [INFO] agent.router: Initializing LAN area manager +31:52.837 [INFO] dc2: Started DNS server: address=127.0.0.1:29967 network=udp +31:52.837 [INFO] dc2.server: Handled event for server in area: event=member-join server=Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 area=wan +31:52.838 [INFO] dc2.server: Adding LAN server: server="Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99 (Addr: tcp/127.0.0.1:30028) (DC: dc2)" +31:52.838 [INFO] dc2: Started DNS server: address=127.0.0.1:29967 network=tcp +31:52.838 [INFO] dc2: Started HTTP server: address=127.0.0.1:30024 network=tcp +31:52.838 [INFO] dc2: started state syncer +31:52.838 [INFO] dc2: Started gRPC server: address=127.0.0.1:30022 network=tcp +31:52.897 [WARN] dc2.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.897 [INFO] dc2.server.raft: entering candidate state: node="Node at 127.0.0.1:30028 [Candidate]" term=2 +31:52.918 [DEBUG] dc2.server.raft: votes: needed=1 +31:52.918 [DEBUG] dc2.server.raft: vote granted: from=d7aa34bd-d7b6-593e-4946-2b559f4b7f99 term=2 tally=1 +31:52.918 [INFO] dc2.server.raft: election won: tally=1 +31:52.918 [INFO] dc2.server.raft: entering leader state: leader="Node at 127.0.0.1:30028 [Leader]" +31:52.918 [INFO] dc2.server: cluster leadership acquired +31:52.919 [INFO] dc2.server: New leader elected: payload=Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99 +31:52.933 [DEBUG] dc2.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false +31:52.961 [WARN] dc2.server.connect: primary datacenter is configured but unreachable - deferring initialization of the secondary datacenter CA +31:52.961 [INFO] dc2.leader: started routine: routine="config entry replication" +31:52.961 [INFO] dc2.leader: started routine: routine="federation state replication" +31:52.961 [INFO] dc2.leader: started routine: routine="federation state anti-entropy" +31:52.961 [INFO] dc2.leader: started routine: routine="secondary CA roots watch" +31:52.961 [INFO] dc2.leader: started routine: routine="intention replication" +31:52.961 [INFO] dc2.leader: started routine: routine="intermediate cert renew watch" +31:52.961 [INFO] dc2.leader: started routine: routine="CA root pruning" +31:52.961 [DEBUG] dc2.server: Skipping self join check for node since the cluster is too small: node=Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99 +31:52.961 [INFO] dc2.server: member joined, marking health alive: member=Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99 +31:52.961 [DEBUG] dc2.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 +31:52.961 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConnectCA.Roots +31:52.961 [ERROR] dc2.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: No path to datacenter" +31:52.961 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConfigEntry.ListAll +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server: federation states are not enabled in the primary dc +31:52.961 [DEBUG] dc2.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 +31:52.962 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=Intention.List +31:52.962 [ERROR] dc2.server.connect: error replicating intentions: routine="intention replication" error="No path to datacenter" +31:53.196 [INFO] dc2: (WAN) joining: wan_addresses=[127.0.0.1:29985] +31:53.200 [DEBUG] dc2.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29985 +31:53.200 [DEBUG] dc2: Skipping remote check since it is managed automatically: check=serfHealth +31:53.205 [INFO] dc2.server.serf.wan: serf: EventMemberJoin: Node-237d687c-e901-a162-14cc-6108db37e68c.dc1 127.0.0.1 +31:53.205 [INFO] dc2: (WAN) joined: number_of_nodes=1 +31:53.208 [INFO] dc2.server: Handled event for server in area: event=member-join server=Node-237d687c-e901-a162-14cc-6108db37e68c.dc1 area=wan +31:53.219 [INFO] dc2: Synced node info +31:53.474 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:53.974 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:53.982 [DEBUG] dc2.server: finished fetching remote objects: amount=1 +31:53.982 [DEBUG] dc2.server: diffing replication state: local_amount=0 remote_amount=1 +31:53.982 [DEBUG] dc2.server: diffed replication state: deletions=0 updates=1 +31:53.982 [DEBUG] dc2.server: performing updates: updates=1 +31:53.994 [DEBUG] dc2.server: finished updates +31:53.994 [DEBUG] dc2.replication.federation_state: replication completed through remote index: index=12 +31:54.060 [DEBUG] dc2.server.replication.config_entry: finished fetching config entries: amount=0 +31:54.060 [DEBUG] dc2.server.replication.config_entry: Config Entry replication: local=0 remote=0 +31:54.060 [DEBUG] dc2.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 +31:54.060 [DEBUG] dc2.server.replication.config_entry: replication completed through remote index: index=1 +31:54.473 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:54.966 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false +31:54.976 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:55.008 [INFO] dc2.server.connect: received new intermediate certificate from primary datacenter +31:55.028 [INFO] dc2.server.connect: updated root certificates from primary datacenter +31:55.072 [DEBUG] dc2: Node info in sync +31:55.097 [INFO] dc2: Synced service: service=foo +31:55.097 [DEBUG] dc2: Check in sync: check=service:foo +31:55.105 [DEBUG] dc2: Skipping remote check since it is managed automatically: check=serfHealth +31:55.105 [DEBUG] dc2: Node info in sync +31:55.105 [DEBUG] dc2: Service in sync: service=foo +31:55.105 [DEBUG] dc2: Check in sync: check=service:foo +31:55.105 [DEBUG] dc2: Node info in sync +31:55.105 [DEBUG] dc2: Service in sync: service=foo +31:55.105 [DEBUG] dc2: Check in sync: check=service:foo +31:57.990 [DEBUG] dc2.server: finished fetching remote objects: amount=2 +31:57.990 [DEBUG] dc2.server: diffing replication state: local_amount=1 remote_amount=2 +31:57.990 [DEBUG] dc2.server: diffed replication state: deletions=0 updates=1 +31:57.990 [DEBUG] dc2.server: performing updates: updates=1 +31:57.991 [INFO] dc2.server: federation state anti-entropy synced +31:57.997 [DEBUG] dc2.server: finished updates +31:57.997 [DEBUG] dc2.replication.federation_state: replication completed through remote index: index=19 +32:00.309 [INFO] dc2.server.connect: received new intermediate certificate from primary datacenter +32:00.317 [INFO] dc2.server.connect: updated root certificates from primary datacenter +32:00.377 [INFO] dc2: Requesting shutdown +32:00.377 [INFO] dc2.server: shutting down server +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="federation state replication" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="federation state anti-entropy" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="secondary CA roots watch" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="intention replication" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="intermediate cert renew watch" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="CA root pruning" +32:00.377 [DEBUG] dc2.leader: stopping routine: routine="config entry replication" +32:00.377 [WARN] dc2.server.serf.lan: serf: Shutdown without a Leave +32:00.377 [DEBUG] dc2.leader: stopped routine: routine="federation state anti-entropy" +32:00.377 [DEBUG] dc2.leader: stopped routine: routine="intermediate cert renew watch" +32:00.377 [DEBUG] dc2.leader: stopped routine: routine="CA root pruning" +32:00.381 [WARN] dc2.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.388Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.388Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.388Z [INFO] agent.router.manager: shutting down +32:00.388 [INFO] dc2: consul server down +32:00.388 [INFO] dc2: shutdown complete +32:00.388 [INFO] dc2: Stopping server: protocol=DNS address=127.0.0.1:29967 network=tcp +32:00.388 [INFO] dc2: Stopping server: protocol=DNS address=127.0.0.1:29967 network=udp +32:00.388 [INFO] dc2: Stopping server: protocol=HTTP address=127.0.0.1:30024 network=tcp +32:00.388 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:29986 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" +32:00.388 [ERROR] dc2.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" +32:00.388 [DEBUG] dc2.leader: stopped routine: routine="secondary CA roots watch" +32:00.388 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:29986 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" +32:00.388 [INFO] dc2.server.replication.config_entry: stopped replication +32:00.388 [DEBUG] dc2.leader: stopped routine: routine="config entry replication" +32:00.388 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:29986 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" +32:00.388 [ERROR] dc2.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" +32:00.388 [DEBUG] dc2.leader: stopped routine: routine="intention replication" +32:00.388 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:29986 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" +32:00.388 [WARN] dc2.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" +32:00.388 [INFO] dc2.replication.federation_state: stopped replication +32:00.388 [DEBUG] dc2.leader: stopped routine: routine="federation state replication" +32:00.890 [INFO] dc2: Waiting for endpoints to shut down +32:00.890 [INFO] dc2: Endpoints down +2021-10-05T16:31:52.367Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:52.387Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:52.472 [INFO] dc1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:237d687c-e901-a162-14cc-6108db37e68c Address:127.0.0.1:29986}]" +31:52.473 [INFO] dc1.server.serf.wan: serf: EventMemberJoin: Node-237d687c-e901-a162-14cc-6108db37e68c.dc1 127.0.0.1 +31:52.473 [INFO] dc1.server.serf.lan: serf: EventMemberJoin: Node-237d687c-e901-a162-14cc-6108db37e68c 127.0.0.1 +2021-10-05T16:31:52.473Z [INFO] agent.router: Initializing LAN area manager +31:52.473 [INFO] dc1.server.raft: entering follower state: follower="Node at 127.0.0.1:29986 [Follower]" leader= +31:52.474 [INFO] dc1: Started DNS server: address=127.0.0.1:29981 network=udp +31:52.474 [INFO] dc1.server: Adding LAN server: server="Node-237d687c-e901-a162-14cc-6108db37e68c (Addr: tcp/127.0.0.1:29986) (DC: dc1)" +31:52.474 [INFO] dc1.server: Handled event for server in area: event=member-join server=Node-237d687c-e901-a162-14cc-6108db37e68c.dc1 area=wan +31:52.474 [INFO] dc1: Started DNS server: address=127.0.0.1:29981 network=tcp +31:52.474 [INFO] dc1: Started HTTP server: address=127.0.0.1:29982 network=tcp +31:52.474 [INFO] dc1: started state syncer +31:52.474 [INFO] dc1: Started gRPC server: address=127.0.0.1:29952 network=tcp +31:52.520 [WARN] dc1.server.raft: heartbeat timeout reached, starting election: last-leader= +31:52.520 [INFO] dc1.server.raft: entering candidate state: node="Node at 127.0.0.1:29986 [Candidate]" term=2 +31:52.552 [DEBUG] dc1.server.raft: votes: needed=1 +31:52.552 [DEBUG] dc1.server.raft: vote granted: from=237d687c-e901-a162-14cc-6108db37e68c term=2 tally=1 +31:52.552 [INFO] dc1.server.raft: election won: tally=1 +31:52.552 [INFO] dc1.server.raft: entering leader state: leader="Node at 127.0.0.1:29986 [Leader]" +31:52.552 [INFO] dc1.server: cluster leadership acquired +31:52.552 [INFO] dc1.server: New leader elected: payload=Node-237d687c-e901-a162-14cc-6108db37e68c +31:52.576 [DEBUG] dc1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29986 +31:52.624 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:52.654 [INFO] dc1: Synced node info +31:52.670 [INFO] dc1.server.connect: initialized primary datacenter CA with provider: provider=consul +31:52.670 [INFO] dc1.leader: started routine: routine="federation state anti-entropy" +31:52.670 [INFO] dc1.leader: started routine: routine="federation state pruning" +31:52.670 [INFO] dc1.leader: started routine: routine="intermediate cert renew watch" +31:52.670 [INFO] dc1.leader: started routine: routine="CA root pruning" +31:52.670 [DEBUG] dc1.server: Skipping self join check for node since the cluster is too small: node=Node-237d687c-e901-a162-14cc-6108db37e68c +31:52.670 [INFO] dc1.server: member joined, marking health alive: member=Node-237d687c-e901-a162-14cc-6108db37e68c +31:52.698 [INFO] dc1.server: federation state anti-entropy synced +31:53.200 [DEBUG] dc1.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:51444 +31:53.204 [INFO] dc1.server.serf.wan: serf: EventMemberJoin: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 127.0.0.1 +31:53.205 [INFO] dc1.server: Handled event for server in area: event=member-join server=Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 area=wan +31:53.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=2a:7e:c2:d0:4c:80:37:7f:d8:70:95:ee:9f:f1:13:15:a7:62:26:ce:6a:3d:75:ff:ef:69:bc:fa:a6:14:76:e1 is_primary=true +31:53.312 [INFO] dc1.server.connect: CA rotated to new root under provider: provider=consul +31:53.338 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:53.840 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:54.338 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +31:54.727 [DEBUG] dc1: Skipping remote check since it is managed automatically: check=serfHealth +31:54.727 [DEBUG] dc1: Node info in sync +31:54.727 [DEBUG] dc1: Node info in sync +31:54.837 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-d7aa34bd-d7b6-593e-4946-2b559f4b7f99.dc2 +32:00.241 [DEBUG] connect.ca.consul: consul CA provider configured: id=86:70:a4:f4:88:0f:7e:4b:96:57:dd:24:dd:e4:33:0c:6d:1c:68:76:e1:b2:c0:41:08:60:d5:90:2a:01:ad:ce is_primary=true +32:00.279 [INFO] dc1.server.connect: CA rotated to new root under provider: provider=consul +32:00.890 [INFO] dc1: Requesting shutdown +32:00.890 [INFO] dc1.server: shutting down server +32:00.890 [DEBUG] dc1.leader: stopping routine: routine="federation state anti-entropy" +32:00.890 [DEBUG] dc1.leader: stopping routine: routine="federation state pruning" +32:00.890 [DEBUG] dc1.leader: stopping routine: routine="intermediate cert renew watch" +32:00.890 [DEBUG] dc1.leader: stopping routine: routine="CA root pruning" +32:00.890 [WARN] dc1.server.serf.lan: serf: Shutdown without a Leave +32:00.890 [ERROR] dc1.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.890 [DEBUG] dc1.leader: stopped routine: routine="federation state anti-entropy" +32:00.890 [DEBUG] dc1.leader: stopped routine: routine="federation state pruning" +32:00.890 [DEBUG] dc1.leader: stopped routine: routine="intermediate cert renew watch" +32:00.890 [DEBUG] dc1.leader: stopped routine: routine="CA root pruning" +32:00.893 [WARN] dc1.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.897Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.897Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.897Z [INFO] agent.router.manager: shutting down +32:00.897 [INFO] dc1: consul server down +32:00.897 [INFO] dc1: shutdown complete +32:00.897 [INFO] dc1: Stopping server: protocol=DNS address=127.0.0.1:29981 network=tcp +32:00.897 [INFO] dc1: Stopping server: protocol=DNS address=127.0.0.1:29981 network=udp +32:00.897 [INFO] dc1: Stopping server: protocol=HTTP address=127.0.0.1:29982 network=tcp +32:01.398 [INFO] dc1: Waiting for endpoints to shut down +32:01.398 [INFO] dc1: Endpoints down +--- PASS: TestAgentConnectCALeafCert_secondaryDC_good (9.05s) +=== CONT TestAgent_Members +2021-10-05T16:32:00.646Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:00.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:00.692 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bb92c42f-7b75-3425-65ec-6a24ddbbdc7f Address:127.0.0.1:29470}]" +32:00.693 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f.dc1 127.0.0.1 +32:00.693 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f 127.0.0.1 +2021-10-05T16:32:00.694Z [INFO] agent.router: Initializing LAN area manager +32:00.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29451 network=udp +32:00.694 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29470 [Follower]" leader= +32:00.694 [INFO] TestAgent.server: Adding LAN server: server="Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f (Addr: tcp/127.0.0.1:29470) (DC: dc1)" +32:00.694 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f.dc1 area=wan +32:00.694 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29451 network=tcp +32:00.695 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29452 network=tcp +32:00.695 [INFO] TestAgent: started state syncer +32:00.695 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29450 network=tcp +32:00.764 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:00.764 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29470 [Candidate]" term=2 +32:00.773 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:00.773 [DEBUG] TestAgent.server.raft: vote granted: from=bb92c42f-7b75-3425-65ec-6a24ddbbdc7f term=2 tally=1 +32:00.773 [INFO] TestAgent.server.raft: election won: tally=1 +32:00.773 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29470 [Leader]" +32:00.774 [INFO] TestAgent.server: cluster leadership acquired +32:00.774 [INFO] TestAgent.server: New leader elected: payload=Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f +32:00.784 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29470 +32:00.797 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:00.842 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:00.842 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:00.842 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:00.842 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:00.842 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:00.842 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f +32:00.842 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bb92c42f-7b75-3425-65ec-6a24ddbbdc7f +32:00.855 [INFO] TestAgent.server: federation state anti-entropy synced +32:00.888 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:00.908 [INFO] TestAgent: Synced node info +32:00.908 [DEBUG] TestAgent: Node info in sync +32:00.920 [INFO] TestAgent: Requesting shutdown +32:00.920 [INFO] TestAgent.server: shutting down server +32:00.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:00.920 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:00.920 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:00.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:00.920 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:00.920 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:00.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:00.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:00.920 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:00.920 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:00.923 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:00.928Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:00.928Z [INFO] agent.router.manager: shutting down +32:00.928 [INFO] TestAgent: consul server down +32:00.928 [INFO] TestAgent: shutdown complete +32:00.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29451 network=tcp +32:00.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29451 network=udp +32:00.928 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29452 network=tcp +32:01.448 [INFO] TestAgent: Waiting for endpoints to shut down +32:01.448 [INFO] TestAgent: Endpoints down +=== RUN TestOperator_RaftPeer/#01 +[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_Members_ACLFilter/no_token +=== RUN TestAgent_Members_ACLFilter/root_token +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:01.088Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.103Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:01.180 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:292b5fa1-717a-5ef6-d612-52ec4f6b1055 Address:127.0.0.1:30385}]" +32:01.180 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055.dc1 127.0.0.1 +32:01.181 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30385 [Follower]" leader= +32:01.181 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 127.0.0.1 +2021-10-05T16:32:01.181Z [INFO] agent.router: Initializing LAN area manager +32:01.181 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29472 network=udp +32:01.181 [INFO] TestAgent.server: Adding LAN server: server="Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 (Addr: tcp/127.0.0.1:30385) (DC: dc1)" +32:01.181 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055.dc1 area=wan +32:01.182 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29472 network=tcp +32:01.182 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29473 network=tcp +32:01.182 [INFO] TestAgent: started state syncer +32:01.182 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29485 network=tcp +32:01.228 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:01.228 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30385 [Candidate]" term=2 +32:01.280 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:01.280 [DEBUG] TestAgent.server.raft: vote granted: from=292b5fa1-717a-5ef6-d612-52ec4f6b1055 term=2 tally=1 +32:01.280 [INFO] TestAgent.server.raft: election won: tally=1 +32:01.280 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30385 [Leader]" +32:01.280 [INFO] TestAgent.server: cluster leadership acquired +32:01.280 [INFO] TestAgent.server: New leader elected: payload=Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 +32:01.297 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30385 +32:01.344 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:01.374 [INFO] TestAgent.server.serf.wan: serf: Received install-key query +32:01.380 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055.dc1 +32:01.380 [INFO] TestAgent.server.serf.lan: serf: Received install-key query +32:01.380 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 +32:01.381 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +32:01.381 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055.dc1 +32:01.381 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +32:01.381 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 +32:01.381 [INFO] TestAgent: Requesting shutdown +32:01.381 [INFO] TestAgent.server: shutting down server +32:01.381 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:01.381 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:01.390 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:01.395Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:01.395Z [INFO] agent.router.manager: shutting down +32:01.396 [INFO] TestAgent: consul server down +32:01.396 [INFO] TestAgent: shutdown complete +32:01.396 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29472 network=tcp +32:01.396 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29472 network=udp +32:01.396 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29473 network=tcp +32:01.396 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:01.396 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:01.396 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:01.396 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:01.396 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:01.396 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +32:01.396 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +32:01.396 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +32:01.396 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +32:01.396 [DEBUG] TestAgent.server: federation states are not enabled in this datacenter: datacenter=dc1 +32:01.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 +32:01.396 [INFO] TestAgent.server: member joined, marking health alive: member=Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 +32:01.396 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-292b5fa1-717a-5ef6-d612-52ec4f6b1055 127.0.0.1 29475 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:292b5fa1-717a-5ef6-d612-52ec4f6b1055 port:30385 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29476] alive 1 5 2 2 5 4}" error="raft is already shutdown" +32:01.396 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:01.396 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:01.396 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:01.396 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:01.396 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:01.396 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:01.396 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:01.396 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:01.897 [INFO] TestAgent: Waiting for endpoints to shut down +32:01.897 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_KeyringInstall (0.83s) +=== CONT TestAgent_Reload_ACLDeny +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:01.386Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.410Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:01.496 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46e7f446-ed50-2b5f-b371-45b7b071dc0f Address:127.0.0.1:29442}]" +32:01.499 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f.dc1 127.0.0.1 +32:01.500 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29442 [Follower]" leader= +32:01.500 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f 127.0.0.1 +2021-10-05T16:32:01.500Z [INFO] agent.router: Initializing LAN area manager +32:01.500 [INFO] TestAgent.server: Adding LAN server: server="Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f (Addr: tcp/127.0.0.1:29442) (DC: dc1)" +32:01.500 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29437 network=udp +32:01.500 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f.dc1 area=wan +32:01.501 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29437 network=tcp +32:01.502 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29438 network=tcp +32:01.502 [INFO] TestAgent: started state syncer +32:01.502 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30330 network=tcp +32:01.557 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:01.557 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29442 [Candidate]" term=2 +32:01.589 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:01.589 [DEBUG] TestAgent.server.raft: vote granted: from=46e7f446-ed50-2b5f-b371-45b7b071dc0f term=2 tally=1 +32:01.589 [INFO] TestAgent.server.raft: election won: tally=1 +32:01.589 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29442 [Leader]" +32:01.589 [INFO] TestAgent.server: cluster leadership acquired +32:01.589 [INFO] TestAgent.server: New leader elected: payload=Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f +32:01.612 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29442 +32:01.642 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:01.668 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:01.668 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:01.668 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:01.668 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:01.668 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:01.668 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f +32:01.668 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46e7f446-ed50-2b5f-b371-45b7b071dc0f +32:01.684 [INFO] TestAgent.server: federation state anti-entropy synced +32:01.692 [INFO] TestAgent: Requesting shutdown +32:01.692 [INFO] TestAgent.server: shutting down server +32:01.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:01.692 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:01.692 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:01.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:01.692 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:01.692 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:01.692 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:01.692 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:01.692 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:01.692 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:01.692 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:01.700 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:01.705Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:01.705Z [INFO] agent.router.manager: shutting down +32:01.705 [INFO] TestAgent: consul server down +32:01.705 [INFO] TestAgent: shutdown complete +32:01.705 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29437 network=tcp +32:01.705 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29437 network=udp +32:01.705 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29438 network=tcp +32:02.205 [INFO] TestAgent: Waiting for endpoints to shut down +32:02.205 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Members_WAN (0.84s) +=== CONT TestAgent_Reload +2021-10-05T16:32:01.502Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.558Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:01.635 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e855136-8c12-7bfc-b7ed-9bd84b93b9c0 Address:127.0.0.1:29449}]" +32:01.636 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29449 [Follower]" leader= +32:01.636 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e855136-8c12-7bfc-b7ed-9bd84b93b9c0.dc1 127.0.0.1 +32:01.637 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e855136-8c12-7bfc-b7ed-9bd84b93b9c0 127.0.0.1 +2021-10-05T16:32:01.637Z [INFO] agent.router: Initializing LAN area manager +32:01.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29514 network=udp +32:01.637 [INFO] TestAgent.server: Adding LAN server: server="Node-1e855136-8c12-7bfc-b7ed-9bd84b93b9c0 (Addr: tcp/127.0.0.1:29449) (DC: dc1)" +32:01.637 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e855136-8c12-7bfc-b7ed-9bd84b93b9c0.dc1 area=wan +32:01.637 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29514 network=tcp +32:01.638 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29515 network=tcp +32:01.638 [INFO] TestAgent: started state syncer +32:01.638 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29492 network=tcp +32:01.701 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:01.701 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29449 [Candidate]" term=2 +32:01.736 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:01.736 [DEBUG] TestAgent.server.raft: vote granted: from=1e855136-8c12-7bfc-b7ed-9bd84b93b9c0 term=2 tally=1 +32:01.736 [INFO] TestAgent.server.raft: election won: tally=1 +32:01.736 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29449 [Leader]" +32:01.736 [INFO] TestAgent: Requesting shutdown +32:01.736 [INFO] TestAgent.server: cluster leadership acquired +32:01.736 [INFO] TestAgent.server: shutting down server +32:01.736 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:01.736 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:01.749 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +32:01.754 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29449 +2021-10-05T16:32:01.767Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:01.767Z [INFO] agent.router.manager: shutting down +32:01.767 [INFO] TestAgent: consul server down +32:01.767 [INFO] TestAgent: shutdown complete +32:01.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29514 network=tcp +32:01.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29514 network=udp +32:01.767 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29515 network=tcp +32:01.767 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +32:01.767 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:02.268 [INFO] TestAgent: Waiting for endpoints to shut down +32:02.268 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_RaftPeer (1.64s) + --- PASS: TestOperator_RaftPeer/#00 (0.82s) + --- PASS: TestOperator_RaftPeer/#01 (0.82s) === CONT TestAgent_Metrics_ACLDeny -2022-11-07T22:46:21.259Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.259Z [WARN] agent: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -2022-11-07T22:46:21.269Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.269Z [WARN] agent.auto_config: using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead, see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ -46:21.311 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bfaba009-510e-cd63-68f8-76262c4c3a50 Address:127.0.0.1:11131}]" -46:21.311 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11131 [Follower]" leader= -46:21.312 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bfaba009-510e-cd63-68f8-76262c4c3a50.dc1 127.0.0.1 -46:21.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bfaba009-510e-cd63-68f8-76262c4c3a50 127.0.0.1 -2022-11-07T22:46:21.312Z [INFO] agent.router: Initializing LAN area manager -46:21.312 [INFO] TestAgent.server: Adding LAN server: server="Node-bfaba009-510e-cd63-68f8-76262c4c3a50 (Addr: tcp/127.0.0.1:11131) (DC: dc1)" -46:21.312 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bfaba009-510e-cd63-68f8-76262c4c3a50.dc1 area=wan -46:21.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11028 network=udp -46:21.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11028 network=tcp -46:21.313 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11078 network=tcp -46:21.313 [INFO] TestAgent: started state syncer -46:21.313 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11118 network=tcp -46:21.367 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.367 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11131 [Candidate]" term=2 -46:21.376 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.376 [DEBUG] TestAgent.server.raft: vote granted: from=bfaba009-510e-cd63-68f8-76262c4c3a50 term=2 tally=1 -46:21.377 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.377 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11131 [Leader]" -46:21.377 [INFO] TestAgent.server: cluster leadership acquired -46:21.377 [INFO] TestAgent.server: New leader elected: payload=Node-bfaba009-510e-cd63-68f8-76262c4c3a50 -46:21.386 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11131 -46:21.413 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.459 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.459 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.459 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.459 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.459 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.459 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bfaba009-510e-cd63-68f8-76262c4c3a50 -46:21.459 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bfaba009-510e-cd63-68f8-76262c4c3a50 -46:21.467 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.518 [INFO] TestAgent: Synced node info -46:21.524 [INFO] TestAgent: Synced check: check=test -46:21.537 [DEBUG] TestAgent: Node info in sync -46:21.546 [INFO] TestAgent: Synced service: service=a -46:21.546 [DEBUG] TestAgent: Check in sync: check=test -46:21.546 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.565 [DEBUG] TestAgent: Node info in sync -46:21.572 [INFO] TestAgent: Synced service: service=a -46:21.572 [DEBUG] TestAgent: Check in sync: check=test -46:21.572 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.578 [DEBUG] TestAgent: Node info in sync -46:21.578 [DEBUG] TestAgent: Service in sync: service=a -46:21.583 [INFO] TestAgent: Synced check: check=test -46:21.583 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.599 [DEBUG] TestAgent: Node info in sync -46:21.610 [INFO] TestAgent: Synced service: service=a -46:21.610 [DEBUG] TestAgent: Check in sync: check=test -46:21.610 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.621 [DEBUG] TestAgent: Node info in sync -46:21.630 [INFO] TestAgent: Synced service: service=a -46:21.630 [DEBUG] TestAgent: Check in sync: check=test -46:21.630 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.635 [DEBUG] TestAgent: Node info in sync -46:21.635 [DEBUG] TestAgent: Service in sync: service=a -46:21.640 [INFO] TestAgent: Synced check: check=test -46:21.640 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.651 [DEBUG] TestAgent: Node info in sync -46:21.654 [INFO] TestAgent: Synced service: service=a -46:21.654 [DEBUG] TestAgent: Check in sync: check=test -46:21.654 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.665 [DEBUG] TestAgent: Node info in sync -46:21.678 [INFO] TestAgent: Synced service: service=a -46:21.678 [DEBUG] TestAgent: Check in sync: check=service:a -46:21.678 [DEBUG] TestAgent: Check in sync: check=test -46:21.678 [INFO] TestAgent: Requesting shutdown -46:21.678 [INFO] TestAgent.server: shutting down server -46:21.678 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.678 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.678 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.678 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.678 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.678 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:21.678 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.678 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.678 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.678 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.678 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.682 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.687Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.687Z [INFO] agent.router.manager: shutting down -46:21.687 [INFO] TestAgent: consul server down -46:21.687 [INFO] TestAgent: shutdown complete -46:21.687 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11028 network=tcp -46:21.687 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11028 network=udp -46:21.687 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11078 network=tcp -46:22.188 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.188 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck_Scripts (0.94s) - --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_node_check (0.02s) - --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_top-level_service_check (0.02s) - --- PASS: TestAgent_RegisterCheck_Scripts/==_Consul_1.0.0_as_slice-based_service_check (0.03s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_node_check (0.01s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_top-level_service_check (0.03s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_(fixup)_as_slice-based_service_check (0.02s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_node_check (0.01s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_top-level_service_check (0.01s) - --- PASS: TestAgent_RegisterCheck_Scripts/>_Consul_1.0.0_as_slice-based_service_check (0.02s) -=== CONT TestAgent_ChecksWithFilter -2022-11-07T22:46:21.306Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.314Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.355 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b3f90de8-260b-2bde-1ef2-e9128c4a533b Address:127.0.0.1:11096}]" -46:21.355 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11096 [Follower]" leader= -46:21.356 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b.dc1 127.0.0.1 -46:21.357 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b 127.0.0.1 -2022-11-07T22:46:21.357Z [INFO] agent.router: Initializing LAN area manager -46:21.357 [INFO] TestAgent.server: Adding LAN server: server="Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b (Addr: tcp/127.0.0.1:11096) (DC: dc1)" -46:21.357 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b.dc1 area=wan -46:21.358 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11126 network=tcp -46:21.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11126 network=udp -46:21.360 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11113 network=tcp -46:21.360 [INFO] TestAgent: started state syncer -46:21.360 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11097 network=tcp -46:21.412 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.412 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11096 [Candidate]" term=2 -46:21.439 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.439 [DEBUG] TestAgent.server.raft: vote granted: from=b3f90de8-260b-2bde-1ef2-e9128c4a533b term=2 tally=1 -46:21.439 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.439 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11096 [Leader]" -46:21.439 [INFO] TestAgent.server: cluster leadership acquired -46:21.439 [INFO] TestAgent.server: New leader elected: payload=Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b -46:21.453 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11096 -46:21.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.504 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.504 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.504 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.504 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.504 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.505 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b -46:21.505 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b3f90de8-260b-2bde-1ef2-e9128c4a533b -46:21.520 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.690 [INFO] TestAgent: Synced node info -46:21.693 [INFO] TestAgent: Synced check: check=test -46:21.693 [INFO] TestAgent: Requesting shutdown -46:21.693 [INFO] TestAgent.server: shutting down server -46:21.693 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.693 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.693 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.693 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.693 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.693 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:21.693 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.693 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.698 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.702Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.702Z [INFO] agent.router.manager: shutting down -46:21.702 [INFO] TestAgent: consul server down -46:21.702 [INFO] TestAgent: shutdown complete -46:21.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11126 network=tcp -46:21.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11126 network=udp -46:21.702 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11113 network=tcp -46:22.202 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.202 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck_Passing (0.91s) -=== CONT TestAgent_HealthServiceByID -2022-11-07T22:46:21.371Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.386Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.437 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b37a8fc3-2d0b-1f32-1ca2-59d200fac113 Address:127.0.0.1:11103}]" -46:21.437 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11103 [Follower]" leader= -46:21.438 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113.dc1 127.0.0.1 -46:21.439 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113 127.0.0.1 -2022-11-07T22:46:21.439Z [INFO] agent.router: Initializing LAN area manager -46:21.439 [INFO] TestAgent.server: Adding LAN server: server="Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113 (Addr: tcp/127.0.0.1:11103) (DC: dc1)" -46:21.439 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113.dc1 area=wan -46:21.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11098 network=udp -46:21.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11098 network=tcp -46:21.439 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11071 network=tcp -46:21.439 [INFO] TestAgent: started state syncer -46:21.439 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11104 network=tcp -46:21.489 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.489 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11103 [Candidate]" term=2 -46:21.507 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.507 [DEBUG] TestAgent.server.raft: vote granted: from=b37a8fc3-2d0b-1f32-1ca2-59d200fac113 term=2 tally=1 -46:21.507 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.507 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11103 [Leader]" -46:21.507 [INFO] TestAgent.server: cluster leadership acquired -46:21.507 [INFO] TestAgent.server: New leader elected: payload=Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113 -46:21.520 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11103 -46:21.537 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.574 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.574 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.574 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.574 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.574 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.574 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113 -46:21.574 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b37a8fc3-2d0b-1f32-1ca2-59d200fac113 -46:21.580 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.744 [INFO] TestAgent: Requesting shutdown -46:21.745 [INFO] TestAgent.server: shutting down server -46:21.745 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.745 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.745 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.745 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.745 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.745 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:21.745 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.745 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.745 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.745 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.745 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.753Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.753Z [INFO] agent.router.manager: shutting down -46:21.753 [INFO] TestAgent: consul server down -46:21.753 [INFO] TestAgent: shutdown complete -46:21.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11098 network=tcp -46:21.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11098 network=udp -46:21.753 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11071 network=tcp -46:22.254 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.254 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheckScriptsExecRemoteDisable (0.91s) -=== CONT TestAgent_FailCheck_ACLDeny -46:21.268 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-f717de55-1efd-10c5-804e-a491d1838d70 127.0.0.1 -2022-11-07T22:46:21.268Z [INFO] agent.router: Initializing LAN area manager -46:21.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11035 network=udp -46:21.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11035 network=tcp -46:21.268 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11134 network=tcp -46:21.268 [INFO] TestAgent: started state syncer -46:21.268 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11027 network=tcp -2022-11-07T22:46:21.269Z [WARN] agent.router.manager: No servers available -46:21.269 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:21.269 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Stream connection from=127.0.0.1:47518 -46:21.269 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea 127.0.0.1 -46:21.269 [INFO] TestAgent: Requesting shutdown -46:21.269 [INFO] TestAgent.client: shutting down client -46:21.269 [WARN] TestAgent.client.serf.lan: serf: Shutdown without a Leave -46:21.269 [INFO] TestAgent.client: adding server: server="Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea (Addr: tcp/127.0.0.1:11857) (DC: dc1)" -46:21.269 [INFO] TestAgent.client: New leader elected: payload=Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea -46:21.277 [INFO] TestAgent: consul client down -46:21.277 [INFO] TestAgent: shutdown complete -46:21.277 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11035 network=tcp -46:21.277 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11035 network=udp -46:21.277 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11134 network=tcp -46:21.778 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.778 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.804Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.817Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.855 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:34e7a39f-e88b-aaf2-db44-e33edc0f92ea Address:127.0.0.1:11857}]" -46:20.855 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11857 [Follower]" leader= -46:20.856 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea.dc1 127.0.0.1 -46:20.856 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea 127.0.0.1 -2022-11-07T22:46:20.856Z [INFO] agent.router: Initializing LAN area manager -46:20.857 [INFO] TestAgent.server: Adding LAN server: server="Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea (Addr: tcp/127.0.0.1:11857) (DC: dc1)" -46:20.857 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea.dc1 area=wan -46:20.857 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11950 network=udp -46:20.857 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11950 network=tcp -46:20.857 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11846 network=tcp -46:20.857 [INFO] TestAgent: started state syncer -46:20.857 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11963 network=tcp -46:20.893 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.893 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11857 [Candidate]" term=2 -46:20.909 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.909 [DEBUG] TestAgent.server.raft: vote granted: from=34e7a39f-e88b-aaf2-db44-e33edc0f92ea term=2 tally=1 -46:20.909 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.909 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11857 [Leader]" -46:20.910 [INFO] TestAgent.server: cluster leadership acquired -46:20.910 [INFO] TestAgent.server: New leader elected: payload=Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea -46:20.920 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11857 -46:20.940 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.973 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.973 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.973 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.973 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.973 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.973 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea -46:20.973 [INFO] TestAgent.server: member joined, marking health alive: member=Node-34e7a39f-e88b-aaf2-db44-e33edc0f92ea -46:20.988 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.010 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.014 [INFO] TestAgent: Synced node info -46:21.014 [DEBUG] TestAgent: Node info in sync -46:21.269 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11108] -46:21.269 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11108 -46:21.269 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f717de55-1efd-10c5-804e-a491d1838d70 127.0.0.1 -46:21.269 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:21.269 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f717de55-1efd-10c5-804e-a491d1838d70 -46:21.778 [INFO] TestAgent: Requesting shutdown -46:21.778 [INFO] TestAgent.server: shutting down server -46:21.778 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.778 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.778 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.778 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.778 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.778 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.778 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.778 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.778 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.778 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.782 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.786Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.786Z [INFO] agent.router.manager: shutting down -46:21.787 [INFO] TestAgent: consul server down -46:21.787 [INFO] TestAgent: shutdown complete -46:21.787 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11950 network=tcp -46:21.787 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11950 network=udp -46:21.787 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11846 network=tcp -46:22.287 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.287 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_JoinLANNotify (1.50s) -=== CONT TestAgent_NodeMaintenance_BadRequest +=== RUN TestAgent_Reload_ACLDeny/no_token +=== RUN TestAgent_Reload_ACLDeny/read-only_token +2021-10-05T16:32:01.357Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:01.357Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.387Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:01.387Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:01.504 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5803f1da-ffb7-4398-3db6-9a7b160b7546 Address:127.0.0.1:29463}]" +32:01.510 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29463 [Follower]" leader= +32:01.513 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5803f1da-ffb7-4398-3db6-9a7b160b7546.dc1 127.0.0.1 +32:01.515 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 127.0.0.1 +2021-10-05T16:32:01.516Z [INFO] agent.router: Initializing LAN area manager +32:01.517 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29465 network=udp +32:01.518 [INFO] TestAgent.server: Adding LAN server: server="Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 (Addr: tcp/127.0.0.1:29463) (DC: dc1)" +32:01.519 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29465 network=tcp +32:01.537 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546.dc1 area=wan +32:01.540 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29466 network=tcp +32:01.540 [INFO] TestAgent: started state syncer +32:01.540 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29457 network=tcp +32:01.560 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:01.560 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29463 [Candidate]" term=2 +32:01.568 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:01.591 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:01.591 [DEBUG] TestAgent.server.raft: vote granted: from=5803f1da-ffb7-4398-3db6-9a7b160b7546 term=2 tally=1 +32:01.591 [INFO] TestAgent.server.raft: election won: tally=1 +32:01.591 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29463 [Leader]" +32:01.592 [INFO] TestAgent.server: cluster leadership acquired +32:01.592 [INFO] TestAgent.server: New leader elected: payload=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.602 [INFO] TestAgent.server: initializing acls +32:01.607 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:01.607 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:01.618 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:01.636 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:01.636 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:01.636 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:01.637 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.637 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5803f1da-ffb7-4398-3db6-9a7b160b7546.dc1 +32:01.638 [INFO] TestAgent.server: Updating LAN server: server="Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 (Addr: tcp/127.0.0.1:29463) (DC: dc1)" +32:01.638 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546.dc1 area=wan +32:01.652 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:01.690 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:01.690 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:01.690 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:01.690 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:01.690 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:01.690 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.690 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.712 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.724 [INFO] TestAgent.server: federation state anti-entropy synced +32:01.754 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:01.762 [INFO] TestAgent: Synced node info +32:01.762 [DEBUG] TestAgent: Node info in sync +32:01.797 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.797 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 +32:01.797 [DEBUG] TestAgent: dropping node from result due to ACLs: node=Node-5803f1da-ffb7-4398-3db6-9a7b160b7546 accessorID=00000000-0000-0000-0000-000000000002 +32:01.797 [INFO] TestAgent: Requesting shutdown +32:01.797 [INFO] TestAgent.server: shutting down server +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:01.797 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:01.797 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:01.797 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:01.797 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:01.803 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:01.809Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:01.809Z [INFO] agent.router.manager: shutting down +32:01.809 [INFO] TestAgent: consul server down +32:01.809 [INFO] TestAgent: shutdown complete +32:01.809 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29465 network=tcp +32:01.809 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29465 network=udp +32:01.809 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29466 network=tcp +32:02.310 [INFO] TestAgent: Waiting for endpoints to shut down +32:02.310 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Members_ACLFilter (0.97s) + --- PASS: TestAgent_Members_ACLFilter/no_token (0.00s) + --- PASS: TestAgent_Members_ACLFilter/root_token (0.00s) +=== CONT TestAgent_Self_ACLDeny +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:01.414Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.504Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:01.595 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c05c83c8-2e72-0291-6a46-994c7056d05d Address:127.0.0.1:29456}]" +32:01.595 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29456 [Follower]" leader= +32:01.596 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c05c83c8-2e72-0291-6a46-994c7056d05d.dc1 127.0.0.1 +32:01.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c05c83c8-2e72-0291-6a46-994c7056d05d 127.0.0.1 +2021-10-05T16:32:01.597Z [INFO] agent.router: Initializing LAN area manager +32:01.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30331 network=udp +32:01.597 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c05c83c8-2e72-0291-6a46-994c7056d05d.dc1 area=wan +32:01.597 [INFO] TestAgent.server: Adding LAN server: server="Node-c05c83c8-2e72-0291-6a46-994c7056d05d (Addr: tcp/127.0.0.1:29456) (DC: dc1)" +32:01.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30331 network=tcp +32:01.598 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30332 network=tcp +32:01.598 [INFO] TestAgent: started state syncer +32:01.598 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29464 network=tcp +32:01.656 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:01.656 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29456 [Candidate]" term=2 +32:01.700 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:01.700 [DEBUG] TestAgent.server.raft: vote granted: from=c05c83c8-2e72-0291-6a46-994c7056d05d term=2 tally=1 +32:01.700 [INFO] TestAgent.server.raft: election won: tally=1 +32:01.700 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29456 [Leader]" +32:01.701 [INFO] TestAgent.server: cluster leadership acquired +32:01.701 [INFO] TestAgent.server: New leader elected: payload=Node-c05c83c8-2e72-0291-6a46-994c7056d05d +32:01.732 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29456 +32:01.755 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:01.784 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:01.784 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:01.784 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:01.784 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:01.784 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:01.784 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c05c83c8-2e72-0291-6a46-994c7056d05d +32:01.784 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c05c83c8-2e72-0291-6a46-994c7056d05d +32:01.803 [INFO] TestAgent.server: federation state anti-entropy synced +32:01.972 [INFO] TestAgent: Requesting shutdown +32:01.972 [INFO] TestAgent.server: shutting down server +32:01.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:01.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:01.972 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:01.972 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:01.972 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:01.972 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:01.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:01.972 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:01.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:01.972 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:01.972 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:01.975 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:01.979Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:01.979Z [INFO] agent.router.manager: shutting down +32:01.979 [INFO] TestAgent: consul server down +32:01.979 [INFO] TestAgent: shutdown complete +32:01.979 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30331 network=tcp +32:01.979 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30331 network=udp +32:01.979 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30332 network=tcp +32:02.479 [INFO] TestAgent: Waiting for endpoints to shut down +32:02.479 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Members (1.08s) +=== CONT TestAgent_Self +=== RUN TestAgent_Self/no_grpc +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestAgent_Metrics_ACLDeny/no_token === RUN TestAgent_Metrics_ACLDeny/agent_master_token === RUN TestAgent_Metrics_ACLDeny/read-only_token -2022-11-07T22:46:21.469Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.479Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.529 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2156226f-823d-d15f-68bf-c7c272403d99 Address:127.0.0.1:11040}]" -46:21.529 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11040 [Follower]" leader= -46:21.529 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2156226f-823d-d15f-68bf-c7c272403d99.dc1 127.0.0.1 -46:21.530 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2156226f-823d-d15f-68bf-c7c272403d99 127.0.0.1 -2022-11-07T22:46:21.530Z [INFO] agent.router: Initializing LAN area manager -46:21.530 [INFO] TestAgent.server: Adding LAN server: server="Node-2156226f-823d-d15f-68bf-c7c272403d99 (Addr: tcp/127.0.0.1:11040) (DC: dc1)" -46:21.530 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2156226f-823d-d15f-68bf-c7c272403d99.dc1 area=wan -46:21.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11105 network=tcp -46:21.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11105 network=udp -46:21.530 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11120 network=tcp -46:21.530 [INFO] TestAgent: started state syncer -46:21.530 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11041 network=tcp -46:21.582 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.582 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11040 [Candidate]" term=2 -46:21.596 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.596 [DEBUG] TestAgent.server.raft: vote granted: from=2156226f-823d-d15f-68bf-c7c272403d99 term=2 tally=1 -46:21.596 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.596 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11040 [Leader]" -46:21.596 [INFO] TestAgent.server: cluster leadership acquired -46:21.596 [INFO] TestAgent.server: New leader elected: payload=Node-2156226f-823d-d15f-68bf-c7c272403d99 -46:21.604 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11040 -46:21.620 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.643 [INFO] TestAgent: Synced node info -46:21.643 [DEBUG] TestAgent: Node info in sync -46:21.650 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.650 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.650 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.650 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.650 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.650 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2156226f-823d-d15f-68bf-c7c272403d99 -46:21.650 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2156226f-823d-d15f-68bf-c7c272403d99 -46:21.677 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.786 [DEBUG] TestAgent: Node info in sync -46:21.790 [INFO] TestAgent: Synced check: check=test -46:21.790 [INFO] TestAgent: Requesting shutdown -46:21.790 [INFO] TestAgent.server: shutting down server -46:21.790 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.790 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.790 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.790 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.790 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.790 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.790 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.790 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.790 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.790 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.793 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.797Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.797Z [INFO] agent.router.manager: shutting down -46:21.797 [INFO] TestAgent: consul server down -46:21.797 [INFO] TestAgent: shutdown complete -46:21.797 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11105 network=tcp -46:21.797 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11105 network=udp -46:21.797 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11120 network=tcp -46:22.297 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.297 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck (0.84s) -=== CONT TestAgent_ServiceMaintenance_ACLDeny -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestAgent_FailCheck_ACLDeny/no_token -=== RUN TestAgent_FailCheck_ACLDeny/root_token -=== RUN TestAgent_HealthServiceByID/passing_checks -=== RUN TestAgent_HealthServiceByID/passing_checks/format=text -=== RUN TestAgent_HealthServiceByID/passing_checks/format=json -=== RUN TestAgent_HealthServiceByID/warning_checks -=== RUN TestAgent_HealthServiceByID/warning_checks/format=text -=== RUN TestAgent_HealthServiceByID/warning_checks/format=json -=== RUN TestAgent_HealthServiceByID/critical_checks -=== RUN TestAgent_HealthServiceByID/critical_checks/format=text -=== RUN TestAgent_HealthServiceByID/critical_checks/format=json -=== RUN TestAgent_HealthServiceByID/unknown_serviceid -=== RUN TestAgent_HealthServiceByID/unknown_serviceid/format=text -=== RUN TestAgent_HealthServiceByID/unknown_serviceid/format=json -=== RUN TestAgent_HealthServiceByID/critical_check_on_node -=== RUN TestAgent_HealthServiceByID/critical_check_on_node/format=text -=== RUN TestAgent_HealthServiceByID/critical_check_on_node/format=json -=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node -=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node/format=text -=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node/format=json -=== RUN TestAgent_ServiceMaintenance_ACLDeny/no_token -=== RUN TestAgent_ServiceMaintenance_ACLDeny/root_token -2022-11-07T22:46:21.288Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.300Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.347 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:328d19ee-7f6a-b366-9022-aa1b3ff34d45 Address:127.0.0.1:11124}]" -46:21.347 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11124 [Follower]" leader= -46:21.348 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45.dc1 127.0.0.1 -46:21.348 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45 127.0.0.1 -2022-11-07T22:46:21.348Z [INFO] agent.router: Initializing LAN area manager -46:21.348 [INFO] TestAgent.server: Adding LAN server: server="Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45 (Addr: tcp/127.0.0.1:11124) (DC: dc1)" -46:21.348 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45.dc1 area=wan -46:21.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11133 network=udp -46:21.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11133 network=tcp -46:21.349 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11141 network=tcp -46:21.349 [INFO] TestAgent: started state syncer -46:21.349 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11125 network=tcp -46:21.383 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.383 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11124 [Candidate]" term=2 -46:21.403 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.403 [DEBUG] TestAgent.server.raft: vote granted: from=328d19ee-7f6a-b366-9022-aa1b3ff34d45 term=2 tally=1 -46:21.403 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.403 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11124 [Leader]" -46:21.403 [INFO] TestAgent.server: cluster leadership acquired -46:21.403 [INFO] TestAgent.server: New leader elected: payload=Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45 -46:21.420 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11124 -46:21.450 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.486 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.486 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.486 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.486 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.486 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.487 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45 -46:21.487 [INFO] TestAgent.server: member joined, marking health alive: member=Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45 -46:21.502 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.532 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:58118 -46:21.533 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8.dc1 127.0.0.1 -46:21.533 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8.dc1 area=wan -46:21.533 [INFO] TestAgent: Requesting shutdown -46:21.533 [INFO] TestAgent.server: shutting down server -46:21.533 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.533 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.533 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.533 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.533 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.533 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.533 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.533 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:21.533 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.533 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.533 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.540 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.547Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.547Z [INFO] agent.router.manager: shutting down -46:21.547 [INFO] TestAgent: consul server down -46:21.547 [INFO] TestAgent: shutdown complete -46:21.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11133 network=tcp -46:21.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11133 network=udp -46:21.547 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11141 network=tcp -46:22.047 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.047 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:21.143Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.152Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.186 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:21119a5a-778f-c58f-0bae-4a85bfaf34f8 Address:127.0.0.1:11962}]" -46:21.186 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11962 [Follower]" leader= -46:21.186 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8.dc1 127.0.0.1 -46:21.186 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8 127.0.0.1 -2022-11-07T22:46:21.186Z [INFO] agent.router: Initializing LAN area manager -46:21.186 [INFO] TestAgent.server: Adding LAN server: server="Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8 (Addr: tcp/127.0.0.1:11962) (DC: dc1)" -46:21.187 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8.dc1 area=wan -46:21.187 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11859 network=udp -46:21.187 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11859 network=tcp -46:21.187 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11958 network=tcp -46:21.187 [INFO] TestAgent: started state syncer -46:21.187 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12033 network=tcp -46:21.242 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.242 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11962 [Candidate]" term=2 -46:21.257 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.257 [DEBUG] TestAgent.server.raft: vote granted: from=21119a5a-778f-c58f-0bae-4a85bfaf34f8 term=2 tally=1 -46:21.257 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.257 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11962 [Leader]" -46:21.258 [INFO] TestAgent.server: cluster leadership acquired -46:21.258 [INFO] TestAgent.server: New leader elected: payload=Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8 -46:21.264 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11962 -46:21.281 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.317 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.317 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.317 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.317 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.317 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.317 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8 -46:21.317 [INFO] TestAgent.server: member joined, marking health alive: member=Node-21119a5a-778f-c58f-0bae-4a85bfaf34f8 -46:21.332 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.420 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.428 [INFO] TestAgent: Synced node info -46:21.532 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11123] -46:21.532 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11123 -46:21.533 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45.dc1 127.0.0.1 -46:21.533 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:21.533 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-328d19ee-7f6a-b366-9022-aa1b3ff34d45.dc1 area=wan -46:22.047 [INFO] TestAgent: Requesting shutdown -46:22.047 [INFO] TestAgent.server: shutting down server -46:22.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.047 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.047 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.047 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.047 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.047 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.047 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.047 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.047 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.051 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.058Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.058Z [INFO] agent.router.manager: shutting down -46:22.058 [INFO] TestAgent: consul server down -46:22.058 [INFO] TestAgent: shutdown complete -46:22.058 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11859 network=tcp -46:22.058 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11859 network=udp -46:22.058 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11958 network=tcp -46:22.558 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.558 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Join_WAN (1.43s) -=== CONT TestAgent_ServiceMaintenance_Disable -2022-11-07T22:46:21.285Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.299Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.348 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 Address:127.0.0.1:11033}]" -46:21.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11033 [Follower]" leader= -46:21.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 127.0.0.1 -46:21.350 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 127.0.0.1 -2022-11-07T22:46:21.350Z [INFO] agent.router: Initializing LAN area manager -46:21.350 [INFO] TestAgent.server: Adding LAN server: server="Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 (Addr: tcp/127.0.0.1:11033) (DC: dc1)" -46:21.350 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 area=wan -46:21.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11119 network=udp -46:21.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11119 network=tcp -46:21.350 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11029 network=tcp -46:21.350 [INFO] TestAgent: started state syncer -46:21.350 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11132 network=tcp -46:21.394 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.394 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11033 [Candidate]" term=2 -46:21.410 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.410 [DEBUG] TestAgent.server.raft: vote granted: from=f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 term=2 tally=1 -46:21.410 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.410 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11033 [Leader]" -46:21.410 [INFO] TestAgent.server: cluster leadership acquired -46:21.410 [INFO] TestAgent.server: New leader elected: payload=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 -46:21.431 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11033 -46:21.458 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.496 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.496 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.496 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.496 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.496 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.496 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 -46:21.496 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 -46:21.508 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.544 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:34850 -46:21.545 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e 127.0.0.1 -46:21.545 [INFO] TestAgent.server: Adding LAN server: server="Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e (Addr: tcp/127.0.0.1:11997) (DC: dc1)" -46:21.545 [INFO] TestAgent.server: New leader elected: payload=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.545 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e other=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 -46:21.545 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.545 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 127.0.0.1:11996 -46:21.545 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:38622 -46:21.546 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 127.0.0.1 -46:21.546 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 address=127.0.0.1:11996 -46:21.546 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 area=wan -46:21.557 [INFO] TestAgent: Requesting shutdown -46:21.557 [INFO] TestAgent.server: shutting down server -46:21.557 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.557 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.557 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.557 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.557 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.557 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.557 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.557 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.557 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:21.557 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.557 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.565 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.570Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.570Z [INFO] agent.router.manager: shutting down -46:21.570 [INFO] TestAgent: consul server down -46:21.570 [INFO] TestAgent: shutdown complete -46:21.570 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11119 network=tcp -46:21.570 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11119 network=udp -46:21.570 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11029 network=tcp -46:22.070 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.070 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:21.044Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:21.044Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.054Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:21.054Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.087 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a865eee4-34aa-5b68-58d0-ace4cd3e134e Address:127.0.0.1:11997}]" -46:21.087 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11997 [Follower]" leader= -46:21.087 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 127.0.0.1 -46:21.088 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e 127.0.0.1 -2022-11-07T22:46:21.088Z [INFO] agent.router: Initializing LAN area manager -46:21.088 [INFO] TestAgent.server: Adding LAN server: server="Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e (Addr: tcp/127.0.0.1:11997) (DC: dc1)" -46:21.088 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 area=wan -46:21.088 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11992 network=udp -46:21.088 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11992 network=tcp -46:21.088 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11993 network=tcp -46:21.088 [INFO] TestAgent: started state syncer -46:21.088 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12012 network=tcp -46:21.138 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:21.143 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.143 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11997 [Candidate]" term=2 -46:21.160 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.160 [DEBUG] TestAgent.server.raft: vote granted: from=a865eee4-34aa-5b68-58d0-ace4cd3e134e term=2 tally=1 -46:21.160 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.160 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11997 [Leader]" -46:21.160 [INFO] TestAgent.server: cluster leadership acquired -46:21.160 [INFO] TestAgent.server: New leader elected: payload=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.175 [INFO] TestAgent.server: initializing acls -46:21.183 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:21.183 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:21.189 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:21.201 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:21.201 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:21.201 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:21.201 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.201 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 -46:21.201 [INFO] TestAgent.server: Updating LAN server: server="Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e (Addr: tcp/127.0.0.1:11997) (DC: dc1)" -46:21.201 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e.dc1 area=wan -46:21.211 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.232 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.232 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.232 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.232 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.232 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.232 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.232 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.248 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.257 [INFO] TestAgent: Synced node info -46:21.271 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.518 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.544 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11031] -46:21.544 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11031 -46:21.545 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 127.0.0.1 -46:21.545 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:21.545 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:21.545 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:55062 -46:21.545 [INFO] TestAgent.server: Adding LAN server: server="Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 (Addr: tcp/127.0.0.1:11033) (DC: dc1)" -46:21.545 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 other=Node-a865eee4-34aa-5b68-58d0-ace4cd3e134e -46:21.545 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3 -46:21.545 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 127.0.0.1:11032 -46:21.546 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 127.0.0.1 -46:21.546 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 area=wan -46:21.547 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-f01d0c7f-65c0-64d0-a3eb-aa8b38015aa3.dc1 address=127.0.0.1:11032 -46:21.550 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader -46:22.070 [INFO] TestAgent: Requesting shutdown -46:22.070 [INFO] TestAgent.server: shutting down server -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.070 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.070 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.070 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.070 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.074 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.078Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.078Z [INFO] agent.router.manager: shutting down -46:22.078 [INFO] TestAgent: consul server down -46:22.078 [INFO] TestAgent: shutdown complete -46:22.078 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11992 network=tcp -46:22.078 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11992 network=udp -46:22.078 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11993 network=tcp -46:22.578 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.578 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Join_ACLDeny (1.54s) - --- PASS: TestAgent_Join_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_Join_ACLDeny/agent_master_token (0.00s) - --- PASS: TestAgent_Join_ACLDeny/read-only_token (0.01s) -=== CONT TestAgent_ServiceMaintenance_Enable -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc2 -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc3 -2022-11-07T22:46:22.128Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.128Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.143Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.143Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.169 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:42b57f28-7871-06a5-827b-8026f2aa4716 Address:127.0.0.1:12004}]" -46:22.169 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12004 [Follower]" leader= -46:22.170 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-42b57f28-7871-06a5-827b-8026f2aa4716.dc1 127.0.0.1 -46:22.170 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-42b57f28-7871-06a5-827b-8026f2aa4716 127.0.0.1 -2022-11-07T22:46:22.170Z [INFO] agent.router: Initializing LAN area manager -46:22.170 [INFO] TestAgent.server: Adding LAN server: server="Node-42b57f28-7871-06a5-827b-8026f2aa4716 (Addr: tcp/127.0.0.1:12004) (DC: dc1)" -46:22.170 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-42b57f28-7871-06a5-827b-8026f2aa4716.dc1 area=wan -46:22.171 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11091 network=udp -46:22.171 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11091 network=tcp -46:22.171 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12000 network=tcp -46:22.171 [INFO] TestAgent: started state syncer -46:22.171 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11139 network=tcp -46:22.221 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:22.230 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.230 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12004 [Candidate]" term=2 -46:22.242 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.242 [DEBUG] TestAgent.server.raft: vote granted: from=42b57f28-7871-06a5-827b-8026f2aa4716 term=2 tally=1 -46:22.242 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.242 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12004 [Leader]" -46:22.242 [INFO] TestAgent.server: cluster leadership acquired -46:22.242 [INFO] TestAgent.server: New leader elected: payload=Node-42b57f28-7871-06a5-827b-8026f2aa4716 -46:22.249 [INFO] TestAgent.server: initializing acls -46:22.252 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:22.252 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:22.259 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:22.268 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:22.268 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:22.268 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:22.268 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-42b57f28-7871-06a5-827b-8026f2aa4716 -46:22.268 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-42b57f28-7871-06a5-827b-8026f2aa4716.dc1 -46:22.268 [INFO] TestAgent.server: Updating LAN server: server="Node-42b57f28-7871-06a5-827b-8026f2aa4716 (Addr: tcp/127.0.0.1:12004) (DC: dc1)" -46:22.268 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-42b57f28-7871-06a5-827b-8026f2aa4716.dc1 area=wan -46:22.277 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.284 [INFO] TestAgent: Synced node info -46:22.284 [DEBUG] TestAgent: Node info in sync -46:22.296 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-42b57f28-7871-06a5-827b-8026f2aa4716 -46:22.301 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.301 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.301 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.301 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.301 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.301 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-42b57f28-7871-06a5-827b-8026f2aa4716 -46:22.301 [INFO] TestAgent.server: member joined, marking health alive: member=Node-42b57f28-7871-06a5-827b-8026f2aa4716 -46:22.308 [INFO] TestAgent: Requesting shutdown -46:22.308 [INFO] TestAgent.server: shutting down server -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.308 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.308 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.309 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:22.309 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:22.309 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.309 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.309 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.314 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:22.315 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.315 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:46:22.320Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.320Z [INFO] agent.router.manager: shutting down -46:22.321 [INFO] TestAgent: consul server down -46:22.321 [INFO] TestAgent: shutdown complete -46:22.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11091 network=tcp -46:22.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11091 network=udp -46:22.321 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12000 network=tcp -46:22.821 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.821 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Metrics_ACLDeny (0.71s) +=== RUN TestAgent_Self_ACLDeny/no_token +=== RUN TestAgent_Self_ACLDeny/agent_master_token +=== RUN TestAgent_Self_ACLDeny/read-only_token +2021-10-05T16:32:01.918Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:01.918Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:01.946Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:01.946Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.028 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ee380f6-77e8-3944-26d4-e27c28212d61 Address:127.0.0.1:29428}]" +32:02.029 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4ee380f6-77e8-3944-26d4-e27c28212d61.dc1 127.0.0.1 +32:02.029 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4ee380f6-77e8-3944-26d4-e27c28212d61 127.0.0.1 +2021-10-05T16:32:02.029Z [INFO] agent.router: Initializing LAN area manager +32:02.030 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29428 [Follower]" leader= +32:02.030 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4ee380f6-77e8-3944-26d4-e27c28212d61.dc1 area=wan +32:02.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29479 network=tcp +32:02.030 [INFO] TestAgent.server: Adding LAN server: server="Node-4ee380f6-77e8-3944-26d4-e27c28212d61 (Addr: tcp/127.0.0.1:29428) (DC: dc1)" +32:02.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29479 network=udp +32:02.031 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29480 network=tcp +32:02.031 [INFO] TestAgent: started state syncer +32:02.031 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30386 network=tcp +32:02.076 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.076 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29428 [Candidate]" term=2 +32:02.080 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:02.094 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:02.094 [DEBUG] TestAgent.server.raft: vote granted: from=4ee380f6-77e8-3944-26d4-e27c28212d61 term=2 tally=1 +32:02.094 [INFO] TestAgent.server.raft: election won: tally=1 +32:02.094 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29428 [Leader]" +32:02.094 [INFO] TestAgent.server: cluster leadership acquired +32:02.094 [INFO] TestAgent.server: New leader elected: payload=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.102 [INFO] TestAgent.server: initializing acls +32:02.106 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:02.106 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:02.115 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:02.126 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:02.126 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:02.126 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:02.126 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.126 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4ee380f6-77e8-3944-26d4-e27c28212d61.dc1 +32:02.127 [INFO] TestAgent.server: Updating LAN server: server="Node-4ee380f6-77e8-3944-26d4-e27c28212d61 (Addr: tcp/127.0.0.1:29428) (DC: dc1)" +32:02.127 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4ee380f6-77e8-3944-26d4-e27c28212d61.dc1 area=wan +32:02.151 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:02.187 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:02.187 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:02.187 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:02.187 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:02.187 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:02.188 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.188 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.197 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.216 [INFO] TestAgent.server: federation state anti-entropy synced +32:02.272 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.272 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4ee380f6-77e8-3944-26d4-e27c28212d61 +32:02.289 [INFO] TestAgent: Requesting shutdown +32:02.289 [INFO] TestAgent.server: shutting down server +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:02.289 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:02.289 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:02.289 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:02.289 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:02.289 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:02.290 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:02.290 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:02.290 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:02.290 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:02.290 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:02.304 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:02.312Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:02.312Z [INFO] agent.router.manager: shutting down +32:02.312 [INFO] TestAgent: consul server down +32:02.312 [INFO] TestAgent: shutdown complete +32:02.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29479 network=tcp +32:02.312 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29479 network=udp +32:02.312 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29480 network=tcp +32:02.812 [INFO] TestAgent: Waiting for endpoints to shut down +32:02.812 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Reload_ACLDeny (0.92s) + --- PASS: TestAgent_Reload_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_Reload_ACLDeny/read-only_token (0.02s) +=== CONT TestAgent_Checks_ACLFilter +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:02.579Z [WARN] TestAgent_Reload: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.229Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.250Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.303 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46181440-f24a-82f1-f970-e9952dcff27a Address:127.0.0.1:30329}]" +32:02.303 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30329 [Follower]" leader= +32:02.304 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46181440-f24a-82f1-f970-e9952dcff27a.dc1 127.0.0.1 +32:02.308 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46181440-f24a-82f1-f970-e9952dcff27a 127.0.0.1 +2021-10-05T16:32:02.308Z [INFO] agent.router: Initializing LAN area manager +32:02.309 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30387 network=udp +32:02.309 [INFO] TestAgent.server: Adding LAN server: server="Node-46181440-f24a-82f1-f970-e9952dcff27a (Addr: tcp/127.0.0.1:30329) (DC: dc1)" +32:02.309 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46181440-f24a-82f1-f970-e9952dcff27a.dc1 area=wan +32:02.309 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30387 network=tcp +32:02.309 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30388 network=tcp +32:02.309 [WARN] TestAgent: The 'handler' field in watches has been deprecated and replaced with the 'args' field. See https://www.consul.io/docs/agent/watches.html +32:02.309 [INFO] TestAgent: started state syncer +32:02.310 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29506 network=tcp +32:02.352 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.352 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30329 [Candidate]" term=2 +32:02.385 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:02.385 [DEBUG] TestAgent.server.raft: vote granted: from=46181440-f24a-82f1-f970-e9952dcff27a term=2 tally=1 +32:02.385 [INFO] TestAgent.server.raft: election won: tally=1 +32:02.385 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30329 [Leader]" +32:02.386 [INFO] TestAgent.server: cluster leadership acquired +32:02.386 [INFO] TestAgent.server: New leader elected: payload=Node-46181440-f24a-82f1-f970-e9952dcff27a +32:02.397 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30329 +32:02.416 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:02.442 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:02.442 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:02.442 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:02.442 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:02.442 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:02.442 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46181440-f24a-82f1-f970-e9952dcff27a +32:02.442 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46181440-f24a-82f1-f970-e9952dcff27a +32:02.446 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test?dc=dc1 from=127.0.0.1:50642 latency=135.370961ms +32:02.475 [DEBUG] TestAgent: watch handler output: watch_handler=true output= +32:02.484 [INFO] TestAgent.server: federation state anti-entropy synced +32:02.500 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:02.507 [INFO] TestAgent: Synced node info +32:02.511 [INFO] TestAgent: Synced service: service=redis +32:02.511 [DEBUG] TestAgent: Node info in sync +32:02.511 [DEBUG] TestAgent: Service in sync: service=redis +32:02.579 [INFO] TestAgent: Requesting shutdown +32:02.579 [INFO] TestAgent.server: shutting down server +32:02.579 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:02.579 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:02.592 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:02.608Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:02.608Z [INFO] agent.router.manager: shutting down +32:02.608 [INFO] TestAgent: consul server down +32:02.608 [INFO] TestAgent: shutdown complete +32:02.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30387 network=tcp +32:02.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30387 network=udp +32:02.608 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30388 network=tcp +32:03.112 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.112 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Reload (0.91s) +=== CONT TestAgent_HealthServicesACLEnforcement +2021-10-05T16:32:02.292Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.292Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.307Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.307Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.408 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:294860bd-eb30-9223-12cf-5a3862ccbc29 Address:127.0.0.1:29491}]" +32:02.408 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29491 [Follower]" leader= +32:02.412 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-294860bd-eb30-9223-12cf-5a3862ccbc29.dc1 127.0.0.1 +32:02.413 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-294860bd-eb30-9223-12cf-5a3862ccbc29 127.0.0.1 +2021-10-05T16:32:02.413Z [INFO] agent.router: Initializing LAN area manager +32:02.413 [INFO] TestAgent.server: Adding LAN server: server="Node-294860bd-eb30-9223-12cf-5a3862ccbc29 (Addr: tcp/127.0.0.1:29491) (DC: dc1)" +32:02.413 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-294860bd-eb30-9223-12cf-5a3862ccbc29.dc1 area=wan +32:02.413 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29500 network=udp +32:02.413 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29500 network=tcp +32:02.414 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29501 network=tcp +32:02.414 [INFO] TestAgent: started state syncer +32:02.414 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29429 network=tcp +32:02.465 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.465 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29491 [Candidate]" term=2 +32:02.467 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:02.489 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:02.489 [DEBUG] TestAgent.server.raft: vote granted: from=294860bd-eb30-9223-12cf-5a3862ccbc29 term=2 tally=1 +32:02.489 [INFO] TestAgent.server.raft: election won: tally=1 +32:02.489 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29491 [Leader]" +32:02.492 [INFO] TestAgent.server: cluster leadership acquired +32:02.492 [INFO] TestAgent.server: New leader elected: payload=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.506 [INFO] TestAgent.server: initializing acls +32:02.510 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:02.510 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:02.517 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:02.526 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:02.526 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:02.526 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:02.526 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.526 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-294860bd-eb30-9223-12cf-5a3862ccbc29.dc1 +32:02.527 [INFO] TestAgent.server: Updating LAN server: server="Node-294860bd-eb30-9223-12cf-5a3862ccbc29 (Addr: tcp/127.0.0.1:29491) (DC: dc1)" +32:02.527 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-294860bd-eb30-9223-12cf-5a3862ccbc29.dc1 area=wan +32:02.541 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:02.579 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:02.579 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:02.579 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:02.579 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:02.579 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:02.579 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.579 [INFO] TestAgent.server: member joined, marking health alive: member=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.598 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.601 [INFO] TestAgent.server: federation state anti-entropy synced +32:02.632 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:02.635 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.635 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-294860bd-eb30-9223-12cf-5a3862ccbc29 +32:02.639 [INFO] TestAgent: Synced node info +32:02.639 [DEBUG] TestAgent: Node info in sync +32:02.672 [INFO] TestAgent: Requesting shutdown +32:02.672 [INFO] TestAgent.server: shutting down server +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:02.672 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:02.673 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:02.673 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:02.673 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:02.688 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:02.695Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:02.695Z [INFO] agent.router.manager: shutting down +32:02.695 [INFO] TestAgent: consul server down +32:02.695 [INFO] TestAgent: shutdown complete +32:02.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29500 network=tcp +32:02.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29500 network=udp +32:02.695 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29501 network=tcp +32:03.195 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.195 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Metrics_ACLDeny (0.93s) --- PASS: TestAgent_Metrics_ACLDeny/no_token (0.00s) --- PASS: TestAgent_Metrics_ACLDeny/agent_master_token (0.00s) - --- PASS: TestAgent_Metrics_ACLDeny/read-only_token (0.01s) -=== CONT TestAgent_ServiceMaintenance_BadRequest -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc1 -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc3 -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc1 -=== RUN TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc2 -2022-11-07T22:46:21.994Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.004Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.028 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:056dd098-be0c-843f-e566-8d3fedeea87f Address:127.0.0.1:11089}]" -46:22.028 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11089 [Follower]" leader= -46:22.029 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-056dd098-be0c-843f-e566-8d3fedeea87f.dc1 127.0.0.1 -46:22.029 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-056dd098-be0c-843f-e566-8d3fedeea87f 127.0.0.1 -2022-11-07T22:46:22.029Z [INFO] agent.router: Initializing LAN area manager -46:22.030 [INFO] TestAgent.server: Adding LAN server: server="Node-056dd098-be0c-843f-e566-8d3fedeea87f (Addr: tcp/127.0.0.1:11089) (DC: dc1)" -46:22.030 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-056dd098-be0c-843f-e566-8d3fedeea87f.dc1 area=wan -46:22.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11049 network=tcp -46:22.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11049 network=udp -46:22.030 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11036 network=tcp -46:22.030 [INFO] TestAgent: started state syncer -46:22.030 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11090 network=tcp -46:22.098 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.098 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11089 [Candidate]" term=2 -46:22.109 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.109 [DEBUG] TestAgent.server.raft: vote granted: from=056dd098-be0c-843f-e566-8d3fedeea87f term=2 tally=1 -46:22.109 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11089 [Leader]" -46:22.109 [INFO] TestAgent.server: cluster leadership acquired -46:22.109 [INFO] TestAgent.server: New leader elected: payload=Node-056dd098-be0c-843f-e566-8d3fedeea87f -46:22.116 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11089 -46:22.128 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.153 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.153 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.153 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.153 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.153 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.153 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-056dd098-be0c-843f-e566-8d3fedeea87f -46:22.153 [INFO] TestAgent.server: member joined, marking health alive: member=Node-056dd098-be0c-843f-e566-8d3fedeea87f -46:22.165 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.404 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.408 [INFO] TestAgent: Synced node info -46:22.408 [DEBUG] TestAgent: Node info in sync -46:22.461 [INFO] TestAgent: Requesting shutdown -46:22.461 [INFO] TestAgent.server: shutting down server -46:22.461 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.461 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.461 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.461 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.461 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.461 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.461 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.461 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.461 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.461 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.469 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.477Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.477Z [INFO] agent.router.manager: shutting down -46:22.477 [INFO] TestAgent: consul server down -46:22.477 [INFO] TestAgent: shutdown complete -46:22.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11049 network=tcp -46:22.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11049 network=udp -46:22.477 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11036 network=tcp -46:22.977 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.977 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheckScriptsExecDisable (0.99s) -=== CONT TestAgent_DeregisterService_ACLDeny -2022-11-07T22:46:21.251Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:21.251Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.266Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:21.266Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.308 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1ca337b3-dff6-7c81-60dc-d34c7ef182b8 Address:127.0.0.1:11117}]" -46:21.308 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11117 [Follower]" leader= -46:21.308 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8.dc1 127.0.0.1 -46:21.309 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 127.0.0.1 -2022-11-07T22:46:21.309Z [INFO] agent.router: Initializing LAN area manager -46:21.309 [INFO] TestAgent.server: Adding LAN server: server="Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 (Addr: tcp/127.0.0.1:11117) (DC: dc1)" -46:21.309 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8.dc1 area=wan -46:21.309 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12034 network=udp -46:21.309 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12034 network=tcp -46:21.309 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11092 network=tcp -46:21.309 [INFO] TestAgent: started state syncer -46:21.309 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11034 network=tcp -46:21.359 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:21.377 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.377 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11117 [Candidate]" term=2 -46:21.397 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:21.397 [DEBUG] TestAgent.server.raft: vote granted: from=1ca337b3-dff6-7c81-60dc-d34c7ef182b8 term=2 tally=1 -46:21.397 [INFO] TestAgent.server.raft: election won: tally=1 -46:21.397 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11117 [Leader]" -46:21.397 [INFO] TestAgent.server: cluster leadership acquired -46:21.397 [INFO] TestAgent.server: New leader elected: payload=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.408 [INFO] TestAgent.server: initializing acls -46:21.415 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:21.415 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:21.431 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:21.453 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:21.453 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:21.453 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:21.453 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.453 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8.dc1 -46:21.453 [INFO] TestAgent.server: Updating LAN server: server="Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 (Addr: tcp/127.0.0.1:11117) (DC: dc1)" -46:21.453 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8.dc1 area=wan -46:21.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.501 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.501 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:21.501 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:21.501 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:21.501 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:21.501 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.501 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.520 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.520 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.561 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -46:21.561 [WARN] TestAgent: Node info update blocked by ACLs: node=1ca337b3-dff6-7c81-60dc-d34c7ef182b8 accessorID=00000000-0000-0000-0000-000000000002 -46:21.561 [DEBUG] TestAgent: Node info in sync -46:21.746 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.746 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-1ca337b3-dff6-7c81-60dc-d34c7ef182b8 -46:21.755 [INFO] TestAgent: Updated agent's ACL token: token=default -46:22.456 [DEBUG] TestAgent.acl: dropping service from result due to ACLs: service="{consul {}}" -46:22.456 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.461 [INFO] TestAgent: Synced node info -46:22.469 [INFO] TestAgent: Requesting shutdown -46:22.469 [INFO] TestAgent.server: shutting down server -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.469 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.469 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.469 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:22.469 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.477 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.492Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.492Z [INFO] agent.router.manager: shutting down -46:22.492 [INFO] TestAgent: consul server down -46:22.492 [INFO] TestAgent: shutdown complete -46:22.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12034 network=tcp -46:22.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12034 network=udp -46:22.492 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11092 network=tcp -46:22.994 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.994 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_TokenTriggersFullSync (6.11s) - --- PASS: TestAgent_TokenTriggersFullSync/acl_agent_token (1.29s) - --- PASS: TestAgent_TokenTriggersFullSync/agent (0.91s) - --- PASS: TestAgent_TokenTriggersFullSync/acl_token (2.15s) - --- PASS: TestAgent_TokenTriggersFullSync/default (1.76s) -=== CONT TestAgent_DeregisterService -2022-11-07T22:46:22.265Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.265Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.275Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.275Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.302 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7312baa6-83ce-82a2-ac7c-704656cf5b51 Address:127.0.0.1:11983}]" -46:22.302 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11983 [Follower]" leader= -46:22.302 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7312baa6-83ce-82a2-ac7c-704656cf5b51.dc1 127.0.0.1 -46:22.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 127.0.0.1 -2022-11-07T22:46:22.303Z [INFO] agent.router: Initializing LAN area manager -46:22.303 [INFO] TestAgent.server: Adding LAN server: server="Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 (Addr: tcp/127.0.0.1:11983) (DC: dc1)" -46:22.303 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51.dc1 area=wan -46:22.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11999 network=tcp -46:22.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11999 network=udp -46:22.303 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11979 network=tcp -46:22.303 [INFO] TestAgent: started state syncer -46:22.303 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11160 network=tcp -46:22.353 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:22.366 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.366 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11983 [Candidate]" term=2 -46:22.377 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.377 [DEBUG] TestAgent.server.raft: vote granted: from=7312baa6-83ce-82a2-ac7c-704656cf5b51 term=2 tally=1 -46:22.377 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.377 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11983 [Leader]" -46:22.377 [INFO] TestAgent.server: cluster leadership acquired -46:22.377 [INFO] TestAgent.server: New leader elected: payload=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.386 [INFO] TestAgent.server: initializing acls -46:22.390 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:22.390 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:22.397 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:22.409 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:22.409 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:22.409 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:22.409 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.409 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7312baa6-83ce-82a2-ac7c-704656cf5b51.dc1 -46:22.409 [INFO] TestAgent.server: Updating LAN server: server="Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 (Addr: tcp/127.0.0.1:11983) (DC: dc1)" -46:22.410 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51.dc1 area=wan -46:22.419 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.449 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.449 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.449 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.449 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.449 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.449 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.449 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.455 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.465 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.469 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.470 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7312baa6-83ce-82a2-ac7c-704656cf5b51 -46:22.478 [INFO] TestAgent: Synced node info -46:22.478 [DEBUG] TestAgent: Node info in sync -46:22.479 [DEBUG] TestAgent: Check status updated: check=test status=critical -46:22.479 [DEBUG] TestAgent: Node info in sync -46:22.479 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 -46:22.479 [INFO] TestAgent: Requesting shutdown -46:22.479 [INFO] TestAgent.server: shutting down server -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:22.479 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.479 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.479 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:22.479 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:22.492 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.499Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.499Z [INFO] agent.router.manager: shutting down -46:22.499 [INFO] TestAgent: consul server down -46:22.499 [INFO] TestAgent: shutdown complete -46:22.499 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11999 network=tcp -46:22.499 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11999 network=udp -46:22.499 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11979 network=tcp -46:22.999 [INFO] TestAgent: Waiting for endpoints to shut down -46:22.999 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_FailCheck_ACLDeny (0.75s) - --- PASS: TestAgent_FailCheck_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_FailCheck_ACLDeny/root_token (0.00s) + --- PASS: TestAgent_Metrics_ACLDeny/read-only_token (0.04s) === CONT TestAgent_HealthServiceByName -2022-11-07T22:46:22.213Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.223Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.249 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f40a0617-a3c8-7d86-cb94-1fee7e51148d Address:127.0.0.1:12032}]" -46:22.249 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12032 [Follower]" leader= -46:22.250 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d.dc1 127.0.0.1 -46:22.250 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d 127.0.0.1 -2022-11-07T22:46:22.250Z [INFO] agent.router: Initializing LAN area manager -46:22.250 [INFO] TestAgent.server: Adding LAN server: server="Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d (Addr: tcp/127.0.0.1:12032) (DC: dc1)" -46:22.250 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d.dc1 area=wan -46:22.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12020 network=udp -46:22.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12020 network=tcp -46:22.251 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12028 network=tcp -46:22.251 [INFO] TestAgent: started state syncer -46:22.251 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11998 network=tcp -46:22.309 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.309 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12032 [Candidate]" term=2 -46:22.329 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.329 [DEBUG] TestAgent.server.raft: vote granted: from=f40a0617-a3c8-7d86-cb94-1fee7e51148d term=2 tally=1 -46:22.329 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.329 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12032 [Leader]" -46:22.329 [INFO] TestAgent.server: cluster leadership acquired -46:22.329 [INFO] TestAgent.server: New leader elected: payload=Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d -46:22.337 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12032 -46:22.354 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.381 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.381 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.381 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.381 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.381 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.381 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d -46:22.381 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f40a0617-a3c8-7d86-cb94-1fee7e51148d -46:22.387 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.492 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.498 [INFO] TestAgent: Synced node info -46:22.498 [DEBUG] TestAgent: Node info in sync -46:22.499 [INFO] TestAgent: Requesting shutdown -46:22.499 [INFO] TestAgent.server: shutting down server -46:22.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.499 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.499 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.499 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.499 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.499 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.499 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.505 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.511Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.511Z [INFO] agent.router.manager: shutting down -46:22.511 [INFO] TestAgent: consul server down -46:22.511 [INFO] TestAgent: shutdown complete -46:22.511 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12020 network=tcp -46:22.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12020 network=udp -46:22.512 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12028 network=tcp -46:23.012 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.012 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_HealthServiceByID (0.81s) - --- PASS: TestAgent_HealthServiceByID/passing_checks (0.00s) - --- PASS: TestAgent_HealthServiceByID/passing_checks/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/passing_checks/format=json (0.00s) - --- PASS: TestAgent_HealthServiceByID/warning_checks (0.00s) - --- PASS: TestAgent_HealthServiceByID/warning_checks/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/warning_checks/format=json (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_checks (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_checks/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_checks/format=json (0.00s) - --- PASS: TestAgent_HealthServiceByID/unknown_serviceid (0.00s) - --- PASS: TestAgent_HealthServiceByID/unknown_serviceid/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/unknown_serviceid/format=json (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_check_on_node (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_check_on_node/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/critical_check_on_node/format=json (0.00s) - --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node (0.00s) - --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node/format=text (0.00s) - --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node/format=json (0.00s) -=== CONT TestAgent_Self +=== RUN TestAgent_Checks_ACLFilter/no_token +=== RUN TestAgent_Checks_ACLFilter/root_token +2021-10-05T16:32:02.329Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.329Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.349Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.349Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.387 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6df3c29c-18dd-9145-b536-2741856b329a Address:127.0.0.1:29498}]" +32:02.388 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29498 [Follower]" leader= +32:02.388 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6df3c29c-18dd-9145-b536-2741856b329a.dc1 127.0.0.1 +32:02.389 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6df3c29c-18dd-9145-b536-2741856b329a 127.0.0.1 +2021-10-05T16:32:02.389Z [INFO] agent.router: Initializing LAN area manager +32:02.389 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29458 network=udp +32:02.389 [INFO] TestAgent.server: Adding LAN server: server="Node-6df3c29c-18dd-9145-b536-2741856b329a (Addr: tcp/127.0.0.1:29498) (DC: dc1)" +32:02.389 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6df3c29c-18dd-9145-b536-2741856b329a.dc1 area=wan +32:02.389 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29458 network=tcp +32:02.390 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29459 network=tcp +32:02.390 [INFO] TestAgent: started state syncer +32:02.390 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29499 network=tcp +32:02.427 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.427 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29498 [Candidate]" term=2 +32:02.440 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:02.441 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:02.441 [DEBUG] TestAgent.server.raft: vote granted: from=6df3c29c-18dd-9145-b536-2741856b329a term=2 tally=1 +32:02.441 [INFO] TestAgent.server.raft: election won: tally=1 +32:02.441 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29498 [Leader]" +32:02.442 [INFO] TestAgent.server: cluster leadership acquired +32:02.442 [INFO] TestAgent.server: New leader elected: payload=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.461 [INFO] TestAgent.server: initializing acls +32:02.469 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:02.469 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:02.485 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:02.516 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:02.516 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:02.516 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:02.516 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.516 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-6df3c29c-18dd-9145-b536-2741856b329a.dc1 +32:02.517 [INFO] TestAgent.server: Updating LAN server: server="Node-6df3c29c-18dd-9145-b536-2741856b329a (Addr: tcp/127.0.0.1:29498) (DC: dc1)" +32:02.517 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-6df3c29c-18dd-9145-b536-2741856b329a.dc1 area=wan +32:02.565 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:02.608 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:02.608 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:02.608 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:02.608 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:02.608 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:02.608 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.608 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.621 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.626 [INFO] TestAgent.server: federation state anti-entropy synced +32:02.626 [INFO] TestAgent: Synced node info +32:02.782 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.782 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-6df3c29c-18dd-9145-b536-2741856b329a +32:02.795 [INFO] TestAgent: Requesting shutdown +32:02.796 [INFO] TestAgent.server: shutting down server +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:02.796 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:02.796 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:02.796 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:02.796 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:02.800 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:02.804Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:02.804Z [INFO] agent.router.manager: shutting down +32:02.804 [INFO] TestAgent: consul server down +32:02.804 [INFO] TestAgent: shutdown complete +32:02.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29458 network=tcp +32:02.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29458 network=udp +32:02.804 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29459 network=tcp +32:03.308 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.308 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Self_ACLDeny (1.00s) + --- PASS: TestAgent_Self_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_Self_ACLDeny/agent_master_token (0.00s) + --- PASS: TestAgent_Self_ACLDeny/read-only_token (0.01s) +=== CONT TestAgent_HealthServiceByID +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_HealthServicesACLEnforcement/no-token-health-by-id +=== RUN TestAgent_HealthServicesACLEnforcement/no-token-health-by-name +=== RUN TestAgent_HealthServicesACLEnforcement/root-token-health-by-id +=== RUN TestAgent_HealthServicesACLEnforcement/root-token-health-by-name +2021-10-05T16:32:02.494Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.555Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.615 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 Address:127.0.0.1:29421}]" +32:02.616 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29421 [Follower]" leader= +32:02.616 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6.dc1 127.0.0.1 +32:02.617 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 127.0.0.1 +2021-10-05T16:32:02.617Z [INFO] agent.router: Initializing LAN area manager +32:02.617 [INFO] TestAgent.server: Adding LAN server: server="Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 (Addr: tcp/127.0.0.1:29421) (DC: dc1)" +32:02.617 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6.dc1 area=wan +32:02.618 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29535 network=udp +32:02.618 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29535 network=tcp +32:02.618 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29536 network=tcp +32:02.618 [INFO] TestAgent: started state syncer +32:02.671 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.671 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29421 [Candidate]" term=2 +32:02.696 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:02.696 [DEBUG] TestAgent.server.raft: vote granted: from=76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 term=2 tally=1 +32:02.696 [INFO] TestAgent.server.raft: election won: tally=1 +32:02.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29421 [Leader]" +32:02.697 [INFO] TestAgent.server: cluster leadership acquired +32:02.697 [INFO] TestAgent.server: New leader elected: payload=Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 +32:02.724 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29421 +32:02.772 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:02.832 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:02.832 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:02.832 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:02.832 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:02.832 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:02.832 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 +32:02.832 [INFO] TestAgent.server: member joined, marking health alive: member=Node-76b618f1-cff4-b1eb-19f9-ae5f027dc1e6 +32:02.855 [INFO] TestAgent.server: federation state anti-entropy synced +32:02.879 [INFO] TestAgent: Requesting shutdown +32:02.879 [INFO] TestAgent.server: shutting down server +32:02.879 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:02.879 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:02.879 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:02.879 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:02.879 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:02.879 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:02.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:02.879 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:02.879 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:02.879 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:02.879 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:02.884 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:02.892Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:02.892Z [INFO] agent.router.manager: shutting down +32:02.892 [INFO] TestAgent: consul server down +32:02.892 [INFO] TestAgent: shutdown complete +32:02.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29535 network=tcp +32:02.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29535 network=udp +32:02.892 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29536 network=tcp +32:03.393 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.393 [INFO] TestAgent: Endpoints down === RUN TestAgent_Self/normal -2022-11-07T22:46:22.326Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.326Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.343Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.343Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.379 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:24e8f198-b38d-2cba-6f45-2ecdb2c19531 Address:127.0.0.1:11145}]" -46:22.380 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11145 [Follower]" leader= -46:22.380 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531.dc1 127.0.0.1 -46:22.380 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 127.0.0.1 -2022-11-07T22:46:22.380Z [INFO] agent.router: Initializing LAN area manager -46:22.381 [INFO] TestAgent.server: Adding LAN server: server="Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 (Addr: tcp/127.0.0.1:11145) (DC: dc1)" -46:22.381 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531.dc1 area=wan -46:22.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12006 network=udp -46:22.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12006 network=tcp -46:22.381 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11085 network=tcp -46:22.381 [INFO] TestAgent: started state syncer -46:22.381 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11083 network=tcp -46:22.419 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.419 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11145 [Candidate]" term=2 -46:22.432 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:22.433 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.433 [DEBUG] TestAgent.server.raft: vote granted: from=24e8f198-b38d-2cba-6f45-2ecdb2c19531 term=2 tally=1 -46:22.433 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.433 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11145 [Leader]" -46:22.433 [INFO] TestAgent.server: cluster leadership acquired -46:22.433 [INFO] TestAgent.server: New leader elected: payload=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.444 [INFO] TestAgent.server: initializing acls -46:22.448 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:22.448 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:22.456 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:22.470 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:22.470 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:22.470 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:22.470 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.471 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531.dc1 -46:22.471 [INFO] TestAgent.server: Updating LAN server: server="Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 (Addr: tcp/127.0.0.1:11145) (DC: dc1)" -46:22.471 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531.dc1 area=wan -46:22.485 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.510 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.510 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.510 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.510 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.510 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.510 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.510 [INFO] TestAgent.server: member joined, marking health alive: member=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.514 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.520 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.521 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.521 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-24e8f198-b38d-2cba-6f45-2ecdb2c19531 -46:22.526 [INFO] TestAgent: Service entered maintenance mode: service=test -46:22.533 [INFO] TestAgent: Synced node info -46:22.533 [WARN] TestAgent: Service registration blocked by ACLs: service=test accessorID=00000000-0000-0000-0000-000000000002 -46:22.536 [INFO] TestAgent: Synced check: check=_service_maintenance:test -46:22.536 [INFO] TestAgent: Requesting shutdown -46:22.536 [INFO] TestAgent.server: shutting down server -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.536 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.536 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:22.536 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.536 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:22.536 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.538 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.544Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.544Z [INFO] agent.router.manager: shutting down -46:22.544 [INFO] TestAgent: consul server down -46:22.544 [INFO] TestAgent: shutdown complete -46:22.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12006 network=tcp -46:22.544 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12006 network=udp -46:22.544 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11085 network=tcp -46:23.044 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.044 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServiceMaintenance_ACLDeny (0.75s) - --- PASS: TestAgent_ServiceMaintenance_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_ServiceMaintenance_ACLDeny/root_token (0.01s) -=== CONT TestAgent_UpdateCheck_ACLDeny -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:22.199Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.209Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.233 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:caa01ee4-27da-11bf-911a-512d71514557 Address:127.0.0.1:11082}]" -46:22.233 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11082 [Follower]" leader= -46:22.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-caa01ee4-27da-11bf-911a-512d71514557.dc1 127.0.0.1 -46:22.235 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-caa01ee4-27da-11bf-911a-512d71514557 127.0.0.1 -2022-11-07T22:46:22.235Z [INFO] agent.router: Initializing LAN area manager -46:22.235 [INFO] TestAgent.server: Adding LAN server: server="Node-caa01ee4-27da-11bf-911a-512d71514557 (Addr: tcp/127.0.0.1:11082) (DC: dc1)" -46:22.235 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-caa01ee4-27da-11bf-911a-512d71514557.dc1 area=wan -46:22.235 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11140 network=udp -46:22.235 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11140 network=tcp -46:22.236 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11050 network=tcp -46:22.236 [INFO] TestAgent: started state syncer -46:22.236 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12019 network=tcp -46:22.302 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.302 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11082 [Candidate]" term=2 -46:22.321 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.321 [DEBUG] TestAgent.server.raft: vote granted: from=caa01ee4-27da-11bf-911a-512d71514557 term=2 tally=1 -46:22.321 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.321 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11082 [Leader]" -46:22.322 [INFO] TestAgent.server: cluster leadership acquired -46:22.322 [INFO] TestAgent.server: New leader elected: payload=Node-caa01ee4-27da-11bf-911a-512d71514557 -46:22.329 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11082 -46:22.344 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.369 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.369 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.369 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.369 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.369 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.369 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-caa01ee4-27da-11bf-911a-512d71514557 -46:22.369 [INFO] TestAgent.server: member joined, marking health alive: member=Node-caa01ee4-27da-11bf-911a-512d71514557 -46:22.377 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.609 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.614 [INFO] TestAgent: Synced node info -46:22.614 [DEBUG] TestAgent: Node info in sync -46:22.655 [INFO] TestAgent: Requesting shutdown -46:22.655 [INFO] TestAgent.server: shutting down server -46:22.655 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.655 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.655 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.655 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.655 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.655 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.655 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.655 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.659 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.663Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.663Z [INFO] agent.router.manager: shutting down -46:22.664 [INFO] TestAgent: consul server down -46:22.664 [INFO] TestAgent: shutdown complete -46:22.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11140 network=tcp -46:22.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11140 network=udp -46:22.664 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11050 network=tcp -46:23.164 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.164 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ChecksWithFilter (0.98s) -=== CONT TestAgent_Self_ACLDeny -2022-11-07T22:46:22.298Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.315Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.349 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:21164713-b4b7-9db3-22be-7b4ab3cad89b Address:127.0.0.1:11138}]" -46:22.349 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11138 [Follower]" leader= -46:22.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-21164713-b4b7-9db3-22be-7b4ab3cad89b.dc1 127.0.0.1 -46:22.350 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-21164713-b4b7-9db3-22be-7b4ab3cad89b 127.0.0.1 -2022-11-07T22:46:22.350Z [INFO] agent.router: Initializing LAN area manager -46:22.350 [INFO] TestAgent.server: Adding LAN server: server="Node-21164713-b4b7-9db3-22be-7b4ab3cad89b (Addr: tcp/127.0.0.1:11138) (DC: dc1)" -46:22.350 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-21164713-b4b7-9db3-22be-7b4ab3cad89b.dc1 area=wan -46:22.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11161 network=udp -46:22.350 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11161 network=tcp -46:22.351 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11043 network=tcp -46:22.351 [INFO] TestAgent: started state syncer -46:22.351 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12005 network=tcp -46:22.395 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.395 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11138 [Candidate]" term=2 -46:22.406 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.406 [DEBUG] TestAgent.server.raft: vote granted: from=21164713-b4b7-9db3-22be-7b4ab3cad89b term=2 tally=1 -46:22.406 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.406 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11138 [Leader]" -46:22.406 [INFO] TestAgent.server: cluster leadership acquired -46:22.406 [INFO] TestAgent.server: New leader elected: payload=Node-21164713-b4b7-9db3-22be-7b4ab3cad89b -46:22.413 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11138 -46:22.428 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.460 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.460 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.460 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.460 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.460 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.460 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-21164713-b4b7-9db3-22be-7b4ab3cad89b -46:22.460 [INFO] TestAgent.server: member joined, marking health alive: member=Node-21164713-b4b7-9db3-22be-7b4ab3cad89b -46:22.467 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.632 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.634 [INFO] TestAgent: Synced node info -46:22.664 [INFO] TestAgent: Requesting shutdown -46:22.665 [INFO] TestAgent.server: shutting down server -46:22.665 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.665 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.665 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.665 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.665 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.665 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.665 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.665 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.665 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.665 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.671 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.674Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.674Z [INFO] agent.router.manager: shutting down -46:22.675 [INFO] TestAgent: consul server down -46:22.675 [INFO] TestAgent: shutdown complete -46:22.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11161 network=tcp -46:22.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11161 network=udp -46:22.675 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11043 network=tcp -46:23.176 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.176 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_NodeMaintenance_BadRequest (0.89s) -=== CONT TestAgent_PassCheck_ACLDeny -=== RUN TestAgent_ServiceMaintenance_BadRequest/not_enabled -=== RUN TestAgent_ServiceMaintenance_BadRequest/no_service_id -=== RUN TestAgent_ServiceMaintenance_BadRequest/bad_service_id -=== RUN TestAgent_DeregisterService_ACLDeny/no_token -=== RUN TestAgent_DeregisterService_ACLDeny/root_token -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:22.569Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.578Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.600 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4c79f976-1402-d3a8-9fb7-64aed350572e Address:127.0.0.1:11068}]" -46:22.600 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11068 [Follower]" leader= -46:22.600 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4c79f976-1402-d3a8-9fb7-64aed350572e.dc1 127.0.0.1 -46:22.601 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4c79f976-1402-d3a8-9fb7-64aed350572e 127.0.0.1 -2022-11-07T22:46:22.601Z [INFO] agent.router: Initializing LAN area manager -46:22.601 [INFO] TestAgent.server: Adding LAN server: server="Node-4c79f976-1402-d3a8-9fb7-64aed350572e (Addr: tcp/127.0.0.1:11068) (DC: dc1)" -46:22.601 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4c79f976-1402-d3a8-9fb7-64aed350572e.dc1 area=wan -46:22.601 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11154 network=udp -46:22.601 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11154 network=tcp -46:22.601 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11148 network=tcp -46:22.602 [INFO] TestAgent: started state syncer -46:22.602 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11069 network=tcp -46:22.650 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.650 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11068 [Candidate]" term=2 -46:22.659 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.659 [DEBUG] TestAgent.server.raft: vote granted: from=4c79f976-1402-d3a8-9fb7-64aed350572e term=2 tally=1 -46:22.659 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.659 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11068 [Leader]" -46:22.660 [INFO] TestAgent.server: cluster leadership acquired -46:22.660 [INFO] TestAgent.server: New leader elected: payload=Node-4c79f976-1402-d3a8-9fb7-64aed350572e -46:22.666 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11068 -46:22.679 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.700 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.700 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.700 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.700 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.700 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.701 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4c79f976-1402-d3a8-9fb7-64aed350572e -46:22.701 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4c79f976-1402-d3a8-9fb7-64aed350572e -46:22.716 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.824 [INFO] TestAgent: Service entered maintenance mode: service=test -46:22.824 [DEBUG] TestAgent: removed check: check=_service_maintenance:test -46:22.824 [INFO] TestAgent: Service left maintenance mode: service=test -46:22.828 [INFO] TestAgent: Synced node info -46:22.833 [INFO] TestAgent: Synced service: service=test -46:22.837 [INFO] TestAgent: Deregistered check: check=_service_maintenance:test -46:22.837 [INFO] TestAgent: Requesting shutdown -46:22.838 [INFO] TestAgent.server: shutting down server -46:22.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.838 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.838 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.838 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.838 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.838 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:22.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.838 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.838 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.841 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.845Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.845Z [INFO] agent.router.manager: shutting down -46:22.845 [INFO] TestAgent: consul server down -46:22.845 [INFO] TestAgent: shutdown complete -46:22.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11154 network=tcp -46:22.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11154 network=udp -46:22.845 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11148 network=tcp -46:23.345 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.345 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServiceMaintenance_Disable (0.79s) -=== CONT TestAgent_FailCheck -=== RUN TestAgent_UpdateCheck_ACLDeny/no_token -=== RUN TestAgent_UpdateCheck_ACLDeny/root_token -2022-11-07T22:46:22.591Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.599Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.622 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bc504d8f-e2bc-d047-99df-8f505646e24e Address:127.0.0.1:11208}]" -46:22.622 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11208 [Follower]" leader= -46:22.622 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bc504d8f-e2bc-d047-99df-8f505646e24e.dc1 127.0.0.1 -46:22.623 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bc504d8f-e2bc-d047-99df-8f505646e24e 127.0.0.1 -2022-11-07T22:46:22.623Z [INFO] agent.router: Initializing LAN area manager -46:22.623 [INFO] TestAgent.server: Adding LAN server: server="Node-bc504d8f-e2bc-d047-99df-8f505646e24e (Addr: tcp/127.0.0.1:11208) (DC: dc1)" -46:22.623 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bc504d8f-e2bc-d047-99df-8f505646e24e.dc1 area=wan -46:22.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11070 network=udp -46:22.623 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11070 network=tcp -46:22.624 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11064 network=tcp -46:22.624 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11209 network=tcp -46:22.624 [INFO] TestAgent: started state syncer -46:22.661 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.661 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11208 [Candidate]" term=2 -46:22.677 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.677 [DEBUG] TestAgent.server.raft: vote granted: from=bc504d8f-e2bc-d047-99df-8f505646e24e term=2 tally=1 -46:22.677 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.677 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11208 [Leader]" -46:22.677 [INFO] TestAgent.server: cluster leadership acquired -46:22.677 [INFO] TestAgent.server: New leader elected: payload=Node-bc504d8f-e2bc-d047-99df-8f505646e24e -46:22.683 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11208 -46:22.694 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.719 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.719 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:22.719 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:22.719 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:22.719 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:22.719 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bc504d8f-e2bc-d047-99df-8f505646e24e -46:22.719 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bc504d8f-e2bc-d047-99df-8f505646e24e -46:22.730 [INFO] TestAgent.server: federation state anti-entropy synced -46:22.897 [INFO] TestAgent: Service entered maintenance mode: service=test -46:22.900 [INFO] TestAgent: Synced node info -46:22.903 [INFO] TestAgent: Synced service: service=test -46:22.908 [INFO] TestAgent: Synced check: check=_service_maintenance:test -46:22.908 [INFO] TestAgent: Requesting shutdown -46:22.908 [INFO] TestAgent.server: shutting down server -46:22.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:22.908 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:22.908 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:22.908 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:22.908 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:22.908 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:22.908 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:22.908 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:22.908 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:22.908 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:22.912 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.916Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.916Z [INFO] agent.router.manager: shutting down -46:22.916 [INFO] TestAgent: consul server down -46:22.916 [INFO] TestAgent: shutdown complete -46:22.916 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11070 network=tcp -46:22.916 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11070 network=udp -46:22.916 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11064 network=tcp -46:23.416 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.416 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServiceMaintenance_Enable (0.84s) -=== CONT TestAgent_WarnCheck_ACLDeny -=== RUN TestAgent_PassCheck_ACLDeny/no_token -=== RUN TestAgent_PassCheck_ACLDeny/root_token === RUN TestAgent_HealthServiceByName/passing_checks === RUN TestAgent_HealthServiceByName/passing_checks/format=text === RUN TestAgent_HealthServiceByName/passing_checks/format=json @@ -49031,503 +49901,230 @@ === RUN TestAgent_HealthServiceByName/maintenance_check_on_node === RUN TestAgent_HealthServiceByName/maintenance_check_on_node/format=text === RUN TestAgent_HealthServiceByName/maintenance_check_on_node/format=json -=== RUN TestAgent_Self_ACLDeny/no_token -=== RUN TestAgent_Self_ACLDeny/agent_master_token -=== RUN TestAgent_Self_ACLDeny/read-only_token -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:22.837Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.846Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.869 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:700f688b-a889-38ec-e77e-654fe1b8addb Address:127.0.0.1:11061}]" -46:22.869 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11061 [Follower]" leader= -46:22.870 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-700f688b-a889-38ec-e77e-654fe1b8addb.dc1 127.0.0.1 -46:22.870 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-700f688b-a889-38ec-e77e-654fe1b8addb 127.0.0.1 -2022-11-07T22:46:22.870Z [INFO] agent.router: Initializing LAN area manager -46:22.870 [INFO] TestAgent.server: Adding LAN server: server="Node-700f688b-a889-38ec-e77e-654fe1b8addb (Addr: tcp/127.0.0.1:11061) (DC: dc1)" -46:22.870 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-700f688b-a889-38ec-e77e-654fe1b8addb.dc1 area=wan -46:22.871 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11210 network=udp -46:22.871 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11210 network=tcp -46:22.871 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11099 network=tcp -46:22.871 [INFO] TestAgent: started state syncer -46:22.871 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11062 network=tcp -46:22.933 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.933 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11061 [Candidate]" term=2 -46:22.943 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:22.943 [DEBUG] TestAgent.server.raft: vote granted: from=700f688b-a889-38ec-e77e-654fe1b8addb term=2 tally=1 -46:22.943 [INFO] TestAgent.server.raft: election won: tally=1 -46:22.943 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11061 [Leader]" -46:22.943 [INFO] TestAgent.server: cluster leadership acquired -46:22.944 [INFO] TestAgent.server: New leader elected: payload=Node-700f688b-a889-38ec-e77e-654fe1b8addb -46:22.948 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11061 -46:22.966 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.004 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.004 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.004 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.004 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.004 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.004 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-700f688b-a889-38ec-e77e-654fe1b8addb -46:23.004 [INFO] TestAgent.server: member joined, marking health alive: member=Node-700f688b-a889-38ec-e77e-654fe1b8addb -46:23.019 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.030 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.034 [INFO] TestAgent: Synced node info -46:23.034 [DEBUG] TestAgent: Node info in sync -46:23.196 [INFO] TestAgent: Requesting shutdown -46:23.196 [INFO] TestAgent.server: shutting down server -46:23.196 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.196 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.196 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.196 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.196 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.196 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.196 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.196 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.196 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.196 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.210Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.210Z [INFO] agent.router.manager: shutting down -46:23.210 [INFO] TestAgent: consul server down -46:23.210 [INFO] TestAgent: shutdown complete -46:23.210 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11210 network=tcp -46:23.210 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11210 network=udp -46:23.210 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11099 network=tcp -46:23.710 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.710 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServiceMaintenance_BadRequest (0.89s) - --- PASS: TestAgent_ServiceMaintenance_BadRequest/not_enabled (0.00s) - --- PASS: TestAgent_ServiceMaintenance_BadRequest/no_service_id (0.00s) - --- PASS: TestAgent_ServiceMaintenance_BadRequest/bad_service_id (0.00s) -=== CONT TestAgent_WarnCheck -2022-11-07T22:46:22.987Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.987Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.997Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:22.997Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.035 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8c6c17b0-781a-1e8c-1360-cf6541b40ffa Address:127.0.0.1:11075}]" -46:23.035 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11075 [Follower]" leader= -46:23.036 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa.dc1 127.0.0.1 -46:23.036 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa 127.0.0.1 -2022-11-07T22:46:23.036Z [INFO] agent.router: Initializing LAN area manager -46:23.036 [INFO] TestAgent.server: Adding LAN server: server="Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa (Addr: tcp/127.0.0.1:11075) (DC: dc1)" -46:23.037 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa.dc1 area=wan -46:23.037 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11063 network=udp -46:23.037 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11063 network=tcp -46:23.037 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11127 network=tcp -46:23.037 [INFO] TestAgent: started state syncer -46:23.037 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11195 network=tcp -46:23.087 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:23.097 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.097 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11075 [Candidate]" term=2 -46:23.111 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.111 [DEBUG] TestAgent.server.raft: vote granted: from=8c6c17b0-781a-1e8c-1360-cf6541b40ffa term=2 tally=1 -46:23.111 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.111 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11075 [Leader]" -46:23.112 [INFO] TestAgent.server: cluster leadership acquired -46:23.112 [INFO] TestAgent.server: New leader elected: payload=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.120 [INFO] TestAgent.server: initializing acls -46:23.123 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.123 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.132 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.145 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.145 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.145 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa.dc1 -46:23.145 [INFO] TestAgent.server: Updating LAN server: server="Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa (Addr: tcp/127.0.0.1:11075) (DC: dc1)" -46:23.145 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa.dc1 area=wan -46:23.158 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.188 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.188 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.188 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.188 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.188 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.188 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.188 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.196 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.197 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.203 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.213 [INFO] TestAgent: Synced node info -46:23.213 [DEBUG] TestAgent: Node info in sync -46:23.214 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8c6c17b0-781a-1e8c-1360-cf6541b40ffa -46:23.214 [DEBUG] TestAgent: removed service: service=test -46:23.214 [DEBUG] TestAgent: Node info in sync -46:23.214 [INFO] TestAgent: Deregistered service: service=test -46:23.214 [INFO] TestAgent: Requesting shutdown -46:23.214 [INFO] TestAgent.server: shutting down server -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.214 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.214 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.215 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.215 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.220 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.230Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.230Z [INFO] agent.router.manager: shutting down -46:23.230 [INFO] TestAgent: consul server down -46:23.230 [INFO] TestAgent: shutdown complete -46:23.230 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11063 network=tcp -46:23.230 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11063 network=udp -46:23.230 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11127 network=tcp -46:23.730 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.730 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_DeregisterService_ACLDeny (0.75s) - --- PASS: TestAgent_DeregisterService_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_DeregisterService_ACLDeny/root_token (0.00s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestAgent_HealthServiceByID/passing_checks +=== RUN TestAgent_HealthServiceByID/passing_checks/format=text +=== RUN TestAgent_HealthServiceByID/passing_checks/format=json +=== RUN TestAgent_HealthServiceByID/warning_checks +=== RUN TestAgent_HealthServiceByID/warning_checks/format=text +=== RUN TestAgent_HealthServiceByID/warning_checks/format=json +=== RUN TestAgent_HealthServiceByID/critical_checks +=== RUN TestAgent_HealthServiceByID/critical_checks/format=text +=== RUN TestAgent_HealthServiceByID/critical_checks/format=json +=== RUN TestAgent_HealthServiceByID/unknown_serviceid +=== RUN TestAgent_HealthServiceByID/unknown_serviceid/format=text +=== RUN TestAgent_HealthServiceByID/unknown_serviceid/format=json +=== RUN TestAgent_HealthServiceByID/critical_check_on_node +=== RUN TestAgent_HealthServiceByID/critical_check_on_node/format=text +=== RUN TestAgent_HealthServiceByID/critical_check_on_node/format=json +=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node +=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node/format=text +=== RUN TestAgent_HealthServiceByID/maintenance_check_on_node/format=json +2021-10-05T16:32:02.830Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.830Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:02.847Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:02.847Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:02.943 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:60ea4d12-b9dd-11e5-d413-9054ad23f46a Address:127.0.0.1:29540}]" +32:02.943 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29540 [Follower]" leader= +32:02.945 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 127.0.0.1 +32:02.946 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a 127.0.0.1 +2021-10-05T16:32:02.946Z [INFO] agent.router: Initializing LAN area manager +32:02.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29416 network=udp +32:02.946 [INFO] TestAgent.server: Adding LAN server: server="Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a (Addr: tcp/127.0.0.1:29540) (DC: dc1)" +32:02.947 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 area=wan +32:02.947 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29416 network=tcp +32:02.947 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29417 network=tcp +32:02.947 [INFO] TestAgent: started state syncer +32:02.947 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29541 network=tcp +32:02.985 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:02.985 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29540 [Candidate]" term=2 +32:03.000 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:03.048 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.048 [DEBUG] TestAgent.server.raft: vote granted: from=60ea4d12-b9dd-11e5-d413-9054ad23f46a term=2 tally=1 +32:03.048 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.048 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29540 [Leader]" +32:03.052 [INFO] TestAgent.server: cluster leadership acquired +32:03.052 [INFO] TestAgent.server: New leader elected: payload=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.070 [INFO] TestAgent.server: initializing acls +32:03.075 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:03.075 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:03.092 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:03.100 [INFO] TestAgent.server: initializing acls +32:03.100 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:03.124 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:03.124 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:03.125 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:03.125 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:03.125 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:03.125 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.125 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 +32:03.125 [INFO] TestAgent.server: Updating LAN server: server="Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a (Addr: tcp/127.0.0.1:29540) (DC: dc1)" +32:03.125 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 area=wan +32:03.125 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.125 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 +32:03.128 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a.dc1 area=wan +32:03.128 [INFO] TestAgent.server: Updating LAN server: server="Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a (Addr: tcp/127.0.0.1:29540) (DC: dc1)" +32:03.155 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:03.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:03.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:03.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:03.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:03.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:03.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.207 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.207 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.212 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-60ea4d12-b9dd-11e5-d413-9054ad23f46a +32:03.212 [DEBUG] TestAgent: dropping check from result due to ACLs: check=mysql +32:03.213 [INFO] TestAgent: Requesting shutdown +32:03.213 [INFO] TestAgent.server: shutting down server +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:03.213 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:03.213 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:03.213 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:03.213 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:03.213 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:03.213 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:03.213 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:03.213 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:03.217 [INFO] TestAgent.server: federation state anti-entropy synced +32:03.217 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:03.218 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:03.222Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:03.222Z [INFO] agent.router.manager: shutting down +32:03.222 [INFO] TestAgent: consul server down +32:03.222 [INFO] TestAgent: shutdown complete +32:03.222 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29416 network=tcp +32:03.222 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29416 network=udp +32:03.222 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29417 network=tcp +32:03.728 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.728 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Checks_ACLFilter (0.92s) + --- PASS: TestAgent_Checks_ACLFilter/no_token (0.00s) + --- PASS: TestAgent_Checks_ACLFilter/root_token (0.00s) +=== CONT TestAgent_ChecksWithFilter +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:03.130Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.155Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:03.228 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7bd5f1c2-efe8-14ef-fb65-40feb274349c Address:127.0.0.1:29512}]" +32:03.230 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c.dc1 127.0.0.1 +32:03.230 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c 127.0.0.1 +2021-10-05T16:32:03.230Z [INFO] agent.router: Initializing LAN area manager +32:03.231 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29493 network=udp +32:03.231 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29512 [Follower]" leader= +32:03.232 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c.dc1 area=wan +32:03.236 [INFO] TestAgent.server: Adding LAN server: server="Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c (Addr: tcp/127.0.0.1:29512) (DC: dc1)" +32:03.236 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29493 network=tcp +32:03.237 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29494 network=tcp +32:03.237 [INFO] TestAgent: started state syncer +32:03.238 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29534 network=tcp +32:03.284 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:03.286 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:03.286 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29512 [Candidate]" term=2 +32:03.297 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.297 [DEBUG] TestAgent.server.raft: vote granted: from=7bd5f1c2-efe8-14ef-fb65-40feb274349c term=2 tally=1 +32:03.297 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.297 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29512 [Leader]" +32:03.300 [INFO] TestAgent.server: cluster leadership acquired +32:03.300 [INFO] TestAgent.server: New leader elected: payload=Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c +32:03.306 [INFO] TestAgent.server: initializing acls +32:03.310 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:03.320 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +32:03.338 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:03.352 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:03.352 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:03.352 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:03.352 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c +32:03.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c.dc1 +32:03.353 [INFO] TestAgent.server: Updating LAN server: server="Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c (Addr: tcp/127.0.0.1:29512) (DC: dc1)" +32:03.353 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7bd5f1c2-efe8-14ef-fb65-40feb274349c.dc1 area=wan +32:03.377 [INFO] TestAgent: Requesting shutdown +32:03.377 [INFO] TestAgent.server: shutting down server +32:03.377 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:03.377 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:03.377 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:03.377 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:03.377 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:03.377 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:03.384 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:03.389Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:03.389Z [INFO] agent.router.manager: shutting down +32:03.409 [INFO] TestAgent: consul server down +32:03.409 [INFO] TestAgent: shutdown complete +32:03.409 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29493 network=tcp +32:03.409 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29493 network=udp +32:03.409 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29494 network=tcp +32:03.409 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +32:03.409 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:03.409 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +32:03.409 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +32:03.409 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +32:03.911 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.911 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_HealthServicesACLEnforcement (0.80s) + --- PASS: TestAgent_HealthServicesACLEnforcement/no-token-health-by-id (0.00s) + --- PASS: TestAgent_HealthServicesACLEnforcement/no-token-health-by-name (0.00s) + --- PASS: TestAgent_HealthServicesACLEnforcement/root-token-health-by-id (0.00s) + --- PASS: TestAgent_HealthServicesACLEnforcement/root-token-health-by-name (0.00s) === CONT TestAgent_Checks -2022-11-07T22:46:23.006Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.017Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.056 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a1e1038-91f1-fbfb-61bb-23fd3dbde72f Address:127.0.0.1:11187}]" -46:23.056 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11187 [Follower]" leader= -46:23.056 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f.dc1 127.0.0.1 -46:23.057 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f 127.0.0.1 -2022-11-07T22:46:23.057Z [INFO] agent.router: Initializing LAN area manager -46:23.057 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11196 network=udp -46:23.057 [INFO] TestAgent.server: Adding LAN server: server="Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f (Addr: tcp/127.0.0.1:11187) (DC: dc1)" -46:23.057 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f.dc1 area=wan -46:23.057 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11196 network=tcp -46:23.057 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11197 network=tcp -46:23.057 [INFO] TestAgent: started state syncer -46:23.057 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11055 network=tcp -46:23.098 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.098 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11187 [Candidate]" term=2 -46:23.111 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.111 [DEBUG] TestAgent.server.raft: vote granted: from=8a1e1038-91f1-fbfb-61bb-23fd3dbde72f term=2 tally=1 -46:23.111 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.111 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11187 [Leader]" -46:23.111 [INFO] TestAgent.server: cluster leadership acquired -46:23.112 [INFO] TestAgent.server: New leader elected: payload=Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f -46:23.120 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11187 -46:23.142 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.171 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.171 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.171 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.171 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.171 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.171 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f -46:23.171 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8a1e1038-91f1-fbfb-61bb-23fd3dbde72f -46:23.194 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.221 [DEBUG] TestAgent: removed service: service=test -46:23.225 [INFO] TestAgent: Synced node info -46:23.229 [INFO] TestAgent: Deregistered service: service=test -46:23.229 [INFO] TestAgent: Requesting shutdown -46:23.229 [INFO] TestAgent.server: shutting down server -46:23.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.229 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.229 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.229 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.229 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.229 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.229 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.229 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:23.229 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.229 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.229 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.237 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.241Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.241Z [INFO] agent.router.manager: shutting down -46:23.241 [INFO] TestAgent: consul server down -46:23.241 [INFO] TestAgent: shutdown complete -46:23.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11196 network=tcp -46:23.241 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11196 network=udp -46:23.241 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11197 network=tcp -46:23.741 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.741 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_DeregisterService (0.75s) -=== CONT TestAgent_DeregisterCheckACLDeny -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestAgent_WarnCheck_ACLDeny/no_token -=== RUN TestAgent_WarnCheck_ACLDeny/root_token -2022-11-07T22:46:23.054Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.054Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.065Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.065Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.090 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:234c2084-731b-f66d-6985-bbfd55a1f899 Address:127.0.0.1:11229}]" -46:23.090 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11229 [Follower]" leader= -46:23.091 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-234c2084-731b-f66d-6985-bbfd55a1f899.dc1 127.0.0.1 -46:23.092 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-234c2084-731b-f66d-6985-bbfd55a1f899 127.0.0.1 -2022-11-07T22:46:23.092Z [INFO] agent.router: Initializing LAN area manager -46:23.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11182 network=udp -46:23.092 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-234c2084-731b-f66d-6985-bbfd55a1f899.dc1 area=wan -46:23.092 [INFO] TestAgent.server: Adding LAN server: server="Node-234c2084-731b-f66d-6985-bbfd55a1f899 (Addr: tcp/127.0.0.1:11229) (DC: dc1)" -46:23.092 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11182 network=tcp -46:23.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11225 network=tcp -46:23.092 [INFO] TestAgent: started state syncer -46:23.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11188 network=tcp -46:23.142 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:23.151 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.151 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11229 [Candidate]" term=2 -46:23.167 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.167 [DEBUG] TestAgent.server.raft: vote granted: from=234c2084-731b-f66d-6985-bbfd55a1f899 term=2 tally=1 -46:23.167 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.167 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11229 [Leader]" -46:23.167 [INFO] TestAgent.server: cluster leadership acquired -46:23.167 [INFO] TestAgent.server: New leader elected: payload=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.174 [INFO] TestAgent.server: initializing acls -46:23.180 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.180 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.188 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.199 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.199 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.199 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.199 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.199 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-234c2084-731b-f66d-6985-bbfd55a1f899.dc1 -46:23.200 [INFO] TestAgent.server: Updating LAN server: server="Node-234c2084-731b-f66d-6985-bbfd55a1f899 (Addr: tcp/127.0.0.1:11229) (DC: dc1)" -46:23.200 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-234c2084-731b-f66d-6985-bbfd55a1f899.dc1 area=wan -46:23.215 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.230 [INFO] TestAgent: Synced node info -46:23.230 [DEBUG] TestAgent: Node info in sync -46:23.247 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.247 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.247 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.247 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.247 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.247 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.247 [INFO] TestAgent.server: member joined, marking health alive: member=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.258 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.258 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.383 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.383 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-234c2084-731b-f66d-6985-bbfd55a1f899 -46:23.383 [DEBUG] TestAgent: Check status updated: check=test status=passing -46:23.383 [DEBUG] TestAgent: Node info in sync -46:23.383 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 -46:23.383 [INFO] TestAgent: Requesting shutdown -46:23.383 [INFO] TestAgent.server: shutting down server -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.383 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.383 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.383 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.383 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.388 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.393Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.393Z [INFO] agent.router.manager: shutting down -46:23.393 [INFO] TestAgent: consul server down -46:23.393 [INFO] TestAgent: shutdown complete -46:23.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11182 network=tcp -46:23.393 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11182 network=udp -46:23.393 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11225 network=tcp -46:23.893 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.893 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_UpdateCheck_ACLDeny (0.85s) - --- PASS: TestAgent_UpdateCheck_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_UpdateCheck_ACLDeny/root_token (0.00s) -=== CONT TestAgent_Checks_ACLFilter -2022-11-07T22:46:23.028Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.043Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.070 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:845bfd4f-68ec-1676-45d3-6aa2c6c25d1f Address:127.0.0.1:11054}]" -46:23.070 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11054 [Follower]" leader= -46:23.071 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f.dc1 127.0.0.1 -46:23.071 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f 127.0.0.1 -2022-11-07T22:46:23.071Z [INFO] agent.router: Initializing LAN area manager -46:23.071 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f.dc1 area=wan -46:23.071 [INFO] TestAgent.server: Adding LAN server: server="Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f (Addr: tcp/127.0.0.1:11054) (DC: dc1)" -46:23.071 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11077 network=udp -46:23.072 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11077 network=tcp -46:23.072 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11155 network=tcp -46:23.072 [INFO] TestAgent: started state syncer -46:23.072 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11181 network=tcp -46:23.106 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.106 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11054 [Candidate]" term=2 -46:23.119 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.119 [DEBUG] TestAgent.server.raft: vote granted: from=845bfd4f-68ec-1676-45d3-6aa2c6c25d1f term=2 tally=1 -46:23.119 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.119 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11054 [Leader]" -46:23.120 [INFO] TestAgent.server: cluster leadership acquired -46:23.120 [INFO] TestAgent.server: New leader elected: payload=Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f -46:23.129 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11054 -46:23.145 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.173 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.173 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.173 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.173 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.173 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.173 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f -46:23.173 [INFO] TestAgent.server: member joined, marking health alive: member=Node-845bfd4f-68ec-1676-45d3-6aa2c6c25d1f -46:23.189 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.397 [INFO] TestAgent: Requesting shutdown -46:23.397 [INFO] TestAgent.server: shutting down server -46:23.397 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.397 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.397 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.397 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.397 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.398 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.398 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.398 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.398 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.398 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.398 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:23.401 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.404Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.404Z [INFO] agent.router.manager: shutting down -46:23.404 [INFO] TestAgent: consul server down -46:23.404 [INFO] TestAgent: shutdown complete -46:23.404 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11077 network=tcp -46:23.404 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11077 network=udp -46:23.405 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11155 network=tcp -46:23.905 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.905 [INFO] TestAgent: Endpoints down -=== RUN TestAgent_Self/no_grpc -=== RUN TestAgent_DeregisterCheckACLDeny/no_token -=== RUN TestAgent_DeregisterCheckACLDeny/root_token -2022-11-07T22:46:23.013Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.060 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5be54dc6-aa94-d50a-e464-90bed3431d15 Address:127.0.0.1:11194}]" -46:23.060 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11194 [Follower]" leader= -46:23.060 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5be54dc6-aa94-d50a-e464-90bed3431d15.dc1 127.0.0.1 -46:23.061 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5be54dc6-aa94-d50a-e464-90bed3431d15 127.0.0.1 -2022-11-07T22:46:23.061Z [INFO] agent.router: Initializing LAN area manager -46:23.061 [INFO] TestAgent.server: Adding LAN server: server="Node-5be54dc6-aa94-d50a-e464-90bed3431d15 (Addr: tcp/127.0.0.1:11194) (DC: dc1)" -46:23.061 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5be54dc6-aa94-d50a-e464-90bed3431d15.dc1 area=wan -46:23.061 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11056 network=udp -46:23.061 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11056 network=tcp -46:23.062 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11176 network=tcp -46:23.062 [INFO] TestAgent: started state syncer -46:23.062 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11076 network=tcp -46:23.123 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.123 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11194 [Candidate]" term=2 -46:23.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.137 [DEBUG] TestAgent.server.raft: vote granted: from=5be54dc6-aa94-d50a-e464-90bed3431d15 term=2 tally=1 -46:23.137 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11194 [Leader]" -46:23.137 [INFO] TestAgent.server: cluster leadership acquired -46:23.137 [INFO] TestAgent.server: New leader elected: payload=Node-5be54dc6-aa94-d50a-e464-90bed3431d15 -46:23.143 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11194 -46:23.164 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.196 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.196 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.196 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.196 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.196 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.196 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5be54dc6-aa94-d50a-e464-90bed3431d15 -46:23.196 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5be54dc6-aa94-d50a-e464-90bed3431d15 -46:23.210 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.213 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.217 [INFO] TestAgent: Synced node info -46:23.448 [INFO] TestAgent: Requesting shutdown -46:23.448 [INFO] TestAgent.server: shutting down server -46:23.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.448 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.448 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.448 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.448 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.448 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.448 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.448 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.448 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.448 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.469 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.491Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.491Z [INFO] agent.router.manager: shutting down -46:23.491 [INFO] TestAgent: consul server down -46:23.491 [INFO] TestAgent: shutdown complete -46:23.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11056 network=tcp -46:23.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11056 network=udp -46:23.491 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11176 network=tcp -46:23.991 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.991 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:23.197Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.197Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.208Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.208Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.245 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:de61997e-8195-a063-902a-a4c98237c08b Address:127.0.0.1:11232}]" -46:23.245 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11232 [Follower]" leader= -46:23.247 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 127.0.0.1 -46:23.247 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-de61997e-8195-a063-902a-a4c98237c08b 127.0.0.1 -2022-11-07T22:46:23.247Z [INFO] agent.router: Initializing LAN area manager -46:23.248 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 area=wan -46:23.248 [INFO] TestAgent.server: Adding LAN server: server="Node-de61997e-8195-a063-902a-a4c98237c08b (Addr: tcp/127.0.0.1:11232) (DC: dc1)" -46:23.248 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11213 network=tcp -46:23.248 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11213 network=udp -46:23.248 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11214 network=tcp -46:23.248 [INFO] TestAgent: started state syncer -46:23.248 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11170 network=tcp -46:23.287 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.287 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11232 [Candidate]" term=2 -46:23.295 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.295 [DEBUG] TestAgent.server.raft: vote granted: from=de61997e-8195-a063-902a-a4c98237c08b term=2 tally=1 -46:23.295 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.295 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11232 [Leader]" -46:23.296 [INFO] TestAgent.server: cluster leadership acquired -46:23.296 [INFO] TestAgent.server: New leader elected: payload=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.298 [INFO] TestAgent.server: initializing acls -46:23.302 [INFO] TestAgent.server: initializing acls -46:23.302 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.302 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.302 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.317 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.317 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.320 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.320 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.320 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.320 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:23.321 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 -46:23.321 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 area=wan -46:23.321 [INFO] TestAgent.server: Updating LAN server: server="Node-de61997e-8195-a063-902a-a4c98237c08b (Addr: tcp/127.0.0.1:11232) (DC: dc1)" -46:23.323 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.323 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.323 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 -46:23.323 [INFO] TestAgent.server: Updating LAN server: server="Node-de61997e-8195-a063-902a-a4c98237c08b (Addr: tcp/127.0.0.1:11232) (DC: dc1)" -46:23.323 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-de61997e-8195-a063-902a-a4c98237c08b.dc1 area=wan -46:23.331 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.350 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.350 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.350 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.350 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.350 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.350 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.350 [INFO] TestAgent.server: member joined, marking health alive: member=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.356 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.356 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.356 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.445 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.445 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-de61997e-8195-a063-902a-a4c98237c08b -46:23.445 [DEBUG] TestAgent: Check status updated: check=test status=passing -46:23.452 [INFO] TestAgent: Synced node info -46:23.452 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 -46:23.452 [INFO] TestAgent: Requesting shutdown -46:23.452 [INFO] TestAgent.server: shutting down server -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.452 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.452 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.452 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.452 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.452 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.469 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.491Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.491Z [INFO] agent.router.manager: shutting down -46:23.491 [INFO] TestAgent: consul server down -46:23.491 [INFO] TestAgent: shutdown complete -46:23.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11213 network=tcp -46:23.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11213 network=udp -46:23.491 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11214 network=tcp -46:23.991 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.991 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_HealthServiceByName (0.99s) +2021-10-05T16:32:03.213Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.230Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:03.268 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:74640e3f-b3cd-5625-9fff-ace66d0540a7 Address:127.0.0.1:29526}]" +32:03.269 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-74640e3f-b3cd-5625-9fff-ace66d0540a7.dc1 127.0.0.1 +32:03.269 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 127.0.0.1 +2021-10-05T16:32:03.269Z [INFO] agent.router: Initializing LAN area manager +32:03.270 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29526 [Follower]" leader= +32:03.270 [INFO] TestAgent.server: Adding LAN server: server="Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 (Addr: tcp/127.0.0.1:29526) (DC: dc1)" +32:03.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-74640e3f-b3cd-5625-9fff-ace66d0540a7.dc1 area=wan +32:03.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29521 network=tcp +32:03.271 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29521 network=udp +32:03.271 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29522 network=tcp +32:03.271 [INFO] TestAgent: started state syncer +32:03.271 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29527 network=tcp +32:03.322 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:03.322 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29526 [Candidate]" term=2 +32:03.344 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.344 [DEBUG] TestAgent.server.raft: vote granted: from=74640e3f-b3cd-5625-9fff-ace66d0540a7 term=2 tally=1 +32:03.344 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.344 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29526 [Leader]" +32:03.344 [INFO] TestAgent.server: cluster leadership acquired +32:03.344 [INFO] TestAgent.server: New leader elected: payload=Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 +32:03.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29526 +32:03.371 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:03.434 [INFO] TestAgent: Requesting shutdown +32:03.434 [INFO] TestAgent.server: shutting down server +32:03.434 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:03.434 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:03.436 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:03.436 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:03.436 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:03.436 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:03.436 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:03.436 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 +32:03.437 [INFO] TestAgent.server: member joined, marking health alive: member=Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 +32:03.438 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:03.442Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:03.442Z [INFO] agent.router.manager: shutting down +32:03.452 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-74640e3f-b3cd-5625-9fff-ace66d0540a7 127.0.0.1 29524 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:74640e3f-b3cd-5625-9fff-ace66d0540a7 port:29526 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29525] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +32:03.452 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:03.452 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:03.452 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:03.452 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:03.452 [INFO] TestAgent: consul server down +32:03.452 [INFO] TestAgent: shutdown complete +32:03.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29521 network=tcp +32:03.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29521 network=udp +32:03.452 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29522 network=tcp +32:03.452 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:03.452 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:03.452 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:03.955 [INFO] TestAgent: Waiting for endpoints to shut down +32:03.955 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_HealthServiceByName (0.76s) --- PASS: TestAgent_HealthServiceByName/passing_checks (0.00s) --- PASS: TestAgent_HealthServiceByName/passing_checks/format=text (0.00s) --- PASS: TestAgent_HealthServiceByName/passing_checks/format=json (0.00s) @@ -49547,1458 +50144,1047 @@ --- PASS: TestAgent_HealthServiceByName/maintenance_check_on_node/format=text (0.00s) --- PASS: TestAgent_HealthServiceByName/maintenance_check_on_node/format=json (0.00s) === CONT TestAgent_Services_ACLFilter ---- PASS: TestAgent_PassCheck_ACLDeny (0.82s) - --- PASS: TestAgent_PassCheck_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_PassCheck_ACLDeny/root_token (0.01s) -=== CONT TestAgent_PassCheck -2022-11-07T22:46:23.174Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.174Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.190Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.190Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.232 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bbde3eff-54a5-4b9f-e76c-02bffb89ce57 Address:127.0.0.1:11180}]" -46:23.232 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11180 [Follower]" leader= -46:23.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 127.0.0.1 -46:23.234 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 127.0.0.1 -2022-11-07T22:46:23.234Z [INFO] agent.router: Initializing LAN area manager -46:23.234 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 area=wan -46:23.234 [INFO] TestAgent.server: Adding LAN server: server="Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 (Addr: tcp/127.0.0.1:11180) (DC: dc1)" -46:23.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11189 network=udp -46:23.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11189 network=tcp -46:23.235 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11211 network=tcp -46:23.235 [INFO] TestAgent: started state syncer -46:23.235 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11184 network=tcp -46:23.274 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.274 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11180 [Candidate]" term=2 -46:23.282 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.283 [DEBUG] TestAgent.server.raft: vote granted: from=bbde3eff-54a5-4b9f-e76c-02bffb89ce57 term=2 tally=1 -46:23.283 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.283 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11180 [Leader]" -46:23.283 [INFO] TestAgent.server: cluster leadership acquired -46:23.283 [INFO] TestAgent.server: New leader elected: payload=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.285 [INFO] TestAgent.server: initializing acls -46:23.289 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.289 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.289 [INFO] TestAgent.server: initializing acls -46:23.289 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.305 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.305 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.311 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.311 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.311 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.311 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.311 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:23.311 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.311 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 -46:23.311 [INFO] TestAgent.server: Updating LAN server: server="Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 (Addr: tcp/127.0.0.1:11180) (DC: dc1)" -46:23.311 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 area=wan -46:23.311 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.311 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 -46:23.311 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57.dc1 area=wan -46:23.311 [INFO] TestAgent.server: Updating LAN server: server="Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 (Addr: tcp/127.0.0.1:11180) (DC: dc1)" -46:23.322 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.341 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.341 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.341 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.341 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.341 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.341 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.341 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.344 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.353 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.353 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.397 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.400 [INFO] TestAgent: Synced node info -46:23.400 [DEBUG] TestAgent: Node info in sync -46:23.488 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.488 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bbde3eff-54a5-4b9f-e76c-02bffb89ce57 -46:23.496 [INFO] TestAgent: Requesting shutdown -46:23.496 [INFO] TestAgent.server: shutting down server -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.496 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.496 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.496 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.496 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.506 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.515Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.515Z [INFO] agent.router.manager: shutting down -46:23.515 [INFO] TestAgent: consul server down -46:23.515 [INFO] TestAgent: shutdown complete -46:23.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11189 network=tcp -46:23.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11189 network=udp -46:23.515 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11211 network=tcp -46:24.016 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.016 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Self_ACLDeny (0.85s) - --- PASS: TestAgent_Self_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_Self_ACLDeny/agent_master_token (0.00s) - --- PASS: TestAgent_Self_ACLDeny/read-only_token (0.01s) -=== CONT TestAgent_HealthServicesACLEnforcement -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:03.330Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.359Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:03.452 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83df522b-37a4-e6aa-3d56-e22f02bbfa5c Address:127.0.0.1:29547}]" +32:03.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29547 [Follower]" leader= +32:03.453 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c.dc1 127.0.0.1 +32:03.460 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c 127.0.0.1 +2021-10-05T16:32:03.460Z [INFO] agent.router: Initializing LAN area manager +32:03.461 [INFO] TestAgent.server: Adding LAN server: server="Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c (Addr: tcp/127.0.0.1:29547) (DC: dc1)" +32:03.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29542 network=udp +32:03.461 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c.dc1 area=wan +32:03.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29542 network=tcp +32:03.461 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29543 network=tcp +32:03.462 [INFO] TestAgent: started state syncer +32:03.462 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29590 network=tcp +32:03.522 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:03.522 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29547 [Candidate]" term=2 +32:03.576 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.576 [DEBUG] TestAgent.server.raft: vote granted: from=83df522b-37a4-e6aa-3d56-e22f02bbfa5c term=2 tally=1 +32:03.576 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.576 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29547 [Leader]" +32:03.576 [INFO] TestAgent.server: cluster leadership acquired +32:03.576 [INFO] TestAgent.server: New leader elected: payload=Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c +32:03.592 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29547 +32:03.618 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:03.660 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:03.660 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:03.660 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:03.660 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:03.660 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:03.660 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c +32:03.660 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83df522b-37a4-e6aa-3d56-e22f02bbfa5c +32:03.689 [INFO] TestAgent: Requesting shutdown +32:03.689 [INFO] TestAgent.server: shutting down server +32:03.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:03.689 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:03.689 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:03.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:03.689 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:03.689 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:03.689 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:03.689 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:03.689 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:03.693 [INFO] TestAgent.server: federation state anti-entropy synced +32:03.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:03.696 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:03.700Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:03.700Z [INFO] agent.router.manager: shutting down +32:03.701 [INFO] TestAgent: consul server down +32:03.701 [INFO] TestAgent: shutdown complete +32:03.701 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29542 network=tcp +32:03.701 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29542 network=udp +32:03.701 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29543 network=tcp +32:04.201 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.201 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_HealthServiceByID (0.89s) + --- PASS: TestAgent_HealthServiceByID/passing_checks (0.00s) + --- PASS: TestAgent_HealthServiceByID/passing_checks/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/passing_checks/format=json (0.00s) + --- PASS: TestAgent_HealthServiceByID/warning_checks (0.00s) + --- PASS: TestAgent_HealthServiceByID/warning_checks/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/warning_checks/format=json (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_checks (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_checks/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_checks/format=json (0.00s) + --- PASS: TestAgent_HealthServiceByID/unknown_serviceid (0.00s) + --- PASS: TestAgent_HealthServiceByID/unknown_serviceid/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/unknown_serviceid/format=json (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_check_on_node (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_check_on_node/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/critical_check_on_node/format=json (0.00s) + --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node (0.00s) + --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node/format=text (0.00s) + --- PASS: TestAgent_HealthServiceByID/maintenance_check_on_node/format=json (0.00s) +=== CONT TestAgent_Services_TerminatingGateway +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:03.432Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.451Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:03.500 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8aa3c885-2b56-af35-559e-aec3111b1e73 Address:127.0.0.1:29561}]" +32:03.500 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29561 [Follower]" leader= +32:03.501 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8aa3c885-2b56-af35-559e-aec3111b1e73.dc1 127.0.0.1 +32:03.501 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8aa3c885-2b56-af35-559e-aec3111b1e73 127.0.0.1 +2021-10-05T16:32:03.502Z [INFO] agent.router: Initializing LAN area manager +32:03.502 [INFO] TestAgent.server: Adding LAN server: server="Node-8aa3c885-2b56-af35-559e-aec3111b1e73 (Addr: tcp/127.0.0.1:29561) (DC: dc1)" +32:03.502 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29570 network=udp +32:03.502 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8aa3c885-2b56-af35-559e-aec3111b1e73.dc1 area=wan +32:03.502 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29570 network=tcp +32:03.503 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29571 network=tcp +32:03.503 [INFO] TestAgent: started state syncer +32:03.503 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29513 network=tcp +32:03.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:03.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29561 [Candidate]" term=2 +32:03.597 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.597 [DEBUG] TestAgent.server.raft: vote granted: from=8aa3c885-2b56-af35-559e-aec3111b1e73 term=2 tally=1 +32:03.597 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.597 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29561 [Leader]" +32:03.597 [INFO] TestAgent.server: cluster leadership acquired +32:03.598 [INFO] TestAgent.server: New leader elected: payload=Node-8aa3c885-2b56-af35-559e-aec3111b1e73 +32:03.606 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29561 +32:03.631 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:03.669 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:03.669 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:03.669 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:03.669 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:03.669 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:03.669 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8aa3c885-2b56-af35-559e-aec3111b1e73 +32:03.669 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8aa3c885-2b56-af35-559e-aec3111b1e73 +32:03.691 [INFO] TestAgent: Synced node info +32:03.691 [DEBUG] TestAgent: Node info in sync +32:03.698 [INFO] TestAgent.server: federation state anti-entropy synced +32:03.857 [INFO] TestAgent: Requesting shutdown +32:03.857 [INFO] TestAgent.server: shutting down server +32:03.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:03.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:03.857 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:03.857 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:03.857 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:03.857 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:03.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:03.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:03.857 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:03.857 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:03.862 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:03.865Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:03.865Z [INFO] agent.router.manager: shutting down +32:03.865 [INFO] TestAgent: consul server down +32:03.865 [INFO] TestAgent: shutdown complete +32:03.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29570 network=tcp +32:03.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29570 network=udp +32:03.865 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29571 network=tcp +32:04.368 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.368 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Self (1.89s) + --- PASS: TestAgent_Self/no_grpc (0.92s) + --- PASS: TestAgent_Self/normal (0.97s) +=== CONT TestAgent_Services_MeshGateway === RUN TestAgent_Services_ACLFilter/no_token === RUN TestAgent_Services_ACLFilter/root_token -=== RUN TestAgent_Checks_ACLFilter/no_token -=== RUN TestAgent_Checks_ACLFilter/root_token -2022-11-07T22:46:23.356Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.365Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.387 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada Address:127.0.0.1:11267}]" -46:23.387 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11267 [Follower]" leader= -46:23.388 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada.dc1 127.0.0.1 -46:23.388 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada 127.0.0.1 -2022-11-07T22:46:23.388Z [INFO] agent.router: Initializing LAN area manager -46:23.388 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada.dc1 area=wan -46:23.388 [INFO] TestAgent.server: Adding LAN server: server="Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada (Addr: tcp/127.0.0.1:11267) (DC: dc1)" -46:23.389 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11164 network=udp -46:23.389 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11164 network=tcp -46:23.389 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11165 network=tcp -46:23.389 [INFO] TestAgent: started state syncer -46:23.389 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11268 network=tcp -46:23.456 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.456 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11267 [Candidate]" term=2 -46:23.486 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.486 [DEBUG] TestAgent.server.raft: vote granted: from=fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada term=2 tally=1 -46:23.486 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.486 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11267 [Leader]" -46:23.487 [INFO] TestAgent.server: cluster leadership acquired -46:23.487 [INFO] TestAgent.server: New leader elected: payload=Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada -46:23.503 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11267 -46:23.520 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.540 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.540 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.540 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.540 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.540 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.540 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada -46:23.540 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fcf8faa0-9a21-7ac7-4c2c-8c306ce92ada -46:23.546 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.612 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.615 [INFO] TestAgent: Synced node info -46:23.615 [DEBUG] TestAgent: Node info in sync -46:23.853 [DEBUG] TestAgent: Check status updated: check=test status=critical -46:23.853 [DEBUG] TestAgent: Node info in sync -46:23.857 [INFO] TestAgent: Synced check: check=test -46:23.857 [INFO] TestAgent: Requesting shutdown -46:23.857 [INFO] TestAgent.server: shutting down server -46:23.857 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.857 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.857 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.857 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.857 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.857 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.857 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.857 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.861 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.868Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.868Z [INFO] agent.router.manager: shutting down -46:23.869 [INFO] TestAgent: consul server down -46:23.869 [INFO] TestAgent: shutdown complete -46:23.869 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11164 network=tcp -46:23.869 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11164 network=udp -46:23.869 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11165 network=tcp -46:24.369 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.369 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_FailCheck (1.02s) -=== CONT TestACLReplicationStatus -2022-11-07T22:46:23.428Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.428Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.437Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.437Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.502 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:47a45d48-7193-3655-cd1e-f2c8499f7158 Address:127.0.0.1:11218}]" -46:23.502 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11218 [Follower]" leader= -46:23.502 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 127.0.0.1 -46:23.503 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-47a45d48-7193-3655-cd1e-f2c8499f7158 127.0.0.1 -2022-11-07T22:46:23.503Z [INFO] agent.router: Initializing LAN area manager -46:23.503 [INFO] TestAgent.server: Adding LAN server: server="Node-47a45d48-7193-3655-cd1e-f2c8499f7158 (Addr: tcp/127.0.0.1:11218) (DC: dc1)" -46:23.503 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 area=wan -46:23.503 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11171 network=tcp -46:23.503 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11171 network=udp -46:23.503 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11172 network=tcp -46:23.503 [INFO] TestAgent: started state syncer -46:23.503 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11289 network=tcp -46:23.543 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.543 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11218 [Candidate]" term=2 -46:23.553 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.553 [DEBUG] TestAgent.server.raft: vote granted: from=47a45d48-7193-3655-cd1e-f2c8499f7158 term=2 tally=1 -46:23.553 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.553 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11218 [Leader]" -46:23.553 [INFO] TestAgent.server: cluster leadership acquired -46:23.553 [INFO] TestAgent.server: initializing acls -46:23.553 [INFO] TestAgent.server: New leader elected: payload=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.559 [INFO] TestAgent.server: initializing acls -46:23.559 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.559 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.559 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.577 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.577 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.580 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.580 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.580 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.580 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:23.580 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.580 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 -46:23.581 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 area=wan -46:23.581 [INFO] TestAgent.server: Updating LAN server: server="Node-47a45d48-7193-3655-cd1e-f2c8499f7158 (Addr: tcp/127.0.0.1:11218) (DC: dc1)" -46:23.583 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.583 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.583 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 -46:23.583 [INFO] TestAgent.server: Updating LAN server: server="Node-47a45d48-7193-3655-cd1e-f2c8499f7158 (Addr: tcp/127.0.0.1:11218) (DC: dc1)" -46:23.583 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-47a45d48-7193-3655-cd1e-f2c8499f7158.dc1 area=wan -46:23.594 [INFO] TestAgent: Synced node info -46:23.594 [DEBUG] TestAgent: Node info in sync -46:23.597 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.614 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.614 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.614 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.614 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.614 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.614 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.614 [INFO] TestAgent.server: member joined, marking health alive: member=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.617 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.623 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.623 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.861 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.861 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-47a45d48-7193-3655-cd1e-f2c8499f7158 -46:23.861 [DEBUG] TestAgent: Check status updated: check=test status=warning -46:23.861 [DEBUG] TestAgent: Node info in sync -46:23.861 [WARN] TestAgent: Check registration blocked by ACLs: check=test accessorID=00000000-0000-0000-0000-000000000002 -46:23.861 [INFO] TestAgent: Requesting shutdown -46:23.861 [INFO] TestAgent.server: shutting down server -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.861 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.861 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.861 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.861 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.861 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.861 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.861 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.861 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.862 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.868 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.877Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.877Z [INFO] agent.router.manager: shutting down -46:23.877 [INFO] TestAgent: consul server down -46:23.877 [INFO] TestAgent: shutdown complete -46:23.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11171 network=tcp -46:23.877 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11171 network=udp -46:23.877 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11172 network=tcp -46:24.378 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.378 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_WarnCheck_ACLDeny (0.96s) - --- PASS: TestAgent_WarnCheck_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_WarnCheck_ACLDeny/root_token (0.00s) +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:03.759Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.791Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:03.866 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1829a01e-b86f-9be0-beb2-a6f07ab295f0 Address:127.0.0.1:29533}]" +32:03.867 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29533 [Follower]" leader= +32:03.867 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0.dc1 127.0.0.1 +32:03.868 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0 127.0.0.1 +2021-10-05T16:32:03.868Z [INFO] agent.router: Initializing LAN area manager +32:03.868 [INFO] TestAgent.server: Adding LAN server: server="Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0 (Addr: tcp/127.0.0.1:29533) (DC: dc1)" +32:03.868 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0.dc1 area=wan +32:03.868 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29528 network=udp +32:03.868 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29528 network=tcp +32:03.869 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29529 network=tcp +32:03.869 [INFO] TestAgent: started state syncer +32:03.869 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29555 network=tcp +32:03.913 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:03.913 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29533 [Candidate]" term=2 +32:03.928 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:03.928 [DEBUG] TestAgent.server.raft: vote granted: from=1829a01e-b86f-9be0-beb2-a6f07ab295f0 term=2 tally=1 +32:03.928 [INFO] TestAgent.server.raft: election won: tally=1 +32:03.928 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29533 [Leader]" +32:03.932 [INFO] TestAgent.server: cluster leadership acquired +32:03.932 [INFO] TestAgent.server: New leader elected: payload=Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0 +32:03.936 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29533 +32:03.973 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.032 [INFO] TestAgent: Synced node info +32:04.032 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.032 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.032 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.032 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.032 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.032 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0 +32:04.032 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1829a01e-b86f-9be0-beb2-a6f07ab295f0 +32:04.044 [INFO] TestAgent.server: federation state anti-entropy synced +32:04.056 [INFO] TestAgent: Requesting shutdown +32:04.056 [INFO] TestAgent.server: shutting down server +32:04.056 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.056 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.056 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.056 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.056 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.057 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.057 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.068 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.076Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.076Z [INFO] agent.router.manager: shutting down +32:04.076 [INFO] TestAgent: consul server down +32:04.076 [INFO] TestAgent: shutdown complete +32:04.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29528 network=tcp +32:04.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29528 network=udp +32:04.076 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29529 network=tcp +32:04.576 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.576 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ChecksWithFilter (0.85s) +=== CONT TestAgent_Services_Sidecar +2021-10-05T16:32:03.935Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.952Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.008 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3df6b201-5a33-f64d-f0ac-3bf28b897b9f Address:127.0.0.1:29568}]" +32:04.009 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29568 [Follower]" leader= +32:04.012 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f.dc1 127.0.0.1 +32:04.013 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f 127.0.0.1 +2021-10-05T16:32:04.014Z [INFO] agent.router: Initializing LAN area manager +32:04.014 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29563 network=udp +32:04.014 [INFO] TestAgent.server: Adding LAN server: server="Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f (Addr: tcp/127.0.0.1:29568) (DC: dc1)" +32:04.014 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f.dc1 area=wan +32:04.014 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29563 network=tcp +32:04.017 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29564 network=tcp +32:04.017 [INFO] TestAgent: started state syncer +32:04.017 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29548 network=tcp +32:04.075 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.075 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29568 [Candidate]" term=2 +32:04.085 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.085 [DEBUG] TestAgent.server.raft: vote granted: from=3df6b201-5a33-f64d-f0ac-3bf28b897b9f term=2 tally=1 +32:04.085 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.085 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29568 [Leader]" +32:04.085 [INFO] TestAgent.server: cluster leadership acquired +32:04.085 [INFO] TestAgent.server: New leader elected: payload=Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f +32:04.095 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29568 +32:04.116 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.169 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.169 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.169 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.169 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.169 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.169 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f +32:04.169 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3df6b201-5a33-f64d-f0ac-3bf28b897b9f +32:04.183 [INFO] TestAgent.server: federation state anti-entropy synced +32:04.216 [INFO] TestAgent: Requesting shutdown +32:04.216 [INFO] TestAgent.server: shutting down server +32:04.216 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.216 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.216 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.216 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.217 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.217 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.217 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.217 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:04.217 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.217 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.217 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.236 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.241Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.241Z [INFO] agent.router.manager: shutting down +32:04.242 [INFO] TestAgent: consul server down +32:04.242 [INFO] TestAgent: shutdown complete +32:04.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29563 network=tcp +32:04.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29563 network=udp +32:04.242 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29564 network=tcp +32:04.742 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.742 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Checks (0.83s) +=== CONT TestAgent_Services_ExternalConnectProxy +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +31:59.541 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 127.0.0.1 +2021-10-05T16:31:59.542Z [INFO] agent.router: Initializing LAN area manager +31:59.552 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29402 network=udp +31:59.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29402 network=tcp +31:59.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29403 network=tcp +31:59.554 [INFO] TestAgent: started state syncer +31:59.555 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30393 network=tcp +2021-10-05T16:31:59.555Z [WARN] agent.router.manager: No servers available +31:59.555 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" +31:59.555 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Stream connection from=127.0.0.1:41480 +31:59.582 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f 127.0.0.1 +31:59.583 [INFO] TestAgent.client: adding server: server="Node-6705c28c-b7e5-7057-b4ae-cea124813a6f (Addr: tcp/127.0.0.1:30378) (DC: dc1)" +31:59.583 [INFO] TestAgent.client: New leader elected: payload=Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.585 [INFO] TestAgent.client: client starting leave +31:59.761 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:59.761 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.761 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +31:59.960 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +31:59.960 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.960 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +32:00.159 [INFO] TestAgent.client.serf.lan: serf: EventMemberLeave: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 127.0.0.1 +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.159 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +32:00.823 [INFO] TestAgent: Synced node info +32:00.823 [DEBUG] TestAgent: Node info in sync +32:00.845 [DEBUG] TestAgent: Node info in sync +32:03.744 [INFO] TestAgent: Requesting shutdown +32:03.744 [INFO] TestAgent.client: shutting down client +32:03.752 [INFO] TestAgent: consul client down +32:03.752 [INFO] TestAgent: shutdown complete +32:03.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29402 network=tcp +32:03.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29402 network=udp +32:03.832 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29403 network=tcp +32:04.332 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.332 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:59.080Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.094Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.156 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6705c28c-b7e5-7057-b4ae-cea124813a6f Address:127.0.0.1:30378}]" +31:59.156 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30378 [Follower]" leader= +31:59.157 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f.dc1 127.0.0.1 +31:59.158 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f 127.0.0.1 +2021-10-05T16:31:59.158Z [INFO] agent.router: Initializing LAN area manager +31:59.158 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30373 network=udp +31:59.158 [INFO] TestAgent.server: Adding LAN server: server="Node-6705c28c-b7e5-7057-b4ae-cea124813a6f (Addr: tcp/127.0.0.1:30378) (DC: dc1)" +31:59.158 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6705c28c-b7e5-7057-b4ae-cea124813a6f.dc1 area=wan +31:59.159 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30373 network=tcp +31:59.159 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30374 network=tcp +31:59.159 [INFO] TestAgent: started state syncer +31:59.159 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30379 network=tcp +31:59.215 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.215 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30378 [Candidate]" term=2 +31:59.240 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.240 [DEBUG] TestAgent.server.raft: vote granted: from=6705c28c-b7e5-7057-b4ae-cea124813a6f term=2 tally=1 +31:59.240 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.240 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30378 [Leader]" +31:59.240 [INFO] TestAgent.server: cluster leadership acquired +31:59.240 [INFO] TestAgent.server: New leader elected: payload=Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.264 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30378 +31:59.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.360 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.360 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.360 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.360 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.360 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.360 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.360 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +31:59.380 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.505 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.513 [INFO] TestAgent: Synced node info +31:59.513 [DEBUG] TestAgent: Node info in sync +31:59.555 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:29405] +31:59.555 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:29405 +31:59.572 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.572 [DEBUG] TestAgent: Node info in sync +31:59.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 127.0.0.1 +31:59.585 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:59.585 [DEBUG] TestAgent: systemd notify failed: error="No socket" +31:59.585 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +31:59.742 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +31:59.944 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +31:59.944 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.144 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +32:00.144 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.159 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 +32:00.159 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.348 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-6705c28c-b7e5-7057-b4ae-cea124813a6f +32:00.348 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 127.0.0.1 +32:00.348 [INFO] TestAgent.server: deregistering member: member=Node-ec53038e-320b-6dcb-eb3f-58a03068e7d6 reason=left +32:04.332 [INFO] TestAgent: Requesting shutdown +32:04.332 [INFO] TestAgent.server: shutting down server +32:04.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.332 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.332 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.332 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.333 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.333 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.336 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.340Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.340Z [INFO] agent.router.manager: shutting down +32:04.340 [INFO] TestAgent: consul server down +32:04.340 [INFO] TestAgent: shutdown complete +32:04.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30373 network=tcp +32:04.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30373 network=udp +32:04.340 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30374 network=tcp +32:04.844 [INFO] TestAgent: Waiting for endpoints to shut down +32:04.844 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Leave (5.78s) +=== CONT TestAgent_ServicesFiltered +2021-10-05T16:32:03.978Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:03.978Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:03.995Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:03.995Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.055 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b4dea1e-3765-371b-2554-49efacbd6e93 Address:127.0.0.1:29554}]" +32:04.056 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29554 [Follower]" leader= +32:04.056 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b4dea1e-3765-371b-2554-49efacbd6e93.dc1 127.0.0.1 +32:04.057 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b4dea1e-3765-371b-2554-49efacbd6e93 127.0.0.1 +2021-10-05T16:32:04.057Z [INFO] agent.router: Initializing LAN area manager +32:04.058 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29549 network=udp +32:04.058 [INFO] TestAgent.server: Adding LAN server: server="Node-4b4dea1e-3765-371b-2554-49efacbd6e93 (Addr: tcp/127.0.0.1:29554) (DC: dc1)" +32:04.058 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b4dea1e-3765-371b-2554-49efacbd6e93.dc1 area=wan +32:04.058 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29549 network=tcp +32:04.059 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29550 network=tcp +32:04.059 [INFO] TestAgent: started state syncer +32:04.059 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29569 network=tcp +32:04.108 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:04.116 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.116 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29554 [Candidate]" term=2 +32:04.143 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.143 [DEBUG] TestAgent.server.raft: vote granted: from=4b4dea1e-3765-371b-2554-49efacbd6e93 term=2 tally=1 +32:04.143 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.143 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29554 [Leader]" +32:04.143 [INFO] TestAgent.server: cluster leadership acquired +32:04.143 [INFO] TestAgent.server: New leader elected: payload=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.151 [INFO] TestAgent.server: initializing acls +32:04.156 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:04.156 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:04.162 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:04.172 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:04.172 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:04.172 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:04.173 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.173 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4b4dea1e-3765-371b-2554-49efacbd6e93.dc1 +32:04.173 [INFO] TestAgent.server: Updating LAN server: server="Node-4b4dea1e-3765-371b-2554-49efacbd6e93 (Addr: tcp/127.0.0.1:29554) (DC: dc1)" +32:04.173 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4b4dea1e-3765-371b-2554-49efacbd6e93.dc1 area=wan +32:04.204 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.247 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.247 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.247 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.247 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.247 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.266 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.284 [INFO] TestAgent.server: federation state anti-entropy synced +32:04.460 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:04.476 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.476 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4b4dea1e-3765-371b-2554-49efacbd6e93 +32:04.480 [INFO] TestAgent: Synced node info +32:04.480 [DEBUG] TestAgent: Node info in sync +32:04.480 [DEBUG] TestAgent: dropping service from result due to ACLs: service=mysql +32:04.480 [INFO] TestAgent: Requesting shutdown +32:04.480 [INFO] TestAgent.server: shutting down server +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.480 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.480 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.480 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.480 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.481 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:04.481 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:04.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.481 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.481 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.489 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.504Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.504Z [INFO] agent.router.manager: shutting down +32:04.504 [INFO] TestAgent: consul server down +32:04.504 [INFO] TestAgent: shutdown complete +32:04.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29549 network=tcp +32:04.504 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29549 network=udp +32:04.504 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29550 network=tcp +32:05.005 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.005 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_ACLFilter (1.05s) + --- PASS: TestAgent_Services_ACLFilter/no_token (0.00s) + --- PASS: TestAgent_Services_ACLFilter/root_token (0.00s) +=== CONT TestAgent_Services +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:04.216Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:04.239Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b2ae965c-63ed-6775-4934-589c37452e54 Address:127.0.0.1:29589}]" +32:04.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b2ae965c-63ed-6775-4934-589c37452e54.dc1 127.0.0.1 +32:04.325 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29589 [Follower]" leader= +32:04.325 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b2ae965c-63ed-6775-4934-589c37452e54 127.0.0.1 +2021-10-05T16:32:04.325Z [INFO] agent.router: Initializing LAN area manager +32:04.325 [INFO] TestAgent.server: Adding LAN server: server="Node-b2ae965c-63ed-6775-4934-589c37452e54 (Addr: tcp/127.0.0.1:29589) (DC: dc1)" +32:04.325 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b2ae965c-63ed-6775-4934-589c37452e54.dc1 area=wan +32:04.325 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29556 network=udp +32:04.326 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29556 network=tcp +32:04.326 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29557 network=tcp +32:04.326 [INFO] TestAgent: started state syncer +32:04.326 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29520 network=tcp +32:04.393 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.393 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29589 [Candidate]" term=2 +32:04.411 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.411 [DEBUG] TestAgent.server.raft: vote granted: from=b2ae965c-63ed-6775-4934-589c37452e54 term=2 tally=1 +32:04.411 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.411 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29589 [Leader]" +32:04.411 [INFO] TestAgent.server: cluster leadership acquired +32:04.411 [INFO] TestAgent.server: New leader elected: payload=Node-b2ae965c-63ed-6775-4934-589c37452e54 +32:04.420 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29589 +32:04.446 [INFO] TestAgent: Synced node info +32:04.446 [DEBUG] TestAgent: Node info in sync +32:04.466 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.521 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.521 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.521 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.521 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.521 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.521 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b2ae965c-63ed-6775-4934-589c37452e54 +32:04.521 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b2ae965c-63ed-6775-4934-589c37452e54 +32:04.552 [INFO] TestAgent.server: federation state anti-entropy synced +32:04.611 [INFO] TestAgent: Requesting shutdown +32:04.611 [INFO] TestAgent.server: shutting down server +32:04.611 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.611 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.611 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.611 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.611 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.611 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.611 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.611 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.611 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.611 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.620 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.626Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.626Z [INFO] agent.router.manager: shutting down +32:04.626 [INFO] TestAgent: consul server down +32:04.626 [INFO] TestAgent: shutdown complete +32:04.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29556 network=tcp +32:04.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29556 network=udp +32:04.626 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29557 network=tcp +32:05.128 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.128 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_TerminatingGateway (0.93s) +=== CONT TestACL_ResolveIdentity +2021-10-05T16:32:05.149Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.149Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_ResolveIdentity (0.02s) +=== CONT TestACLEndpoint_LoginLogout_jwt +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:59.248Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.261Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.311 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9ca824e-8c15-6620-55bf-2184b91721fb Address:127.0.0.1:29477}]" +31:59.312 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9ca824e-8c15-6620-55bf-2184b91721fb.dc1 127.0.0.1 +31:59.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9ca824e-8c15-6620-55bf-2184b91721fb 127.0.0.1 +2021-10-05T16:31:59.312Z [INFO] agent.router: Initializing LAN area manager +31:59.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30415 network=udp +31:59.313 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29477 [Follower]" leader= +31:59.313 [INFO] TestAgent.server: Adding LAN server: server="Node-c9ca824e-8c15-6620-55bf-2184b91721fb (Addr: tcp/127.0.0.1:29477) (DC: dc1)" +31:59.313 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb.dc1 area=wan +31:59.313 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30415 network=tcp +31:59.314 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30416 network=tcp +31:59.314 [INFO] TestAgent: started state syncer +31:59.314 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30407 network=tcp +31:59.354 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.354 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29477 [Candidate]" term=2 +31:59.403 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.403 [DEBUG] TestAgent.server.raft: vote granted: from=c9ca824e-8c15-6620-55bf-2184b91721fb term=2 tally=1 +31:59.403 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.403 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29477 [Leader]" +31:59.404 [INFO] TestAgent.server: cluster leadership acquired +31:59.404 [INFO] TestAgent.server: New leader elected: payload=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +31:59.415 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29477 +31:59.456 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.528 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.528 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.528 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.528 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.528 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.529 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +31:59.529 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +31:59.568 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.585 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:33324 +31:59.586 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a 127.0.0.1 +31:59.587 [INFO] TestAgent.server: Adding LAN server: server="Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a (Addr: tcp/127.0.0.1:30371) (DC: dc1)" +31:59.587 [INFO] TestAgent.server: New leader elected: payload=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:59.587 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a other=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +31:59.587 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:59.588 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 127.0.0.1:30370 +31:59.589 [INFO] TestAgent: Requesting shutdown +31:59.589 [INFO] TestAgent.server: shutting down server +31:59.589 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:59.589 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:59.589 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:59.589 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:59.589 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:59.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 127.0.0.1 +31:59.589 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 address=127.0.0.1:30370 +31:59.589 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 area=wan +31:59.592 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:59.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:59.592 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:59.592 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:59.592 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:59.592 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:59.616 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:59.622Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:59.622Z [INFO] agent.router.manager: shutting down +31:59.622 [INFO] TestAgent: consul server down +31:59.622 [INFO] TestAgent: shutdown complete +31:59.622 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30415 network=tcp +31:59.622 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30415 network=udp +31:59.622 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30416 network=tcp +32:00.124 [INFO] TestAgent: Waiting for endpoints to shut down +32:00.124 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:58.716Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.752Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.822 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b60fa6ee-8db1-f548-5f43-eb11fb71f15a Address:127.0.0.1:30371}]" +31:58.822 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30371 [Follower]" leader= +31:58.822 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 127.0.0.1 +31:58.823 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a 127.0.0.1 +2021-10-05T16:31:58.823Z [INFO] agent.router: Initializing LAN area manager +31:58.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30366 network=udp +31:58.824 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a.dc1 area=wan +31:58.824 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30366 network=tcp +31:58.824 [INFO] TestAgent.server: Adding LAN server: server="Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a (Addr: tcp/127.0.0.1:30371) (DC: dc1)" +31:58.824 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30367 network=tcp +31:58.824 [INFO] TestAgent: started state syncer +31:58.824 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30323 network=tcp +31:58.862 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.862 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30371 [Candidate]" term=2 +31:58.900 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.900 [DEBUG] TestAgent.server.raft: vote granted: from=b60fa6ee-8db1-f548-5f43-eb11fb71f15a term=2 tally=1 +31:58.900 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.900 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30371 [Leader]" +31:58.900 [INFO] TestAgent.server: cluster leadership acquired +31:58.900 [INFO] TestAgent.server: New leader elected: payload=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:58.933 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30371 +31:58.964 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.026 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.026 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.026 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.026 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.026 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.026 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:59.026 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:59.040 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.221 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.235 [INFO] TestAgent: Synced node info +31:59.585 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:30418] +31:59.585 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.585 [DEBUG] TestAgent: Node info in sync +31:59.585 [DEBUG] TestAgent: Node info in sync +31:59.585 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30418 +31:59.588 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9ca824e-8c15-6620-55bf-2184b91721fb 127.0.0.1 +31:59.588 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:59.589 [DEBUG] TestAgent: systemd notify failed: error="No socket" +31:59.589 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:60968 +31:59.589 [INFO] TestAgent.server: Adding LAN server: server="Node-c9ca824e-8c15-6620-55bf-2184b91721fb (Addr: tcp/127.0.0.1:29477) (DC: dc1)" +31:59.589 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-c9ca824e-8c15-6620-55bf-2184b91721fb other=Node-b60fa6ee-8db1-f548-5f43-eb11fb71f15a +31:59.589 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +31:59.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9ca824e-8c15-6620-55bf-2184b91721fb.dc1 127.0.0.1 +31:59.589 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb.dc1 area=wan +32:00.324 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-c9ca824e-8c15-6620-55bf-2184b91721fb (timeout reached) +32:00.825 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-c9ca824e-8c15-6620-55bf-2184b91721fb has failed, no acks received +32:00.941 [WARN] TestAgent: error getting server health from server: server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:29477: connect: connection refused" +32:01.940 [WARN] TestAgent: error getting server health from server: server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb error="context deadline exceeded" +32:02.324 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-c9ca824e-8c15-6620-55bf-2184b91721fb (timeout reached) +32:02.944 [WARN] TestAgent: error getting server health from server: server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:29477: connect: connection refused" +32:03.824 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-c9ca824e-8c15-6620-55bf-2184b91721fb has failed, no acks received +32:03.944 [WARN] TestAgent: error getting server health from server: server=Node-c9ca824e-8c15-6620-55bf-2184b91721fb error="context deadline exceeded" +32:04.826 [INFO] TestAgent.server.memberlist.lan: memberlist: Marking Node-c9ca824e-8c15-6620-55bf-2184b91721fb as failed, suspect timeout reached (0 peer confirmations) +32:04.826 [INFO] TestAgent.server.serf.lan: serf: EventMemberFailed: Node-c9ca824e-8c15-6620-55bf-2184b91721fb 127.0.0.1 +32:04.826 [INFO] TestAgent.server: Removing LAN server: server="Node-c9ca824e-8c15-6620-55bf-2184b91721fb (Addr: tcp/127.0.0.1:29477) (DC: dc1)" +32:04.826 [INFO] TestAgent.server: member failed, marking health critical: member=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +32:04.839 [INFO] TestAgent: Force leaving node: node=Node-c9ca824e-8c15-6620-55bf-2184b91721fb +32:04.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave (forced): Node-c9ca824e-8c15-6620-55bf-2184b91721fb 127.0.0.1 +32:04.839 [INFO] TestAgent.server: Removing LAN server: server="Node-c9ca824e-8c15-6620-55bf-2184b91721fb (Addr: tcp/127.0.0.1:29477) (DC: dc1)" +32:04.839 [INFO] TestAgent.server: deregistering member: member=Node-c9ca824e-8c15-6620-55bf-2184b91721fb reason=left +32:04.864 [INFO] TestAgent: Requesting shutdown +32:04.865 [INFO] TestAgent.server: shutting down server +32:04.865 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.865 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.865 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.865 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.865 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.865 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.865 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.865 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.865 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.865 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.869 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.878Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.878Z [INFO] agent.router.manager: shutting down +32:04.878 [INFO] TestAgent: consul server down +32:04.878 [INFO] TestAgent: shutdown complete +32:04.878 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30366 network=tcp +32:04.878 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30366 network=udp +32:04.878 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30367 network=tcp +32:05.324 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-c9ca824e-8c15-6620-55bf-2184b91721fb (timeout reached) +32:05.380 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.380 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ForceLeave (6.70s) +=== CONT TestACL_filterServices +2021-10-05T16:32:04.412Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:04.439Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.516 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:32c2aa7d-6e57-7c10-2545-881e7462a3e4 Address:127.0.0.1:29610}]" +32:04.517 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4.dc1 127.0.0.1 +32:04.517 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4 127.0.0.1 +2021-10-05T16:32:04.517Z [INFO] agent.router: Initializing LAN area manager +32:04.518 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29605 network=udp +32:04.518 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29610 [Follower]" leader= +32:04.518 [INFO] TestAgent.server: Adding LAN server: server="Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4 (Addr: tcp/127.0.0.1:29610) (DC: dc1)" +32:04.518 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4.dc1 area=wan +32:04.518 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29605 network=tcp +32:04.519 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29606 network=tcp +32:04.519 [INFO] TestAgent: started state syncer +32:04.519 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29562 network=tcp +32:04.584 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.584 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29610 [Candidate]" term=2 +32:04.606 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.606 [DEBUG] TestAgent.server.raft: vote granted: from=32c2aa7d-6e57-7c10-2545-881e7462a3e4 term=2 tally=1 +32:04.606 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.606 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29610 [Leader]" +32:04.607 [INFO] TestAgent.server: cluster leadership acquired +32:04.607 [INFO] TestAgent.server: New leader elected: payload=Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4 +32:04.633 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29610 +32:04.655 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.688 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.688 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.688 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.688 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.688 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.688 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4 +32:04.688 [INFO] TestAgent.server: member joined, marking health alive: member=Node-32c2aa7d-6e57-7c10-2545-881e7462a3e4 +32:04.707 [INFO] TestAgent.server: federation state anti-entropy synced +32:04.869 [INFO] TestAgent: Requesting shutdown +32:04.869 [INFO] TestAgent.server: shutting down server +32:04.869 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:04.869 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:04.869 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:04.869 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:04.869 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:04.869 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:04.869 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:04.869 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:04.869 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:04.869 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:04.869 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:04.874 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:04.890Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:04.890Z [INFO] agent.router.manager: shutting down +32:04.890 [INFO] TestAgent: consul server down +32:04.890 [INFO] TestAgent: shutdown complete +32:04.890 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29605 network=tcp +32:04.890 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29605 network=udp +32:04.890 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29606 network=tcp +32:05.396 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.396 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_MeshGateway (1.03s) +=== CONT TestACL_filterMembers +2021-10-05T16:32:05.404Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.404Z [WARN] agent: bootstrap = true: do not enable unless necessary +32:05.404 [DEBUG] TestACL_filterServices: dropping service from result due to ACLs: service=my-other +--- PASS: TestACL_filterServices (0.02s) +=== CONT TestACL_vetCheckUpdate +2021-10-05T16:32:05.431Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.431Z [WARN] agent: bootstrap = true: do not enable unless necessary +32:05.431 [DEBUG] TestACL_filterMembers: dropping node from result due to ACLs: node=Nope accessorID=9df2d1a4-2d07-414e-8ead-6053f56ed2eb +--- PASS: TestACL_filterMembers (0.03s) +=== CONT TestACL_vetCheckRegister +2021-10-05T16:32:05.434Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.434Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_vetCheckUpdate (0.03s) +=== CONT TestACL_vetServiceUpdate +2021-10-05T16:32:05.471Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.471Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_vetServiceUpdate (0.04s) +=== CONT TestACL_vetServiceRegister +2021-10-05T16:32:05.490Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.490Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_vetCheckRegister (0.06s) +=== CONT TestACL_RootAuthorizersDenied +2021-10-05T16:32:05.518Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.518Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_vetServiceRegister (0.04s) === CONT TestACL_AgentMasterToken -2022-11-07T22:46:24.387Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.387Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_AgentMasterToken (0.01s) +2021-10-05T16:32:05.534Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.535Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_RootAuthorizersDenied (0.04s) === CONT TestACL_Version8EnabledByDefault -2022-11-07T22:46:24.395Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.395Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_Version8EnabledByDefault (0.01s) +2021-10-05T16:32:04.601Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:04.629Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.716 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:647aad21-ca57-dc22-13a3-e7243e81c60e Address:127.0.0.1:29582}]" +32:04.716 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29582 [Follower]" leader= +32:04.717 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-647aad21-ca57-dc22-13a3-e7243e81c60e.dc1 127.0.0.1 +32:04.717 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-647aad21-ca57-dc22-13a3-e7243e81c60e 127.0.0.1 +2021-10-05T16:32:04.717Z [INFO] agent.router: Initializing LAN area manager +32:04.717 [INFO] TestAgent.server: Adding LAN server: server="Node-647aad21-ca57-dc22-13a3-e7243e81c60e (Addr: tcp/127.0.0.1:29582) (DC: dc1)" +32:04.718 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-647aad21-ca57-dc22-13a3-e7243e81c60e.dc1 area=wan +32:04.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29584 network=udp +32:04.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29584 network=tcp +32:04.718 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29585 network=tcp +32:04.718 [INFO] TestAgent: started state syncer +32:04.718 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29625 network=tcp +32:04.774 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.774 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29582 [Candidate]" term=2 +32:04.787 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.787 [DEBUG] TestAgent.server.raft: vote granted: from=647aad21-ca57-dc22-13a3-e7243e81c60e term=2 tally=1 +32:04.787 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.787 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29582 [Leader]" +32:04.787 [INFO] TestAgent.server: cluster leadership acquired +32:04.787 [INFO] TestAgent.server: New leader elected: payload=Node-647aad21-ca57-dc22-13a3-e7243e81c60e +32:04.795 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29582 +32:04.816 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:04.844 [INFO] TestAgent: Synced node info +32:04.854 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:04.854 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:04.854 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:04.854 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:04.854 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:04.855 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-647aad21-ca57-dc22-13a3-e7243e81c60e +32:04.855 [INFO] TestAgent.server: member joined, marking health alive: member=Node-647aad21-ca57-dc22-13a3-e7243e81c60e +32:04.867 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.025 [INFO] TestAgent: Requesting shutdown +32:05.025 [INFO] TestAgent.server: shutting down server +32:05.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:05.025 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:05.025 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:05.025 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:05.025 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:05.025 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:05.025 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:05.026 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:05.026 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:05.026 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:05.031 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:05.038Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:05.038Z [INFO] agent.router.manager: shutting down +32:05.038 [INFO] TestAgent: consul server down +32:05.038 [INFO] TestAgent: shutdown complete +32:05.038 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29584 network=tcp +32:05.038 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29584 network=udp +32:05.038 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29585 network=tcp +32:05.540 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.540 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_Sidecar (0.96s) === CONT TestACL_Authorize -=== RUN TestAgent_HealthServicesACLEnforcement/no-token-health-by-id -=== RUN TestAgent_HealthServicesACLEnforcement/no-token-health-by-name -=== RUN TestAgent_HealthServicesACLEnforcement/root-token-health-by-id -=== RUN TestAgent_HealthServicesACLEnforcement/root-token-health-by-name -2022-11-07T22:46:23.753Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.753Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.763Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.763Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.789 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f Address:127.0.0.1:11337}]" -46:23.789 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11337 [Follower]" leader= -46:23.789 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 127.0.0.1 -46:23.790 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f 127.0.0.1 -2022-11-07T22:46:23.790Z [INFO] agent.router: Initializing LAN area manager -46:23.790 [INFO] TestAgent.server: Adding LAN server: server="Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f (Addr: tcp/127.0.0.1:11337) (DC: dc1)" -46:23.790 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 area=wan -46:23.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11304 network=tcp -46:23.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11304 network=udp -46:23.790 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11305 network=tcp -46:23.790 [INFO] TestAgent: started state syncer -46:23.790 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11338 network=tcp -46:23.829 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.829 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11337 [Candidate]" term=2 -46:23.840 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.840 [DEBUG] TestAgent.server.raft: vote granted: from=cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f term=2 tally=1 -46:23.840 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.840 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11337 [Leader]" -46:23.840 [INFO] TestAgent.server: cluster leadership acquired -46:23.840 [INFO] TestAgent.server: initializing acls -46:23.840 [INFO] TestAgent.server: New leader elected: payload=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.847 [INFO] TestAgent.server: initializing acls -46:23.847 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.847 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:23.847 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:23.867 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.868 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:23.876 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.876 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:23.876 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:23.876 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:23.876 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:23.876 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.876 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 -46:23.876 [INFO] TestAgent.server: Updating LAN server: server="Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f (Addr: tcp/127.0.0.1:11337) (DC: dc1)" -46:23.876 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 area=wan -46:23.876 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.876 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 -46:23.876 [INFO] TestAgent.server: Updating LAN server: server="Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f (Addr: tcp/127.0.0.1:11337) (DC: dc1)" -46:23.876 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f.dc1 area=wan -46:23.886 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.904 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.904 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.904 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.904 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.904 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.904 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.904 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.914 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.914 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.914 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cee0c86c-b9c7-6c60-c37f-c103b6b2bc6f -46:23.939 [DEBUG] TestAgent: removed check: check=test -46:23.944 [INFO] TestAgent: Synced node info -46:23.944 [INFO] TestAgent: Deregistered check: check=test -46:23.944 [INFO] TestAgent: Requesting shutdown -46:23.944 [INFO] TestAgent.server: shutting down server -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:23.944 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:23.944 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.944 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.944 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:23.944 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:23.949 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.955Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.955Z [INFO] agent.router.manager: shutting down -46:23.955 [INFO] TestAgent: consul server down -46:23.955 [INFO] TestAgent: shutdown complete -46:23.955 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11304 network=tcp -46:23.955 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11304 network=udp -46:23.955 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11305 network=tcp -46:24.455 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.455 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_DeregisterCheckACLDeny (0.71s) - --- PASS: TestAgent_DeregisterCheckACLDeny/no_token (0.00s) - --- PASS: TestAgent_DeregisterCheckACLDeny/root_token (0.00s) -=== CONT TestACL_vetServiceUpdate -2022-11-07T22:46:24.464Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.464Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_vetServiceUpdate (0.01s) -=== CONT TestACLEndpoint_LoginLogout_jwt -2022-11-07T22:46:17.577Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.589Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.589 [INFO] bonnie.server.gateway_locator: will dial the primary datacenter through its mesh gateways -46:17.620 [INFO] bonnie.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:465aaa0a-6b61-e0bd-8cca-7c64ad70a4b0 Address:127.0.0.1:11274}]" -46:17.620 [INFO] bonnie.server.raft: entering follower state: follower="Node at 127.0.0.1:11274 [Follower]" leader= -46:17.620 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 -46:17.621 [INFO] bonnie.server.serf.lan: serf: EventMemberJoin: bonnie 127.0.0.1 -2022-11-07T22:46:17.621Z [INFO] agent.router: Initializing LAN area manager -46:17.621 [INFO] bonnie.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan -46:17.621 [INFO] bonnie.server: Adding LAN server: server="bonnie (Addr: tcp/127.0.0.1:11274) (DC: dc3)" -46:17.621 [INFO] bonnie: Started DNS server: address=127.0.0.1:11492 network=tcp -46:17.621 [INFO] bonnie: Started DNS server: address=127.0.0.1:11492 network=udp -46:17.622 [INFO] bonnie: Started HTTP server: address=127.0.0.1:11458 network=tcp -46:17.622 [INFO] bonnie: started state syncer -46:17.622 [INFO] bonnie: Started gRPC server: address=127.0.0.1:11539 network=tcp -46:17.622 [INFO] bonnie: Refreshing mesh gateways is supported for the following discovery methods: discovery_methods="aliyun aws gce mdns os packet" -46:17.622 [INFO] bonnie: Refreshing mesh gateways... -46:17.622 [INFO] bonnie.server.gateway_locator: updated fallback list of primary mesh gateways: mesh_gateways=[127.0.0.1:11230] -46:17.622 [INFO] bonnie: Refreshing mesh gateways completed -46:17.622 [INFO] bonnie: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" -46:17.622 [INFO] bonnie: Joining cluster...: cluster=WAN -46:17.622 [INFO] bonnie: (WAN) joining: wan_addresses=[*.dc1/192.0.2.2] -46:17.652 [DEBUG] bonnie.server.memberlist.wan: memberlist: Initiating push/pull sync with: *.dc1 127.0.0.1:11230 -46:17.653 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 -46:17.654 [INFO] bonnie.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 -46:17.654 [INFO] bonnie: (WAN) joined: number_of_nodes=1 -46:17.654 [INFO] bonnie: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 -46:17.654 [INFO] bonnie.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan -46:17.654 [INFO] bonnie.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan -46:17.665 [WARN] bonnie.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.665 [INFO] bonnie.server.raft: entering candidate state: node="Node at 127.0.0.1:11274 [Candidate]" term=2 -46:17.676 [DEBUG] bonnie.server.raft: votes: needed=1 -46:17.676 [DEBUG] bonnie.server.raft: vote granted: from=465aaa0a-6b61-e0bd-8cca-7c64ad70a4b0 term=2 tally=1 -46:17.676 [INFO] bonnie.server.raft: election won: tally=1 -46:17.676 [INFO] bonnie.server.raft: entering leader state: leader="Node at 127.0.0.1:11274 [Leader]" -46:17.676 [INFO] bonnie.server: cluster leadership acquired -46:17.676 [INFO] bonnie.server: New leader elected: payload=bonnie -46:17.683 [DEBUG] bonnie.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true -46:17.730 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false -46:17.755 [INFO] bonnie.server.connect: received new intermediate certificate from primary datacenter -46:17.762 [INFO] bonnie.server.connect: updated root certificates from primary datacenter -46:17.762 [INFO] bonnie.server.connect: initialized secondary datacenter CA with provider: provider=consul -46:17.762 [INFO] bonnie.leader: started routine: routine="config entry replication" -46:17.762 [INFO] bonnie.leader: started routine: routine="federation state replication" -46:17.762 [INFO] bonnie.leader: started routine: routine="federation state anti-entropy" -46:17.762 [INFO] bonnie.leader: started routine: routine="secondary CA roots watch" -46:17.762 [INFO] bonnie.leader: started routine: routine="intention replication" -46:17.762 [INFO] bonnie.leader: started routine: routine="intermediate cert renew watch" -46:17.762 [INFO] bonnie.leader: started routine: routine="CA root pruning" -46:17.762 [DEBUG] bonnie.server: Skipping self join check for node since the cluster is too small: node=bonnie -46:17.762 [INFO] bonnie.server: member joined, marking health alive: member=bonnie -46:17.762 [DEBUG] bonnie.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 -46:17.762 [DEBUG] bonnie.server.replication.config_entry: finished fetching config entries: amount=0 -46:17.762 [DEBUG] bonnie.server.replication.config_entry: Config Entry replication: local=0 remote=0 -46:17.762 [DEBUG] bonnie.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 -46:17.762 [DEBUG] bonnie.server.replication.config_entry: replication completed through remote index: index=1 -46:17.762 [DEBUG] bonnie.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 -46:17.763 [INFO] bonnie.server.gateway_locator: will dial the primary datacenter using our local mesh gateways if possible -46:17.763 [DEBUG] bonnie.server: finished fetching remote objects: amount=2 -46:17.763 [DEBUG] bonnie.server: diffing replication state: local_amount=0 remote_amount=2 -46:17.763 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=2 -46:17.763 [DEBUG] bonnie.server: performing updates: updates=2 -46:17.774 [INFO] bonnie.server.connect: updated root certificates from primary datacenter -46:17.779 [INFO] bonnie.server: federation state anti-entropy synced -46:17.779 [DEBUG] bonnie.server: finished updates -46:17.779 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=17 -46:17.780 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 -46:17.780 [DEBUG] bonnie.server: diffing replication state: local_amount=2 remote_amount=3 -46:17.780 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 -46:17.780 [DEBUG] bonnie.server: performing updates: updates=1 -46:17.785 [INFO] bonnie.server: federation state anti-entropy synced -46:17.789 [DEBUG] bonnie.server: finished updates -46:17.789 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=19 -46:17.790 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 -46:17.790 [DEBUG] bonnie.server: diffing replication state: local_amount=3 remote_amount=3 -46:17.790 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 -46:17.790 [DEBUG] bonnie.server: performing updates: updates=1 -46:17.796 [INFO] bonnie.server: federation state anti-entropy synced -46:17.801 [DEBUG] bonnie.server: finished updates -46:17.801 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=20 -46:17.802 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 -46:17.802 [DEBUG] bonnie.server: diffing replication state: local_amount=3 remote_amount=3 -46:17.802 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 -46:17.802 [DEBUG] bonnie.server: performing updates: updates=1 -46:17.807 [DEBUG] bonnie.server: finished updates -46:17.807 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=21 -46:17.863 [DEBUG] bonnie: Skipping remote check since it is managed automatically: check=serfHealth -46:17.866 [INFO] bonnie: Synced node info -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.877 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:17.881 [DEBUG] bonnie: Node info in sync -46:17.887 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 -46:17.887 [DEBUG] bonnie.server: diffing replication state: local_amount=3 remote_amount=3 -46:17.887 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 -46:17.887 [DEBUG] bonnie.server: performing updates: updates=1 -46:17.887 [INFO] bonnie: Synced service: service=mesh-gateway -46:17.894 [DEBUG] bonnie.server: finished updates -46:17.894 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=22 -46:18.150 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11398: Remote DC has no server currently reachable -46:18.315 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.315 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.621 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11398: Remote DC has no server currently reachable -46:18.819 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.819 [DEBUG] bonnie.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:19.122 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11398: Remote DC has no server currently reachable -46:19.413 [DEBUG] bonnie: Skipping remote check since it is managed automatically: check=serfHealth -46:19.413 [DEBUG] bonnie: Node info in sync -46:19.413 [DEBUG] bonnie: Service in sync: service=mesh-gateway -46:19.413 [DEBUG] bonnie: Node info in sync -46:19.414 [DEBUG] bonnie: Service in sync: service=mesh-gateway -46:19.621 [INFO] bonnie.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:11230] local=[] -46:20.075 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.505 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.545 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.626 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.626 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.627 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.627 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11231 datacenter=dc2 method=Internal.ServiceDump error="rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:20.627 [ERROR] bonnie.proxycfg: watch error: service_id=mesh-gateway id=mesh-gateway:dc2 error="error filling agent cache: rpc error getting client: failed to get conn: Remote DC has no server currently reachable" -46:22.621 [ERROR] bonnie.server.memberlist.wan: memberlist: Failed to send ping: Remote DC has no server currently reachable -46:22.768 [INFO] bonnie.server: federation state anti-entropy synced -46:22.768 [DEBUG] bonnie.server: finished fetching remote objects: amount=3 -46:22.768 [DEBUG] bonnie.server: diffing replication state: local_amount=3 remote_amount=3 -46:22.768 [DEBUG] bonnie.server: diffed replication state: deletions=0 updates=1 -46:22.768 [DEBUG] bonnie.server: performing updates: updates=1 -46:22.773 [DEBUG] bonnie.server: finished updates -46:22.773 [DEBUG] bonnie.replication.federation_state: replication completed through remote index: index=23 -46:22.773 [INFO] bonnie.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:11230] local=[127.0.0.1:11230] -46:22.929 [INFO] bonnie: Requesting shutdown -46:22.930 [INFO] bonnie.server: shutting down server -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="federation state anti-entropy" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="secondary CA roots watch" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="intention replication" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="intermediate cert renew watch" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="CA root pruning" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="config entry replication" -46:22.930 [DEBUG] bonnie.leader: stopping routine: routine="federation state replication" -46:22.930 [WARN] bonnie.server.serf.lan: serf: Shutdown without a Leave -46:22.930 [DEBUG] bonnie.leader: stopped routine: routine="CA root pruning" -46:22.930 [DEBUG] bonnie.leader: stopped routine: routine="federation state anti-entropy" -46:22.930 [DEBUG] bonnie.leader: stopped routine: routine="intermediate cert renew watch" -46:22.936 [WARN] bonnie.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.941Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.941Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.941Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.941Z [INFO] agent.router.manager: shutting down -46:22.942 [INFO] bonnie: consul server down -46:22.942 [INFO] bonnie: shutdown complete -46:22.942 [INFO] bonnie: Stopping server: protocol=DNS address=127.0.0.1:11492 network=tcp -46:22.942 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=Internal.ServiceDump error="rpc error making call: EOF" -46:22.942 [INFO] bonnie: Stopping server: protocol=DNS address=127.0.0.1:11492 network=udp -46:22.942 [INFO] bonnie: Stopping server: protocol=HTTP address=127.0.0.1:11458 network=tcp -46:22.942 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" -46:22.942 [INFO] bonnie.server.replication.config_entry: stopped replication -46:22.942 [DEBUG] bonnie.leader: stopped routine: routine="config entry replication" -46:22.942 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" -46:22.942 [ERROR] bonnie.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" -46:22.942 [DEBUG] bonnie.leader: stopped routine: routine="secondary CA roots watch" -46:22.942 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" -46:22.942 [ERROR] bonnie.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" -46:22.942 [ERROR] bonnie.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" -46:22.942 [DEBUG] bonnie.leader: stopped routine: routine="intention replication" -46:22.942 [WARN] bonnie.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" -46:22.942 [INFO] bonnie.replication.federation_state: stopped replication -46:22.942 [DEBUG] bonnie.leader: stopped routine: routine="federation state replication" -46:23.442 [INFO] bonnie: Waiting for endpoints to shut down -46:23.442 [INFO] bonnie: Endpoints down -2022-11-07T22:46:17.348Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:17.358Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:17.359 [INFO] betty.server.gateway_locator: will dial the primary datacenter through its mesh gateways -46:17.395 [INFO] betty.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca86d984-f11b-3d78-1764-19adaef5fda5 Address:127.0.0.1:11231}]" -46:17.395 [INFO] betty.server.raft: entering follower state: follower="Node at 127.0.0.1:11231 [Follower]" leader= -46:17.396 [INFO] betty.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 -46:17.396 [INFO] betty.server.serf.lan: serf: EventMemberJoin: betty 127.0.0.1 -2022-11-07T22:46:17.396Z [INFO] agent.router: Initializing LAN area manager -46:17.397 [INFO] betty: Started DNS server: address=127.0.0.1:11415 network=udp -46:17.397 [INFO] betty.server: Adding LAN server: server="betty (Addr: tcp/127.0.0.1:11231) (DC: dc2)" -46:17.397 [INFO] betty.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan -46:17.397 [INFO] betty: Started DNS server: address=127.0.0.1:11415 network=tcp -46:17.398 [INFO] betty: Started HTTP server: address=127.0.0.1:11395 network=tcp -46:17.398 [INFO] betty: started state syncer -46:17.398 [INFO] betty: Started gRPC server: address=127.0.0.1:11532 network=tcp -46:17.398 [INFO] betty: Refreshing mesh gateways is supported for the following discovery methods: discovery_methods="aliyun aws gce mdns os packet" -46:17.398 [INFO] betty: Refreshing mesh gateways... -46:17.398 [INFO] betty.server.gateway_locator: updated fallback list of primary mesh gateways: mesh_gateways=[127.0.0.1:11230] -46:17.398 [INFO] betty: Refreshing mesh gateways completed -46:17.398 [INFO] betty: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" -46:17.398 [INFO] betty: Joining cluster...: cluster=WAN -46:17.398 [INFO] betty: (WAN) joining: wan_addresses=[*.dc1/192.0.2.2] -46:17.433 [DEBUG] betty.server.memberlist.wan: memberlist: Initiating push/pull sync with: *.dc1 127.0.0.1:11230 -46:17.435 [WARN] betty.server.raft: heartbeat timeout reached, starting election: last-leader= -46:17.435 [INFO] betty.server.raft: entering candidate state: node="Node at 127.0.0.1:11231 [Candidate]" term=2 -46:17.435 [INFO] betty.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 -46:17.435 [INFO] betty: (WAN) joined: number_of_nodes=1 -46:17.435 [INFO] betty: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 -46:17.435 [INFO] betty.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan -46:17.445 [DEBUG] betty.server.raft: votes: needed=1 -46:17.445 [DEBUG] betty.server.raft: vote granted: from=ca86d984-f11b-3d78-1764-19adaef5fda5 term=2 tally=1 -46:17.445 [INFO] betty.server.raft: election won: tally=1 -46:17.445 [INFO] betty.server.raft: entering leader state: leader="Node at 127.0.0.1:11231 [Leader]" -46:17.445 [INFO] betty.server: cluster leadership acquired -46:17.445 [INFO] betty.server: New leader elected: payload=betty -46:17.453 [DEBUG] betty.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=0 found=true -46:17.503 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false -46:17.527 [INFO] betty.server.connect: received new intermediate certificate from primary datacenter -46:17.533 [INFO] betty.server.connect: updated root certificates from primary datacenter -46:17.533 [INFO] betty.server.connect: initialized secondary datacenter CA with provider: provider=consul -46:17.533 [INFO] betty.leader: started routine: routine="config entry replication" -46:17.533 [INFO] betty.leader: started routine: routine="federation state replication" -46:17.533 [INFO] betty.leader: started routine: routine="federation state anti-entropy" -46:17.533 [INFO] betty.leader: started routine: routine="secondary CA roots watch" -46:17.533 [INFO] betty.leader: started routine: routine="intention replication" -46:17.533 [INFO] betty.leader: started routine: routine="intermediate cert renew watch" -46:17.533 [INFO] betty.leader: started routine: routine="CA root pruning" -46:17.533 [DEBUG] betty.server: Skipping self join check for node since the cluster is too small: node=betty -46:17.533 [INFO] betty.server: member joined, marking health alive: member=betty -46:17.533 [DEBUG] betty.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 -46:17.533 [DEBUG] betty.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 -46:17.534 [INFO] betty.server.gateway_locator: will dial the primary datacenter using our local mesh gateways if possible -46:17.534 [DEBUG] betty.server: finished fetching remote objects: amount=1 -46:17.534 [DEBUG] betty.server: diffing replication state: local_amount=0 remote_amount=1 -46:17.534 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:17.534 [DEBUG] betty.server: performing updates: updates=1 -46:17.534 [DEBUG] betty.server.replication.config_entry: finished fetching config entries: amount=0 -46:17.534 [DEBUG] betty.server.replication.config_entry: Config Entry replication: local=0 remote=0 -46:17.534 [DEBUG] betty.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 -46:17.534 [DEBUG] betty.server.replication.config_entry: replication completed through remote index: index=1 -46:17.542 [INFO] betty.server: federation state anti-entropy synced -46:17.544 [INFO] betty.server.connect: updated root certificates from primary datacenter -46:17.544 [DEBUG] betty.server: finished updates -46:17.544 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=14 -46:17.545 [DEBUG] betty.server: finished fetching remote objects: amount=2 -46:17.545 [DEBUG] betty.server: diffing replication state: local_amount=1 remote_amount=2 -46:17.545 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:17.545 [DEBUG] betty.server: performing updates: updates=1 -46:17.548 [DEBUG] betty.server: finished updates -46:17.548 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=16 -46:17.549 [INFO] betty.server: federation state anti-entropy synced -46:17.549 [DEBUG] betty.server: finished fetching remote objects: amount=2 -46:17.549 [DEBUG] betty.server: diffing replication state: local_amount=2 remote_amount=2 -46:17.549 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:17.549 [DEBUG] betty.server: performing updates: updates=1 -46:17.553 [DEBUG] betty.server: finished updates -46:17.553 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=17 -46:17.743 [DEBUG] betty: Skipping remote check since it is managed automatically: check=serfHealth -46:17.751 [INFO] betty: Synced node info -46:17.751 [DEBUG] betty: Node info in sync -46:17.779 [DEBUG] betty.server: finished fetching remote objects: amount=3 -46:17.779 [DEBUG] betty.server: diffing replication state: local_amount=2 remote_amount=3 -46:17.779 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:17.779 [DEBUG] betty.server: performing updates: updates=1 -46:17.787 [DEBUG] betty.server: finished updates -46:17.787 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=19 -46:17.788 [DEBUG] betty.server: finished fetching remote objects: amount=3 -46:17.788 [DEBUG] betty.server: diffing replication state: local_amount=3 remote_amount=3 -46:17.788 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:17.788 [DEBUG] betty.server: performing updates: updates=1 -46:17.796 [DEBUG] betty.server: finished updates -46:17.796 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=20 -46:17.873 [DEBUG] betty: Node info in sync -46:17.876 [INFO] betty: Synced service: service=mesh-gateway -46:17.877 [INFO] betty.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 -46:17.877 [INFO] betty.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan -46:17.886 [INFO] betty.server: federation state anti-entropy synced -46:17.896 [ERROR] betty.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11461: Remote DC has no server currently reachable -46:18.315 [DEBUG] betty.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.315 [DEBUG] betty.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.396 [ERROR] betty.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11461: Remote DC has no server currently reachable -46:18.536 [DEBUG] betty.server: finished fetching remote objects: amount=3 -46:18.536 [DEBUG] betty.server: diffing replication state: local_amount=3 remote_amount=3 -46:18.536 [DEBUG] betty.server: diffed replication state: deletions=0 updates=2 -46:18.536 [DEBUG] betty.server: performing updates: updates=2 -46:18.558 [DEBUG] betty.server: finished updates -46:18.558 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=22 -46:18.819 [DEBUG] betty.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.819 [DEBUG] betty.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.896 [ERROR] betty.server.memberlist.wan: memberlist: Failed to send gossip to 127.0.0.1:11461: Remote DC has no server currently reachable -46:19.395 [INFO] betty.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:11230] local=[127.0.0.1:11230] -46:19.672 [DEBUG] betty: Skipping remote check since it is managed automatically: check=serfHealth -46:19.672 [DEBUG] betty: Node info in sync -46:19.672 [DEBUG] betty: Service in sync: service=mesh-gateway -46:19.672 [DEBUG] betty: Node info in sync -46:19.672 [DEBUG] betty: Service in sync: service=mesh-gateway -46:22.768 [DEBUG] betty.server: finished fetching remote objects: amount=3 -46:22.768 [DEBUG] betty.server: diffing replication state: local_amount=3 remote_amount=3 -46:22.768 [DEBUG] betty.server: diffed replication state: deletions=0 updates=1 -46:22.768 [DEBUG] betty.server: performing updates: updates=1 -46:22.773 [DEBUG] betty.server: finished updates -46:22.773 [DEBUG] betty.replication.federation_state: replication completed through remote index: index=23 -46:23.442 [INFO] betty: Requesting shutdown -46:23.442 [INFO] betty.server: shutting down server -46:23.442 [DEBUG] betty.leader: stopping routine: routine="config entry replication" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="federation state replication" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="federation state anti-entropy" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="secondary CA roots watch" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="intention replication" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="intermediate cert renew watch" -46:23.442 [DEBUG] betty.leader: stopping routine: routine="CA root pruning" -46:23.442 [WARN] betty.server.serf.lan: serf: Shutdown without a Leave -46:23.442 [DEBUG] betty.leader: stopped routine: routine="federation state anti-entropy" -46:23.442 [DEBUG] betty.leader: stopped routine: routine="intermediate cert renew watch" -46:23.442 [DEBUG] betty.leader: stopped routine: routine="CA root pruning" -46:23.446 [WARN] betty.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.469Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.469Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.469Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.469Z [INFO] agent.router.manager: shutting down -46:23.469 [INFO] betty: consul server down -46:23.469 [INFO] betty: shutdown complete -46:23.469 [INFO] betty: Stopping server: protocol=DNS address=127.0.0.1:11415 network=tcp -46:23.469 [INFO] betty: Stopping server: protocol=DNS address=127.0.0.1:11415 network=udp -46:23.469 [INFO] betty: Stopping server: protocol=HTTP address=127.0.0.1:11395 network=tcp -46:23.469 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" -46:23.469 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" -46:23.469 [ERROR] betty.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" -46:23.469 [DEBUG] betty.leader: stopped routine: routine="secondary CA roots watch" -46:23.469 [WARN] betty.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" -46:23.469 [INFO] betty.replication.federation_state: stopped replication -46:23.469 [DEBUG] betty.leader: stopped routine: routine="federation state replication" -46:23.469 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" -46:23.469 [ERROR] betty.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" -46:23.469 [DEBUG] betty.leader: stopped routine: routine="intention replication" -46:23.469 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" -46:23.469 [ERROR] betty.server.rpc: RPC failed to server in DC: server=127.0.0.1:11252 datacenter=dc1 method=Internal.ServiceDump error="rpc error making call: EOF" -46:23.469 [INFO] betty.server.replication.config_entry: stopped replication -46:23.469 [DEBUG] betty.leader: stopped routine: routine="config entry replication" -46:23.970 [INFO] betty: Waiting for endpoints to shut down -46:23.970 [INFO] betty: Endpoints down -2022-11-07T22:46:15.267Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:15.279Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.314 [INFO] bob.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8251e3b1-a757-4a76-3d30-38da5242c077 Address:127.0.0.1:11252}]" -46:15.314 [INFO] bob.server.raft: entering follower state: follower="Node at 127.0.0.1:11252 [Follower]" leader= -46:15.315 [INFO] bob.server.serf.wan: serf: EventMemberJoin: bob.dc1 127.0.0.1 -46:15.316 [INFO] bob.server.serf.lan: serf: EventMemberJoin: bob 127.0.0.1 -2022-11-07T22:46:15.316Z [INFO] agent.router: Initializing LAN area manager -46:15.317 [INFO] bob: Started DNS server: address=127.0.0.1:11275 network=udp -46:15.318 [INFO] bob.server: Adding LAN server: server="bob (Addr: tcp/127.0.0.1:11252) (DC: dc1)" -46:15.318 [INFO] bob.server: Handled event for server in area: event=member-join server=bob.dc1 area=wan -46:15.318 [INFO] bob: Started DNS server: address=127.0.0.1:11275 network=tcp -46:15.320 [INFO] bob: Started HTTP server: address=127.0.0.1:11248 network=tcp -46:15.320 [INFO] bob: started state syncer -46:15.320 [INFO] bob: Started gRPC server: address=127.0.0.1:11169 network=tcp -46:15.374 [WARN] bob.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.374 [INFO] bob.server.raft: entering candidate state: node="Node at 127.0.0.1:11252 [Candidate]" term=2 -46:15.384 [DEBUG] bob.server.raft: votes: needed=1 -46:15.384 [DEBUG] bob.server.raft: vote granted: from=8251e3b1-a757-4a76-3d30-38da5242c077 term=2 tally=1 -46:15.384 [INFO] bob.server.raft: election won: tally=1 -46:15.384 [INFO] bob.server.raft: entering leader state: leader="Node at 127.0.0.1:11252 [Leader]" -46:15.384 [INFO] bob.server: cluster leadership acquired -46:15.384 [INFO] bob.server: New leader elected: payload=bob -46:15.393 [DEBUG] bob.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11252 -46:15.409 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.445 [INFO] bob.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.445 [INFO] bob.leader: started routine: routine="federation state anti-entropy" -46:15.445 [INFO] bob.leader: started routine: routine="federation state pruning" -46:15.445 [INFO] bob.leader: started routine: routine="intermediate cert renew watch" -46:15.445 [INFO] bob.leader: started routine: routine="CA root pruning" -46:15.445 [DEBUG] bob.server: Skipping self join check for node since the cluster is too small: node=bob -46:15.445 [INFO] bob.server: member joined, marking health alive: member=bob -46:15.471 [INFO] bob.server: federation state anti-entropy synced -46:15.471 [INFO] bob: Synced node info -46:15.633 [DEBUG] bob: Node info in sync -46:15.637 [INFO] bob: Synced service: service=mesh-gateway -46:15.645 [INFO] bob.server: federation state anti-entropy synced -46:16.891 [DEBUG] bob: Skipping remote check since it is managed automatically: check=serfHealth -46:16.891 [DEBUG] bob: Node info in sync -46:16.891 [DEBUG] bob: Service in sync: service=mesh-gateway -46:16.891 [DEBUG] bob: Node info in sync -46:16.891 [DEBUG] bob: Service in sync: service=mesh-gateway -46:17.315 [INFO] bob.server.gateway_locator: new cached locations of mesh gateways: primary=[127.0.0.1:11230] local=[127.0.0.1:11230] -46:17.434 [DEBUG] bob.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:50508 -46:17.435 [INFO] bob.server.serf.wan: serf: EventMemberJoin: betty.dc2 127.0.0.1 -46:17.435 [INFO] bob.server: Handled event for server in area: event=member-join server=betty.dc2 area=wan -46:17.653 [DEBUG] bob.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:50514 -46:17.653 [INFO] bob.server.serf.wan: serf: EventMemberJoin: bonnie.dc3 127.0.0.1 -46:17.653 [INFO] bob.server: Handled event for server in area: event=member-join server=bonnie.dc3 area=wan -46:17.941 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.151 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.397 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.397 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.621 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:18.621 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 -46:18.896 [DEBUG] bob.server.serf.wan: serf: messageJoinType: bonnie.dc3 -46:19.122 [DEBUG] bob.server.serf.wan: serf: messageJoinType: betty.dc2 -46:23.970 [INFO] bob: Requesting shutdown -46:23.970 [INFO] bob.server: shutting down server -46:23.970 [DEBUG] bob.leader: stopping routine: routine="intermediate cert renew watch" -46:23.970 [DEBUG] bob.leader: stopping routine: routine="CA root pruning" -46:23.970 [DEBUG] bob.leader: stopping routine: routine="federation state anti-entropy" -46:23.970 [DEBUG] bob.leader: stopping routine: routine="federation state pruning" -46:23.970 [WARN] bob.server.serf.lan: serf: Shutdown without a Leave -46:23.970 [DEBUG] bob.leader: stopped routine: routine="intermediate cert renew watch" -46:23.970 [DEBUG] bob.leader: stopped routine: routine="CA root pruning" -46:23.970 [DEBUG] bob.leader: stopped routine: routine="federation state pruning" -46:23.970 [ERROR] bob.server: error performing anti-entropy sync of federation state: error="context canceled" -46:23.970 [DEBUG] bob.leader: stopped routine: routine="federation state anti-entropy" -46:23.973 [WARN] bob.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.976Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.976Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.976Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.976Z [INFO] agent.router.manager: shutting down -46:23.977 [INFO] bob: consul server down -46:23.977 [INFO] bob: shutdown complete -46:23.977 [INFO] bob: Stopping server: protocol=DNS address=127.0.0.1:11275 network=tcp -46:23.977 [INFO] bob: Stopping server: protocol=DNS address=127.0.0.1:11275 network=udp -46:23.977 [INFO] bob: Stopping server: protocol=HTTP address=127.0.0.1:11248 network=tcp -46:24.477 [INFO] bob: Waiting for endpoints to shut down -46:24.477 [INFO] bob: Endpoints down ---- PASS: TestAgent_JoinWAN_viaMeshGateway (9.23s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc2 (0.03s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc1_to_dc3 (0.04s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc1 (0.00s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc2_to_dc3 (0.03s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc1 (0.00s) - --- PASS: TestAgent_JoinWAN_viaMeshGateway/dc3_to_dc2 (0.03s) -=== CONT TestACL_vetServiceRegister -2022-11-07T22:46:24.489Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.489Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_vetServiceRegister (0.01s) +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:05.574Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.574Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_Version8EnabledByDefault (0.04s) +=== CONT TestACL_Legacy_List +2021-10-05T16:32:05.589Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.589Z [WARN] agent: bootstrap = true: do not enable unless necessary +--- PASS: TestACL_AgentMasterToken (0.07s) === CONT TestACL_LoginProcedure_HTTP -[WARN] freeport: 10 out of 55 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:23.741Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.750Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.776 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d Address:127.0.0.1:11183}]" -46:23.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11183 [Follower]" leader= -46:23.777 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d.dc1 127.0.0.1 -46:23.777 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d 127.0.0.1 -2022-11-07T22:46:23.778Z [INFO] agent.router: Initializing LAN area manager -46:23.778 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d.dc1 area=wan -46:23.778 [INFO] TestAgent.server: Adding LAN server: server="Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d (Addr: tcp/127.0.0.1:11183) (DC: dc1)" -46:23.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11262 network=udp -46:23.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11262 network=tcp -46:23.778 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11263 network=tcp -46:23.778 [INFO] TestAgent: started state syncer -46:23.778 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11240 network=tcp -46:23.846 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.846 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11183 [Candidate]" term=2 -46:23.858 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.858 [DEBUG] TestAgent.server.raft: vote granted: from=ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d term=2 tally=1 -46:23.858 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.858 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11183 [Leader]" -46:23.858 [INFO] TestAgent.server: cluster leadership acquired -46:23.858 [INFO] TestAgent.server: New leader elected: payload=Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d -46:23.866 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11183 -46:23.883 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.904 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.904 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.904 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.904 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.904 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.904 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d -46:23.904 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec7bd9c9-c47f-cb46-ba00-3ebd3f73e55d -46:23.912 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.119 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.122 [INFO] TestAgent: Synced node info -46:24.182 [INFO] TestAgent: Requesting shutdown -46:24.182 [INFO] TestAgent.server: shutting down server -46:24.182 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.182 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.182 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.182 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.182 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.182 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.182 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.182 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.182 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.182 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.187 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.195Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.195Z [INFO] agent.router.manager: shutting down -46:24.195 [INFO] TestAgent: consul server down -46:24.195 [INFO] TestAgent: shutdown complete -46:24.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11262 network=tcp -46:24.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11262 network=udp -46:24.195 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11263 network=tcp -46:24.695 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.695 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Checks (0.96s) -=== CONT TestACL_HTTP -2022-11-07T22:46:23.721Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.734Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.757 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3bcaa9f2-23ad-c998-be52-cb664cd4f270 Address:127.0.0.1:11253}]" -46:23.757 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11253 [Follower]" leader= -46:23.758 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270.dc1 127.0.0.1 -46:23.758 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270 127.0.0.1 -2022-11-07T22:46:23.758Z [INFO] agent.router: Initializing LAN area manager -46:23.758 [INFO] TestAgent.server: Adding LAN server: server="Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270 (Addr: tcp/127.0.0.1:11253) (DC: dc1)" -46:23.758 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270.dc1 area=wan -46:23.758 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11255 network=udp -46:23.759 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11255 network=tcp -46:23.759 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11256 network=tcp -46:23.759 [INFO] TestAgent: started state syncer -46:23.759 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11219 network=tcp -46:23.795 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.795 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11253 [Candidate]" term=2 -46:23.805 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.805 [DEBUG] TestAgent.server.raft: vote granted: from=3bcaa9f2-23ad-c998-be52-cb664cd4f270 term=2 tally=1 -46:23.805 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.805 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11253 [Leader]" -46:23.805 [INFO] TestAgent.server: cluster leadership acquired -46:23.805 [INFO] TestAgent.server: New leader elected: payload=Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270 -46:23.811 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11253 -46:23.823 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.848 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:23.848 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:23.848 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:23.848 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:23.848 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:23.848 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270 -46:23.848 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3bcaa9f2-23ad-c998-be52-cb664cd4f270 -46:23.858 [INFO] TestAgent.server: federation state anti-entropy synced -46:23.946 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.950 [INFO] TestAgent: Synced node info -46:24.186 [DEBUG] TestAgent: Check status updated: check=test status=warning -46:24.186 [DEBUG] TestAgent: Node info in sync -46:24.190 [INFO] TestAgent: Synced check: check=test -46:24.190 [INFO] TestAgent: Requesting shutdown -46:24.190 [INFO] TestAgent.server: shutting down server -46:24.190 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.190 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.190 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.190 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.190 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.190 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.190 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.190 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.190 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.190 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.195 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.201Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.201Z [INFO] agent.router.manager: shutting down -46:24.201 [INFO] TestAgent: consul server down -46:24.201 [INFO] TestAgent: shutdown complete -46:24.201 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11255 network=tcp -46:24.202 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11255 network=udp -46:24.202 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11256 network=tcp -46:24.702 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.702 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_WarnCheck (0.99s) -=== CONT TestACL_RootAuthorizersDenied -2022-11-07T22:46:24.714Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.714Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_RootAuthorizersDenied (0.01s) -=== CONT TestACL_Bootstrap === RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/invalid_bearer_token === RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_no_bindings -=== RUN TestACL_LoginProcedure_HTTP/AuthMethod -=== RUN TestACL_LoginProcedure_HTTP/AuthMethod/Create === RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_1_service_binding -2022-11-07T22:46:24.008Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.008Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.023Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.023Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.070 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b11a3344-ff69-c333-042f-95c8b38d7d1d Address:127.0.0.1:11260}]" -46:24.070 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11260 [Follower]" leader= -46:24.070 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b11a3344-ff69-c333-042f-95c8b38d7d1d.dc1 127.0.0.1 -46:24.071 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b11a3344-ff69-c333-042f-95c8b38d7d1d 127.0.0.1 -2022-11-07T22:46:24.071Z [INFO] agent.router: Initializing LAN area manager -46:24.071 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d.dc1 area=wan -46:24.071 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11199 network=udp -46:24.071 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11199 network=tcp -46:24.071 [INFO] TestAgent.server: Adding LAN server: server="Node-b11a3344-ff69-c333-042f-95c8b38d7d1d (Addr: tcp/127.0.0.1:11260) (DC: dc1)" -46:24.071 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11200 network=tcp -46:24.071 [INFO] TestAgent: started state syncer -46:24.071 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11261 network=tcp -46:24.121 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.124 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.124 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11260 [Candidate]" term=2 -46:24.133 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.133 [DEBUG] TestAgent.server.raft: vote granted: from=b11a3344-ff69-c333-042f-95c8b38d7d1d term=2 tally=1 -46:24.133 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.133 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11260 [Leader]" -46:24.133 [INFO] TestAgent.server: cluster leadership acquired -46:24.133 [INFO] TestAgent.server: New leader elected: payload=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.141 [INFO] TestAgent.server: initializing acls -46:24.146 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.146 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:24.153 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.164 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.164 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.164 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.164 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.164 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b11a3344-ff69-c333-042f-95c8b38d7d1d.dc1 -46:24.164 [INFO] TestAgent.server: Updating LAN server: server="Node-b11a3344-ff69-c333-042f-95c8b38d7d1d (Addr: tcp/127.0.0.1:11260) (DC: dc1)" -46:24.164 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d.dc1 area=wan -46:24.179 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.197 [INFO] TestAgent: Synced node info -46:24.206 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.206 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.206 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.206 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.206 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.206 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.206 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.215 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.215 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.341 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.341 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b11a3344-ff69-c333-042f-95c8b38d7d1d -46:24.341 [DEBUG] TestAgent: dropping service from result due to ACLs: service=mysql -46:24.341 [INFO] TestAgent: Requesting shutdown -46:24.341 [INFO] TestAgent.server: shutting down server -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.341 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.341 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.341 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.341 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.346 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.349Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.349Z [INFO] agent.router.manager: shutting down -46:24.349 [INFO] TestAgent: consul server down -46:24.349 [INFO] TestAgent: shutdown complete -46:24.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11199 network=tcp -46:24.349 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11199 network=udp -46:24.349 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11200 network=tcp -46:24.849 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.849 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_ACLFilter (0.86s) - --- PASS: TestAgent_Services_ACLFilter/no_token (0.00s) - --- PASS: TestAgent_Services_ACLFilter/root_token (0.00s) +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/invalid_bearer_token +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_no_bindings +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_1_service_binding +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/invalid_bearer_token +2021-10-05T16:32:04.773Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:04.788Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.883 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 Address:127.0.0.1:29617}]" +32:04.883 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565.dc1 127.0.0.1 +32:04.884 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29617 [Follower]" leader= +32:04.884 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 127.0.0.1 +2021-10-05T16:32:04.884Z [INFO] agent.router: Initializing LAN area manager +32:04.885 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29626 network=udp +32:04.885 [INFO] TestAgent.server: Adding LAN server: server="Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 (Addr: tcp/127.0.0.1:29617) (DC: dc1)" +32:04.885 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565.dc1 area=wan +32:04.885 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29626 network=tcp +32:04.885 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29627 network=tcp +32:04.885 [INFO] TestAgent: started state syncer +32:04.885 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29618 network=tcp +32:04.954 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:04.954 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29617 [Candidate]" term=2 +32:04.981 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:04.981 [DEBUG] TestAgent.server.raft: vote granted: from=ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 term=2 tally=1 +32:04.981 [INFO] TestAgent.server.raft: election won: tally=1 +32:04.981 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29617 [Leader]" +32:04.981 [INFO] TestAgent.server: cluster leadership acquired +32:04.982 [INFO] TestAgent.server: New leader elected: payload=Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 +32:04.993 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29617 +32:05.010 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.051 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.051 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.051 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.051 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.051 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.051 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 +32:05.051 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef94035a-dbe9-7ccc-0f9e-1b0dfa913565 +32:05.078 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.110 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:05.115 [INFO] TestAgent: Synced node info +32:05.335 [INFO] TestAgent: Requesting shutdown +32:05.335 [INFO] TestAgent.server: shutting down server +32:05.335 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:05.335 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:05.335 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:05.335 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:05.335 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:05.336 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:05.336 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:05.336 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:05.336 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:05.336 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:05.344 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:05.356Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:05.356Z [INFO] agent.router.manager: shutting down +32:05.356 [INFO] TestAgent: consul server down +32:05.356 [INFO] TestAgent: shutdown complete +32:05.356 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29626 network=tcp +32:05.356 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29626 network=udp +32:05.356 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29627 network=tcp +32:05.857 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.857 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services_ExternalConnectProxy (1.11s) +=== CONT TestACL_HTTP +2021-10-05T16:32:04.873Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:04.929Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:04.985 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:92cb5a03-db02-a5e9-b82a-900fe9379e47 Address:127.0.0.1:29575}]" +32:04.986 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-92cb5a03-db02-a5e9-b82a-900fe9379e47.dc1 127.0.0.1 +32:04.987 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-92cb5a03-db02-a5e9-b82a-900fe9379e47 127.0.0.1 +2021-10-05T16:32:04.987Z [INFO] agent.router: Initializing LAN area manager +32:04.987 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29575 [Follower]" leader= +32:04.987 [INFO] TestAgent.server: Adding LAN server: server="Node-92cb5a03-db02-a5e9-b82a-900fe9379e47 (Addr: tcp/127.0.0.1:29575) (DC: dc1)" +32:04.987 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-92cb5a03-db02-a5e9-b82a-900fe9379e47.dc1 area=wan +32:04.987 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29598 network=tcp +32:04.987 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29598 network=udp +32:04.988 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29599 network=tcp +32:04.988 [INFO] TestAgent: started state syncer +32:04.988 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29611 network=tcp +32:05.033 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.033 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29575 [Candidate]" term=2 +32:05.060 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.060 [DEBUG] TestAgent.server.raft: vote granted: from=92cb5a03-db02-a5e9-b82a-900fe9379e47 term=2 tally=1 +32:05.060 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.060 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29575 [Leader]" +32:05.061 [INFO] TestAgent.server: cluster leadership acquired +32:05.061 [INFO] TestAgent.server: New leader elected: payload=Node-92cb5a03-db02-a5e9-b82a-900fe9379e47 +32:05.074 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29575 +32:05.094 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.160 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.160 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.160 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.160 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.160 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.160 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92cb5a03-db02-a5e9-b82a-900fe9379e47 +32:05.160 [INFO] TestAgent.server: member joined, marking health alive: member=Node-92cb5a03-db02-a5e9-b82a-900fe9379e47 +32:05.200 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.335 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:05.343 [INFO] TestAgent: Synced node info +32:05.345 [INFO] TestAgent: Requesting shutdown +32:05.345 [INFO] TestAgent.server: shutting down server +32:05.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:05.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:05.345 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:05.345 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:05.345 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:05.346 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:05.346 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:05.346 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:05.346 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:05.346 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:05.351 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:05.356Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:05.356Z [INFO] agent.router.manager: shutting down +32:05.357 [INFO] TestAgent: consul server down +32:05.357 [INFO] TestAgent: shutdown complete +32:05.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29598 network=tcp +32:05.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29598 network=udp +32:05.357 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29599 network=tcp +32:05.858 [INFO] TestAgent: Waiting for endpoints to shut down +32:05.858 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServicesFiltered (1.02s) +=== CONT TestACL_Bootstrap +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_no_bindings +=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_1_service_binding +2021-10-05T16:32:05.025Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.047Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:05.085 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a Address:127.0.0.1:29624}]" +32:05.086 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a.dc1 127.0.0.1 +32:05.087 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29624 [Follower]" leader= +32:05.089 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a 127.0.0.1 +2021-10-05T16:32:05.089Z [INFO] agent.router: Initializing LAN area manager +32:05.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29612 network=udp +32:05.089 [INFO] TestAgent.server: Adding LAN server: server="Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a (Addr: tcp/127.0.0.1:29624) (DC: dc1)" +32:05.089 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a.dc1 area=wan +32:05.089 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29612 network=tcp +32:05.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29613 network=tcp +32:05.092 [INFO] TestAgent: started state syncer +32:05.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29583 network=tcp +32:05.160 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.160 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29624 [Candidate]" term=2 +32:05.196 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.196 [DEBUG] TestAgent.server.raft: vote granted: from=38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a term=2 tally=1 +32:05.196 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.196 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29624 [Leader]" +32:05.197 [INFO] TestAgent.server: cluster leadership acquired +32:05.197 [INFO] TestAgent.server: New leader elected: payload=Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a +32:05.224 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29624 +32:05.248 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.270 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.270 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.270 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.270 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.270 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.270 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a +32:05.270 [INFO] TestAgent.server: member joined, marking health alive: member=Node-38a6fd4f-2cc5-755b-d68f-9dd9a8472f8a +32:05.284 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.537 [INFO] TestAgent: Requesting shutdown +32:05.537 [INFO] TestAgent.server: shutting down server +32:05.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:05.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:05.537 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:05.537 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:05.537 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:05.537 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:05.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:05.538 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:05.538 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:05.538 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:05.538 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:05.552 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:05.564Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:05.564Z [INFO] agent.router.manager: shutting down +32:05.564 [INFO] TestAgent: consul server down +32:05.564 [INFO] TestAgent: shutdown complete +32:05.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29612 network=tcp +32:05.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29612 network=udp +32:05.564 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29613 network=tcp +32:06.064 [INFO] TestAgent: Waiting for endpoints to shut down +32:06.064 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Services (1.06s) === CONT TestACL_Disabled_Response +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestACL_LoginProcedure_HTTP/AuthMethod +=== RUN TestACL_LoginProcedure_HTTP/AuthMethod/Create === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Create_other === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Update_Name_URL_Mismatch === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Update -2022-11-07T22:46:23.904Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.904Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.919Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:23.919Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.946 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1353ab7c-ce6e-54eb-810d-53269bcf2c7d Address:127.0.0.1:11365}]" -46:23.946 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11365 [Follower]" leader= -46:23.947 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d.dc1 127.0.0.1 -46:23.947 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d 127.0.0.1 -2022-11-07T22:46:23.947Z [INFO] agent.router: Initializing LAN area manager -46:23.947 [INFO] TestAgent.server: Adding LAN server: server="Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d (Addr: tcp/127.0.0.1:11365) (DC: dc1)" -46:23.947 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d.dc1 area=wan -46:23.948 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11360 network=udp -46:23.948 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11360 network=tcp -46:23.948 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11361 network=tcp -46:23.948 [INFO] TestAgent: started state syncer -46:23.948 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11254 network=tcp -46:23.993 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:23.993 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11365 [Candidate]" term=2 -46:23.998 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.002 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.002 [DEBUG] TestAgent.server.raft: vote granted: from=1353ab7c-ce6e-54eb-810d-53269bcf2c7d term=2 tally=1 -46:24.002 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.002 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11365 [Leader]" -46:24.002 [INFO] TestAgent.server: cluster leadership acquired -46:24.002 [INFO] TestAgent.server: New leader elected: payload=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.008 [INFO] TestAgent.server: initializing acls -46:24.011 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.011 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:24.016 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.027 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.027 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.027 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.027 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.027 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d.dc1 -46:24.027 [INFO] TestAgent.server: Updating LAN server: server="Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d (Addr: tcp/127.0.0.1:11365) (DC: dc1)" -46:24.027 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d.dc1 area=wan -46:24.040 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.071 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.071 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.071 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.071 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.071 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.071 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.071 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.076 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.083 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.267 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.273 [INFO] TestAgent: Synced node info -46:24.355 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.355 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-1353ab7c-ce6e-54eb-810d-53269bcf2c7d -46:24.355 [DEBUG] TestAgent: dropping check from result due to ACLs: check=mysql -46:24.356 [INFO] TestAgent: Requesting shutdown -46:24.356 [INFO] TestAgent.server: shutting down server -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.356 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.356 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.356 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.356 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.360 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.363Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.363Z [INFO] agent.router.manager: shutting down -46:24.363 [INFO] TestAgent: consul server down -46:24.363 [INFO] TestAgent: shutdown complete -46:24.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11360 network=tcp -46:24.363 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11360 network=udp -46:24.363 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11361 network=tcp -46:24.863 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.864 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Checks_ACLFilter (0.97s) - --- PASS: TestAgent_Checks_ACLFilter/no_token (0.00s) - --- PASS: TestAgent_Checks_ACLFilter/root_token (0.00s) -=== CONT TestACL_vetCheckUpdate -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Invalid_payload === RUN TestACL_LoginProcedure_HTTP/AuthMethod/List === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Delete === RUN TestACL_LoginProcedure_HTTP/AuthMethod/Read === RUN TestACL_LoginProcedure_HTTP/BindingRule === RUN TestACL_LoginProcedure_HTTP/BindingRule/Create -2022-11-07T22:46:23.921Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:23.931Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:23.961 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6c4d3be7-7c92-011d-0ffa-e6b79f305de5 Address:127.0.0.1:11239}]" -46:23.961 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11239 [Follower]" leader= -46:23.961 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5.dc1 127.0.0.1 -46:23.962 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5 127.0.0.1 -2022-11-07T22:46:23.962Z [INFO] agent.router: Initializing LAN area manager -46:23.962 [INFO] TestAgent.server: Adding LAN server: server="Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5 (Addr: tcp/127.0.0.1:11239) (DC: dc1)" -46:23.962 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5.dc1 area=wan -46:23.962 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11241 network=udp -46:23.962 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11241 network=tcp -46:23.962 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11242 network=tcp -46:23.962 [INFO] TestAgent: started state syncer -46:24.020 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.020 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11239 [Candidate]" term=2 -46:24.028 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.028 [DEBUG] TestAgent.server.raft: vote granted: from=6c4d3be7-7c92-011d-0ffa-e6b79f305de5 term=2 tally=1 -46:24.028 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.028 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11239 [Leader]" -46:24.028 [INFO] TestAgent.server: cluster leadership acquired -46:24.028 [INFO] TestAgent.server: New leader elected: payload=Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5 -46:24.040 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11239 -46:24.065 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.099 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.099 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.099 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.099 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.099 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.099 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5 -46:24.099 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6c4d3be7-7c92-011d-0ffa-e6b79f305de5 -46:24.111 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.149 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.153 [INFO] TestAgent: Synced node info -46:24.153 [DEBUG] TestAgent: Node info in sync -46:24.368 [INFO] TestAgent: Requesting shutdown -46:24.368 [INFO] TestAgent.server: shutting down server -46:24.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.368 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.368 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.368 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.368 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.368 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.368 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.375Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.375Z [INFO] agent.router.manager: shutting down -46:24.375 [INFO] TestAgent: consul server down -46:24.375 [INFO] TestAgent: shutdown complete -46:24.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11241 network=tcp -46:24.376 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11241 network=udp -46:24.376 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11242 network=tcp -46:24.876 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.876 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:24.875Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.875Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_vetCheckUpdate (0.01s) -=== CONT TestAgent_DeregisterCheck ---- PASS: TestAgent_Self (1.86s) - --- PASS: TestAgent_Self/normal (0.89s) - --- PASS: TestAgent_Self/no_grpc (0.97s) -=== CONT TestACL_filterMembers === RUN TestACL_LoginProcedure_HTTP/BindingRule/Create_other -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/invalid_bearer_token === RUN TestACL_LoginProcedure_HTTP/BindingRule/BindingRule_CRUD_Missing_ID_in_URL === RUN TestACL_LoginProcedure_HTTP/BindingRule/Update -2022-11-07T22:46:24.884Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.884Z [WARN] agent: bootstrap = true: do not enable unless necessary -46:24.885 [DEBUG] TestACL_filterMembers: dropping node from result due to ACLs: node=Nope accessorID=9df2d1a4-2d07-414e-8ead-6053f56ed2eb ---- PASS: TestACL_filterMembers (0.01s) -=== CONT TestACL_filterServices === RUN TestACL_LoginProcedure_HTTP/BindingRule/ID_Supplied === RUN TestACL_LoginProcedure_HTTP/BindingRule/Invalid_payload === RUN TestACL_LoginProcedure_HTTP/BindingRule/List === RUN TestACL_LoginProcedure_HTTP/BindingRule/Delete -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_no_bindings === RUN TestACL_LoginProcedure_HTTP/BindingRule/Read === RUN TestACL_LoginProcedure_HTTP/Login === RUN TestACL_LoginProcedure_HTTP/Login/Create_Token_1 -2022-11-07T22:46:24.903Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.903Z [WARN] agent: bootstrap = true: do not enable unless necessary -46:24.905 [DEBUG] TestACL_filterServices: dropping service from result due to ACLs: service=my-other ---- PASS: TestACL_filterServices (0.02s) -=== CONT TestAgent_RegisterCheck_ACLDeny === RUN TestACL_LoginProcedure_HTTP/Login/Create_Token_2 -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_1_service_binding === RUN TestACL_LoginProcedure_HTTP/Login/List_Tokens_by_(incorrect)_Method === RUN TestACL_LoginProcedure_HTTP/Login/List_Tokens_by_(correct)_Method === RUN TestACL_LoginProcedure_HTTP/Login/Logout === RUN TestACL_LoginProcedure_HTTP/Login/Token_is_gone_after_Logout -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/invalid_bearer_token -2022-11-07T22:46:24.027Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.036Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:db76144f-84f6-60fd-5ba7-6893ae1b8061 Address:127.0.0.1:11159}]" -46:24.080 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11159 [Follower]" leader= -46:24.081 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-db76144f-84f6-60fd-5ba7-6893ae1b8061.dc1 127.0.0.1 -46:24.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 127.0.0.1 -2022-11-07T22:46:24.081Z [INFO] agent.router: Initializing LAN area manager -46:24.081 [INFO] TestAgent.server: Adding LAN server: server="Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 (Addr: tcp/127.0.0.1:11159) (DC: dc1)" -46:24.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11147 network=udp -46:24.081 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061.dc1 area=wan -46:24.082 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11147 network=tcp -46:24.082 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11106 network=tcp -46:24.082 [INFO] TestAgent: started state syncer -46:24.082 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11111 network=tcp -46:24.126 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.126 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11159 [Candidate]" term=2 -46:24.132 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.138 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.138 [DEBUG] TestAgent.server.raft: vote granted: from=db76144f-84f6-60fd-5ba7-6893ae1b8061 term=2 tally=1 -46:24.138 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11159 [Leader]" -46:24.138 [INFO] TestAgent.server: cluster leadership acquired -46:24.138 [INFO] TestAgent.server: New leader elected: payload=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.146 [INFO] TestAgent.server: initializing acls -46:24.153 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.160 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.170 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.170 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.170 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.170 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.170 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-db76144f-84f6-60fd-5ba7-6893ae1b8061.dc1 -46:24.171 [INFO] TestAgent.server: Updating LAN server: server="Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 (Addr: tcp/127.0.0.1:11159) (DC: dc1)" -46:24.171 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061.dc1 area=wan -46:24.181 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.201 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.201 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.201 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.201 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.201 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.201 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.201 [INFO] TestAgent.server: member joined, marking health alive: member=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.215 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.215 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.402 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.408 [INFO] TestAgent: Synced node info -46:24.408 [DEBUG] TestAgent: Node info in sync -46:24.446 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-db76144f-84f6-60fd-5ba7-6893ae1b8061 -46:24.446 [INFO] TestAgent: Requesting shutdown -46:24.446 [INFO] TestAgent.server: shutting down server -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.446 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.446 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.446 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.446 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.451 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.454Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.454Z [INFO] agent.router.manager: shutting down -46:24.454 [INFO] TestAgent: consul server down -46:24.454 [INFO] TestAgent: shutdown complete -46:24.454 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11147 network=tcp -46:24.454 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11147 network=udp -46:24.454 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11106 network=tcp -46:24.955 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.955 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_HealthServicesACLEnforcement (0.94s) - --- PASS: TestAgent_HealthServicesACLEnforcement/no-token-health-by-id (0.00s) - --- PASS: TestAgent_HealthServicesACLEnforcement/no-token-health-by-name (0.00s) - --- PASS: TestAgent_HealthServicesACLEnforcement/root-token-health-by-id (0.00s) - --- PASS: TestAgent_HealthServicesACLEnforcement/root-token-health-by-name (0.00s) -=== CONT TestAgent_Services_MeshGateway +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestACL_Bootstrap/bootstrap -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_no_bindings -=== RUN TestACL_Bootstrap/not_again -=== RUN TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_1_service_binding -2022-11-07T22:46:24.007Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.022Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.067 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6475194d-63ec-0bc1-2a24-bfb08a650a3b Address:127.0.0.1:11288}]" -46:24.067 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11288 [Follower]" leader= -46:24.067 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b.dc1 127.0.0.1 -46:24.068 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b 127.0.0.1 -2022-11-07T22:46:24.068Z [INFO] agent.router: Initializing LAN area manager -46:24.068 [INFO] TestAgent.server: Adding LAN server: server="Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b (Addr: tcp/127.0.0.1:11288) (DC: dc1)" -46:24.068 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b.dc1 area=wan -46:24.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11234 network=udp -46:24.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11234 network=tcp -46:24.068 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11235 network=tcp -46:24.068 [INFO] TestAgent: started state syncer -46:24.068 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11146 network=tcp -46:24.126 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.126 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11288 [Candidate]" term=2 -46:24.137 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.137 [DEBUG] TestAgent.server.raft: vote granted: from=6475194d-63ec-0bc1-2a24-bfb08a650a3b term=2 tally=1 -46:24.137 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.137 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11288 [Leader]" -46:24.137 [INFO] TestAgent.server: cluster leadership acquired -46:24.137 [INFO] TestAgent.server: New leader elected: payload=Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b -46:24.146 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11288 -46:24.160 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.186 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.186 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.186 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.186 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.186 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.186 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b -46:24.186 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6475194d-63ec-0bc1-2a24-bfb08a650a3b -46:24.199 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.387 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.390 [INFO] TestAgent: Synced node info -46:24.390 [DEBUG] TestAgent: Node info in sync -46:24.477 [DEBUG] TestAgent: Check status updated: check=test status=passing -46:24.477 [DEBUG] TestAgent: Node info in sync -46:24.480 [INFO] TestAgent: Synced check: check=test -46:24.480 [INFO] TestAgent: Requesting shutdown -46:24.480 [INFO] TestAgent.server: shutting down server -46:24.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.480 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.480 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.480 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.480 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.480 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.481 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.481 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.484 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.491Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.491Z [INFO] agent.router.manager: shutting down -46:24.491 [INFO] TestAgent: consul server down -46:24.491 [INFO] TestAgent: shutdown complete -46:24.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11234 network=tcp -46:24.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11234 network=udp -46:24.491 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11235 network=tcp -46:24.992 [INFO] TestAgent: Waiting for endpoints to shut down -46:24.992 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_PassCheck (1.00s) -=== CONT TestACL_filterChecks -2022-11-07T22:46:25.000Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.000Z [WARN] agent: bootstrap = true: do not enable unless necessary -46:25.000 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-node -46:25.000 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-other -46:25.000 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-service -46:25.000 [DEBUG] TestACL_filterChecks: dropping check from result due to ACLs: check=my-other ---- PASS: TestACL_filterChecks (0.01s) -=== CONT TestACL_ResolveIdentity -2022-11-07T22:46:25.008Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.008Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_ResolveIdentity (0.01s) -=== CONT TestACL_Legacy_Destroy -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestACL_Disabled_Response/ACLBootstrap -=== RUN TestACL_Disabled_Response/ACLReplicationStatus -=== RUN TestACL_Disabled_Response/AgentToken -=== RUN TestACL_Disabled_Response/ACLRulesTranslate -=== RUN TestACL_Disabled_Response/ACLRulesTranslateLegacyToken -=== RUN TestACL_Disabled_Response/ACLPolicyList -=== RUN TestACL_Disabled_Response/ACLPolicyCRUD -=== RUN TestACL_Disabled_Response/ACLPolicyCreate -=== RUN TestACL_Disabled_Response/ACLTokenList -=== RUN TestACL_Disabled_Response/ACLTokenCreate -=== RUN TestACL_Disabled_Response/ACLTokenSelf -=== RUN TestACL_Disabled_Response/ACLTokenCRUD -=== RUN TestACL_Disabled_Response/ACLRoleList -=== RUN TestACL_Disabled_Response/ACLRoleCreate -=== RUN TestACL_Disabled_Response/ACLRoleCRUD -=== RUN TestACL_Disabled_Response/ACLBindingRuleList -=== RUN TestACL_Disabled_Response/ACLBindingRuleCreate -=== RUN TestACL_Disabled_Response/ACLBindingRuleCRUD -=== RUN TestACL_Disabled_Response/ACLAuthMethodList -=== RUN TestACL_Disabled_Response/ACLAuthMethodCreate -=== RUN TestACL_Disabled_Response/ACLAuthMethodCRUD -=== RUN TestACL_Disabled_Response/ACLLogin -=== RUN TestACL_Disabled_Response/ACLLogout -=== RUN TestACL_Disabled_Response/ACLAuthorize -2022-11-07T22:46:24.380Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.380Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.389Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.389Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.410 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:afec5077-4fe5-1aff-98fd-290fd48f112a Address:127.0.0.1:11026}]" -46:24.411 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11026 [Follower]" leader= -46:24.411 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-afec5077-4fe5-1aff-98fd-290fd48f112a.dc1 127.0.0.1 -46:24.412 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-afec5077-4fe5-1aff-98fd-290fd48f112a 127.0.0.1 -2022-11-07T22:46:24.412Z [INFO] agent.router: Initializing LAN area manager -46:24.412 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-afec5077-4fe5-1aff-98fd-290fd48f112a.dc1 area=wan -46:24.412 [INFO] TestAgent.server: Adding LAN server: server="Node-afec5077-4fe5-1aff-98fd-290fd48f112a (Addr: tcp/127.0.0.1:11026) (DC: dc1)" -46:24.412 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11112 network=udp -46:24.412 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11112 network=tcp -46:24.412 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11022 network=tcp -46:24.412 [INFO] TestAgent: started state syncer -46:24.412 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11921 network=tcp -46:24.462 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.464 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.464 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11026 [Candidate]" term=2 -46:24.474 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.474 [DEBUG] TestAgent.server.raft: vote granted: from=afec5077-4fe5-1aff-98fd-290fd48f112a term=2 tally=1 -46:24.474 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.474 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11026 [Leader]" -46:24.474 [INFO] TestAgent.server: cluster leadership acquired -46:24.474 [INFO] TestAgent.server: New leader elected: payload=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.479 [INFO] TestAgent.server: initializing acls -46:24.483 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.483 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:24.490 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.502 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.502 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.502 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.502 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.502 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-afec5077-4fe5-1aff-98fd-290fd48f112a.dc1 -46:24.502 [INFO] TestAgent.server: Updating LAN server: server="Node-afec5077-4fe5-1aff-98fd-290fd48f112a (Addr: tcp/127.0.0.1:11026) (DC: dc1)" -46:24.503 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-afec5077-4fe5-1aff-98fd-290fd48f112a.dc1 area=wan -46:24.513 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.538 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.538 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.538 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.538 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.538 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.538 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.538 [INFO] TestAgent.server: member joined, marking health alive: member=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.551 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.551 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.566 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.571 [INFO] TestAgent: Synced node info -46:24.631 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.631 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-afec5077-4fe5-1aff-98fd-290fd48f112a -46:24.631 [INFO] TestAgent: Requesting shutdown -46:24.631 [INFO] TestAgent.server: shutting down server -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.631 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.631 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.631 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.631 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.639 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.648Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.648Z [INFO] agent.router.manager: shutting down -46:24.648 [INFO] TestAgent: consul server down -46:24.648 [INFO] TestAgent: shutdown complete -46:24.648 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11112 network=tcp -46:24.648 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11112 network=udp -46:24.648 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11022 network=tcp -46:25.148 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.148 [INFO] TestAgent: Endpoints down ---- PASS: TestACLReplicationStatus (0.78s) -=== CONT TestACL_Legacy_Clone === RUN TestACL_HTTP/Policy === RUN TestACL_HTTP/Policy/Create +=== RUN TestACL_Bootstrap/not_again === RUN TestACL_HTTP/Policy/Minimal === RUN TestACL_HTTP/Policy/Name_Chars === RUN TestACL_HTTP/Policy/Update_Name_ID_Mismatch @@ -51013,30 +51199,109 @@ === RUN TestACL_HTTP/Role === RUN TestACL_HTTP/Role/Create === RUN TestACL_HTTP/Role/Name_Chars +2021-10-05T16:32:05.189Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.216Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:05.264 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fa9efb1a-3167-e449-6859-3040570ced53 Address:127.0.0.1:29519}]" +32:05.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29519 [Follower]" leader= +32:05.265 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fa9efb1a-3167-e449-6859-3040570ced53.dc1 127.0.0.1 +32:05.272 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fa9efb1a-3167-e449-6859-3040570ced53 127.0.0.1 +2021-10-05T16:32:05.272Z [INFO] agent.router: Initializing LAN area manager +32:05.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29507 network=udp +32:05.273 [INFO] TestAgent.server: Adding LAN server: server="Node-fa9efb1a-3167-e449-6859-3040570ced53 (Addr: tcp/127.0.0.1:29519) (DC: dc1)" +32:05.273 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fa9efb1a-3167-e449-6859-3040570ced53.dc1 area=wan +32:05.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29507 network=tcp +32:05.273 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29508 network=tcp +32:05.273 [INFO] TestAgent: started state syncer +32:05.273 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29576 network=tcp +32:05.324 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:05.332 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.332 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29519 [Candidate]" term=2 +32:05.352 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.352 [DEBUG] TestAgent.server.raft: vote granted: from=fa9efb1a-3167-e449-6859-3040570ced53 term=2 tally=1 +32:05.352 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.352 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29519 [Leader]" +32:05.353 [INFO] TestAgent.server: cluster leadership acquired +32:05.353 [INFO] TestAgent.server: New leader elected: payload=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.360 [INFO] TestAgent.server: initializing acls +32:05.363 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:05.384 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:05.412 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:05.412 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:05.412 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:05.412 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.412 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-fa9efb1a-3167-e449-6859-3040570ced53.dc1 +32:05.416 [INFO] TestAgent.server: Updating LAN server: server="Node-fa9efb1a-3167-e449-6859-3040570ced53 (Addr: tcp/127.0.0.1:29519) (DC: dc1)" +32:05.416 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-fa9efb1a-3167-e449-6859-3040570ced53.dc1 area=wan +32:05.468 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.545 [INFO] TestAgent: Synced node info +32:05.545 [DEBUG] TestAgent: Node info in sync +32:05.568 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.568 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.568 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.568 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.568 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.568 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.568 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.597 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.616 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.640 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:05.640 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fa9efb1a-3167-e449-6859-3040570ced53 +32:06.021 [INFO] TestAgent: Requesting shutdown +32:06.021 [INFO] TestAgent.server: shutting down server +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.021 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:06.021 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.021 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.021 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:06.035 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.036Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.036Z [INFO] agent.router.manager: shutting down +32:06.036 [INFO] TestAgent: consul server down +32:06.036 [INFO] TestAgent: shutdown complete +32:06.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29507 network=tcp +32:06.036 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29507 network=udp +32:06.036 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29508 network=tcp +32:06.536 [INFO] TestAgent: Waiting for endpoints to shut down +32:06.536 [INFO] TestAgent: Endpoints down +--- PASS: TestACLEndpoint_LoginLogout_jwt (1.39s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys (0.06s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/invalid_bearer_token (0.00s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_no_bindings (0.01s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_1_service_binding (0.02s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks (0.10s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/invalid_bearer_token (0.01s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_no_bindings (0.03s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_1_service_binding (0.02s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery (0.21s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/invalid_bearer_token (0.04s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_no_bindings (0.01s) + --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_1_service_binding (0.10s) +=== CONT TestACLReplicationStatus === RUN TestACL_HTTP/Role/Update_Name_ID_Mismatch === RUN TestACL_HTTP/Role/Role_CRUD_Missing_ID_in_URL === RUN TestACL_HTTP/Role/Update -=== RUN TestAgent_RegisterCheck_ACLDeny/no_token_-_node_check -=== RUN TestAgent_RegisterCheck_ACLDeny/svc_token_-_node_check -=== RUN TestAgent_RegisterCheck_ACLDeny/node_token_-_node_check === RUN TestACL_HTTP/Role/ID_Supplied === RUN TestACL_HTTP/Role/Invalid_payload === RUN TestACL_HTTP/Role/Delete +[WARN] freeport: 10 out of 18 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestACL_HTTP/Role/List === RUN TestACL_HTTP/Role/Read === RUN TestACL_HTTP/Token === RUN TestACL_HTTP/Token/Create -=== RUN TestAgent_RegisterCheck_ACLDeny/no_token_-_svc_check -=== RUN TestAgent_RegisterCheck_ACLDeny/node_token_-_svc_check -=== RUN TestAgent_RegisterCheck_ACLDeny/svc_token_-_svc_check === RUN TestACL_HTTP/Token/Create_Local === RUN TestACL_HTTP/Token/Read === RUN TestACL_HTTP/Token/Self === RUN TestACL_HTTP/Token/Clone === RUN TestACL_HTTP/Token/Update -=== RUN TestACL_HTTP/Token/CRUD_Missing_Token_Accessor_ID -=== RUN TestACL_HTTP/Token/Update_Accessor_Mismatch -=== RUN TestACL_HTTP/Token/Delete === RUN TestACL_Authorize/master-token === RUN TestACL_Authorize/master-token/dc1 === RUN TestACL_Authorize/master-token/dc2 @@ -51047,10 +51312,37 @@ === RUN TestACL_Authorize/decode-failure === RUN TestACL_Authorize/acl-not-found === RUN TestACL_Authorize/local-token-in-secondary-dc +=== RUN TestACL_Authorize/local-token-wrong-dc +=== RUN TestACL_HTTP/Token/CRUD_Missing_Token_Accessor_ID +=== RUN TestACL_HTTP/Token/Update_Accessor_Mismatch +=== RUN TestACL_HTTP/Token/Delete +=== RUN TestACL_Disabled_Response/ACLBootstrap +=== RUN TestACL_Disabled_Response/ACLReplicationStatus +=== RUN TestACL_Disabled_Response/AgentToken +=== RUN TestACL_Disabled_Response/ACLRulesTranslate +=== RUN TestACL_Disabled_Response/ACLRulesTranslateLegacyToken +=== RUN TestACL_Disabled_Response/ACLPolicyList +=== RUN TestACL_Disabled_Response/ACLPolicyCRUD +=== RUN TestACL_Disabled_Response/ACLPolicyCreate +=== RUN TestACL_Disabled_Response/ACLTokenList +=== RUN TestACL_Disabled_Response/ACLTokenCreate +=== RUN TestACL_Disabled_Response/ACLTokenSelf +=== RUN TestACL_Disabled_Response/ACLTokenCRUD +=== RUN TestACL_Disabled_Response/ACLRoleList +=== RUN TestACL_Disabled_Response/ACLRoleCreate +=== RUN TestACL_Disabled_Response/ACLRoleCRUD +=== RUN TestACL_Disabled_Response/ACLBindingRuleList +=== RUN TestACL_Disabled_Response/ACLBindingRuleCreate +=== RUN TestACL_Disabled_Response/ACLBindingRuleCRUD +=== RUN TestACL_Disabled_Response/ACLAuthMethodList +=== RUN TestACL_Disabled_Response/ACLAuthMethodCreate +=== RUN TestACL_Disabled_Response/ACLAuthMethodCRUD +=== RUN TestACL_Disabled_Response/ACLLogin +=== RUN TestACL_Disabled_Response/ACLLogout +=== RUN TestACL_Disabled_Response/ACLAuthorize === RUN TestACL_HTTP/Token/List === RUN TestACL_HTTP/Token/List_by_Policy === RUN TestACL_HTTP/Token/Create_with_Accessor -=== RUN TestACL_Authorize/local-token-wrong-dc === RUN TestACL_HTTP/Token/Create_with_Secret === RUN TestACL_HTTP/Token/Create_with_Accessor_and_Secret === RUN TestACL_HTTP/Token/Create_with_Accessor_Dup @@ -51059,337 +51351,335 @@ === RUN TestACL_HTTP/Token/Create_with_Accessor_as_Secret_Dup === RUN TestACL_HTTP/Token/Create_with_Reserved_Accessor === RUN TestACL_HTTP/Token/Create_with_Reserved_Secret -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:24.500Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.500Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.510Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.510Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.556 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 Address:127.0.0.1:11316}]" -46:24.556 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11316 [Follower]" leader= -46:24.556 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8.dc1 127.0.0.1 -46:24.556 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 127.0.0.1 -2022-11-07T22:46:24.556Z [INFO] agent.router: Initializing LAN area manager -46:24.557 [INFO] TestAgent.server: Adding LAN server: server="Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 (Addr: tcp/127.0.0.1:11316) (DC: dc1)" -46:24.557 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8.dc1 area=wan -46:24.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11220 network=udp -46:24.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11220 network=tcp -46:24.557 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11221 network=tcp -46:24.557 [INFO] TestAgent: started state syncer -46:24.557 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11317 network=tcp -46:24.607 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.621 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.621 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11316 [Candidate]" term=2 -46:24.636 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.636 [DEBUG] TestAgent.server.raft: vote granted: from=755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 term=2 tally=1 -46:24.636 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.636 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11316 [Leader]" -46:24.636 [INFO] TestAgent.server: cluster leadership acquired -46:24.637 [INFO] TestAgent.server: New leader elected: payload=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.647 [INFO] TestAgent.server: initializing acls -46:24.650 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.650 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:24.659 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.674 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.674 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.674 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.674 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.674 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8.dc1 -46:24.674 [INFO] TestAgent.server: Updating LAN server: server="Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 (Addr: tcp/127.0.0.1:11316) (DC: dc1)" -46:24.674 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8.dc1 area=wan -46:24.696 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.716 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.716 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.716 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.716 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.716 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.716 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.716 [INFO] TestAgent.server: member joined, marking health alive: member=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.721 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.729 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.778 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.782 [INFO] TestAgent: Synced node info -46:24.782 [DEBUG] TestAgent: Node info in sync -46:24.846 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.846 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-755f8d67-00b9-0b1a-ff9d-89ee84b2faa8 -46:24.924 [INFO] TestAgent: Requesting shutdown -46:24.924 [INFO] TestAgent.server: shutting down server -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.924 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.924 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.924 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.924 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.931 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.939Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.939Z [INFO] agent.router.manager: shutting down -46:24.939 [INFO] TestAgent: consul server down -46:24.939 [INFO] TestAgent: shutdown complete -46:24.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11220 network=tcp -46:24.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11220 network=udp -46:24.939 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11221 network=tcp -46:25.440 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.440 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_LoginProcedure_HTTP (0.95s) - --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod (0.03s) - --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Create (0.00s) +2021-10-05T16:32:05.623Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.623Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.661Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.661Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:05.740 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 Address:127.0.0.1:30284}]" +32:05.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49.dc1 127.0.0.1 +32:05.741 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 127.0.0.1 +2021-10-05T16:32:05.741Z [INFO] agent.router: Initializing LAN area manager +32:05.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30129 network=udp +32:05.742 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30284 [Follower]" leader= +32:05.742 [INFO] TestAgent.server: Adding LAN server: server="Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 (Addr: tcp/127.0.0.1:30284) (DC: dc1)" +32:05.742 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49.dc1 area=wan +32:05.742 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30129 network=tcp +32:05.743 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30127 network=tcp +32:05.743 [INFO] TestAgent: started state syncer +32:05.743 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30285 network=tcp +32:05.792 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:05.811 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.811 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30284 [Candidate]" term=2 +32:05.853 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.853 [DEBUG] TestAgent.server.raft: vote granted: from=7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 term=2 tally=1 +32:05.853 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.853 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30284 [Leader]" +32:05.854 [INFO] TestAgent.server: cluster leadership acquired +32:05.854 [INFO] TestAgent.server: New leader elected: payload=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:05.867 [INFO] TestAgent.server: initializing acls +32:05.873 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:05.873 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:05.881 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:05.890 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:05.890 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:05.890 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:05.891 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:05.891 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49.dc1 +32:05.892 [INFO] TestAgent.server: Updating LAN server: server="Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 (Addr: tcp/127.0.0.1:30284) (DC: dc1)" +32:05.892 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49.dc1 area=wan +32:05.902 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.956 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.956 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.956 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.956 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.956 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.956 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:05.956 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:05.992 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:05.993 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.012 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-7cfc1a3c-57ba-caee-b10b-9a5dbac97f49 +32:06.096 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:06.151 [INFO] TestAgent: Synced node info +32:06.151 [DEBUG] TestAgent: Node info in sync +32:06.218 [INFO] TestAgent: Requesting shutdown +32:06.218 [INFO] TestAgent.server: shutting down server +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.218 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.218 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:06.218 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.224 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.225Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.225Z [INFO] agent.router.manager: shutting down +32:06.225 [INFO] TestAgent: consul server down +32:06.225 [INFO] TestAgent: shutdown complete +32:06.225 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30129 network=tcp +32:06.225 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30129 network=udp +32:06.226 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30127 network=tcp +32:06.728 [INFO] TestAgent: Waiting for endpoints to shut down +32:06.728 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_List (1.16s) +=== CONT TestRemoteExecGetSpec_ACLAgentToken +2021-10-05T16:32:05.634Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.634Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.681Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.681Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:05.772 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3f4a91e3-7919-4e5b-59b8-74aa805862c5 Address:127.0.0.1:30203}]" +32:05.772 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30203 [Follower]" leader= +32:05.773 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5.dc1 127.0.0.1 +32:05.773 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 127.0.0.1 +2021-10-05T16:32:05.773Z [INFO] agent.router: Initializing LAN area manager +32:05.774 [INFO] TestAgent.server: Adding LAN server: server="Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 (Addr: tcp/127.0.0.1:30203) (DC: dc1)" +32:05.774 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5.dc1 area=wan +32:05.774 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30286 network=udp +32:05.774 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30286 network=tcp +32:05.774 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30287 network=tcp +32:05.774 [INFO] TestAgent: started state syncer +32:05.774 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30204 network=tcp +32:05.820 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.820 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30203 [Candidate]" term=2 +32:05.824 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:05.860 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.860 [DEBUG] TestAgent.server.raft: vote granted: from=3f4a91e3-7919-4e5b-59b8-74aa805862c5 term=2 tally=1 +32:05.860 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.860 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30203 [Leader]" +32:05.864 [INFO] TestAgent.server: cluster leadership acquired +32:05.864 [INFO] TestAgent.server: New leader elected: payload=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:05.876 [INFO] TestAgent.server: initializing acls +32:05.880 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:05.880 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:05.894 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +32:05.894 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:05.901 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:05.901 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:05.901 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:05.901 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:05.902 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5.dc1 +32:05.902 [INFO] TestAgent.server: Updating LAN server: server="Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 (Addr: tcp/127.0.0.1:30203) (DC: dc1)" +32:05.902 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5.dc1 area=wan +32:05.976 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:06.043 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:06.043 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:06.043 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:06.043 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:06.043 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:06.043 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:06.043 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:06.053 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.068 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:06.081 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:06.081 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3f4a91e3-7919-4e5b-59b8-74aa805862c5 +32:06.231 [INFO] TestAgent: Requesting shutdown +32:06.231 [INFO] TestAgent.server: shutting down server +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.231 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.231 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.231 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.231 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.231 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.231 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.231 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.232 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:06.232 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.248 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.248Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.248Z [INFO] agent.router.manager: shutting down +32:06.248 [INFO] TestAgent: consul server down +32:06.248 [INFO] TestAgent: shutdown complete +32:06.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30286 network=tcp +32:06.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30286 network=udp +32:06.248 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30287 network=tcp +32:06.749 [INFO] TestAgent: Waiting for endpoints to shut down +32:06.749 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_LoginProcedure_HTTP (1.16s) + --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod (0.07s) + --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Create (0.02s) --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Create_other (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Update_Name_URL_Mismatch (0.00s) - --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Update (0.00s) + --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Update (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Invalid_payload (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/List (0.00s) - --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Delete (0.01s) + --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Delete (0.03s) --- PASS: TestACL_LoginProcedure_HTTP/AuthMethod/Read (0.00s) - --- PASS: TestACL_LoginProcedure_HTTP/BindingRule (0.03s) - --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Create (0.00s) + --- PASS: TestACL_LoginProcedure_HTTP/BindingRule (0.05s) + --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Create (0.03s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Create_other (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/BindingRule_CRUD_Missing_ID_in_URL (0.00s) - --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Update (0.00s) + --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Update (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/ID_Supplied (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Invalid_payload (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/List (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Delete (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/BindingRule/Read (0.00s) - --- PASS: TestACL_LoginProcedure_HTTP/Login (0.02s) + --- PASS: TestACL_LoginProcedure_HTTP/Login (0.03s) --- PASS: TestACL_LoginProcedure_HTTP/Login/Create_Token_1 (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/Login/Create_Token_2 (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/Login/List_Tokens_by_(incorrect)_Method (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/Login/List_Tokens_by_(correct)_Method (0.00s) --- PASS: TestACL_LoginProcedure_HTTP/Login/Logout (0.01s) --- PASS: TestACL_LoginProcedure_HTTP/Login/Token_is_gone_after_Logout (0.00s) -=== CONT TestACL_Legacy_Get -=== RUN TestACL_Legacy_Get/wrong_id -2022-11-07T22:46:24.729Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.729Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.744Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.744Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.767 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:79c42fc4-f450-dd34-f57a-8fd3d77aaca7 Address:127.0.0.1:11358}]" -46:24.767 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11358 [Follower]" leader= -46:24.768 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7.dc1 127.0.0.1 -46:24.768 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 127.0.0.1 -2022-11-07T22:46:24.768Z [INFO] agent.router: Initializing LAN area manager -46:24.768 [INFO] TestAgent.server: Adding LAN server: server="Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 (Addr: tcp/127.0.0.1:11358) (DC: dc1)" -46:24.769 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7.dc1 area=wan -46:24.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11290 network=udp -46:24.769 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11290 network=tcp -46:24.769 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11291 network=tcp -46:24.769 [INFO] TestAgent: started state syncer -46:24.769 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11296 network=tcp -46:24.817 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.817 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11358 [Candidate]" term=2 -46:24.819 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.826 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.826 [DEBUG] TestAgent.server.raft: vote granted: from=79c42fc4-f450-dd34-f57a-8fd3d77aaca7 term=2 tally=1 -46:24.826 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.826 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11358 [Leader]" -46:24.827 [INFO] TestAgent.server: cluster leadership acquired -46:24.827 [INFO] TestAgent.server: New leader elected: payload=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.833 [INFO] TestAgent.server: initializing acls -46:24.837 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.845 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.845 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.845 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.845 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.845 [INFO] TestAgent.server: Updating LAN server: server="Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 (Addr: tcp/127.0.0.1:11358) (DC: dc1)" -46:24.845 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7.dc1 -46:24.845 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7.dc1 area=wan -46:24.866 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.889 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.889 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.889 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.889 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.889 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.889 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.889 [INFO] TestAgent.server: member joined, marking health alive: member=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.903 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.909 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.966 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.966 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-79c42fc4-f450-dd34-f57a-8fd3d77aaca7 -46:24.966 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestACL_Bootstrap-TestAgent-agent214681130/acl-bootstrap-reset: no such file or directory" -46:24.972 [INFO] TestAgent.server.acl: ACL bootstrap completed -46:24.972 [INFO] TestAgent: Requesting shutdown -46:24.972 [INFO] TestAgent.server: shutting down server -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.972 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.972 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.972 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.972 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.972 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.978 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.988Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.988Z [INFO] agent.router.manager: shutting down -46:24.988 [INFO] TestAgent: consul server down -46:24.988 [INFO] TestAgent: shutdown complete -46:24.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11290 network=tcp -46:24.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11290 network=udp -46:24.988 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11291 network=tcp -46:25.489 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.489 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Bootstrap (0.77s) - --- PASS: TestACL_Bootstrap/bootstrap (0.01s) +=== CONT TestRemoteExecWrites_ACLAgentToken +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:05.894Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.895Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.944Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.944Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.052 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0f9b12d-57be-cfbd-341f-9a291380d84d Address:127.0.0.1:29636}]" +32:06.053 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29636 [Follower]" leader= +32:06.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 127.0.0.1 +32:06.054 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d 127.0.0.1 +2021-10-05T16:32:06.054Z [INFO] agent.router: Initializing LAN area manager +32:06.054 [INFO] TestAgent.server: Adding LAN server: server="Node-b0f9b12d-57be-cfbd-341f-9a291380d84d (Addr: tcp/127.0.0.1:29636) (DC: dc1)" +32:06.054 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 area=wan +32:06.055 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30073 network=udp +32:06.055 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30073 network=tcp +32:06.055 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29639 network=tcp +32:06.055 [INFO] TestAgent: started state syncer +32:06.055 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29637 network=tcp +32:06.105 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:06.108 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:06.108 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29636 [Candidate]" term=2 +32:06.176 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:06.176 [DEBUG] TestAgent.server.raft: vote granted: from=b0f9b12d-57be-cfbd-341f-9a291380d84d term=2 tally=1 +32:06.176 [INFO] TestAgent.server.raft: election won: tally=1 +32:06.176 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29636 [Leader]" +32:06.177 [INFO] TestAgent.server: cluster leadership acquired +32:06.177 [INFO] TestAgent.server: New leader elected: payload=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.188 [INFO] TestAgent.server: initializing acls +32:06.196 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:06.205 [INFO] TestAgent.server: initializing acls +32:06.218 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:06.218 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:06.218 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:06.218 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 +32:06.218 [INFO] TestAgent.server: Updating LAN server: server="Node-b0f9b12d-57be-cfbd-341f-9a291380d84d (Addr: tcp/127.0.0.1:29636) (DC: dc1)" +32:06.218 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 area=wan +32:06.249 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:06.249 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:06.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 +32:06.249 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d.dc1 area=wan +32:06.249 [INFO] TestAgent.server: Updating LAN server: server="Node-b0f9b12d-57be-cfbd-341f-9a291380d84d (Addr: tcp/127.0.0.1:29636) (DC: dc1)" +32:06.283 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:06.388 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:06.388 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:06.388 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:06.388 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:06.388 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:06.388 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.388 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.413 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.413 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.425 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b0f9b12d-57be-cfbd-341f-9a291380d84d +32:06.425 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestACL_Bootstrap-TestAgent-agent274675526/acl-bootstrap-reset: no such file or directory" +32:06.438 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.449 [INFO] TestAgent.server.acl: ACL bootstrap completed +32:06.449 [INFO] TestAgent: Requesting shutdown +32:06.449 [INFO] TestAgent.server: shutting down server +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.449 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.449 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.449 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.450 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.450 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.457 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.458Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.458Z [INFO] agent.router.manager: shutting down +32:06.458 [INFO] TestAgent: consul server down +32:06.458 [INFO] TestAgent: shutdown complete +32:06.458 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30073 network=tcp +32:06.458 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30073 network=udp +32:06.458 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29639 network=tcp +32:06.958 [INFO] TestAgent: Waiting for endpoints to shut down +32:06.958 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Bootstrap (1.10s) + --- PASS: TestACL_Bootstrap/bootstrap (0.02s) --- PASS: TestACL_Bootstrap/not_again (0.00s) -=== CONT TestACL_Legacy_List -2022-11-07T22:46:24.480Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.492Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.529 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0cadd44d-6ba3-ed7f-c297-2bded75df283 Address:127.0.0.1:11190}]" -46:24.530 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11190 [Follower]" leader= -46:24.530 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0cadd44d-6ba3-ed7f-c297-2bded75df283.dc1 127.0.0.1 -46:24.530 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 127.0.0.1 -2022-11-07T22:46:24.530Z [INFO] agent.router: Initializing LAN area manager -46:24.530 [INFO] TestAgent.server: Adding LAN server: server="Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 (Addr: tcp/127.0.0.1:11190) (DC: dc1)" -46:24.530 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283.dc1 area=wan -46:24.531 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11311 network=udp -46:24.531 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11311 network=tcp -46:24.531 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11312 network=tcp -46:24.531 [INFO] TestAgent: started state syncer -46:24.531 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11191 network=tcp -46:24.581 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.595 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.595 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11190 [Candidate]" term=2 -46:24.608 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.608 [DEBUG] TestAgent.server.raft: vote granted: from=0cadd44d-6ba3-ed7f-c297-2bded75df283 term=2 tally=1 -46:24.608 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.608 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11190 [Leader]" -46:24.608 [INFO] TestAgent.server: cluster leadership acquired -46:24.608 [INFO] TestAgent.server: New leader elected: payload=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.616 [INFO] TestAgent.server: initializing acls -46:24.624 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.632 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.650 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.650 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.650 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.650 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.650 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0cadd44d-6ba3-ed7f-c297-2bded75df283.dc1 -46:24.650 [INFO] TestAgent.server: Updating LAN server: server="Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 (Addr: tcp/127.0.0.1:11190) (DC: dc1)" -46:24.650 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283.dc1 area=wan -46:24.662 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.689 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.689 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.689 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.689 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.689 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.689 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.689 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.700 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.711 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.829 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.829 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0cadd44d-6ba3-ed7f-c297-2bded75df283 -46:24.940 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.950 [INFO] TestAgent: Synced node info -46:24.950 [DEBUG] TestAgent: Node info in sync -46:24.993 [INFO] TestAgent: Requesting shutdown -46:24.993 [INFO] TestAgent.server: shutting down server -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.993 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.993 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.993 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.993 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.999 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.008Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.008Z [INFO] agent.router.manager: shutting down -46:25.009 [INFO] TestAgent: consul server down -46:25.009 [INFO] TestAgent: shutdown complete -46:25.009 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11311 network=tcp -46:25.009 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11311 network=udp -46:25.009 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11312 network=tcp -46:25.509 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.509 [INFO] TestAgent: Endpoints down ---- PASS: TestACLEndpoint_LoginLogout_jwt (1.05s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys (0.04s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/invalid_bearer_token (0.00s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_no_bindings (0.00s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_static_keys/valid_bearer_token_1_service_binding (0.02s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks (0.06s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/invalid_bearer_token (0.01s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_no_bindings (0.01s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_jwks/valid_bearer_token_1_service_binding (0.01s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery (0.06s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/invalid_bearer_token (0.01s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_no_bindings (0.00s) - --- PASS: TestACLEndpoint_LoginLogout_jwt/success_-_jwt_oidc_discovery/valid_bearer_token_1_service_binding (0.01s) -=== CONT TestACL_Legacy_Update -[WARN] freeport: 10 out of 32 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:24.859Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.868Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.912 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ccc9875a-42b2-13f4-fbe6-6b35a68ad745 Address:127.0.0.1:11359}]" -46:24.912 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11359 [Follower]" leader= -46:24.912 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745.dc1 127.0.0.1 -46:24.913 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745 127.0.0.1 -2022-11-07T22:46:24.913Z [INFO] agent.router: Initializing LAN area manager -46:24.913 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745.dc1 area=wan -46:24.913 [INFO] TestAgent.server: Adding LAN server: server="Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745 (Addr: tcp/127.0.0.1:11359) (DC: dc1)" -46:24.913 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11354 network=tcp -46:24.913 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11354 network=udp -46:24.914 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11355 network=tcp -46:24.914 [INFO] TestAgent: started state syncer -46:24.914 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11339 network=tcp -46:24.969 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.969 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11359 [Candidate]" term=2 -46:24.982 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.982 [DEBUG] TestAgent.server.raft: vote granted: from=ccc9875a-42b2-13f4-fbe6-6b35a68ad745 term=2 tally=1 -46:24.982 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.982 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11359 [Leader]" -46:24.982 [INFO] TestAgent.server: cluster leadership acquired -46:24.982 [INFO] TestAgent.server: New leader elected: payload=Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745 -46:24.990 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11359 -46:25.007 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.030 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.030 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.030 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.030 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.030 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.030 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745 -46:25.031 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ccc9875a-42b2-13f4-fbe6-6b35a68ad745 -46:25.044 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.139 [INFO] TestAgent: Requesting shutdown -46:25.139 [INFO] TestAgent.server: shutting down server -46:25.139 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.139 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.139 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.139 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.139 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.140 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:25.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.140 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.140 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.140 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.140 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.144 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.148Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.148Z [INFO] agent.router.manager: shutting down -46:25.148 [INFO] TestAgent: consul server down -46:25.148 [INFO] TestAgent: shutdown complete -46:25.148 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11354 network=tcp -46:25.148 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11354 network=udp -46:25.148 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11355 network=tcp -46:25.649 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.649 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Disabled_Response (0.80s) +=== CONT TestPreparedQuery_parseLimit +--- PASS: TestPreparedQuery_parseLimit (0.00s) +=== CONT TestRemoteExecWrites_ACLToken +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:06.090Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:06.113Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.219 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5a632a31-706d-cb2d-4f06-c46ce620da50 Address:127.0.0.1:29594}]" +32:06.219 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29594 [Follower]" leader= +32:06.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5a632a31-706d-cb2d-4f06-c46ce620da50.dc1 127.0.0.1 +32:06.223 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5a632a31-706d-cb2d-4f06-c46ce620da50 127.0.0.1 +2021-10-05T16:32:06.223Z [INFO] agent.router: Initializing LAN area manager +32:06.223 [INFO] TestAgent.server: Adding LAN server: server="Node-5a632a31-706d-cb2d-4f06-c46ce620da50 (Addr: tcp/127.0.0.1:29594) (DC: dc1)" +32:06.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=udp +32:06.223 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5a632a31-706d-cb2d-4f06-c46ce620da50.dc1 area=wan +32:06.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29645 network=tcp +32:06.224 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29632 network=tcp +32:06.224 [INFO] TestAgent: started state syncer +32:06.224 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29595 network=tcp +32:06.272 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:06.272 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29594 [Candidate]" term=2 +32:06.388 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:06.388 [DEBUG] TestAgent.server.raft: vote granted: from=5a632a31-706d-cb2d-4f06-c46ce620da50 term=2 tally=1 +32:06.388 [INFO] TestAgent.server.raft: election won: tally=1 +32:06.388 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29594 [Leader]" +32:06.388 [INFO] TestAgent.server: cluster leadership acquired +32:06.389 [INFO] TestAgent.server: New leader elected: payload=Node-5a632a31-706d-cb2d-4f06-c46ce620da50 +32:06.413 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29594 +32:06.447 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:06.517 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:06.517 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:06.517 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:06.517 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:06.517 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:06.517 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5a632a31-706d-cb2d-4f06-c46ce620da50 +32:06.517 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5a632a31-706d-cb2d-4f06-c46ce620da50 +32:06.538 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.620 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:06.640 [INFO] TestAgent: Synced node info +32:06.640 [DEBUG] TestAgent: Node info in sync +32:06.653 [INFO] TestAgent: Requesting shutdown +32:06.653 [INFO] TestAgent.server: shutting down server +32:06.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.653 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.653 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.653 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.653 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.653 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.653 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.653 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.653 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.680 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.685Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.685Z [INFO] agent.router.manager: shutting down +32:06.685 [INFO] TestAgent: consul server down +32:06.685 [INFO] TestAgent: shutdown complete +32:06.685 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=tcp +32:06.685 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29645 network=udp +32:06.685 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29632 network=tcp +32:07.185 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.185 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Disabled_Response (1.12s) --- PASS: TestACL_Disabled_Response/ACLBootstrap (0.00s) --- PASS: TestACL_Disabled_Response/ACLReplicationStatus (0.00s) --- PASS: TestACL_Disabled_Response/AgentToken (0.00s) @@ -51414,369 +51704,131 @@ --- PASS: TestACL_Disabled_Response/ACLLogin (0.00s) --- PASS: TestACL_Disabled_Response/ACLLogout (0.00s) --- PASS: TestACL_Disabled_Response/ACLAuthorize (0.00s) -=== CONT TestACL_Legacy_UpdateUpsert -2022-11-07T22:46:24.918Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.927Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.961 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:643c2dd2-5c70-1987-5d38-c9eb3a5b602e Address:127.0.0.1:12022}]" -46:24.961 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12022 [Follower]" leader= -46:24.962 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e.dc1 127.0.0.1 -46:24.963 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e 127.0.0.1 -2022-11-07T22:46:24.963Z [INFO] agent.router: Initializing LAN area manager -46:24.963 [INFO] TestAgent.server: Adding LAN server: server="Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e (Addr: tcp/127.0.0.1:12022) (DC: dc1)" -46:24.963 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e.dc1 area=wan -46:24.963 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11389 network=udp -46:24.963 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11389 network=tcp -46:24.964 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11390 network=tcp -46:24.964 [INFO] TestAgent: started state syncer -46:24.964 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12023 network=tcp -46:25.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12022 [Candidate]" term=2 -46:25.013 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.017 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.017 [DEBUG] TestAgent.server.raft: vote granted: from=643c2dd2-5c70-1987-5d38-c9eb3a5b602e term=2 tally=1 -46:25.017 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.017 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12022 [Leader]" -46:25.017 [INFO] TestAgent.server: cluster leadership acquired -46:25.017 [INFO] TestAgent.server: New leader elected: payload=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.027 [INFO] TestAgent.server: initializing acls -46:25.031 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.031 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.037 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.048 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.048 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.048 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.048 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.049 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e.dc1 -46:25.049 [INFO] TestAgent.server: Updating LAN server: server="Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e (Addr: tcp/127.0.0.1:12022) (DC: dc1)" -46:25.049 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e.dc1 area=wan -46:25.064 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.086 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.086 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.086 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.086 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.086 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.086 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.086 [INFO] TestAgent.server: member joined, marking health alive: member=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.089 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.096 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.161 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.161 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-643c2dd2-5c70-1987-5d38-c9eb3a5b602e -46:25.171 [INFO] TestAgent: Synced node info -46:25.175 [INFO] TestAgent: Synced service: service=foo:1234 -46:25.187 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.187 [DEBUG] TestAgent: Node info in sync -46:25.187 [DEBUG] TestAgent: Service in sync: service=foo:1234 -46:25.187 [DEBUG] TestAgent: Node info in sync -46:25.187 [DEBUG] TestAgent: Service in sync: service=foo:1234 -46:25.205 [DEBUG] TestAgent: Node info in sync -46:25.205 [DEBUG] TestAgent: Service in sync: service=foo:1234 -46:25.210 [INFO] TestAgent: Synced check: check=test -46:25.214 [DEBUG] TestAgent: Node info in sync -46:25.214 [DEBUG] TestAgent: Service in sync: service=foo:1234 -46:25.214 [DEBUG] TestAgent: Check in sync: check=test -46:25.220 [INFO] TestAgent: Synced check: check=test2 -46:25.220 [INFO] TestAgent: Requesting shutdown -46:25.220 [INFO] TestAgent.server: shutting down server -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.220 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.220 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.226 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.232Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.232Z [INFO] agent.router.manager: shutting down -46:25.232 [INFO] TestAgent: consul server down -46:25.232 [INFO] TestAgent: shutdown complete -46:25.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11389 network=tcp -46:25.232 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11389 network=udp -46:25.232 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11390 network=tcp -46:25.732 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.732 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_RegisterCheck_ACLDeny (0.83s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/no_token_-_node_check (0.00s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/svc_token_-_node_check (0.00s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/node_token_-_node_check (0.01s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/no_token_-_svc_check (0.00s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/node_token_-_svc_check (0.00s) - --- PASS: TestAgent_RegisterCheck_ACLDeny/svc_token_-_svc_check (0.01s) -=== CONT TestACL_vetCheckRegister -2022-11-07T22:46:25.746Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.746Z [WARN] agent: bootstrap = true: do not enable unless necessary ---- PASS: TestACL_vetCheckRegister (0.01s) -=== CONT TestAgent_Services_Sidecar -2022-11-07T22:46:25.018Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.018Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.027Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.027Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.064 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3e654b69-0f91-5a76-f063-2a626ca213fd Address:127.0.0.1:11373}]" -46:25.064 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11373 [Follower]" leader= -46:25.065 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 127.0.0.1 -46:25.065 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e654b69-0f91-5a76-f063-2a626ca213fd 127.0.0.1 -2022-11-07T22:46:25.065Z [INFO] agent.router: Initializing LAN area manager -46:25.065 [INFO] TestAgent.server: Adding LAN server: server="Node-3e654b69-0f91-5a76-f063-2a626ca213fd (Addr: tcp/127.0.0.1:11373) (DC: dc1)" -46:25.065 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 area=wan -46:25.065 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11368 network=udp -46:25.065 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11368 network=tcp -46:25.066 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11369 network=tcp -46:25.066 [INFO] TestAgent: started state syncer -46:25.066 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11409 network=tcp -46:25.105 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.105 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11373 [Candidate]" term=2 -46:25.114 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.114 [DEBUG] TestAgent.server.raft: vote granted: from=3e654b69-0f91-5a76-f063-2a626ca213fd term=2 tally=1 -46:25.114 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.114 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11373 [Leader]" -46:25.114 [INFO] TestAgent.server: cluster leadership acquired -46:25.114 [INFO] TestAgent.server: New leader elected: payload=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.115 [INFO] TestAgent.server: initializing acls -46:25.121 [INFO] TestAgent.server: initializing acls -46:25.121 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.121 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.126 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.126 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.135 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.137 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.144 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.144 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.144 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.144 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:25.144 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.145 [INFO] TestAgent.server: Updating LAN server: server="Node-3e654b69-0f91-5a76-f063-2a626ca213fd (Addr: tcp/127.0.0.1:11373) (DC: dc1)" -46:25.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 -46:25.145 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 area=wan -46:25.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 -46:25.145 [INFO] TestAgent.server: Updating LAN server: server="Node-3e654b69-0f91-5a76-f063-2a626ca213fd (Addr: tcp/127.0.0.1:11373) (DC: dc1)" -46:25.145 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3e654b69-0f91-5a76-f063-2a626ca213fd.dc1 area=wan -46:25.157 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.176 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.176 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.176 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.176 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.176 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.176 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.183 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.183 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.193 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.226 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.234 [INFO] TestAgent: Synced node info -46:25.241 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.241 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-3e654b69-0f91-5a76-f063-2a626ca213fd -46:25.248 [INFO] TestAgent: Requesting shutdown -46:25.248 [INFO] TestAgent.server: shutting down server -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.248 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.248 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.248 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.253 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.260Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.260Z [INFO] agent.router.manager: shutting down -46:25.261 [INFO] TestAgent: consul server down -46:25.261 [INFO] TestAgent: shutdown complete -46:25.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11368 network=tcp -46:25.261 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11368 network=udp -46:25.261 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11369 network=tcp -46:25.761 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.761 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_Destroy (0.75s) -=== CONT TestAgent_Services_ExternalConnectProxy -2022-11-07T22:46:24.892Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.912Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.948 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ac45f1e2-1092-8e2c-34c1-df4a23f50445 Address:127.0.0.1:11387}]" -46:24.949 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11387 [Follower]" leader= -46:24.949 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445.dc1 127.0.0.1 -46:24.949 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445 127.0.0.1 -2022-11-07T22:46:24.949Z [INFO] agent.router: Initializing LAN area manager -46:24.950 [INFO] TestAgent.server: Adding LAN server: server="Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445 (Addr: tcp/127.0.0.1:11387) (DC: dc1)" -46:24.950 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445.dc1 area=wan -46:24.950 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11340 network=udp -46:24.950 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11340 network=tcp -46:24.950 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11341 network=tcp -46:24.950 [INFO] TestAgent: started state syncer -46:24.950 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11388 network=tcp -46:24.989 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.989 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11387 [Candidate]" term=2 -46:24.999 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.999 [DEBUG] TestAgent.server.raft: vote granted: from=ac45f1e2-1092-8e2c-34c1-df4a23f50445 term=2 tally=1 -46:24.999 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.999 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11387 [Leader]" -46:24.999 [INFO] TestAgent.server: cluster leadership acquired -46:25.000 [INFO] TestAgent.server: New leader elected: payload=Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445 -46:25.009 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11387 -46:25.026 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.053 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.053 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.053 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.053 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.053 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.053 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445 -46:25.053 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ac45f1e2-1092-8e2c-34c1-df4a23f50445 -46:25.068 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.215 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.222 [INFO] TestAgent: Synced node info -46:25.256 [DEBUG] TestAgent: removed check: check=test -46:25.256 [DEBUG] TestAgent: Node info in sync -46:25.260 [INFO] TestAgent: Deregistered check: check=test -46:25.260 [INFO] TestAgent: Requesting shutdown -46:25.260 [INFO] TestAgent.server: shutting down server -46:25.260 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.260 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.260 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.260 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.260 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.260 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.260 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.260 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.260 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.260 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.263 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.271Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.271Z [INFO] agent.router.manager: shutting down -46:25.271 [INFO] TestAgent: consul server down -46:25.271 [INFO] TestAgent: shutdown complete -46:25.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11340 network=tcp -46:25.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11340 network=udp -46:25.271 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11341 network=tcp -46:25.771 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.771 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:24.708Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.708Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.720Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:24.720Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.752 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:741cf3bc-41fe-a716-4697-edfb0b64fc92 Address:127.0.0.1:11295}]" -46:24.752 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11295 [Follower]" leader= -46:24.752 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-741cf3bc-41fe-a716-4697-edfb0b64fc92.dc1 127.0.0.1 -46:24.753 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 127.0.0.1 -2022-11-07T22:46:24.753Z [INFO] agent.router: Initializing LAN area manager -46:24.753 [INFO] TestAgent.server: Adding LAN server: server="Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 (Addr: tcp/127.0.0.1:11295) (DC: dc1)" -46:24.753 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92.dc1 area=wan -46:24.753 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11297 network=udp -46:24.753 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11297 network=tcp -46:24.754 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11298 network=tcp -46:24.754 [INFO] TestAgent: started state syncer -46:24.754 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11310 network=tcp -46:24.801 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.801 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11295 [Candidate]" term=2 -46:24.803 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.812 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.812 [DEBUG] TestAgent.server.raft: vote granted: from=741cf3bc-41fe-a716-4697-edfb0b64fc92 term=2 tally=1 -46:24.812 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11295 [Leader]" -46:24.812 [INFO] TestAgent.server: cluster leadership acquired -46:24.813 [INFO] TestAgent.server: New leader elected: payload=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:24.822 [INFO] TestAgent.server: initializing acls -46:24.827 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.827 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:24.837 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.848 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.848 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.848 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.849 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:24.849 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-741cf3bc-41fe-a716-4697-edfb0b64fc92.dc1 -46:24.849 [INFO] TestAgent.server: Updating LAN server: server="Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 (Addr: tcp/127.0.0.1:11295) (DC: dc1)" -46:24.849 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92.dc1 area=wan -46:24.869 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.908 [INFO] TestAgent: Synced node info -46:24.908 [DEBUG] TestAgent: Node info in sync -46:24.908 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.908 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.908 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.908 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.908 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.908 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:24.908 [INFO] TestAgent.server: member joined, marking health alive: member=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:24.914 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.926 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:25.162 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:25.162 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-741cf3bc-41fe-a716-4697-edfb0b64fc92 -46:25.262 [INFO] TestAgent: Requesting shutdown -46:25.262 [INFO] TestAgent.server: shutting down server -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.262 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.262 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.262 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.262 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.266 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.274Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.274Z [INFO] agent.router.manager: shutting down -46:25.274 [INFO] TestAgent: consul server down -46:25.274 [INFO] TestAgent: shutdown complete -46:25.274 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11297 network=tcp -46:25.274 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11297 network=udp -46:25.274 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11298 network=tcp -46:25.774 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.774 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_DeregisterCheck (0.90s) -=== CONT TestAgent_ServicesFiltered ---- PASS: TestACL_HTTP (1.08s) - --- PASS: TestACL_HTTP/Policy (0.02s) - --- PASS: TestACL_HTTP/Policy/Create (0.00s) - --- PASS: TestACL_HTTP/Policy/Minimal (0.00s) - --- PASS: TestACL_HTTP/Policy/Name_Chars (0.00s) +=== CONT TestRemoteExecWrites +2021-10-05T16:32:05.870Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.870Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.944Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:05.944Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.045 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88c46067-cf27-844d-8bc4-9749c86f9935 Address:127.0.0.1:30134}]" +32:06.045 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 127.0.0.1 +32:06.046 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88c46067-cf27-844d-8bc4-9749c86f9935 127.0.0.1 +2021-10-05T16:32:06.046Z [INFO] agent.router: Initializing LAN area manager +32:06.046 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30266 network=udp +32:06.046 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30134 [Follower]" leader= +32:06.047 [INFO] TestAgent.server: Adding LAN server: server="Node-88c46067-cf27-844d-8bc4-9749c86f9935 (Addr: tcp/127.0.0.1:30134) (DC: dc1)" +32:06.047 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 area=wan +32:06.047 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30266 network=tcp +32:06.048 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30130 network=tcp +32:06.048 [INFO] TestAgent: started state syncer +32:06.048 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30142 network=tcp +32:06.098 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:06.108 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:06.108 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30134 [Candidate]" term=2 +32:06.178 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:06.178 [DEBUG] TestAgent.server.raft: vote granted: from=88c46067-cf27-844d-8bc4-9749c86f9935 term=2 tally=1 +32:06.178 [INFO] TestAgent.server.raft: election won: tally=1 +32:06.178 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30134 [Leader]" +32:06.179 [INFO] TestAgent.server: cluster leadership acquired +32:06.179 [INFO] TestAgent.server: New leader elected: payload=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.188 [INFO] TestAgent.server: initializing acls +32:06.193 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:06.193 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:06.199 [INFO] TestAgent.server: initializing acls +32:06.199 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:06.238 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:06.238 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:06.265 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:06.265 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:06.265 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:06.266 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.266 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 +32:06.268 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:06.268 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:06.268 [INFO] TestAgent.server: Updating LAN server: server="Node-88c46067-cf27-844d-8bc4-9749c86f9935 (Addr: tcp/127.0.0.1:30134) (DC: dc1)" +32:06.268 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.268 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 +32:06.268 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 area=wan +32:06.268 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-88c46067-cf27-844d-8bc4-9749c86f9935.dc1 area=wan +32:06.268 [INFO] TestAgent.server: Updating LAN server: server="Node-88c46067-cf27-844d-8bc4-9749c86f9935 (Addr: tcp/127.0.0.1:30134) (DC: dc1)" +32:06.302 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:06.408 [INFO] TestAgent: Synced node info +32:06.408 [DEBUG] TestAgent: Node info in sync +32:06.424 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:06.424 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:06.424 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:06.424 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:06.424 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:06.424 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.424 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.428 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.498 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.498 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.498 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88c46067-cf27-844d-8bc4-9749c86f9935 +32:06.718 [INFO] TestAgent: Requesting shutdown +32:06.718 [INFO] TestAgent.server: shutting down server +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:06.718 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.718 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.718 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:06.718 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.732 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.740Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.740Z [INFO] agent.router.manager: shutting down +32:06.740 [INFO] TestAgent: consul server down +32:06.740 [INFO] TestAgent: shutdown complete +32:06.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30266 network=tcp +32:06.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30266 network=udp +32:06.740 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30130 network=tcp +32:07.244 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.244 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_HTTP (1.39s) + --- PASS: TestACL_HTTP/Policy (0.10s) + --- PASS: TestACL_HTTP/Policy/Create (0.07s) + --- PASS: TestACL_HTTP/Policy/Minimal (0.01s) + --- PASS: TestACL_HTTP/Policy/Name_Chars (0.01s) --- PASS: TestACL_HTTP/Policy/Update_Name_ID_Mismatch (0.00s) --- PASS: TestACL_HTTP/Policy/Policy_CRUD_Missing_ID_in_URL (0.00s) - --- PASS: TestACL_HTTP/Policy/Update (0.00s) + --- PASS: TestACL_HTTP/Policy/Update (0.01s) --- PASS: TestACL_HTTP/Policy/ID_Supplied (0.00s) --- PASS: TestACL_HTTP/Policy/Invalid_payload (0.00s) --- PASS: TestACL_HTTP/Policy/Delete (0.00s) --- PASS: TestACL_HTTP/Policy/List (0.00s) --- PASS: TestACL_HTTP/Policy/Read (0.00s) --- PASS: TestACL_HTTP/Policy/Read_Name (0.00s) - --- PASS: TestACL_HTTP/Role (0.03s) - --- PASS: TestACL_HTTP/Role/Create (0.00s) + --- PASS: TestACL_HTTP/Role (0.06s) + --- PASS: TestACL_HTTP/Role/Create (0.01s) --- PASS: TestACL_HTTP/Role/Name_Chars (0.01s) --- PASS: TestACL_HTTP/Role/Update_Name_ID_Mismatch (0.00s) --- PASS: TestACL_HTTP/Role/Role_CRUD_Missing_ID_in_URL (0.00s) - --- PASS: TestACL_HTTP/Role/Update (0.01s) + --- PASS: TestACL_HTTP/Role/Update (0.02s) --- PASS: TestACL_HTTP/Role/ID_Supplied (0.00s) --- PASS: TestACL_HTTP/Role/Invalid_payload (0.00s) - --- PASS: TestACL_HTTP/Role/Delete (0.00s) + --- PASS: TestACL_HTTP/Role/Delete (0.02s) --- PASS: TestACL_HTTP/Role/List (0.00s) --- PASS: TestACL_HTTP/Role/Read (0.00s) - --- PASS: TestACL_HTTP/Token (0.05s) - --- PASS: TestACL_HTTP/Token/Create (0.00s) - --- PASS: TestACL_HTTP/Token/Create_Local (0.00s) + --- PASS: TestACL_HTTP/Token (0.13s) + --- PASS: TestACL_HTTP/Token/Create (0.01s) + --- PASS: TestACL_HTTP/Token/Create_Local (0.01s) --- PASS: TestACL_HTTP/Token/Read (0.00s) --- PASS: TestACL_HTTP/Token/Self (0.00s) - --- PASS: TestACL_HTTP/Token/Clone (0.01s) - --- PASS: TestACL_HTTP/Token/Update (0.00s) + --- PASS: TestACL_HTTP/Token/Clone (0.03s) + --- PASS: TestACL_HTTP/Token/Update (0.02s) --- PASS: TestACL_HTTP/Token/CRUD_Missing_Token_Accessor_ID (0.00s) --- PASS: TestACL_HTTP/Token/Update_Accessor_Mismatch (0.00s) - --- PASS: TestACL_HTTP/Token/Delete (0.01s) + --- PASS: TestACL_HTTP/Token/Delete (0.03s) --- PASS: TestACL_HTTP/Token/List (0.00s) --- PASS: TestACL_HTTP/Token/List_by_Policy (0.00s) - --- PASS: TestACL_HTTP/Token/Create_with_Accessor (0.00s) - --- PASS: TestACL_HTTP/Token/Create_with_Secret (0.00s) + --- PASS: TestACL_HTTP/Token/Create_with_Accessor (0.01s) + --- PASS: TestACL_HTTP/Token/Create_with_Secret (0.03s) --- PASS: TestACL_HTTP/Token/Create_with_Accessor_and_Secret (0.01s) --- PASS: TestACL_HTTP/Token/Create_with_Accessor_Dup (0.00s) --- PASS: TestACL_HTTP/Token/Create_with_Secret_as_Accessor_Dup (0.00s) @@ -51784,645 +51836,248 @@ --- PASS: TestACL_HTTP/Token/Create_with_Accessor_as_Secret_Dup (0.00s) --- PASS: TestACL_HTTP/Token/Create_with_Reserved_Accessor (0.00s) --- PASS: TestACL_HTTP/Token/Create_with_Reserved_Secret (0.00s) -=== CONT TestAgent_Services -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:24.965Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.974Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.012 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7907f154-f25c-1ea2-31fe-0197cbbd2a73 Address:127.0.0.1:11422}]" -46:25.012 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11422 [Follower]" leader= -46:25.013 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73.dc1 127.0.0.1 -46:25.013 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73 127.0.0.1 -2022-11-07T22:46:25.013Z [INFO] agent.router: Initializing LAN area manager -46:25.013 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73.dc1 area=wan -46:25.013 [INFO] TestAgent.server: Adding LAN server: server="Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73 (Addr: tcp/127.0.0.1:11422) (DC: dc1)" -46:25.013 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12024 network=udp -46:25.014 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12024 network=tcp -46:25.014 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12025 network=tcp -46:25.014 [INFO] TestAgent: started state syncer -46:25.014 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11367 network=tcp -46:25.054 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.054 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11422 [Candidate]" term=2 -46:25.073 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.073 [DEBUG] TestAgent.server.raft: vote granted: from=7907f154-f25c-1ea2-31fe-0197cbbd2a73 term=2 tally=1 -46:25.073 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.073 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11422 [Leader]" -46:25.073 [INFO] TestAgent.server: cluster leadership acquired -46:25.073 [INFO] TestAgent.server: New leader elected: payload=Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73 -46:25.078 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11422 -46:25.093 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.113 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.113 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.113 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.113 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.113 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.113 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73 -46:25.113 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7907f154-f25c-1ea2-31fe-0197cbbd2a73 -46:25.129 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.201 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.206 [INFO] TestAgent: Synced node info -46:25.350 [INFO] TestAgent: Requesting shutdown -46:25.350 [INFO] TestAgent.server: shutting down server -46:25.350 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.350 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.350 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.350 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.350 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.350 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.362Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.362Z [INFO] agent.router.manager: shutting down -46:25.362 [INFO] TestAgent: consul server down -46:25.362 [INFO] TestAgent: shutdown complete -46:25.362 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12024 network=tcp -46:25.362 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12024 network=udp -46:25.362 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12025 network=tcp -46:25.863 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.863 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_MeshGateway (0.91s) -=== CONT TestAgent_NodeMaintenance_Enable -2022-11-07T22:46:25.158Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.158Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.168Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.168Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d83a39c1-7c18-16a9-1545-81759df360bf Address:127.0.0.1:11380}]" -46:25.212 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11380 [Follower]" leader= -46:25.212 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d83a39c1-7c18-16a9-1545-81759df360bf.dc1 127.0.0.1 -46:25.213 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d83a39c1-7c18-16a9-1545-81759df360bf 127.0.0.1 -2022-11-07T22:46:25.213Z [INFO] agent.router: Initializing LAN area manager -46:25.213 [INFO] TestAgent.server: Adding LAN server: server="Node-d83a39c1-7c18-16a9-1545-81759df360bf (Addr: tcp/127.0.0.1:11380) (DC: dc1)" -46:25.213 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d83a39c1-7c18-16a9-1545-81759df360bf.dc1 area=wan -46:25.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11410 network=tcp -46:25.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11410 network=udp -46:25.213 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11411 network=tcp -46:25.213 [INFO] TestAgent: started state syncer -46:25.213 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11325 network=tcp -46:25.263 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.281 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.281 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11380 [Candidate]" term=2 -46:25.290 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.290 [DEBUG] TestAgent.server.raft: vote granted: from=d83a39c1-7c18-16a9-1545-81759df360bf term=2 tally=1 -46:25.290 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.290 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11380 [Leader]" -46:25.290 [INFO] TestAgent.server: cluster leadership acquired -46:25.290 [INFO] TestAgent.server: New leader elected: payload=Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.296 [INFO] TestAgent.server: initializing acls -46:25.299 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.300 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.304 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.313 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.313 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.313 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.313 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.313 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d83a39c1-7c18-16a9-1545-81759df360bf.dc1 -46:25.314 [INFO] TestAgent.server: Updating LAN server: server="Node-d83a39c1-7c18-16a9-1545-81759df360bf (Addr: tcp/127.0.0.1:11380) (DC: dc1)" -46:25.314 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d83a39c1-7c18-16a9-1545-81759df360bf.dc1 area=wan -46:25.323 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.340 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.340 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.340 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.340 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.340 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.340 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.340 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.344 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.349 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.352 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-d83a39c1-7c18-16a9-1545-81759df360bf -46:25.364 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.364 [INFO] TestAgent: Requesting shutdown -46:25.364 [INFO] TestAgent.server: shutting down server -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.364 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.364 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.364 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.364 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.365 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.365 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.365 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.365 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.365 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.368 [INFO] TestAgent: Synced node info -46:25.368 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.374Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.374Z [INFO] agent.router.manager: shutting down -46:25.374 [INFO] TestAgent: consul server down -46:25.374 [INFO] TestAgent: shutdown complete -46:25.374 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11410 network=tcp -46:25.374 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11410 network=udp -46:25.374 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11411 network=tcp -46:25.874 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.874 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_Clone (0.73s) -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} +=== CONT TestRemoteExecGetSpec_ACLDeny [WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:25.451Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.451Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.462Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.462Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.482 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5095b3e6-0293-0cb6-3828-4a3b06170440 Address:127.0.0.1:11345}]" -46:25.483 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11345 [Follower]" leader= -46:25.483 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5095b3e6-0293-0cb6-3828-4a3b06170440.dc1 127.0.0.1 -46:25.484 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5095b3e6-0293-0cb6-3828-4a3b06170440 127.0.0.1 -2022-11-07T22:46:25.484Z [INFO] agent.router: Initializing LAN area manager -46:25.484 [INFO] TestAgent.server: Adding LAN server: server="Node-5095b3e6-0293-0cb6-3828-4a3b06170440 (Addr: tcp/127.0.0.1:11345) (DC: dc1)" -46:25.484 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5095b3e6-0293-0cb6-3828-4a3b06170440.dc1 area=wan -46:25.484 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11326 network=udp -46:25.484 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11326 network=tcp -46:25.484 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11327 network=tcp -46:25.484 [INFO] TestAgent: started state syncer -46:25.484 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11318 network=tcp -46:25.534 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.551 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.551 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11345 [Candidate]" term=2 -46:25.561 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.561 [DEBUG] TestAgent.server.raft: vote granted: from=5095b3e6-0293-0cb6-3828-4a3b06170440 term=2 tally=1 -46:25.561 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.561 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11345 [Leader]" -46:25.561 [INFO] TestAgent.server: cluster leadership acquired -46:25.561 [INFO] TestAgent.server: New leader elected: payload=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.568 [INFO] TestAgent.server: initializing acls -46:25.572 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.572 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.577 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.589 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.589 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.589 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.589 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.589 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5095b3e6-0293-0cb6-3828-4a3b06170440.dc1 -46:25.589 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5095b3e6-0293-0cb6-3828-4a3b06170440.dc1 area=wan -46:25.589 [INFO] TestAgent.server: Updating LAN server: server="Node-5095b3e6-0293-0cb6-3828-4a3b06170440 (Addr: tcp/127.0.0.1:11345) (DC: dc1)" -46:25.600 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.620 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.620 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.620 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.620 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.620 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.620 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.620 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.624 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.631 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.633 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.634 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-5095b3e6-0293-0cb6-3828-4a3b06170440 -46:25.634 [INFO] TestAgent: Requesting shutdown -46:25.634 [INFO] TestAgent.server: shutting down server -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.634 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.634 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.634 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.634 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.634 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.642 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.647Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.647Z [INFO] agent.router.manager: shutting down -46:25.647 [INFO] TestAgent: consul server down -46:25.647 [INFO] TestAgent: shutdown complete -46:25.647 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11326 network=tcp -46:25.647 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11326 network=udp -46:25.647 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11327 network=tcp -46:26.148 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.148 [INFO] TestAgent: Endpoints down -=== RUN TestACL_Legacy_Get/right_id -46:21.010 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d 127.0.0.1 -2022-11-07T22:46:21.010Z [INFO] agent.router: Initializing LAN area manager -46:21.010 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11985 network=tcp -46:21.010 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11985 network=udp -46:21.010 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12014 network=tcp -46:21.010 [INFO] TestAgent: started state syncer -46:21.010 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11991 network=tcp -2022-11-07T22:46:21.011Z [WARN] agent.router.manager: No servers available -46:21.011 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No known Consul servers" -46:21.011 [DEBUG] TestAgent.client.memberlist.lan: memberlist: Stream connection from=127.0.0.1:38478 -46:21.011 [INFO] TestAgent.client.serf.lan: serf: EventMemberJoin: Node-823a7a65-d716-8058-7963-4e6605c03e0a 127.0.0.1 -46:21.012 [INFO] TestAgent.client: adding server: server="Node-823a7a65-d716-8058-7963-4e6605c03e0a (Addr: tcp/127.0.0.1:11955) (DC: dc1)" -46:21.012 [INFO] TestAgent.client: New leader elected: payload=Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.012 [INFO] TestAgent.client: client starting leave -46:21.187 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:21.187 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.386 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:21.386 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.387 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.587 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:21.587 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.587 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.787 [INFO] TestAgent.client.serf.lan: serf: EventMemberLeave: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d 127.0.0.1 -46:21.787 [DEBUG] TestAgent.client.serf.lan: serf: messageUserEventType: consul:new-leader -46:21.787 [DEBUG] TestAgent.client.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.787 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.787 [DEBUG] TestAgent.client.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:22.082 [INFO] TestAgent: Synced node info -46:22.082 [DEBUG] TestAgent: Node info in sync -46:23.868 [DEBUG] TestAgent: Node info in sync -46:25.211 [INFO] TestAgent: Requesting shutdown -46:25.211 [INFO] TestAgent.client: shutting down client -46:25.213 [INFO] TestAgent: consul client down -46:25.213 [INFO] TestAgent: shutdown complete -46:25.213 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11985 network=tcp -46:25.213 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11985 network=udp -46:25.213 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12014 network=tcp -46:25.713 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.713 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.744Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.757Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.785 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:823a7a65-d716-8058-7963-4e6605c03e0a Address:127.0.0.1:11955}]" -46:20.785 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11955 [Follower]" leader= -46:20.786 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-823a7a65-d716-8058-7963-4e6605c03e0a.dc1 127.0.0.1 -46:20.786 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-823a7a65-d716-8058-7963-4e6605c03e0a 127.0.0.1 -2022-11-07T22:46:20.786Z [INFO] agent.router: Initializing LAN area manager -46:20.786 [INFO] TestAgent.server: Adding LAN server: server="Node-823a7a65-d716-8058-7963-4e6605c03e0a (Addr: tcp/127.0.0.1:11955) (DC: dc1)" -46:20.786 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-823a7a65-d716-8058-7963-4e6605c03e0a.dc1 area=wan -46:20.786 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11929 network=udp -46:20.786 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11929 network=tcp -46:20.787 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11951 network=tcp -46:20.787 [INFO] TestAgent: started state syncer -46:20.787 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11949 network=tcp -46:20.833 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.833 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11955 [Candidate]" term=2 -46:20.851 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.851 [DEBUG] TestAgent.server.raft: vote granted: from=823a7a65-d716-8058-7963-4e6605c03e0a term=2 tally=1 -46:20.851 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.851 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11955 [Leader]" -46:20.851 [INFO] TestAgent.server: cluster leadership acquired -46:20.851 [INFO] TestAgent.server: New leader elected: payload=Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:20.860 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11955 -46:20.875 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.909 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.909 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.909 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.909 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.909 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.909 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:20.909 [INFO] TestAgent.server: member joined, marking health alive: member=Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:20.931 [INFO] TestAgent.server: federation state anti-entropy synced -46:21.011 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:12016] -46:21.011 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:12016 -46:21.012 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d 127.0.0.1 -46:21.012 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:21.012 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:21.012 [INFO] TestAgent.server: member joined, marking health alive: member=Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.081 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.087 [INFO] TestAgent: Synced node info -46:21.210 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.210 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.410 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.410 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.520 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:21.520 [DEBUG] TestAgent: Node info in sync -46:21.520 [DEBUG] TestAgent: Node info in sync -46:21.610 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.610 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.787 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-823a7a65-d716-8058-7963-4e6605c03e0a -46:21.787 [DEBUG] TestAgent.server.serf.lan: serf: messageLeaveType: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d -46:21.811 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d 127.0.0.1 -46:21.811 [INFO] TestAgent.server: deregistering member: member=Node-db2a653a-c7b3-d286-71d5-cf3c597d1c4d reason=left -46:25.713 [INFO] TestAgent: Requesting shutdown -46:25.713 [INFO] TestAgent.server: shutting down server -46:25.713 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.713 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.713 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.713 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.713 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.713 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.713 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.713 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.713 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.713 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.721 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.725Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.725Z [INFO] agent.router.manager: shutting down -46:25.725 [INFO] TestAgent: consul server down -46:25.725 [INFO] TestAgent: shutdown complete -46:25.725 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11929 network=tcp -46:25.725 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11929 network=udp -46:25.725 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11951 network=tcp -46:26.226 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.226 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Leave (5.49s) -=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{9_9_8_8_6_8_8_5_8_8_-5} -2022-11-07T22:46:25.499Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.499Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.509Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.509Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.532 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fccadfa4-9005-f9eb-f484-fa5315d4af69 Address:127.0.0.1:11408}]" -46:25.532 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11408 [Follower]" leader= -46:25.534 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fccadfa4-9005-f9eb-f484-fa5315d4af69.dc1 127.0.0.1 -46:25.534 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 127.0.0.1 -2022-11-07T22:46:25.534Z [INFO] agent.router: Initializing LAN area manager -46:25.534 [INFO] TestAgent.server: Adding LAN server: server="Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 (Addr: tcp/127.0.0.1:11408) (DC: dc1)" -46:25.534 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69.dc1 area=wan -46:25.534 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11319 network=udp -46:25.534 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11319 network=tcp -46:25.535 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11320 network=tcp -46:25.535 [INFO] TestAgent: started state syncer -46:25.535 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11402 network=tcp -46:25.584 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.586 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.586 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11408 [Candidate]" term=2 -46:25.601 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.601 [DEBUG] TestAgent.server.raft: vote granted: from=fccadfa4-9005-f9eb-f484-fa5315d4af69 term=2 tally=1 -46:25.601 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.601 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11408 [Leader]" -46:25.601 [INFO] TestAgent.server: cluster leadership acquired -46:25.601 [INFO] TestAgent.server: New leader elected: payload=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.609 [INFO] TestAgent.server: initializing acls -46:25.612 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.612 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.616 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.627 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.627 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.627 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.628 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.628 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-fccadfa4-9005-f9eb-f484-fa5315d4af69.dc1 -46:25.628 [INFO] TestAgent.server: Updating LAN server: server="Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 (Addr: tcp/127.0.0.1:11408) (DC: dc1)" -46:25.628 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69.dc1 area=wan -46:25.641 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.660 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.660 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.660 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.660 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.660 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.661 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.661 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.670 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.670 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.695 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-fccadfa4-9005-f9eb-f484-fa5315d4af69 -46:25.734 [INFO] TestAgent: Requesting shutdown -46:25.734 [INFO] TestAgent.server: shutting down server -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.734 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.734 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.734 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.734 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.734 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.738 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.741Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.741Z [INFO] agent.router.manager: shutting down -46:25.741 [INFO] TestAgent: consul server down -46:25.741 [INFO] TestAgent: shutdown complete -46:25.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11319 network=tcp -46:25.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11319 network=udp -46:25.741 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11320 network=tcp -46:26.242 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.242 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_List (0.75s) -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{9_9_8_8_6_8_8_5_8_8_-5} -2022-11-07T22:46:24.785Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.794Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.820 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c92bbe61-bb4f-b6c0-7e23-818602ea8515 Address:127.0.0.1:11309}]" -46:24.820 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11309 [Follower]" leader= -46:24.821 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 127.0.0.1 -46:24.821 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 127.0.0.1 -2022-11-07T22:46:24.821Z [INFO] agent.router: Initializing LAN area manager -46:24.821 [INFO] TestAgent.server: Adding LAN server: server="Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 (Addr: tcp/127.0.0.1:11309) (DC: dc2)" -46:24.821 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 area=wan -46:24.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11332 network=udp -46:24.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11332 network=tcp -46:24.822 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11333 network=tcp -46:24.822 [INFO] TestAgent: started state syncer -46:24.822 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11303 network=tcp -46:24.857 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.857 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11309 [Candidate]" term=2 -46:24.871 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.871 [DEBUG] TestAgent.server.raft: vote granted: from=c92bbe61-bb4f-b6c0-7e23-818602ea8515 term=2 tally=1 -46:24.871 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.871 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11309 [Leader]" -46:24.871 [INFO] TestAgent.server: cluster leadership acquired -46:24.871 [INFO] TestAgent.server: New leader elected: payload=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:24.872 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false -46:24.879 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false -46:24.879 [INFO] TestAgent.leader: started routine: routine="legacy ACL replication" -46:24.879 [INFO] TestAgent.server: started legacy ACL replication -46:24.879 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List -46:24.879 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" -46:24.879 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List -46:24.879 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" -46:24.880 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List -46:24.880 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" -46:24.880 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List -46:24.880 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" -46:24.880 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List -46:24.880 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" -46:24.886 [WARN] TestAgent.server.connect: primary datacenter is configured but unreachable - deferring initialization of the secondary datacenter CA -46:24.886 [INFO] TestAgent.leader: started routine: routine="config entry replication" -46:24.886 [INFO] TestAgent.leader: started routine: routine="federation state replication" -46:24.886 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.886 [INFO] TestAgent.leader: started routine: routine="secondary CA roots watch" -46:24.886 [INFO] TestAgent.leader: started routine: routine="intention replication" -46:24.886 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.886 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.886 [DEBUG] TestAgent.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 -46:24.886 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConnectCA.Roots -46:24.886 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:24.886 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [DEBUG] TestAgent.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 -46:24.886 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=Intention.List -46:24.886 [ERROR] TestAgent.server.connect: error replicating intentions: routine="intention replication" error="No path to datacenter" -46:24.886 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConfigEntry.ListAll -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [DEBUG] TestAgent.server: federation states are not enabled in the primary dc -46:24.886 [ERROR] TestAgent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: No path to datacenter" -46:24.894 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.GetPolicy -46:24.894 [WARN] TestAgent: Node info update blocked by ACLs: node=c92bbe61-bb4f-b6c0-7e23-818602ea8515 accessorID=legacy-token -46:24.972 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false -46:25.021 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.GetPolicy -46:25.021 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.021 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:11968] -46:25.021 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11968 -46:25.022 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 127.0.0.1 -46:25.022 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:25.022 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 area=wan -46:25.022 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.047 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.073 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.099 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.124 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.150 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.172 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL replication" -46:25.172 [INFO] TestAgent.leader: started routine: routine="ACL policy replication" -46:25.172 [INFO] TestAgent.leader: started routine: routine="ACL role replication" -46:25.172 [INFO] TestAgent.leader: started routine: routine="ACL token replication" -46:25.172 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL replication" -46:25.172 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.172 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:25.172 [INFO] TestAgent.server.replication.acl.policy: started ACL Policy replication -46:25.172 [INFO] TestAgent.server.replication.acl.role: started ACL Role replication -46:25.172 [INFO] TestAgent.server.replication.acl.token: started ACL Token replication -46:25.172 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.172 [INFO] TestAgent.server: Updating LAN server: server="Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 (Addr: tcp/127.0.0.1:11309) (DC: dc2)" -46:25.172 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 -46:25.172 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 area=wan -46:25.172 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515 -46:25.173 [DEBUG] TestAgent.server.replication.acl.policy: finished fetching acls: amount=2 -46:25.173 [DEBUG] TestAgent.server.replication.acl.policy: acl replication: local=0 remote=2 -46:25.173 [DEBUG] TestAgent.server.replication.acl.policy: acl replication: deletions=0 updates=2 -46:25.173 [DEBUG] TestAgent.server.replication.acl.token: finished fetching acls: amount=3 -46:25.173 [DEBUG] TestAgent.server.replication.acl.token: acl replication: local=0 remote=3 -46:25.173 [DEBUG] TestAgent.server.replication.acl.token: acl replication: deletions=0 updates=3 -46:25.173 [DEBUG] TestAgent.server.replication.acl.role: finished fetching acls: amount=0 -46:25.173 [DEBUG] TestAgent.server.replication.acl.role: acl replication: local=0 remote=0 -46:25.173 [DEBUG] TestAgent.server.replication.acl.role: acl replication: deletions=0 updates=0 -46:25.173 [DEBUG] TestAgent.server.replication.acl.role: ACL replication completed through remote index: index=1 -46:25.173 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - downloaded updates: amount=2 -46:25.174 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - performing updates -46:25.174 [DEBUG] TestAgent.server.replication.acl.token: acl replication - downloaded updates: amount=3 -46:25.174 [DEBUG] TestAgent.server.replication.acl.token: acl replication - performing updates -46:25.203 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - upserted batch: number_upserted=2 batch_size=675 -46:25.203 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - finished updates -46:25.203 [DEBUG] TestAgent.server.replication.acl.policy: ACL replication completed through remote index: index=16 -46:25.203 [DEBUG] TestAgent.server.replication.acl.token: acl replication - upserted batch: number_upserted=3 batch_size=442 -46:25.203 [DEBUG] TestAgent.server.replication.acl.token: acl replication - finished updates -46:25.203 [DEBUG] TestAgent.server.replication.acl.token: ACL replication completed through remote index: index=17 -46:25.238 [INFO] TestAgent: Requesting shutdown -46:25.238 [INFO] TestAgent.server: shutting down server -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="ACL role replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="ACL token replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="ACL policy replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="config entry replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="secondary CA roots watch" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="intention replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="federation state replication" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.238 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.238 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.238 [INFO] TestAgent.server.replication.config_entry: stopped replication -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="config entry replication" -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="intention replication" -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="secondary CA roots watch" -46:25.238 [INFO] TestAgent.replication.federation_state: stopped replication -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="federation state replication" -46:25.238 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.242 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.247Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.247Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.247Z [INFO] agent.router.manager: shutting down -46:25.247 [INFO] TestAgent: consul server down -46:25.247 [INFO] TestAgent: shutdown complete -46:25.247 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11332 network=tcp -46:25.247 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11332 network=udp -46:25.247 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11333 network=tcp -46:25.247 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11969 datacenter=dc1 method=ACL.RoleList error="rpc error making call: EOF" -46:25.247 [WARN] TestAgent.server.replication.acl.role: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL roles: rpc error making call: EOF" -46:25.247 [DEBUG] TestAgent.leader: stopped routine: routine="ACL role replication" -46:25.247 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11969 datacenter=dc1 method=ACL.TokenList error="rpc error making call: EOF" -46:25.247 [WARN] TestAgent.server.replication.acl.token: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL tokens: rpc error making call: EOF" -46:25.247 [DEBUG] TestAgent.leader: stopped routine: routine="ACL token replication" -46:25.247 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:11969 datacenter=dc1 method=ACL.PolicyList error="rpc error making call: EOF" -46:25.247 [WARN] TestAgent.server.replication.acl.policy: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL policies: rpc error making call: EOF" -46:25.247 [DEBUG] TestAgent.leader: stopped routine: routine="ACL policy replication" -46:25.747 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.747 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:24.407Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.416Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.438 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:503acb31-3d7d-81b9-91a8-926446181a13 Address:127.0.0.1:11969}]" -46:24.438 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11969 [Follower]" leader= -46:24.439 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 127.0.0.1 -46:24.439 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-503acb31-3d7d-81b9-91a8-926446181a13 127.0.0.1 -2022-11-07T22:46:24.439Z [INFO] agent.router: Initializing LAN area manager -46:24.439 [INFO] TestAgent.server: Adding LAN server: server="Node-503acb31-3d7d-81b9-91a8-926446181a13 (Addr: tcp/127.0.0.1:11969) (DC: dc1)" -46:24.439 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 area=wan -46:24.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11922 network=udp -46:24.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11922 network=tcp -46:24.440 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11965 network=tcp -46:24.440 [INFO] TestAgent: started state syncer -46:24.440 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11366 network=tcp -46:24.484 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.484 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11969 [Candidate]" term=2 -46:24.489 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:24.494 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.494 [DEBUG] TestAgent.server.raft: vote granted: from=503acb31-3d7d-81b9-91a8-926446181a13 term=2 tally=1 -46:24.494 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.494 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11969 [Leader]" -46:24.494 [INFO] TestAgent.server: cluster leadership acquired -46:24.494 [INFO] TestAgent.server: New leader elected: payload=Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:24.500 [INFO] TestAgent.server: initializing acls -46:24.505 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:24.514 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:24.533 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:24.533 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:24.533 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:24.533 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:24.533 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 -46:24.533 [INFO] TestAgent.server: Updating LAN server: server="Node-503acb31-3d7d-81b9-91a8-926446181a13 (Addr: tcp/127.0.0.1:11969) (DC: dc1)" -46:24.533 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-503acb31-3d7d-81b9-91a8-926446181a13.dc1 area=wan -46:24.550 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.568 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.568 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.568 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.568 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.568 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.568 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:24.568 [INFO] TestAgent.server: member joined, marking health alive: member=Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:24.580 [INFO] TestAgent: Synced node info -46:24.580 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:24.586 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.769 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-503acb31-3d7d-81b9-91a8-926446181a13 -46:25.021 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:60158 -46:25.022 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 127.0.0.1 -46:25.022 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c92bbe61-bb4f-b6c0-7e23-818602ea8515.dc2 area=wan -46:25.747 [INFO] TestAgent: Requesting shutdown -46:25.747 [INFO] TestAgent.server: shutting down server -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.747 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.747 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.747 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.747 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.754 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.758Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.758Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.758Z [INFO] agent.router.manager: shutting down -46:25.758 [INFO] TestAgent: consul server down -46:25.758 [INFO] TestAgent: shutdown complete -46:25.758 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11922 network=tcp -46:25.758 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11922 network=udp -46:25.758 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11965 network=tcp -46:26.258 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.258 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Authorize (1.86s) - --- PASS: TestACL_Authorize/master-token (0.00s) +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:06.053Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:06.088Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.220 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:143f3d21-1b65-f04c-7a23-9829293f5345 Address:127.0.0.1:29622}]" +32:06.220 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29622 [Follower]" leader= +32:06.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 127.0.0.1 +32:06.221 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-143f3d21-1b65-f04c-7a23-9829293f5345 127.0.0.1 +2021-10-05T16:32:06.221Z [INFO] agent.router: Initializing LAN area manager +32:06.221 [INFO] TestAgent.server: Adding LAN server: server="Node-143f3d21-1b65-f04c-7a23-9829293f5345 (Addr: tcp/127.0.0.1:29622) (DC: dc2)" +32:06.221 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 area=wan +32:06.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=udp +32:06.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29603 network=tcp +32:06.223 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29604 network=tcp +32:06.223 [INFO] TestAgent: started state syncer +32:06.223 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29623 network=tcp +32:06.264 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:06.264 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29622 [Candidate]" term=2 +32:06.272 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false +32:06.300 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:06.300 [DEBUG] TestAgent.server.raft: vote granted: from=143f3d21-1b65-f04c-7a23-9829293f5345 term=2 tally=1 +32:06.300 [INFO] TestAgent.server.raft: election won: tally=1 +32:06.300 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29622 [Leader]" +32:06.300 [INFO] TestAgent.server: cluster leadership acquired +32:06.301 [INFO] TestAgent.server: New leader elected: payload=Node-143f3d21-1b65-f04c-7a23-9829293f5345 +32:06.368 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false +32:06.368 [INFO] TestAgent.leader: started routine: routine="legacy ACL replication" +32:06.368 [INFO] TestAgent.server: started legacy ACL replication +32:06.371 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List +32:06.371 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" +32:06.371 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List +32:06.371 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" +32:06.371 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List +32:06.371 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" +32:06.371 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List +32:06.372 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" +32:06.372 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.List +32:06.372 [WARN] TestAgent.server.replication.acl.legacy: Legacy ACL replication error (will retry if still leader): error="failed to retrieve remote ACLs: No path to datacenter" +32:06.372 [WARN] TestAgent.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ACL.GetPolicy +32:06.372 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:29581] +32:06.372 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false +32:06.372 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:29581 +32:06.374 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 127.0.0.1 +32:06.374 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +32:06.374 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 area=wan +32:06.410 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false +32:06.492 [INFO] TestAgent.server.connect: received new intermediate certificate from primary datacenter +32:06.503 [INFO] TestAgent.server.connect: updated root certificates from primary datacenter +32:06.503 [INFO] TestAgent.server.connect: initialized secondary datacenter CA with provider: provider=consul +32:06.503 [INFO] TestAgent.leader: started routine: routine="config entry replication" +32:06.503 [INFO] TestAgent.leader: started routine: routine="federation state replication" +32:06.503 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:06.503 [INFO] TestAgent.leader: started routine: routine="secondary CA roots watch" +32:06.503 [INFO] TestAgent.leader: started routine: routine="intention replication" +32:06.503 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:06.503 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:06.504 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-143f3d21-1b65-f04c-7a23-9829293f5345 +32:06.504 [INFO] TestAgent.server: member joined, marking health alive: member=Node-143f3d21-1b65-f04c-7a23-9829293f5345 +32:06.504 [DEBUG] TestAgent.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 +32:06.504 [DEBUG] TestAgent.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 +32:06.505 [DEBUG] TestAgent.server.replication.config_entry: finished fetching config entries: amount=0 +32:06.505 [DEBUG] TestAgent.server.replication.config_entry: Config Entry replication: local=0 remote=0 +32:06.505 [DEBUG] TestAgent.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 +32:06.505 [DEBUG] TestAgent.server.replication.config_entry: replication completed through remote index: index=1 +32:06.505 [DEBUG] TestAgent.server: finished fetching remote objects: amount=1 +32:06.505 [DEBUG] TestAgent.server: diffing replication state: local_amount=0 remote_amount=1 +32:06.505 [DEBUG] TestAgent.server: diffed replication state: deletions=0 updates=1 +32:06.505 [DEBUG] TestAgent.server: performing updates: updates=1 +32:06.572 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL replication" +32:06.572 [INFO] TestAgent.leader: started routine: routine="ACL policy replication" +32:06.572 [INFO] TestAgent.leader: started routine: routine="ACL role replication" +32:06.572 [INFO] TestAgent.leader: started routine: routine="ACL token replication" +32:06.572 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:06.572 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:06.572 [INFO] TestAgent.server.replication.acl.policy: started ACL Policy replication +32:06.572 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-143f3d21-1b65-f04c-7a23-9829293f5345 +32:06.572 [INFO] TestAgent.server: Updating LAN server: server="Node-143f3d21-1b65-f04c-7a23-9829293f5345 (Addr: tcp/127.0.0.1:29622) (DC: dc2)" +32:06.573 [INFO] TestAgent.server.replication.acl.role: started ACL Role replication +32:06.573 [INFO] TestAgent.server.replication.acl.token: started ACL Token replication +32:06.573 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL replication" +32:06.573 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 +32:06.573 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 area=wan +32:06.573 [DEBUG] TestAgent.server.replication.acl.policy: finished fetching acls: amount=2 +32:06.573 [DEBUG] TestAgent.server.replication.acl.policy: acl replication: local=0 remote=2 +32:06.573 [DEBUG] TestAgent.server.replication.acl.policy: acl replication: deletions=0 updates=2 +32:06.574 [DEBUG] TestAgent.server.replication.acl.role: finished fetching acls: amount=0 +32:06.574 [DEBUG] TestAgent.server.replication.acl.role: acl replication: local=0 remote=0 +32:06.574 [DEBUG] TestAgent.server.replication.acl.role: acl replication: deletions=0 updates=0 +32:06.574 [DEBUG] TestAgent.server.replication.acl.role: ACL replication completed through remote index: index=1 +32:06.574 [DEBUG] TestAgent.server.replication.acl.token: finished fetching acls: amount=3 +32:06.574 [DEBUG] TestAgent.server.replication.acl.token: acl replication: local=0 remote=3 +32:06.574 [DEBUG] TestAgent.server.replication.acl.token: acl replication: deletions=0 updates=3 +32:06.574 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - downloaded updates: amount=2 +32:06.574 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - performing updates +32:06.575 [DEBUG] TestAgent.server.replication.acl.token: acl replication - downloaded updates: amount=3 +32:06.575 [DEBUG] TestAgent.server.replication.acl.token: acl replication - performing updates +32:06.631 [DEBUG] TestAgent.server: finished updates +32:06.631 [DEBUG] TestAgent.replication.federation_state: replication completed through remote index: index=14 +32:06.631 [INFO] TestAgent.server.connect: updated root certificates from primary datacenter +32:06.632 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-143f3d21-1b65-f04c-7a23-9829293f5345 +32:06.632 [INFO] TestAgent: Synced node info +32:06.632 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - upserted batch: number_upserted=2 batch_size=675 +32:06.632 [DEBUG] TestAgent.server.replication.acl.policy: acl replication - finished updates +32:06.632 [DEBUG] TestAgent.server.replication.acl.policy: ACL replication completed through remote index: index=15 +32:06.632 [DEBUG] TestAgent.server.replication.acl.token: acl replication - upserted batch: number_upserted=3 batch_size=442 +32:06.632 [DEBUG] TestAgent.server.replication.acl.token: acl replication - finished updates +32:06.632 [DEBUG] TestAgent.server.replication.acl.token: ACL replication completed through remote index: index=16 +32:06.647 [INFO] TestAgent: Requesting shutdown +32:06.647 [INFO] TestAgent.server: shutting down server +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="intention replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="ACL role replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="ACL policy replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="config entry replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="federation state replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="secondary CA roots watch" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="ACL token replication" +32:06.648 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:06.648 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:06.648 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:06.648 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:06.648 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:06.656 [INFO] TestAgent.server: federation state anti-entropy synced +32:06.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:06.656 [DEBUG] TestAgent.server: finished fetching remote objects: amount=2 +32:06.656 [INFO] TestAgent.replication.federation_state: stopped replication +32:06.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state replication" +32:06.676 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:06.684Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.684Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:06.684Z [INFO] agent.router.manager: shutting down +32:06.684 [INFO] TestAgent: consul server down +32:06.684 [INFO] TestAgent: shutdown complete +32:06.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=tcp +32:06.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29603 network=udp +32:06.684 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29604 network=tcp +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" +32:06.684 [ERROR] TestAgent.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="intention replication" +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" +32:06.684 [INFO] TestAgent.server.replication.config_entry: stopped replication +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="config entry replication" +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=ACL.RoleList error="rpc error making call: EOF" +32:06.684 [WARN] TestAgent.server.replication.acl.role: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL roles: rpc error making call: EOF" +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="ACL role replication" +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" +32:06.684 [ERROR] TestAgent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="secondary CA roots watch" +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=ACL.PolicyList error="rpc error making call: EOF" +32:06.684 [WARN] TestAgent.server.replication.acl.policy: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL policies: rpc error making call: EOF" +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="ACL policy replication" +32:06.684 [ERROR] TestAgent.server.rpc: RPC failed to server in DC: server=127.0.0.1:29638 datacenter=dc1 method=ACL.TokenList error="rpc error making call: EOF" +32:06.684 [WARN] TestAgent.server.replication.acl.token: ACL replication error (will retry if still leader): error="failed to retrieve remote ACL tokens: rpc error making call: EOF" +32:06.684 [DEBUG] TestAgent.leader: stopped routine: routine="ACL token replication" +32:07.184 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.184 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:05.601Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:05.633Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:05.697 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:73f30748-39f8-9ff3-06be-96a612992e72 Address:127.0.0.1:29638}]" +32:05.698 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 127.0.0.1 +32:05.699 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-73f30748-39f8-9ff3-06be-96a612992e72 127.0.0.1 +2021-10-05T16:32:05.699Z [INFO] agent.router: Initializing LAN area manager +32:05.699 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29577 network=udp +32:05.699 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29638 [Follower]" leader= +32:05.699 [INFO] TestAgent.server: Adding LAN server: server="Node-73f30748-39f8-9ff3-06be-96a612992e72 (Addr: tcp/127.0.0.1:29638) (DC: dc1)" +32:05.699 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 area=wan +32:05.700 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29577 network=tcp +32:05.700 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29578 network=tcp +32:05.700 [INFO] TestAgent: started state syncer +32:05.700 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30128 network=tcp +32:05.751 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:05.764 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:05.764 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29638 [Candidate]" term=2 +32:05.788 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:05.788 [DEBUG] TestAgent.server.raft: vote granted: from=73f30748-39f8-9ff3-06be-96a612992e72 term=2 tally=1 +32:05.788 [INFO] TestAgent.server.raft: election won: tally=1 +32:05.788 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29638 [Leader]" +32:05.788 [INFO] TestAgent.server: cluster leadership acquired +32:05.788 [INFO] TestAgent.server: New leader elected: payload=Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:05.797 [INFO] TestAgent.server: initializing acls +32:05.803 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:05.820 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:05.844 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:05.844 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:05.844 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:05.844 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:05.844 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 +32:05.845 [INFO] TestAgent.server: Updating LAN server: server="Node-73f30748-39f8-9ff3-06be-96a612992e72 (Addr: tcp/127.0.0.1:29638) (DC: dc1)" +32:05.845 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-73f30748-39f8-9ff3-06be-96a612992e72.dc1 area=wan +32:05.855 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:05.882 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:05.882 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:05.882 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:05.882 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:05.882 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:05.882 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:05.882 [INFO] TestAgent.server: member joined, marking health alive: member=Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:05.889 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:05.895 [INFO] TestAgent.server: federation state anti-entropy synced +32:05.964 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-73f30748-39f8-9ff3-06be-96a612992e72 +32:06.106 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:06.114 [INFO] TestAgent: Synced node info +32:06.372 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:49552 +32:06.373 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 127.0.0.1 +32:06.373 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-143f3d21-1b65-f04c-7a23-9829293f5345.dc2 area=wan +32:06.831 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:06.831 [DEBUG] TestAgent: Node info in sync +32:06.831 [DEBUG] TestAgent: Node info in sync +32:07.184 [INFO] TestAgent: Requesting shutdown +32:07.184 [INFO] TestAgent.server: shutting down server +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.184 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.184 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.185 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.185 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.200Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.200Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.200Z [INFO] agent.router.manager: shutting down +32:07.200 [INFO] TestAgent: consul server down +32:07.200 [INFO] TestAgent: shutdown complete +32:07.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29577 network=tcp +32:07.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29577 network=udp +32:07.200 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29578 network=tcp +32:07.701 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.701 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Authorize (2.16s) + --- PASS: TestACL_Authorize/master-token (0.01s) --- PASS: TestACL_Authorize/master-token/dc1 (0.00s) - --- PASS: TestACL_Authorize/master-token/dc2 (0.00s) + --- PASS: TestACL_Authorize/master-token/dc2 (0.01s) --- PASS: TestACL_Authorize/custom-token (0.00s) --- PASS: TestACL_Authorize/custom-token/dc1 (0.00s) --- PASS: TestACL_Authorize/custom-token/dc2 (0.00s) @@ -52431,3866 +52086,4442 @@ --- PASS: TestACL_Authorize/acl-not-found (0.00s) --- PASS: TestACL_Authorize/local-token-in-secondary-dc (0.00s) --- PASS: TestACL_Authorize/local-token-wrong-dc (0.00s) +=== CONT TestPProfHandlers_ACLs +2021-10-05T16:32:06.551Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.551Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:06.565Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.565Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.684 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e127a0a0-e28a-c9ca-9414-5420b5ceeee5 Address:127.0.0.1:29664}]" +32:06.684 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29664 [Follower]" leader= +32:06.685 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5.dc1 127.0.0.1 +32:06.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 127.0.0.1 +2021-10-05T16:32:06.692Z [INFO] agent.router: Initializing LAN area manager +32:06.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=udp +32:06.693 [INFO] TestAgent.server: Adding LAN server: server="Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 (Addr: tcp/127.0.0.1:29664) (DC: dc1)" +32:06.693 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5.dc1 area=wan +32:06.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29680 network=tcp +32:06.694 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29681 network=tcp +32:06.694 [INFO] TestAgent: started state syncer +32:06.694 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29665 network=tcp +32:06.744 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:06.744 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29664 [Candidate]" term=2 +32:06.745 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:06.846 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:06.957 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:06.957 [DEBUG] TestAgent.server.raft: vote granted: from=e127a0a0-e28a-c9ca-9414-5420b5ceeee5 term=2 tally=1 +32:06.957 [INFO] TestAgent.server.raft: election won: tally=1 +32:06.957 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29664 [Leader]" +32:06.958 [INFO] TestAgent.server: cluster leadership acquired +32:06.958 [INFO] TestAgent.server: New leader elected: payload=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:06.979 [INFO] TestAgent.server: initializing acls +32:06.996 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:06.996 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.007 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.017 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.017 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.017 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.017 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.017 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5.dc1 +32:07.018 [INFO] TestAgent.server: Updating LAN server: server="Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 (Addr: tcp/127.0.0.1:29664) (DC: dc1)" +32:07.018 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5.dc1 area=wan +32:07.035 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.076 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.076 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.076 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.076 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.076 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.076 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.076 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.090 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.112 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.214 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.214 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-e127a0a0-e28a-c9ca-9414-5420b5ceeee5 +32:07.214 [INFO] TestAgent: Requesting shutdown +32:07.214 [INFO] TestAgent.server: shutting down server +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.214 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.214 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.215 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.215 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.215 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.220 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.220Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.220Z [INFO] agent.router.manager: shutting down +32:07.220 [INFO] TestAgent: consul server down +32:07.220 [INFO] TestAgent: shutdown complete +32:07.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=tcp +32:07.221 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29680 network=udp +32:07.221 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29681 network=tcp +32:07.721 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.721 [INFO] TestAgent: Endpoints down +--- PASS: TestACLReplicationStatus (1.18s) +=== CONT TestACL_Legacy_Destroy +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:06.769Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.769Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:06.789Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.789Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.981 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:032756d7-8b7c-48e5-0526-1d346e53a8ab Address:127.0.0.1:29678}]" +32:06.981 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-032756d7-8b7c-48e5-0526-1d346e53a8ab.dc1 127.0.0.1 +32:06.982 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-032756d7-8b7c-48e5-0526-1d346e53a8ab 127.0.0.1 +2021-10-05T16:32:06.982Z [INFO] agent.router: Initializing LAN area manager +32:06.982 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=udp +32:06.982 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29678 [Follower]" leader= +32:06.983 [INFO] TestAgent.server: Adding LAN server: server="Node-032756d7-8b7c-48e5-0526-1d346e53a8ab (Addr: tcp/127.0.0.1:29678) (DC: dc1)" +32:06.983 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab.dc1 area=wan +32:06.983 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29652 network=tcp +32:06.984 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29653 network=tcp +32:06.984 [INFO] TestAgent: started state syncer +32:06.984 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29679 network=tcp +32:07.023 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.023 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29678 [Candidate]" term=2 +32:07.033 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:07.038 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.038 [DEBUG] TestAgent.server.raft: vote granted: from=032756d7-8b7c-48e5-0526-1d346e53a8ab term=2 tally=1 +32:07.038 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.038 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29678 [Leader]" +32:07.038 [INFO] TestAgent.server: cluster leadership acquired +32:07.038 [INFO] TestAgent.server: New leader elected: payload=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.051 [INFO] TestAgent.server: initializing acls +32:07.054 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.054 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.070 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.086 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.086 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.086 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.086 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.086 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-032756d7-8b7c-48e5-0526-1d346e53a8ab.dc1 +32:07.087 [INFO] TestAgent.server: Updating LAN server: server="Node-032756d7-8b7c-48e5-0526-1d346e53a8ab (Addr: tcp/127.0.0.1:29678) (DC: dc1)" +32:07.087 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab.dc1 area=wan +32:07.120 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.182 [INFO] TestAgent: Synced node info +32:07.182 [DEBUG] TestAgent: Node info in sync +32:07.182 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.182 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.182 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.182 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.182 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.182 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.182 [INFO] TestAgent.server: member joined, marking health alive: member=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.196 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.233 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-032756d7-8b7c-48e5-0526-1d346e53a8ab +32:07.280 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.343 [INFO] TestAgent: Requesting shutdown +32:07.343 [INFO] TestAgent.server: shutting down server +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.343 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.343 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.343 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.343 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.349 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.355Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.355Z [INFO] agent.router.manager: shutting down +32:07.355 [INFO] TestAgent: consul server down +32:07.355 [INFO] TestAgent: shutdown complete +32:07.355 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=tcp +32:07.355 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29652 network=udp +32:07.355 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29653 network=tcp +32:07.855 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.855 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecWrites_ACLAgentToken (1.11s) +=== CONT TestACL_Legacy_Get +=== RUN TestACL_Legacy_Get/wrong_id +2021-10-05T16:32:06.990Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.990Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.007Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.008Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.042 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:209ddb91-a166-c1f9-90fb-a5d4444c859f Address:127.0.0.1:29643}]" +32:07.043 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 127.0.0.1 +32:07.043 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f 127.0.0.1 +2021-10-05T16:32:07.043Z [INFO] agent.router: Initializing LAN area manager +32:07.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=udp +32:07.044 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29643 [Follower]" leader= +32:07.044 [INFO] TestAgent.server: Adding LAN server: server="Node-209ddb91-a166-c1f9-90fb-a5d4444c859f (Addr: tcp/127.0.0.1:29643) (DC: dc1)" +32:07.044 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 area=wan +32:07.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29631 network=tcp +32:07.045 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29688 network=tcp +32:07.045 [INFO] TestAgent: started state syncer +32:07.045 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29644 network=tcp +32:07.094 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:07.104 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.104 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29643 [Candidate]" term=2 +32:07.148 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.148 [DEBUG] TestAgent.server.raft: vote granted: from=209ddb91-a166-c1f9-90fb-a5d4444c859f term=2 tally=1 +32:07.148 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.148 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29643 [Leader]" +32:07.152 [INFO] TestAgent.server: cluster leadership acquired +32:07.152 [INFO] TestAgent.server: New leader elected: payload=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.180 [INFO] TestAgent.server: initializing acls +32:07.196 [INFO] TestAgent.server: initializing acls +32:07.199 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.199 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.212 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.212 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.225 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.230 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.230 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.230 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.230 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.230 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 +32:07.231 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.231 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:07.232 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.232 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 +32:07.232 [INFO] TestAgent.server: Updating LAN server: server="Node-209ddb91-a166-c1f9-90fb-a5d4444c859f (Addr: tcp/127.0.0.1:29643) (DC: dc1)" +32:07.232 [INFO] TestAgent.server: Updating LAN server: server="Node-209ddb91-a166-c1f9-90fb-a5d4444c859f (Addr: tcp/127.0.0.1:29643) (DC: dc1)" +32:07.232 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 area=wan +32:07.232 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f.dc1 area=wan +32:07.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.341 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.341 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.341 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.341 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.341 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.342 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.342 [INFO] TestAgent.server: member joined, marking health alive: member=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.352 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.352 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.361 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-209ddb91-a166-c1f9-90fb-a5d4444c859f +32:07.368 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.405 [INFO] TestAgent: Requesting shutdown +32:07.405 [INFO] TestAgent.server: shutting down server +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.405 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.405 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.405 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.405 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.405 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.408 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.417Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.417Z [INFO] agent.router.manager: shutting down +32:07.418 [INFO] TestAgent: consul server down +32:07.418 [INFO] TestAgent: shutdown complete +32:07.418 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=tcp +32:07.418 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29631 network=udp +32:07.418 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29688 network=tcp +32:07.918 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.918 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecWrites_ACLToken (0.95s) +=== CONT TestACL_Legacy_Clone +2021-10-05T16:32:06.758Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.758Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:06.789Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:06.789Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:06.993 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c727f468-e518-4048-8b1e-6871f96b6250 Address:127.0.0.1:29657}]" +32:06.993 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29657 [Follower]" leader= +32:06.994 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c727f468-e518-4048-8b1e-6871f96b6250.dc1 127.0.0.1 +32:06.995 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c727f468-e518-4048-8b1e-6871f96b6250 127.0.0.1 +2021-10-05T16:32:06.995Z [INFO] agent.router: Initializing LAN area manager +32:06.995 [INFO] TestAgent.server: Adding LAN server: server="Node-c727f468-e518-4048-8b1e-6871f96b6250 (Addr: tcp/127.0.0.1:29657) (DC: dc1)" +32:06.995 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c727f468-e518-4048-8b1e-6871f96b6250.dc1 area=wan +32:06.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=udp +32:06.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29687 network=tcp +32:06.996 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29646 network=tcp +32:06.996 [INFO] TestAgent: started state syncer +32:06.996 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29658 network=tcp +32:07.047 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:07.056 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.056 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29657 [Candidate]" term=2 +32:07.079 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.079 [DEBUG] TestAgent.server.raft: vote granted: from=c727f468-e518-4048-8b1e-6871f96b6250 term=2 tally=1 +32:07.079 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.079 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29657 [Leader]" +32:07.079 [INFO] TestAgent.server: cluster leadership acquired +32:07.080 [INFO] TestAgent.server: New leader elected: payload=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.088 [INFO] TestAgent.server: initializing acls +32:07.093 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.093 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.104 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.134 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.134 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.134 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.134 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.134 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c727f468-e518-4048-8b1e-6871f96b6250.dc1 +32:07.136 [INFO] TestAgent.server: Updating LAN server: server="Node-c727f468-e518-4048-8b1e-6871f96b6250 (Addr: tcp/127.0.0.1:29657) (DC: dc1)" +32:07.136 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c727f468-e518-4048-8b1e-6871f96b6250.dc1 area=wan +32:07.157 [INFO] TestAgent: Synced node info +32:07.157 [DEBUG] TestAgent: Node info in sync +32:07.157 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.207 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.217 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.432 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.432 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c727f468-e518-4048-8b1e-6871f96b6250 +32:07.460 [INFO] TestAgent: Requesting shutdown +32:07.460 [INFO] TestAgent.server: shutting down server +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.460 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.460 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.460 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.460 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.467 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.479Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.479Z [INFO] agent.router.manager: shutting down +32:07.479 [INFO] TestAgent: consul server down +32:07.479 [INFO] TestAgent: shutdown complete +32:07.479 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=tcp +32:07.479 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29687 network=udp +32:07.479 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29646 network=tcp +32:07.980 [INFO] TestAgent: Waiting for endpoints to shut down +32:07.980 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecGetSpec_ACLAgentToken (1.26s) +=== CONT TestPreparedQuery_Integration +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestPProfHandlers_ACLs/case_0_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) +=== RUN TestPProfHandlers_ACLs/case_1_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/heap",_nilResponse:true}) +=== RUN TestPProfHandlers_ACLs/case_2_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/",_nilResponse:true}) +=== RUN TestPProfHandlers_ACLs/case_3_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"",_endpoint:"/debug/pprof/",_nilResponse:true}) +=== RUN TestPProfHandlers_ACLs/case_4_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) +2021-10-05T16:32:07.299Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.299Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.356Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.356Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.401 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0f1d3741-9147-1f69-7b43-1db6650245ee Address:127.0.0.1:29727}]" +32:07.402 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0f1d3741-9147-1f69-7b43-1db6650245ee.dc1 127.0.0.1 +32:07.403 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0f1d3741-9147-1f69-7b43-1db6650245ee 127.0.0.1 +2021-10-05T16:32:07.403Z [INFO] agent.router: Initializing LAN area manager +32:07.403 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29727 [Follower]" leader= +32:07.403 [INFO] TestAgent.server: Adding LAN server: server="Node-0f1d3741-9147-1f69-7b43-1db6650245ee (Addr: tcp/127.0.0.1:29727) (DC: dc1)" +32:07.403 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0f1d3741-9147-1f69-7b43-1db6650245ee.dc1 area=wan +32:07.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=udp +32:07.403 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29701 network=tcp +32:07.404 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29702 network=tcp +32:07.404 [INFO] TestAgent: started state syncer +32:07.404 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29728 network=tcp +32:07.456 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:07.468 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.468 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29727 [Candidate]" term=2 +32:07.501 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.501 [DEBUG] TestAgent.server.raft: vote granted: from=0f1d3741-9147-1f69-7b43-1db6650245ee term=2 tally=1 +32:07.501 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.501 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29727 [Leader]" +32:07.503 [INFO] TestAgent.server: cluster leadership acquired +32:07.503 [INFO] TestAgent.server: New leader elected: payload=Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.511 [INFO] TestAgent.server: initializing acls +32:07.516 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.516 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.528 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.541 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.541 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.541 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.542 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.542 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0f1d3741-9147-1f69-7b43-1db6650245ee.dc1 +32:07.542 [INFO] TestAgent.server: Updating LAN server: server="Node-0f1d3741-9147-1f69-7b43-1db6650245ee (Addr: tcp/127.0.0.1:29727) (DC: dc1)" +32:07.542 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0f1d3741-9147-1f69-7b43-1db6650245ee.dc1 area=wan +32:07.558 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.578 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.578 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.578 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.578 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.578 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.578 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.578 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.585 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.594 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.594 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0f1d3741-9147-1f69-7b43-1db6650245ee +32:07.606 [ERROR] TestAgent: failed to get remote exec job: error="Permission denied" +32:07.610 [INFO] TestAgent: Requesting shutdown +32:07.610 [INFO] TestAgent.server: shutting down server +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.610 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.610 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.610 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.610 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.610 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.615 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.621Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.621Z [INFO] agent.router.manager: shutting down +32:07.621 [INFO] TestAgent: consul server down +32:07.621 [INFO] TestAgent: shutdown complete +32:07.621 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=tcp +32:07.621 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29701 network=udp +32:07.621 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29702 network=tcp +32:08.124 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.124 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecGetSpec_ACLDeny (0.88s) +=== CONT TestRemoteExecGetSpec_ACLToken +2021-10-05T16:32:07.237Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.256Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.357 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:379b6c11-e4e8-21a2-0133-827c134161a3 Address:127.0.0.1:29699}]" +32:07.358 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-379b6c11-e4e8-21a2-0133-827c134161a3.dc1 127.0.0.1 +32:07.358 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29699 [Follower]" leader= +32:07.359 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-379b6c11-e4e8-21a2-0133-827c134161a3 127.0.0.1 +2021-10-05T16:32:07.359Z [INFO] agent.router: Initializing LAN area manager +32:07.359 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-379b6c11-e4e8-21a2-0133-827c134161a3.dc1 area=wan +32:07.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=udp +32:07.359 [INFO] TestAgent.server: Adding LAN server: server="Node-379b6c11-e4e8-21a2-0133-827c134161a3 (Addr: tcp/127.0.0.1:29699) (DC: dc1)" +32:07.359 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29694 network=tcp +32:07.360 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29695 network=tcp +32:07.360 [INFO] TestAgent: started state syncer +32:07.360 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29700 network=tcp +32:07.404 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.404 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29699 [Candidate]" term=2 +32:07.452 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.452 [DEBUG] TestAgent.server.raft: vote granted: from=379b6c11-e4e8-21a2-0133-827c134161a3 term=2 tally=1 +32:07.452 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.452 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29699 [Leader]" +32:07.452 [INFO] TestAgent.server: cluster leadership acquired +32:07.452 [INFO] TestAgent.server: New leader elected: payload=Node-379b6c11-e4e8-21a2-0133-827c134161a3 +32:07.479 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29699 +32:07.506 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.536 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.536 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.536 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.536 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.536 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.536 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-379b6c11-e4e8-21a2-0133-827c134161a3 +32:07.536 [INFO] TestAgent.server: member joined, marking health alive: member=Node-379b6c11-e4e8-21a2-0133-827c134161a3 +32:07.554 [INFO] TestAgent.server: federation state anti-entropy synced +32:07.586 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:07.591 [INFO] TestAgent: Synced node info +32:07.591 [DEBUG] TestAgent: Node info in sync +32:07.641 [INFO] TestAgent: Requesting shutdown +32:07.641 [INFO] TestAgent.server: shutting down server +32:07.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:07.641 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:07.641 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:07.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:07.641 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:07.641 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:07.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:07.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:07.641 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:07.641 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:07.645 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:07.649Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:07.649Z [INFO] agent.router.manager: shutting down +32:07.649 [INFO] TestAgent: consul server down +32:07.649 [INFO] TestAgent: shutdown complete +32:07.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=tcp +32:07.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29694 network=udp +32:07.649 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29695 network=tcp +32:08.149 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.149 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecWrites (0.96s) +=== CONT TestRemoteExecGetSpec +=== RUN TestPProfHandlers_ACLs/case_5_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"towel",_endpoint:"/debug/pprof/heap",_nilResponse:true}) +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:07.881Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.881Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.911Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.911Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.970 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:72e17581-ebfa-46f3-a382-5fb1c49046a6 Address:127.0.0.1:29692}]" +32:07.970 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 127.0.0.1 +32:07.971 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 127.0.0.1 +2021-10-05T16:32:07.971Z [INFO] agent.router: Initializing LAN area manager +32:07.971 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29692 [Follower]" leader= +32:07.972 [INFO] TestAgent.server: Adding LAN server: server="Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 (Addr: tcp/127.0.0.1:29692) (DC: dc1)" +32:07.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=tcp +32:07.973 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29715 network=udp +32:07.974 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29597 network=tcp +32:07.974 [INFO] TestAgent: started state syncer +32:07.974 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29693 network=tcp +32:07.974 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 area=wan +32:08.025 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:08.030 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.030 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29692 [Candidate]" term=2 +32:08.064 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.064 [DEBUG] TestAgent.server.raft: vote granted: from=72e17581-ebfa-46f3-a382-5fb1c49046a6 term=2 tally=1 +32:08.064 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.064 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29692 [Leader]" +32:08.064 [INFO] TestAgent.server: cluster leadership acquired +32:08.064 [INFO] TestAgent.server: New leader elected: payload=Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 +32:08.096 [INFO] TestAgent.server: initializing acls +32:08.108 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:08.108 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:08.119 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:08.125 [INFO] TestAgent.server: initializing acls +32:08.125 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:08.133 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:08.133 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:08.133 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:08.133 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 +32:08.133 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 +32:08.134 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 area=wan +32:08.134 [INFO] TestAgent.server: Updating LAN server: server="Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 (Addr: tcp/127.0.0.1:29692) (DC: dc1)" +32:08.138 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:08.138 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:08.139 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 +32:08.139 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 +32:08.139 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-72e17581-ebfa-46f3-a382-5fb1c49046a6.dc1 area=wan +32:08.139 [INFO] TestAgent.server: Updating LAN server: server="Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 (Addr: tcp/127.0.0.1:29692) (DC: dc1)" +32:08.163 [INFO] TestAgent: Synced node info +32:08.163 [DEBUG] TestAgent: Node info in sync +32:08.163 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.175 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-72e17581-ebfa-46f3-a382-5fb1c49046a6 +32:08.175 [INFO] TestAgent: Requesting shutdown +32:08.175 [INFO] TestAgent.server: shutting down server +32:08.175 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:08.175 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:08.175 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.176 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:08.176 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:08.181 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.188Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.188Z [INFO] agent.router.manager: shutting down +32:08.192 [INFO] TestAgent: consul server down +32:08.192 [INFO] TestAgent: shutdown complete +32:08.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=tcp +32:08.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29715 network=udp +32:08.192 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29597 network=tcp +32:08.192 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +32:08.192 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +32:08.696 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.696 [INFO] TestAgent: Endpoints down +=== RUN TestACL_Legacy_Get/right_id +2021-10-05T16:32:07.719Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.719Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.757Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.757Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.822 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c23bcce3-d141-7f24-3254-70af31166101 Address:127.0.0.1:29720}]" +32:07.822 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29720 [Follower]" leader= +32:07.822 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 127.0.0.1 +32:07.823 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c23bcce3-d141-7f24-3254-70af31166101 127.0.0.1 +2021-10-05T16:32:07.823Z [INFO] agent.router: Initializing LAN area manager +32:07.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=udp +32:07.823 [INFO] TestAgent.server: Adding LAN server: server="Node-c23bcce3-d141-7f24-3254-70af31166101 (Addr: tcp/127.0.0.1:29720) (DC: dc1)" +32:07.823 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 area=wan +32:07.824 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29596 network=tcp +32:07.824 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29716 network=tcp +32:07.824 [INFO] TestAgent: started state syncer +32:07.825 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29721 network=tcp +32:07.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29720 [Candidate]" term=2 +32:07.875 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:07.920 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.920 [DEBUG] TestAgent.server.raft: vote granted: from=c23bcce3-d141-7f24-3254-70af31166101 term=2 tally=1 +32:07.920 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.920 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29720 [Leader]" +32:07.920 [INFO] TestAgent.server: cluster leadership acquired +32:07.920 [INFO] TestAgent.server: New leader elected: payload=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:07.944 [INFO] TestAgent.server: initializing acls +32:07.958 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.958 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.972 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.975 [INFO] TestAgent.server: initializing acls +32:07.975 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.996 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.996 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.996 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.996 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:07.996 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.996 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c23bcce3-d141-7f24-3254-70af31166101 +32:07.996 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 +32:07.996 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 area=wan +32:07.996 [INFO] TestAgent.server: Updating LAN server: server="Node-c23bcce3-d141-7f24-3254-70af31166101 (Addr: tcp/127.0.0.1:29720) (DC: dc1)" +32:07.997 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c23bcce3-d141-7f24-3254-70af31166101 +32:07.997 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 +32:07.997 [INFO] TestAgent.server: Updating LAN server: server="Node-c23bcce3-d141-7f24-3254-70af31166101 (Addr: tcp/127.0.0.1:29720) (DC: dc1)" +32:07.997 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c23bcce3-d141-7f24-3254-70af31166101.dc1 area=wan +32:08.020 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +32:08.021 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.066 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:08.066 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:08.066 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:08.066 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:08.066 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:08.066 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:08.066 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:08.072 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:08.073 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:08.081 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.089 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-c23bcce3-d141-7f24-3254-70af31166101 +32:08.189 [INFO] TestAgent: Requesting shutdown +32:08.189 [INFO] TestAgent.server: shutting down server +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.189 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.189 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.189 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.189 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.196 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.200Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.200Z [INFO] agent.router.manager: shutting down +32:08.200 [INFO] TestAgent: consul server down +32:08.200 [INFO] TestAgent: shutdown complete +32:08.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=tcp +32:08.200 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29596 network=udp +32:08.200 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29716 network=tcp +32:08.704 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.704 [INFO] TestAgent: Endpoints down +--- PASS: TestPProfHandlers_ACLs (1.00s) + --- PASS: TestPProfHandlers_ACLs/case_0_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) (0.03s) + --- PASS: TestPProfHandlers_ACLs/case_1_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/heap",_nilResponse:true}) (0.00s) + --- PASS: TestPProfHandlers_ACLs/case_2_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"agent",_endpoint:"/debug/pprof/",_nilResponse:true}) (0.00s) + --- PASS: TestPProfHandlers_ACLs/case_3_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"",_endpoint:"/debug/pprof/",_nilResponse:true}) (0.00s) + --- PASS: TestPProfHandlers_ACLs/case_4_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:200,_token:"master",_endpoint:"/debug/pprof/heap",_nilResponse:false}) (0.07s) + --- PASS: TestPProfHandlers_ACLs/case_5_(struct_{_code_int;_token_string;_endpoint_string;_nilResponse_bool_}{code:403,_token:"towel",_endpoint:"/debug/pprof/heap",_nilResponse:true}) (0.00s) +=== CONT TestOperator_RaftConfiguration +2021-10-05T16:32:07.736Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.736Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.751Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.751Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:07.787 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bcd6577b-b23f-0971-79ce-50199002d9e5 Address:127.0.0.1:29713}]" +32:07.788 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29713 [Follower]" leader= +32:07.788 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 127.0.0.1 +32:07.789 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bcd6577b-b23f-0971-79ce-50199002d9e5 127.0.0.1 +2021-10-05T16:32:07.789Z [INFO] agent.router: Initializing LAN area manager +32:07.789 [INFO] TestAgent.server: Adding LAN server: server="Node-bcd6577b-b23f-0971-79ce-50199002d9e5 (Addr: tcp/127.0.0.1:29713) (DC: dc1)" +32:07.789 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 area=wan +32:07.789 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=udp +32:07.789 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29743 network=tcp +32:07.790 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29737 network=tcp +32:07.790 [INFO] TestAgent: started state syncer +32:07.790 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29714 network=tcp +32:07.825 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:07.825 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29713 [Candidate]" term=2 +32:07.837 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:07.837 [DEBUG] TestAgent.server.raft: vote granted: from=bcd6577b-b23f-0971-79ce-50199002d9e5 term=2 tally=1 +32:07.837 [INFO] TestAgent.server.raft: election won: tally=1 +32:07.837 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29713 [Leader]" +32:07.837 [INFO] TestAgent.server: cluster leadership acquired +32:07.838 [INFO] TestAgent.server: New leader elected: payload=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.840 [INFO] TestAgent.server: initializing acls +32:07.842 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +32:07.844 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:07.844 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.845 [INFO] TestAgent.server: initializing acls +32:07.845 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:07.851 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.867 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:07.872 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.872 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:07.872 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:07.872 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:07.872 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.872 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 +32:07.872 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 area=wan +32:07.872 [INFO] TestAgent.server: Updating LAN server: server="Node-bcd6577b-b23f-0971-79ce-50199002d9e5 (Addr: tcp/127.0.0.1:29713) (DC: dc1)" +32:07.877 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:07.877 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.877 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 +32:07.877 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bcd6577b-b23f-0971-79ce-50199002d9e5.dc1 area=wan +32:07.877 [INFO] TestAgent.server: Updating LAN server: server="Node-bcd6577b-b23f-0971-79ce-50199002d9e5 (Addr: tcp/127.0.0.1:29713) (DC: dc1)" +32:07.880 [INFO] TestAgent: Synced node info +32:07.880 [DEBUG] TestAgent: Node info in sync +32:07.904 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:07.956 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:07.956 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:07.956 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:07.956 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:07.956 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:07.956 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.956 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.976 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.976 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:07.985 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.228 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:08.228 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-bcd6577b-b23f-0971-79ce-50199002d9e5 +32:08.264 [INFO] TestAgent: Requesting shutdown +32:08.264 [INFO] TestAgent.server: shutting down server +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:08.264 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.264 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.264 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:08.264 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:08.271 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.275Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.275Z [INFO] agent.router.manager: shutting down +32:08.275 [INFO] TestAgent: consul server down +32:08.275 [INFO] TestAgent: shutdown complete +32:08.275 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=tcp +32:08.275 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29743 network=udp +32:08.275 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29737 network=tcp +32:08.776 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.776 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_Destroy (1.06s) +=== CONT TestACL_Legacy_UpdateUpsert +[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:07.937Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.937Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:07.952Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:07.952Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.045 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f9171f1-1169-b416-19fe-baaf29ee5728 Address:127.0.0.1:29748}]" +32:08.046 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f9171f1-1169-b416-19fe-baaf29ee5728.dc1 127.0.0.1 +32:08.046 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29748 [Follower]" leader= +32:08.047 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f9171f1-1169-b416-19fe-baaf29ee5728 127.0.0.1 +2021-10-05T16:32:08.047Z [INFO] agent.router: Initializing LAN area manager +32:08.048 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=udp +32:08.048 [INFO] TestAgent.server: Adding LAN server: server="Node-8f9171f1-1169-b416-19fe-baaf29ee5728 (Addr: tcp/127.0.0.1:29748) (DC: dc1)" +32:08.048 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f9171f1-1169-b416-19fe-baaf29ee5728.dc1 area=wan +32:08.048 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29736 network=tcp +32:08.049 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29730 network=tcp +32:08.049 [INFO] TestAgent: started state syncer +32:08.049 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29749 network=tcp +32:08.098 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:08.107 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.107 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29748 [Candidate]" term=2 +32:08.125 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.125 [DEBUG] TestAgent.server.raft: vote granted: from=8f9171f1-1169-b416-19fe-baaf29ee5728 term=2 tally=1 +32:08.125 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.125 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29748 [Leader]" +32:08.125 [INFO] TestAgent.server: cluster leadership acquired +32:08.125 [INFO] TestAgent.server: New leader elected: payload=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.132 [INFO] TestAgent.server: initializing acls +32:08.136 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:08.136 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:08.144 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:08.164 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:08.164 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:08.164 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:08.164 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.164 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8f9171f1-1169-b416-19fe-baaf29ee5728.dc1 +32:08.165 [INFO] TestAgent.server: Updating LAN server: server="Node-8f9171f1-1169-b416-19fe-baaf29ee5728 (Addr: tcp/127.0.0.1:29748) (DC: dc1)" +32:08.165 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8f9171f1-1169-b416-19fe-baaf29ee5728.dc1 area=wan +32:08.178 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.215 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:08.215 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:08.215 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:08.215 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:08.215 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:08.215 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.215 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.223 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.230 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.281 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:08.289 [INFO] TestAgent: Synced node info +32:08.289 [DEBUG] TestAgent: Node info in sync +32:08.344 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:08.344 [DEBUG] TestAgent: Node info in sync +32:08.384 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.384 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-8f9171f1-1169-b416-19fe-baaf29ee5728 +32:08.416 [INFO] TestAgent: Requesting shutdown +32:08.416 [INFO] TestAgent.server: shutting down server +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:08.416 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:08.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:08.416 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:08.424 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.436Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.436Z [INFO] agent.router.manager: shutting down +32:08.436 [INFO] TestAgent: consul server down +32:08.436 [INFO] TestAgent: shutdown complete +32:08.436 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=tcp +32:08.436 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29736 network=udp +32:08.436 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29730 network=tcp +32:08.940 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.940 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_Clone (1.02s) +=== CONT TestPreparedQuery_Delete +2021-10-05T16:32:08.018Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.057Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.144 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:818e7689-26da-a4f9-f658-958765f80de5 Address:127.0.0.1:29755}]" +32:08.144 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29755 [Follower]" leader= +32:08.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-818e7689-26da-a4f9-f658-958765f80de5.dc1 127.0.0.1 +32:08.150 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-818e7689-26da-a4f9-f658-958765f80de5 127.0.0.1 +2021-10-05T16:32:08.150Z [INFO] agent.router: Initializing LAN area manager +32:08.151 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=udp +32:08.151 [INFO] TestAgent.server: Adding LAN server: server="Node-818e7689-26da-a4f9-f658-958765f80de5 (Addr: tcp/127.0.0.1:29755) (DC: dc1)" +32:08.151 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-818e7689-26da-a4f9-f658-958765f80de5.dc1 area=wan +32:08.151 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29757 network=tcp +32:08.152 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29744 network=tcp +32:08.152 [INFO] TestAgent: started state syncer +32:08.152 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29756 network=tcp +32:08.209 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.209 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29755 [Candidate]" term=2 +32:08.240 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.240 [DEBUG] TestAgent.server.raft: vote granted: from=818e7689-26da-a4f9-f658-958765f80de5 term=2 tally=1 +32:08.240 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.240 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29755 [Leader]" +32:08.240 [INFO] TestAgent.server: cluster leadership acquired +32:08.240 [INFO] TestAgent.server: New leader elected: payload=Node-818e7689-26da-a4f9-f658-958765f80de5 +32:08.261 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29755 +32:08.280 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.302 [INFO] TestAgent: Synced node info +32:08.356 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:08.356 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:08.356 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:08.356 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:08.356 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:08.356 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-818e7689-26da-a4f9-f658-958765f80de5 +32:08.357 [INFO] TestAgent.server: member joined, marking health alive: member=Node-818e7689-26da-a4f9-f658-958765f80de5 +32:08.374 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.431 [INFO] TestAgent: Requesting shutdown +32:08.431 [INFO] TestAgent.server: shutting down server +32:08.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.431 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.431 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.431 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.432 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.432 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.432 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.432 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.432 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.437 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.445Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.446Z [INFO] agent.router.manager: shutting down +32:08.446 [INFO] TestAgent: consul server down +32:08.446 [INFO] TestAgent: shutdown complete +32:08.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=tcp +32:08.446 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29757 network=udp +32:08.446 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29744 network=tcp +32:08.946 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.946 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Integration (0.95s) +=== CONT TestPreparedQuery_Update +2021-10-05T16:32:08.164Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.176Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.220 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab950b36-f231-5237-e588-0c8ce728aefc Address:127.0.0.1:29762}]" +32:08.221 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29762 [Follower]" leader= +32:08.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab950b36-f231-5237-e588-0c8ce728aefc.dc1 127.0.0.1 +32:08.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab950b36-f231-5237-e588-0c8ce728aefc 127.0.0.1 +2021-10-05T16:32:08.222Z [INFO] agent.router: Initializing LAN area manager +32:08.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=udp +32:08.222 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab950b36-f231-5237-e588-0c8ce728aefc.dc1 area=wan +32:08.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29673 network=tcp +32:08.223 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29751 network=tcp +32:08.223 [INFO] TestAgent: started state syncer +32:08.223 [INFO] TestAgent.server: Adding LAN server: server="Node-ab950b36-f231-5237-e588-0c8ce728aefc (Addr: tcp/127.0.0.1:29762) (DC: dc1)" +32:08.223 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29763 network=tcp +32:08.285 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.285 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29762 [Candidate]" term=2 +32:08.298 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.298 [DEBUG] TestAgent.server.raft: vote granted: from=ab950b36-f231-5237-e588-0c8ce728aefc term=2 tally=1 +32:08.298 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.298 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29762 [Leader]" +32:08.298 [INFO] TestAgent.server: cluster leadership acquired +32:08.299 [INFO] TestAgent.server: New leader elected: payload=Node-ab950b36-f231-5237-e588-0c8ce728aefc +32:08.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29762 +32:08.327 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.360 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:08.360 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:08.360 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:08.360 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:08.360 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:08.360 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab950b36-f231-5237-e588-0c8ce728aefc +32:08.360 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab950b36-f231-5237-e588-0c8ce728aefc +32:08.377 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.436 [INFO] TestAgent: Requesting shutdown +32:08.436 [INFO] TestAgent.server: shutting down server +32:08.436 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.436 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.436 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.436 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.436 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.436 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.436 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.436 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:08.436 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.436 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.436 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.442 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.451Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.451Z [INFO] agent.router.manager: shutting down +32:08.451 [INFO] TestAgent: consul server down +32:08.451 [INFO] TestAgent: shutdown complete +32:08.451 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=tcp +32:08.451 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29673 network=udp +32:08.451 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29751 network=tcp +32:08.952 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.952 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecGetSpec (0.80s) +=== CONT TestKVSEndpoint_DELETE_ConflictingFlags +2021-10-05T16:32:08.151Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.151Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.183Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.183Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.261 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:232038f4-31b5-46aa-fe80-0df225aaaa7f Address:127.0.0.1:29706}]" +32:08.261 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29706 [Follower]" leader= +32:08.262 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 127.0.0.1 +32:08.263 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f 127.0.0.1 +2021-10-05T16:32:08.263Z [INFO] agent.router: Initializing LAN area manager +32:08.263 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=udp +32:08.263 [INFO] TestAgent.server: Adding LAN server: server="Node-232038f4-31b5-46aa-fe80-0df225aaaa7f (Addr: tcp/127.0.0.1:29706) (DC: dc1)" +32:08.263 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29729 network=tcp +32:08.263 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 area=wan +32:08.264 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29758 network=tcp +32:08.264 [INFO] TestAgent: started state syncer +32:08.264 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29707 network=tcp +32:08.297 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.297 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29706 [Candidate]" term=2 +32:08.312 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.312 [DEBUG] TestAgent.server.raft: vote granted: from=232038f4-31b5-46aa-fe80-0df225aaaa7f term=2 tally=1 +32:08.312 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.312 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29706 [Leader]" +32:08.313 [INFO] TestAgent.server: cluster leadership acquired +32:08.313 [INFO] TestAgent.server: New leader elected: payload=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.314 [INFO] TestAgent.server: initializing acls +32:08.322 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:08.322 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:08.323 [INFO] TestAgent.server: initializing acls +32:08.323 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:08.349 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:08.351 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:08.352 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:08.352 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:08.352 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:08.352 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:08.352 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.352 [INFO] TestAgent.server: Updating LAN server: server="Node-232038f4-31b5-46aa-fe80-0df225aaaa7f (Addr: tcp/127.0.0.1:29706) (DC: dc1)" +32:08.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 +32:08.352 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 +32:08.352 [INFO] TestAgent.server: Updating LAN server: server="Node-232038f4-31b5-46aa-fe80-0df225aaaa7f (Addr: tcp/127.0.0.1:29706) (DC: dc1)" +32:08.352 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 area=wan +32:08.352 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f.dc1 area=wan +32:08.370 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:08.408 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:08.408 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:08.408 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:08.408 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:08.408 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:08.408 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.408 [INFO] TestAgent.server: member joined, marking health alive: member=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.425 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.425 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.426 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-232038f4-31b5-46aa-fe80-0df225aaaa7f +32:08.432 [INFO] TestAgent.server: federation state anti-entropy synced +32:08.444 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:08.454 [INFO] TestAgent: Synced node info +32:08.454 [DEBUG] TestAgent: Node info in sync +32:08.454 [INFO] TestAgent: Requesting shutdown +32:08.454 [INFO] TestAgent.server: shutting down server +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:08.454 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:08.454 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:08.455 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:08.455 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:08.465 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:08.474Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:08.474Z [INFO] agent.router.manager: shutting down +32:08.474 [INFO] TestAgent: consul server down +32:08.474 [INFO] TestAgent: shutdown complete +32:08.474 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=tcp +32:08.474 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29729 network=udp +32:08.474 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29758 network=tcp +32:08.976 [INFO] TestAgent: Waiting for endpoints to shut down +32:08.976 [INFO] TestAgent: Endpoints down +--- PASS: TestRemoteExecGetSpec_ACLToken (0.85s) +=== CONT TestACL_Legacy_Update +[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:08.733Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.764Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.845 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:579ad28a-3123-d5b0-a2c9-940e4f055ef1 Address:127.0.0.1:29741}]" +32:08.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-579ad28a-3123-d5b0-a2c9-940e4f055ef1.dc1 127.0.0.1 +32:08.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29741 [Follower]" leader= +32:08.847 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-579ad28a-3123-d5b0-a2c9-940e4f055ef1 127.0.0.1 +2021-10-05T16:32:08.847Z [INFO] agent.router: Initializing LAN area manager +32:08.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=udp +32:08.847 [INFO] TestAgent.server: Adding LAN server: server="Node-579ad28a-3123-d5b0-a2c9-940e4f055ef1 (Addr: tcp/127.0.0.1:29741) (DC: dc1)" +32:08.847 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-579ad28a-3123-d5b0-a2c9-940e4f055ef1.dc1 area=wan +32:08.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29764 network=tcp +32:08.848 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29674 network=tcp +32:08.848 [INFO] TestAgent: started state syncer +32:08.848 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29742 network=tcp +32:08.904 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.904 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29741 [Candidate]" term=2 +32:08.955 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.955 [DEBUG] TestAgent.server.raft: vote granted: from=579ad28a-3123-d5b0-a2c9-940e4f055ef1 term=2 tally=1 +32:08.955 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.955 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29741 [Leader]" +32:08.956 [INFO] TestAgent.server: cluster leadership acquired +32:08.956 [INFO] TestAgent.server: New leader elected: payload=Node-579ad28a-3123-d5b0-a2c9-940e4f055ef1 +32:08.977 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29741 +32:09.028 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.048 [INFO] TestAgent: Requesting shutdown +32:09.048 [INFO] TestAgent.server: shutting down server +32:09.048 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.048 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:09.056 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.068Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.068Z [INFO] agent.router.manager: shutting down +32:09.080 [INFO] TestAgent: consul server down +32:09.080 [INFO] TestAgent: shutdown complete +32:09.080 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=tcp +32:09.080 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29764 network=udp +32:09.080 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29674 network=tcp +32:09.080 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +32:09.080 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:09.080 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +32:09.580 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.580 [INFO] TestAgent: Endpoints down +--- PASS: TestOperator_RaftConfiguration (0.88s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} +2021-10-05T16:32:09.025Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.055Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.124 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e5df06b5-a50a-a754-cae8-f7fc12124ec7 Address:127.0.0.1:29839}]" +32:09.125 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e5df06b5-a50a-a754-cae8-f7fc12124ec7.dc1 127.0.0.1 +32:09.125 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e5df06b5-a50a-a754-cae8-f7fc12124ec7 127.0.0.1 +2021-10-05T16:32:09.125Z [INFO] agent.router: Initializing LAN area manager +32:09.125 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=udp +32:09.125 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29839 [Follower]" leader= +32:09.126 [INFO] TestAgent.server: Adding LAN server: server="Node-e5df06b5-a50a-a754-cae8-f7fc12124ec7 (Addr: tcp/127.0.0.1:29839) (DC: dc1)" +32:09.126 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e5df06b5-a50a-a754-cae8-f7fc12124ec7.dc1 area=wan +32:09.126 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29750 network=tcp +32:09.126 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29779 network=tcp +32:09.126 [INFO] TestAgent: started state syncer +32:09.126 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29840 network=tcp +32:09.164 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.164 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29839 [Candidate]" term=2 +32:09.198 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.198 [DEBUG] TestAgent.server.raft: vote granted: from=e5df06b5-a50a-a754-cae8-f7fc12124ec7 term=2 tally=1 +32:09.198 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.198 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29839 [Leader]" +32:09.198 [INFO] TestAgent.server: cluster leadership acquired +32:09.198 [INFO] TestAgent.server: New leader elected: payload=Node-e5df06b5-a50a-a754-cae8-f7fc12124ec7 +32:09.232 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29839 +32:09.261 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.278 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +32:09.278 [INFO] TestAgent: Requesting shutdown +32:09.278 [INFO] TestAgent.server: shutting down server +32:09.278 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.279 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:09.285 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.298Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.298Z [INFO] agent.router.manager: shutting down +32:09.300 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +32:09.300 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:09.300 [INFO] TestAgent: consul server down +32:09.300 [INFO] TestAgent: shutdown complete +32:09.300 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=tcp +32:09.300 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29750 network=udp +32:09.300 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29779 network=tcp +32:09.800 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.800 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Update (0.85s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{6_6_6_6_6_6_6_5_6_6_-5} +[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:08.729Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.729Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.769Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.769Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.856 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:461e0e0a-d543-8c01-6e2c-05dcfb9818af Address:127.0.0.1:29671}]" +32:08.856 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29671 [Follower]" leader= +32:08.857 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 127.0.0.1 +32:08.857 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af 127.0.0.1 +2021-10-05T16:32:08.857Z [INFO] agent.router: Initializing LAN area manager +32:08.857 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=udp +32:08.858 [INFO] TestAgent.server: Adding LAN server: server="Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af (Addr: tcp/127.0.0.1:29671) (DC: dc1)" +32:08.858 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 area=wan +32:08.858 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29785 network=tcp +32:08.858 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29709 network=tcp +32:08.859 [INFO] TestAgent: started state syncer +32:08.859 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29672 network=tcp +32:08.920 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:08.924 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:08.924 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29671 [Candidate]" term=2 +32:08.980 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:08.980 [DEBUG] TestAgent.server.raft: vote granted: from=461e0e0a-d543-8c01-6e2c-05dcfb9818af term=2 tally=1 +32:08.980 [INFO] TestAgent.server.raft: election won: tally=1 +32:08.980 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29671 [Leader]" +32:08.984 [INFO] TestAgent.server: cluster leadership acquired +32:08.984 [INFO] TestAgent.server: New leader elected: payload=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.016 [INFO] TestAgent.server: initializing acls +32:09.028 [INFO] TestAgent.server: initializing acls +32:09.040 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:09.040 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:09.056 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:09.056 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:09.092 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:09.112 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:09.112 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:09.112 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:09.112 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.113 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 +32:09.113 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:09.113 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:09.113 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.113 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 +32:09.113 [INFO] TestAgent.server: Updating LAN server: server="Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af (Addr: tcp/127.0.0.1:29671) (DC: dc1)" +32:09.113 [INFO] TestAgent.server: Updating LAN server: server="Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af (Addr: tcp/127.0.0.1:29671) (DC: dc1)" +32:09.113 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 area=wan +32:09.113 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af.dc1 area=wan +32:09.140 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.168 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.168 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.168 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.168 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.168 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.168 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.168 [INFO] TestAgent.server: member joined, marking health alive: member=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.177 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.183 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.238 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:09.247 [INFO] TestAgent: Synced node info +32:09.247 [DEBUG] TestAgent: Node info in sync +32:09.309 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.309 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-461e0e0a-d543-8c01-6e2c-05dcfb9818af +32:09.320 [INFO] TestAgent: Requesting shutdown +32:09.320 [INFO] TestAgent.server: shutting down server +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:09.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.320 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:09.320 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:09.332 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.344Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.344Z [INFO] agent.router.manager: shutting down +32:09.344 [INFO] TestAgent: consul server down +32:09.344 [INFO] TestAgent: shutdown complete +32:09.344 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=tcp +32:09.344 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29785 network=udp +32:09.344 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29709 network=tcp +32:09.847 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.847 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:09.008Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.046Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.120 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:80f88e1b-b70e-43cb-844d-00e0c3db5a80 Address:127.0.0.1:29783}]" +32:09.121 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-80f88e1b-b70e-43cb-844d-00e0c3db5a80.dc1 127.0.0.1 +32:09.121 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-80f88e1b-b70e-43cb-844d-00e0c3db5a80 127.0.0.1 +2021-10-05T16:32:09.121Z [INFO] agent.router: Initializing LAN area manager +32:09.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=udp +32:09.121 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29783 [Follower]" leader= +32:09.122 [INFO] TestAgent.server: Adding LAN server: server="Node-80f88e1b-b70e-43cb-844d-00e0c3db5a80 (Addr: tcp/127.0.0.1:29783) (DC: dc1)" +32:09.122 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-80f88e1b-b70e-43cb-844d-00e0c3db5a80.dc1 area=wan +32:09.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29778 network=tcp +32:09.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29765 network=tcp +32:09.122 [INFO] TestAgent: started state syncer +32:09.122 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29784 network=tcp +32:09.175 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.175 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29783 [Candidate]" term=2 +32:09.204 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.204 [DEBUG] TestAgent.server.raft: vote granted: from=80f88e1b-b70e-43cb-844d-00e0c3db5a80 term=2 tally=1 +32:09.204 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.204 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29783 [Leader]" +32:09.205 [INFO] TestAgent.server: cluster leadership acquired +32:09.205 [INFO] TestAgent.server: New leader elected: payload=Node-80f88e1b-b70e-43cb-844d-00e0c3db5a80 +32:09.238 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29783 +32:09.288 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.308 [WARN] TestAgent.server: endpoint injected; this should only be used for testing +32:09.308 [INFO] TestAgent: Requesting shutdown +32:09.308 [INFO] TestAgent.server: shutting down server +32:09.308 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.308 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:09.320 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.332Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.332Z [INFO] agent.router.manager: shutting down +32:09.342 [INFO] TestAgent: consul server down +32:09.342 [INFO] TestAgent: shutdown complete +32:09.342 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=tcp +32:09.342 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29778 network=udp +32:09.342 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29765 network=tcp +32:09.343 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +32:09.343 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:09.343 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +32:09.343 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +32:09.343 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +32:09.848 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.848 [INFO] TestAgent: Endpoints down +--- PASS: TestPreparedQuery_Delete (0.91s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} +2021-10-05T16:32:08.801Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.801Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:08.828Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:08.828Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:08.928 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ced2f13c-0306-051e-7660-2c74d1fc2449 Address:127.0.0.1:29790}]" +32:08.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 127.0.0.1 +32:08.929 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ced2f13c-0306-051e-7660-2c74d1fc2449 127.0.0.1 +2021-10-05T16:32:08.930Z [INFO] agent.router: Initializing LAN area manager +32:08.930 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=udp +32:08.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29790 [Follower]" leader= +32:08.930 [INFO] TestAgent.server: Adding LAN server: server="Node-ced2f13c-0306-051e-7660-2c74d1fc2449 (Addr: tcp/127.0.0.1:29790) (DC: dc1)" +32:08.930 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 area=wan +32:08.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29708 network=tcp +32:08.940 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29786 network=tcp +32:08.940 [INFO] TestAgent: started state syncer +32:08.940 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29791 network=tcp +32:08.984 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:09.004 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.004 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29790 [Candidate]" term=2 +32:09.055 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.055 [DEBUG] TestAgent.server.raft: vote granted: from=ced2f13c-0306-051e-7660-2c74d1fc2449 term=2 tally=1 +32:09.055 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.055 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29790 [Leader]" +32:09.056 [INFO] TestAgent.server: cluster leadership acquired +32:09.056 [INFO] TestAgent.server: New leader elected: payload=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.080 [INFO] TestAgent.server: initializing acls +32:09.084 [INFO] TestAgent.server: initializing acls +32:09.092 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:09.092 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:09.106 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:09.106 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:09.133 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:09.134 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:09.145 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:09.145 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:09.145 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:09.146 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.146 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 +32:09.146 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:09.146 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +32:09.147 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.147 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 +32:09.147 [INFO] TestAgent.server: Updating LAN server: server="Node-ced2f13c-0306-051e-7660-2c74d1fc2449 (Addr: tcp/127.0.0.1:29790) (DC: dc1)" +32:09.147 [INFO] TestAgent.server: Updating LAN server: server="Node-ced2f13c-0306-051e-7660-2c74d1fc2449 (Addr: tcp/127.0.0.1:29790) (DC: dc1)" +32:09.147 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 area=wan +32:09.147 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ced2f13c-0306-051e-7660-2c74d1fc2449.dc1 area=wan +32:09.163 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.192 [INFO] TestAgent: Synced node info +32:09.192 [DEBUG] TestAgent: Node info in sync +32:09.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.202 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.202 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.207 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.320 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.320 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-ced2f13c-0306-051e-7660-2c74d1fc2449 +32:09.332 [INFO] TestAgent: Requesting shutdown +32:09.332 [INFO] TestAgent.server: shutting down server +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.332 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.332 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.332 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.332 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.343 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.352Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.352Z [INFO] agent.router.manager: shutting down +32:09.352 [INFO] TestAgent: consul server down +32:09.352 [INFO] TestAgent: shutdown complete +32:09.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=tcp +32:09.352 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29708 network=udp +32:09.352 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29786 network=tcp +32:09.853 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.853 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_UpdateUpsert (1.08s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} +--- PASS: TestACL_Legacy_Get (2.00s) + --- PASS: TestACL_Legacy_Get/wrong_id (0.84s) + --- PASS: TestACL_Legacy_Get/right_id (1.16s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} +2021-10-05T16:32:09.005Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.051Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.135 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d447744d-30ed-f8d8-f095-53557f56407b Address:127.0.0.1:29825}]" +32:09.135 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d447744d-30ed-f8d8-f095-53557f56407b.dc1 127.0.0.1 +32:09.136 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29825 [Follower]" leader= +32:09.136 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d447744d-30ed-f8d8-f095-53557f56407b 127.0.0.1 +2021-10-05T16:32:09.136Z [INFO] agent.router: Initializing LAN area manager +32:09.137 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=udp +32:09.137 [INFO] TestAgent.server: Adding LAN server: server="Node-d447744d-30ed-f8d8-f095-53557f56407b (Addr: tcp/127.0.0.1:29825) (DC: dc1)" +32:09.137 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d447744d-30ed-f8d8-f095-53557f56407b.dc1 area=wan +32:09.137 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29820 network=tcp +32:09.137 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29821 network=tcp +32:09.138 [INFO] TestAgent: started state syncer +32:09.138 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29826 network=tcp +32:09.198 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.198 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29825 [Candidate]" term=2 +32:09.218 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.218 [DEBUG] TestAgent.server.raft: vote granted: from=d447744d-30ed-f8d8-f095-53557f56407b term=2 tally=1 +32:09.218 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.218 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29825 [Leader]" +32:09.218 [INFO] TestAgent.server: cluster leadership acquired +32:09.218 [INFO] TestAgent.server: New leader elected: payload=Node-d447744d-30ed-f8d8-f095-53557f56407b +32:09.228 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29825 +32:09.288 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.345 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.345 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.345 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.345 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.345 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d447744d-30ed-f8d8-f095-53557f56407b +32:09.345 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d447744d-30ed-f8d8-f095-53557f56407b +32:09.372 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.376 [INFO] TestAgent: Requesting shutdown +32:09.376 [INFO] TestAgent.server: shutting down server +32:09.376 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.376 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.376 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.376 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.376 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.376 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.376 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.376 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:09.376 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.376 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.376 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.388 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.400Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.400Z [INFO] agent.router.manager: shutting down +32:09.400 [INFO] TestAgent: consul server down +32:09.400 [INFO] TestAgent: shutdown complete +32:09.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=tcp +32:09.400 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29820 network=udp +32:09.400 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29821 network=tcp +32:09.900 [INFO] TestAgent: Waiting for endpoints to shut down +32:09.900 [INFO] TestAgent: Endpoints down +--- PASS: TestKVSEndpoint_DELETE_ConflictingFlags (0.95s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} +2021-10-05T16:32:09.021Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:09.021Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.041Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:32:09.041Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.107 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 Address:127.0.0.1:29818}]" +32:09.107 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2.dc1 127.0.0.1 +32:09.108 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29818 [Follower]" leader= +32:09.108 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 127.0.0.1 +2021-10-05T16:32:09.108Z [INFO] agent.router: Initializing LAN area manager +32:09.108 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=udp +32:09.108 [INFO] TestAgent.server: Adding LAN server: server="Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 (Addr: tcp/127.0.0.1:29818) (DC: dc1)" +32:09.108 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2.dc1 area=wan +32:09.108 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29813 network=tcp +32:09.109 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29814 network=tcp +32:09.109 [INFO] TestAgent: started state syncer +32:09.109 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29819 network=tcp +32:09.158 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +32:09.171 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.171 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29818 [Candidate]" term=2 +32:09.186 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.186 [DEBUG] TestAgent.server.raft: vote granted: from=0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 term=2 tally=1 +32:09.186 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.186 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29818 [Leader]" +32:09.186 [INFO] TestAgent.server: cluster leadership acquired +32:09.186 [INFO] TestAgent.server: New leader elected: payload=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.187 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +32:09.196 [INFO] TestAgent.server: initializing acls +32:09.201 [INFO] TestAgent.server: Created ACL 'global-management' policy +32:09.201 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +32:09.212 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +32:09.224 [INFO] TestAgent.server: Created ACL anonymous token from configuration +32:09.224 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +32:09.224 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +32:09.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.224 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2.dc1 +32:09.224 [INFO] TestAgent.server: Updating LAN server: server="Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 (Addr: tcp/127.0.0.1:29818) (DC: dc1)" +32:09.224 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2.dc1 area=wan +32:09.260 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.303 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.303 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.303 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.303 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.303 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.304 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.304 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.314 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.327 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.480 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.480 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-0682fda9-f8c3-cdd3-439e-fe47bfec3ef2 +32:09.524 [INFO] TestAgent: Requesting shutdown +32:09.524 [INFO] TestAgent.server: shutting down server +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:09.524 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:09.524 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.524 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:09.524 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:09.530 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.538Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.538Z [INFO] agent.router.manager: shutting down +32:09.538 [INFO] TestAgent: consul server down +32:09.538 [INFO] TestAgent: shutdown complete +32:09.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=tcp +32:09.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29813 network=udp +32:09.538 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29814 network=tcp +32:10.038 [INFO] TestAgent: Waiting for endpoints to shut down +32:10.038 [INFO] TestAgent: Endpoints down +--- PASS: TestACL_Legacy_Update (1.06s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} +[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:59.045Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.045Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:59.080Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:31:59.080Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:59.156 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:72f1e40f-8238-5d7a-d13a-113a42c0f84a Address:127.0.0.1:30392}]" +31:59.156 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30392 [Follower]" leader= +31:59.160 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 127.0.0.1 +31:59.161 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a 127.0.0.1 +2021-10-05T16:31:59.161Z [INFO] agent.router: Initializing LAN area manager +31:59.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30401 network=udp +31:59.161 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 area=wan +31:59.161 [INFO] TestAgent.server: Adding LAN server: server="Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a (Addr: tcp/127.0.0.1:30392) (DC: dc1)" +31:59.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30401 network=tcp +31:59.162 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30402 network=tcp +31:59.162 [INFO] TestAgent: started state syncer +31:59.162 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30372 network=tcp +31:59.211 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:59.216 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:59.216 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30392 [Candidate]" term=2 +31:59.260 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:59.260 [DEBUG] TestAgent.server.raft: vote granted: from=72f1e40f-8238-5d7a-d13a-113a42c0f84a term=2 tally=1 +31:59.261 [INFO] TestAgent.server.raft: election won: tally=1 +31:59.261 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30392 [Leader]" +31:59.264 [INFO] TestAgent.server: cluster leadership acquired +31:59.264 [INFO] TestAgent.server: New leader elected: payload=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.286 [INFO] TestAgent.server: initializing acls +31:59.290 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:59.290 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:59.297 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:59.314 [INFO] TestAgent.server: initializing acls +31:59.314 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:59.330 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:59.330 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:59.330 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:59.330 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.330 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 +31:59.330 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 area=wan +31:59.331 [INFO] TestAgent.server: Updating LAN server: server="Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a (Addr: tcp/127.0.0.1:30392) (DC: dc1)" +31:59.345 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:59.345 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:59.345 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.346 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 +31:59.346 [INFO] TestAgent.server: Updating LAN server: server="Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a (Addr: tcp/127.0.0.1:30392) (DC: dc1)" +31:59.346 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 area=wan +31:59.377 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.428 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.428 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:59.428 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:59.428 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:59.428 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:59.428 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.428 [INFO] TestAgent.server: member joined, marking health alive: member=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.432 [INFO] TestAgent.server: federation state anti-entropy synced +31:59.448 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.448 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.470 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.470 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +31:59.508 [INFO] TestAgent.server: server starting leave +31:59.508 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 127.0.0.1 +31:59.508 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a.dc1 area=wan +2021-10-05T16:31:59.508Z [INFO] agent.router.manager: shutting down +31:59.543 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:59.552 [INFO] TestAgent: Synced node info +32:01.538 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:01.538 [DEBUG] TestAgent: Node info in sync +32:01.538 [DEBUG] TestAgent: Node info in sync +32:02.508 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a 127.0.0.1 +32:02.508 [INFO] TestAgent.server: Removing LAN server: server="Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a (Addr: tcp/127.0.0.1:30392) (DC: dc1)" +32:02.508 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-72f1e40f-8238-5d7a-d13a-113a42c0f84a +32:03.355 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +32:05.354 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +32:05.512 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s +32:07.354 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +32:09.355 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +32:09.356 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" +32:10.512 [INFO] TestAgent: Requesting shutdown +32:10.512 [INFO] TestAgent.server: shutting down server +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:10.512 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:10.512 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:10.512 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +2021-10-05T16:32:10.532Z [INFO] agent.router.manager: shutting down +32:10.532 [INFO] TestAgent: consul server down +32:10.532 [INFO] TestAgent: shutdown complete +32:10.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30401 network=tcp +32:10.532 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30401 network=udp +32:10.532 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30402 network=tcp +32:11.033 [INFO] TestAgent: Waiting for endpoints to shut down +32:11.033 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_Leave_ACLDeny (12.02s) + --- PASS: TestAgent_Leave_ACLDeny/no_token (0.00s) + --- PASS: TestAgent_Leave_ACLDeny/read-only_token (0.03s) + --- PASS: TestAgent_Leave_ACLDeny/agent_master_token (11.02s) +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:09.593Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.610Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.665 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:435df1fc-d44d-76f6-9450-876cbad61828 Address:127.0.0.1:29797}]" +32:09.666 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:09.666 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:09.666Z [INFO] agent.router: Initializing LAN area manager +32:09.667 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=udp +32:09.667 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29797 [Follower]" leader= +32:09.667 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29797) (DC: dc1)" +32:09.667 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:09.667 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29827 network=tcp +32:09.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29828 network=tcp +32:09.668 [INFO] TestAgent: started state syncer +32:09.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29798 network=tcp +32:09.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29797 [Candidate]" term=2 +32:09.727 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.727 [DEBUG] TestAgent.server.raft: vote granted: from=435df1fc-d44d-76f6-9450-876cbad61828 term=2 tally=1 +32:09.727 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.727 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29797 [Leader]" +32:09.728 [INFO] TestAgent.server: cluster leadership acquired +32:09.729 [INFO] TestAgent.server: New leader elected: payload=test-node +32:09.758 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29797 +32:09.769 [INFO] TestAgent: Synced node info +32:09.769 [DEBUG] TestAgent: Node info in sync +32:09.828 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.888 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.888 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.888 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.888 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.888 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.888 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:09.888 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:09.949 [INFO] TestAgent.server: federation state anti-entropy synced +32:11.217 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.333058ms client=127.0.0.1:49790 client_network=udp +32:11.219 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.03074ms client=127.0.0.1:48581 client_network=udp +32:11.225 [DEBUG] TestAgent.dns: request served from client: name=e9417644-4402-1eee-2351-e49b722d86f0.query.consul. type=A class=IN latency=1.169031ms client=127.0.0.1:48806 client_network=udp +32:11.228 [INFO] TestAgent: Requesting shutdown +32:11.228 [INFO] TestAgent.server: shutting down server +32:11.228 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.228 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.228 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.228 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.228 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.228 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.240 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.248Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.248Z [INFO] agent.router.manager: shutting down +32:11.248 [INFO] TestAgent: consul server down +32:11.248 [INFO] TestAgent: shutdown complete +32:11.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=tcp +32:11.248 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29827 network=udp +32:11.248 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29828 network=tcp +32:11.748 [INFO] TestAgent: Waiting for endpoints to shut down +32:11.748 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{9_9_8_8_6_8_8_5_8_8_-5} +[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:09.874Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.889Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.954 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8a181f1a-2ff9-617e-2090-1aae51266786 Address:127.0.0.1:29811}]" +32:09.955 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:09.955 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29811 [Follower]" leader= +32:09.955 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:09.955Z [INFO] agent.router: Initializing LAN area manager +32:09.955 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=udp +32:09.956 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29811) (DC: dc1)" +32:09.956 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:09.956 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29806 network=tcp +32:09.956 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29807 network=tcp +32:09.956 [INFO] TestAgent: started state syncer +32:09.956 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29812 network=tcp +32:10.020 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.020 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29811 [Candidate]" term=2 +32:10.041 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.041 [DEBUG] TestAgent.server.raft: vote granted: from=8a181f1a-2ff9-617e-2090-1aae51266786 term=2 tally=1 +32:10.041 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.041 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29811 [Leader]" +32:10.041 [INFO] TestAgent.server: cluster leadership acquired +32:10.041 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.058 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29811 +32:10.101 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.168 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.168 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.168 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.168 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.168 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.168 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.168 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.204 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.204 [INFO] TestAgent: Synced node info +32:11.232 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:11.232 [DEBUG] TestAgent: Node info in sync +32:11.232 [DEBUG] TestAgent: Node info in sync +32:11.611 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.043441ms client=127.0.0.1:53527 client_network=udp +32:11.617 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.395899ms client=127.0.0.1:38375 client_network=udp +32:11.619 [DEBUG] TestAgent.dns: request served from client: name=f4643f1a-530b-0154-7e95-634304823ed1.query.consul. type=A class=IN latency=1.266307ms client=127.0.0.1:60357 client_network=udp +32:11.619 [INFO] TestAgent: Requesting shutdown +32:11.619 [INFO] TestAgent.server: shutting down server +32:11.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.619 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.619 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.619 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.619 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.619 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.619 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.628 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.633Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.633Z [INFO] agent.router.manager: shutting down +32:11.633 [INFO] TestAgent: consul server down +32:11.633 [INFO] TestAgent: shutdown complete +32:11.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=tcp +32:11.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29806 network=udp +32:11.633 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29807 network=tcp +32:12.136 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.136 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{9_9_8_8_6_8_8_5_8_8_-5} +2021-10-05T16:32:10.080Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:10.104Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:10.180 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:acd0c60f-0b7b-302e-da0f-8c1e269bbe98 Address:127.0.0.1:29888}]" +32:10.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:10.181 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:10.181Z [INFO] agent.router: Initializing LAN area manager +32:10.182 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=udp +32:10.182 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29888 [Follower]" leader= +32:10.182 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29888) (DC: dc1)" +32:10.182 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:10.182 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29841 network=tcp +32:10.183 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29884 network=tcp +32:10.183 [INFO] TestAgent: started state syncer +32:10.183 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29889 network=tcp +32:10.252 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.252 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29888 [Candidate]" term=2 +32:10.292 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.292 [DEBUG] TestAgent.server.raft: vote granted: from=acd0c60f-0b7b-302e-da0f-8c1e269bbe98 term=2 tally=1 +32:10.292 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.292 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29888 [Leader]" +32:10.292 [INFO] TestAgent.server: cluster leadership acquired +32:10.292 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.300 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29888 +32:10.323 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.389 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.389 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.389 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.389 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.389 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.389 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.389 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.417 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.488 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.511 [INFO] TestAgent: Synced node info +32:10.511 [DEBUG] TestAgent: Node info in sync +32:10.816 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.816 [DEBUG] TestAgent: Node info in sync +32:11.653 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.242055ms client=127.0.0.1:58647 client_network=udp +32:11.655 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.07404ms client=127.0.0.1:41831 client_network=udp +32:11.659 [DEBUG] TestAgent.dns: request served from client: name=6adac4ef-b8df-28c5-f149-bf1060361d50.query.consul. type=AAAA class=IN latency=3.583158ms client=127.0.0.1:46248 client_network=udp +32:11.659 [INFO] TestAgent: Requesting shutdown +32:11.659 [INFO] TestAgent.server: shutting down server +32:11.659 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.659 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.659 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.659 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.659 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.659 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.659 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.659 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.659 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.659 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.667 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.667Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.667Z [INFO] agent.router.manager: shutting down +32:11.667 [INFO] TestAgent: consul server down +32:11.667 [INFO] TestAgent: shutdown complete +32:11.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=tcp +32:11.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29841 network=udp +32:11.667 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29884 network=tcp +32:12.167 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.167 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{9_9_8_8_6_8_8_5_8_8_-5} +2021-10-05T16:32:09.929Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.946Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:10.000 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:44ef99ee-8f13-4dff-fc27-a868c4c400dc Address:127.0.0.1:29650}]" +32:10.001 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:10.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:10.002Z [INFO] agent.router: Initializing LAN area manager +32:10.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29855 network=udp +32:10.002 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29650 [Follower]" leader= +32:10.002 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:10.002 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29650) (DC: dc1)" +32:10.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29855 network=tcp +32:10.003 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29660 network=tcp +32:10.003 [INFO] TestAgent: started state syncer +32:10.003 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29651 network=tcp +32:10.043 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.043 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29650 [Candidate]" term=2 +32:10.064 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.064 [DEBUG] TestAgent.server.raft: vote granted: from=44ef99ee-8f13-4dff-fc27-a868c4c400dc term=2 tally=1 +32:10.064 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.064 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29650 [Leader]" +32:10.064 [INFO] TestAgent.server: cluster leadership acquired +32:10.064 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.090 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29650 +32:10.133 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.184 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.184 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.184 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.184 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.184 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.184 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.184 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.217 [INFO] TestAgent: Synced node info +32:10.234 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.538 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.538 [DEBUG] TestAgent: Node info in sync +32:10.538 [DEBUG] TestAgent: Node info in sync +32:11.682 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.702217ms client=127.0.0.1:41722 client_network=udp +32:11.684 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.609524ms client=127.0.0.1:44058 client_network=udp +32:11.687 [DEBUG] TestAgent.dns: request served from client: name=3b2bf2e8-ade3-1aa3-a0dd-ab15a6d7990d.query.consul. type=ANY class=IN latency=2.134685ms client=127.0.0.1:38729 client_network=udp +32:11.687 [INFO] TestAgent: Requesting shutdown +32:11.687 [INFO] TestAgent.server: shutting down server +32:11.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.687 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.687 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.687 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.687 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.687 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.687 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.687 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.687 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.694 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.694Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.694Z [INFO] agent.router.manager: shutting down +32:11.695 [INFO] TestAgent: consul server down +32:11.695 [INFO] TestAgent: shutdown complete +32:11.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29855 network=tcp +32:11.695 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29855 network=udp +32:11.695 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29660 network=tcp +32:12.195 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.195 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{8_8_8_8_6_8_8_5_8_8_-5} -[WARN] freeport: 10 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:25.521Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.521Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.530Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.530Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.552 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 Address:127.0.0.1:11956}]" -46:25.552 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11956 [Follower]" leader= -46:25.553 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1.dc1 127.0.0.1 -46:25.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 127.0.0.1 -2022-11-07T22:46:25.554Z [INFO] agent.router: Initializing LAN area manager -46:25.554 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1.dc1 area=wan -46:25.554 [INFO] TestAgent.server: Adding LAN server: server="Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 (Addr: tcp/127.0.0.1:11956) (DC: dc1)" -46:25.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11403 network=tcp -46:25.554 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11403 network=udp -46:25.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11404 network=tcp -46:25.554 [INFO] TestAgent: started state syncer -46:25.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11957 network=tcp -46:25.604 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.607 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.607 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11956 [Candidate]" term=2 -46:25.619 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.619 [DEBUG] TestAgent.server.raft: vote granted: from=61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 term=2 tally=1 -46:25.619 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.619 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11956 [Leader]" -46:25.619 [INFO] TestAgent.server: cluster leadership acquired -46:25.619 [INFO] TestAgent.server: New leader elected: payload=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.625 [INFO] TestAgent.server: initializing acls -46:25.629 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.629 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.640 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.651 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.651 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.651 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.651 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.651 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1.dc1 -46:25.652 [INFO] TestAgent.server: Updating LAN server: server="Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 (Addr: tcp/127.0.0.1:11956) (DC: dc1)" -46:25.652 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1.dc1 area=wan -46:25.661 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.679 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.679 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.679 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.679 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.679 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.679 [INFO] TestAgent.server: member joined, marking health alive: member=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.688 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.697 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.821 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.821 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-61cf15cf-5f1c-40d0-e2ba-0a9187a783c1 -46:25.837 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:25.838 [INFO] TestAgent: Requesting shutdown -46:25.838 [INFO] TestAgent.server: shutting down server -46:25.838 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.838 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.838 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.838 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.839 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.839 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.839 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.839 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.847 [INFO] TestAgent: Synced node info -46:25.847 [DEBUG] TestAgent: Node info in sync -46:25.850 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.853Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.853Z [INFO] agent.router.manager: shutting down -46:25.853 [INFO] TestAgent: consul server down -46:25.853 [INFO] TestAgent: shutdown complete -46:25.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11403 network=tcp -46:25.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11403 network=udp -46:25.853 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11404 network=tcp -46:26.353 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.353 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_Update (0.84s) +2021-10-05T16:32:09.871Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.887Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.986 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:938a4845-9f08-a28c-a826-8feae8d9ca9b Address:127.0.0.1:29769}]" +32:09.986 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29769 [Follower]" leader= +32:09.987 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:09.987 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:09.987Z [INFO] agent.router: Initializing LAN area manager +32:09.988 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=udp +32:09.988 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29769) (DC: dc1)" +32:09.988 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:09.988 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29792 network=tcp +32:09.988 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29835 network=tcp +32:09.988 [INFO] TestAgent: started state syncer +32:09.988 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29770 network=tcp +32:10.035 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.035 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29769 [Candidate]" term=2 +32:10.056 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.056 [DEBUG] TestAgent.server.raft: vote granted: from=938a4845-9f08-a28c-a826-8feae8d9ca9b term=2 tally=1 +32:10.056 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.056 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29769 [Leader]" +32:10.057 [INFO] TestAgent.server: cluster leadership acquired +32:10.057 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.076 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29769 +32:10.124 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.184 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.184 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.184 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.184 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.184 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.184 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.184 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.212 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.216 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.235 [INFO] TestAgent: Synced node info +32:10.918 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.918 [DEBUG] TestAgent: Node info in sync +32:10.918 [DEBUG] TestAgent: Node info in sync +32:11.720 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.633738ms client=127.0.0.1:49042 client_network=udp +32:11.723 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.801938ms client=127.0.0.1:49642 client_network=udp +32:11.725 [DEBUG] TestAgent.dns: request served from client: name=04a9419d-5744-f194-deb3-893e80d0fe34.query.consul. type=ANY class=IN latency=2.420013ms client=127.0.0.1:46032 client_network=udp +32:11.726 [INFO] TestAgent: Requesting shutdown +32:11.726 [INFO] TestAgent.server: shutting down server +32:11.726 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.726 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.726 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.726 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.726 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.726 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.726 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.726 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.736 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.739Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.739Z [INFO] agent.router.manager: shutting down +32:11.739 [INFO] TestAgent: consul server down +32:11.739 [INFO] TestAgent: shutdown complete +32:11.739 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=tcp +32:11.739 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29792 network=udp +32:11.739 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29835 network=tcp +32:12.241 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.241 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{8_8_8_8_6_8_8_5_8_8_-5} -2022-11-07T22:46:25.659Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.659Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.669Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:25.669Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.696 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e01a9b7-4227-8cbe-968e-d4af620e91af Address:127.0.0.1:11352}]" -46:25.696 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11352 [Follower]" leader= -46:25.697 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9e01a9b7-4227-8cbe-968e-d4af620e91af.dc1 127.0.0.1 -46:25.698 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9e01a9b7-4227-8cbe-968e-d4af620e91af 127.0.0.1 -2022-11-07T22:46:25.698Z [INFO] agent.router: Initializing LAN area manager -46:25.698 [INFO] TestAgent.server: Adding LAN server: server="Node-9e01a9b7-4227-8cbe-968e-d4af620e91af (Addr: tcp/127.0.0.1:11352) (DC: dc1)" -46:25.698 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af.dc1 area=wan -46:25.698 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12007 network=udp -46:25.698 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12007 network=tcp -46:25.699 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12008 network=tcp -46:25.699 [INFO] TestAgent: started state syncer -46:25.699 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11381 network=tcp -46:25.748 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.749 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11352 [Candidate]" term=2 -46:25.749 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:25.768 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.768 [DEBUG] TestAgent.server.raft: vote granted: from=9e01a9b7-4227-8cbe-968e-d4af620e91af term=2 tally=1 -46:25.768 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.768 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11352 [Leader]" -46:25.768 [INFO] TestAgent.server: cluster leadership acquired -46:25.768 [INFO] TestAgent.server: New leader elected: payload=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.776 [INFO] TestAgent.server: initializing acls -46:25.779 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:25.779 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:25.783 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:25.794 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:25.794 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:25.794 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:25.794 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.794 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-9e01a9b7-4227-8cbe-968e-d4af620e91af.dc1 -46:25.794 [INFO] TestAgent.server: Updating LAN server: server="Node-9e01a9b7-4227-8cbe-968e-d4af620e91af (Addr: tcp/127.0.0.1:11352) (DC: dc1)" -46:25.794 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af.dc1 area=wan -46:25.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.832 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.832 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.832 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.832 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.832 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.833 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.833 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.842 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.851 [INFO] TestAgent.server: federation state anti-entropy synced -46:25.901 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.901 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-9e01a9b7-4227-8cbe-968e-d4af620e91af -46:25.914 [INFO] TestAgent: Requesting shutdown -46:25.914 [INFO] TestAgent.server: shutting down server -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:25.914 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:25.914 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:25.914 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:25.915 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:25.915 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:25.922 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:25.934Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:25.934Z [INFO] agent.router.manager: shutting down -46:25.934 [INFO] TestAgent: consul server down -46:25.934 [INFO] TestAgent: shutdown complete -46:25.934 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12007 network=tcp -46:25.934 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12007 network=udp -46:25.934 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12008 network=tcp -46:26.435 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.435 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_UpdateUpsert (0.79s) +2021-10-05T16:32:09.869Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.886Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.966 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ebdec16b-61fe-0466-e1fc-9df672542fd6 Address:127.0.0.1:29804}]" +32:09.966 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29804 [Follower]" leader= +32:09.967 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:09.967 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:09.967Z [INFO] agent.router: Initializing LAN area manager +32:09.968 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29804) (DC: dc1)" +32:09.968 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:09.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=udp +32:09.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29799 network=tcp +32:09.968 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29793 network=tcp +32:09.968 [INFO] TestAgent: started state syncer +32:09.968 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29805 network=tcp +32:10.021 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.021 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29804 [Candidate]" term=2 +32:10.040 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.040 [DEBUG] TestAgent.server.raft: vote granted: from=ebdec16b-61fe-0466-e1fc-9df672542fd6 term=2 tally=1 +32:10.041 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.041 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29804 [Leader]" +32:10.041 [INFO] TestAgent.server: cluster leadership acquired +32:10.041 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.058 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29804 +32:10.100 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.156 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.156 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.156 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.156 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.156 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.156 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.156 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.180 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.185 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.198 [INFO] TestAgent: Synced node info +32:11.789 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.425113ms client=127.0.0.1:39495 client_network=udp +32:11.802 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.387482ms client=127.0.0.1:51549 client_network=udp +32:11.804 [DEBUG] TestAgent.dns: request served from client: name=b0d0bc6a-65e9-3913-d4a1-4ca8a878ae7b.query.consul. type=AAAA class=IN latency=1.054105ms client=127.0.0.1:59924 client_network=udp +32:11.804 [INFO] TestAgent: Requesting shutdown +32:11.804 [INFO] TestAgent.server: shutting down server +32:11.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.804 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.804 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.804 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.804 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.804 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.805 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.810 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.811Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.811Z [INFO] agent.router.manager: shutting down +32:11.811 [INFO] TestAgent: consul server down +32:11.811 [INFO] TestAgent: shutdown complete +32:11.811 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=tcp +32:11.811 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29799 network=udp +32:11.811 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29793 network=tcp +32:12.312 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.312 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{8_8_8_8_6_8_8_5_8_8_-5} -2022-11-07T22:46:25.784Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.792Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.835 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 Address:127.0.0.1:11478}]" -46:25.835 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11478 [Follower]" leader= -46:25.835 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349.dc1 127.0.0.1 -46:25.835 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 127.0.0.1 -2022-11-07T22:46:25.835Z [INFO] agent.router: Initializing LAN area manager -46:25.836 [INFO] TestAgent.server: Adding LAN server: server="Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 (Addr: tcp/127.0.0.1:11478) (DC: dc1)" -46:25.836 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349.dc1 area=wan -46:25.836 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11424 network=udp -46:25.836 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11424 network=tcp -46:25.836 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11425 network=tcp -46:25.836 [INFO] TestAgent: started state syncer -46:25.836 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11507 network=tcp -46:25.899 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.899 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11478 [Candidate]" term=2 -46:25.918 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.918 [DEBUG] TestAgent.server.raft: vote granted: from=dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 term=2 tally=1 -46:25.918 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.918 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11478 [Leader]" -46:25.918 [INFO] TestAgent.server: cluster leadership acquired -46:25.918 [INFO] TestAgent.server: New leader elected: payload=Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 -46:25.936 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11478 -46:25.950 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.973 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.973 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.973 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.973 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.973 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.973 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 -46:25.973 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dadb7f53-a4f8-f1ba-c5c0-47fb72e70349 -46:25.990 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.052 [INFO] TestAgent: Requesting shutdown -46:26.052 [INFO] TestAgent.server: shutting down server -46:26.052 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.052 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.052 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.052 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.052 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.052 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.052 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.052 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:26.052 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.052 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.052 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.057 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.063Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.063Z [INFO] agent.router.manager: shutting down -46:26.063 [INFO] TestAgent: consul server down -46:26.063 [INFO] TestAgent: shutdown complete -46:26.063 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11424 network=tcp -46:26.063 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11424 network=udp -46:26.063 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11425 network=tcp -46:26.564 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.564 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services (0.79s) +2021-10-05T16:32:09.849Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.873Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.961 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8955620f-91be-d49f-2706-0fda4e8af10d Address:127.0.0.1:29832}]" +32:09.961 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29832 [Follower]" leader= +32:09.964 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:09.965 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:09.965Z [INFO] agent.router: Initializing LAN area manager +32:09.965 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=udp +32:09.965 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29832) (DC: dc1)" +32:09.965 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:09.965 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29834 network=tcp +32:09.966 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29800 network=tcp +32:09.966 [INFO] TestAgent: started state syncer +32:09.966 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29833 network=tcp +32:10.014 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:10.014 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29832 [Candidate]" term=2 +32:10.036 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:10.036 [DEBUG] TestAgent.server.raft: vote granted: from=8955620f-91be-d49f-2706-0fda4e8af10d term=2 tally=1 +32:10.036 [INFO] TestAgent.server.raft: election won: tally=1 +32:10.036 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29832 [Leader]" +32:10.036 [INFO] TestAgent.server: cluster leadership acquired +32:10.036 [INFO] TestAgent.server: New leader elected: payload=test-node +32:10.054 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29832 +32:10.088 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:10.152 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:10.152 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:10.152 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:10.152 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:10.152 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:10.152 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:10.152 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:10.188 [INFO] TestAgent.server: federation state anti-entropy synced +32:10.300 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:10.318 [INFO] TestAgent: Synced node info +32:10.318 [DEBUG] TestAgent: Node info in sync +32:11.802 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.218891ms client=127.0.0.1:58838 client_network=udp +32:11.804 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=986.331µs client=127.0.0.1:52722 client_network=udp +32:11.806 [DEBUG] TestAgent.dns: request served from client: name=a139e3ae-81f6-108e-bb8c-173eb6291863.query.consul. type=A class=IN latency=1.479758ms client=127.0.0.1:52264 client_network=udp +32:11.808 [INFO] TestAgent: Requesting shutdown +32:11.808 [INFO] TestAgent.server: shutting down server +32:11.808 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:11.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:11.808 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:11.808 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:11.808 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:11.808 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:11.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:11.808 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:11.808 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:11.808 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:11.814 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:11.815Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:11.815Z [INFO] agent.router.manager: shutting down +32:11.815 [INFO] TestAgent: consul server down +32:11.815 [INFO] TestAgent: shutdown complete +32:11.815 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=tcp +32:11.815 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29834 network=udp +32:11.815 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29800 network=tcp +32:12.316 [INFO] TestAgent: Waiting for endpoints to shut down +32:12.316 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{7_7_7_7_6_7_7_5_7_7_-5} -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:25.872Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.881Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.918 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:078dd028-1c08-2266-ec15-ceb1c27686f3 Address:127.0.0.1:11436}]" -46:25.918 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11436 [Follower]" leader= -46:25.919 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-078dd028-1c08-2266-ec15-ceb1c27686f3.dc1 127.0.0.1 -46:25.919 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-078dd028-1c08-2266-ec15-ceb1c27686f3 127.0.0.1 -2022-11-07T22:46:25.919Z [INFO] agent.router: Initializing LAN area manager -46:25.919 [INFO] TestAgent.server: Adding LAN server: server="Node-078dd028-1c08-2266-ec15-ceb1c27686f3 (Addr: tcp/127.0.0.1:11436) (DC: dc1)" -46:25.919 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-078dd028-1c08-2266-ec15-ceb1c27686f3.dc1 area=wan -46:25.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11508 network=tcp -46:25.919 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11508 network=udp -46:25.920 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11509 network=tcp -46:25.920 [INFO] TestAgent: started state syncer -46:25.920 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11451 network=tcp -46:25.975 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.975 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11436 [Candidate]" term=2 -46:25.985 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.985 [DEBUG] TestAgent.server.raft: vote granted: from=078dd028-1c08-2266-ec15-ceb1c27686f3 term=2 tally=1 -46:25.985 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.985 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11436 [Leader]" -46:25.986 [INFO] TestAgent.server: cluster leadership acquired -46:25.986 [INFO] TestAgent.server: New leader elected: payload=Node-078dd028-1c08-2266-ec15-ceb1c27686f3 -46:25.995 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11436 -46:26.016 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.042 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.042 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.042 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.042 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.042 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.042 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-078dd028-1c08-2266-ec15-ceb1c27686f3 -46:26.042 [INFO] TestAgent.server: member joined, marking health alive: member=Node-078dd028-1c08-2266-ec15-ceb1c27686f3 -46:26.058 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.074 [INFO] TestAgent: Node entered maintenance mode -46:26.077 [INFO] TestAgent: Synced node info -46:26.081 [INFO] TestAgent: Synced check: check=_node_maintenance -46:26.081 [INFO] TestAgent: Requesting shutdown -46:26.081 [INFO] TestAgent.server: shutting down server -46:26.081 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.081 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.081 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.081 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.082 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.082 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.082 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.082 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:26.082 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.082 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.082 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.087 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.090Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.090Z [INFO] agent.router.manager: shutting down -46:26.090 [INFO] TestAgent: consul server down -46:26.090 [INFO] TestAgent: shutdown complete -46:26.090 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11508 network=tcp -46:26.090 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11508 network=udp -46:26.090 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11509 network=tcp -46:26.590 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.590 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_NodeMaintenance_Enable (0.73s) +[WARN] freeport: 4 out of 52 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:58.809Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.853Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.906 [INFO] Agent2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:af776d73-a83f-1d79-6f88-a8045824d8e4 Address:127.0.0.1:30343}]" +31:58.906 [INFO] Agent2.server.raft: entering follower state: follower="Node at 127.0.0.1:30343 [Follower]" leader= +31:58.907 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 127.0.0.1 +31:58.907 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-af776d73-a83f-1d79-6f88-a8045824d8e4 127.0.0.1 +2021-10-05T16:31:58.907Z [INFO] agent.router: Initializing LAN area manager +31:58.907 [INFO] Agent2: Started DNS server: address=127.0.0.1:30324 network=udp +31:58.908 [INFO] Agent2.server: Adding LAN server: server="Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +31:58.908 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 area=wan +31:58.908 [INFO] Agent2: Started DNS server: address=127.0.0.1:30324 network=tcp +31:58.908 [INFO] Agent2: Started HTTP server: address=127.0.0.1:30325 network=tcp +31:58.908 [INFO] Agent2: started state syncer +31:58.908 [INFO] Agent2: Started gRPC server: address=127.0.0.1:30400 network=tcp +31:58.965 [WARN] Agent2.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.965 [INFO] Agent2.server.raft: entering candidate state: node="Node at 127.0.0.1:30343 [Candidate]" term=2 +31:58.993 [DEBUG] Agent2.server.raft: votes: needed=1 +31:58.993 [DEBUG] Agent2.server.raft: vote granted: from=af776d73-a83f-1d79-6f88-a8045824d8e4 term=2 tally=1 +31:58.993 [INFO] Agent2.server.raft: election won: tally=1 +31:58.993 [INFO] Agent2.server.raft: entering leader state: leader="Node at 127.0.0.1:30343 [Leader]" +31:58.993 [INFO] Agent2.server: cluster leadership acquired +31:58.993 [INFO] Agent2.server: New leader elected: payload=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +31:59.032 [DEBUG] Agent2.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30343 +31:59.059 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:59.122 [INFO] Agent2: Synced node info +31:59.144 [INFO] Agent2.server.connect: initialized primary datacenter CA with provider: provider=consul +31:59.144 [INFO] Agent2.leader: started routine: routine="federation state anti-entropy" +31:59.144 [INFO] Agent2.leader: started routine: routine="federation state pruning" +31:59.144 [INFO] Agent2.leader: started routine: routine="intermediate cert renew watch" +31:59.144 [INFO] Agent2.leader: started routine: routine="CA root pruning" +31:59.144 [DEBUG] Agent2.server: Skipping self join check for node since the cluster is too small: node=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +31:59.144 [INFO] Agent2.server: member joined, marking health alive: member=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +31:59.167 [INFO] Agent2.server: federation state anti-entropy synced +31:59.317 [DEBUG] Agent2.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:44506 +31:59.318 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d 127.0.0.1 +31:59.318 [INFO] Agent2.server: Adding LAN server: server="Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d (Addr: tcp/127.0.0.1:30308) (DC: dc1)" +31:59.318 [INFO] Agent2.server: New leader elected: payload=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:59.318 [ERROR] Agent2.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d other=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +31:59.318 [INFO] Agent2.server: member joined, marking health alive: member=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:59.318 [INFO] Agent2: Requesting shutdown +31:59.318 [INFO] Agent2.server: shutting down server +31:59.318 [DEBUG] Agent2.leader: stopping routine: routine="CA root pruning" +31:59.318 [DEBUG] Agent2.leader: stopping routine: routine="federation state anti-entropy" +31:59.318 [DEBUG] Agent2.leader: stopping routine: routine="federation state pruning" +31:59.318 [DEBUG] Agent2.leader: stopping routine: routine="intermediate cert renew watch" +31:59.318 [WARN] Agent2.server.serf.lan: serf: Shutdown without a Leave +31:59.319 [DEBUG] Agent2.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 127.0.0.1:30307 +31:59.319 [ERROR] Agent2.server: error performing anti-entropy sync of federation state: error="context canceled" +31:59.319 [DEBUG] Agent2.leader: stopped routine: routine="federation state anti-entropy" +31:59.319 [DEBUG] Agent2.leader: stopped routine: routine="CA root pruning" +31:59.319 [DEBUG] Agent2.leader: stopped routine: routine="federation state pruning" +31:59.319 [DEBUG] Agent2.leader: stopped routine: routine="intermediate cert renew watch" +31:59.320 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 127.0.0.1 +31:59.320 [DEBUG] Agent2.server: Successfully performed flood-join for server at address: server=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 address=127.0.0.1:30307 +31:59.320 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 area=wan +31:59.328 [WARN] Agent2.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:59.336Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:59.336Z [INFO] agent.router.manager: shutting down +31:59.336 [INFO] Agent2: consul server down +31:59.336 [INFO] Agent2: shutdown complete +31:59.336 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30324 network=tcp +31:59.336 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:30324 network=udp +31:59.336 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:30325 network=tcp +31:59.836 [INFO] Agent2: Waiting for endpoints to shut down +31:59.836 [INFO] Agent2: Endpoints down +2021-10-05T16:31:58.290Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.311Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.372 [INFO] Agent1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e16b1ca7-ac90-850e-14c8-07e1e08b8f4d Address:127.0.0.1:30308}]" +31:58.372 [INFO] Agent1.server.raft: entering follower state: follower="Node at 127.0.0.1:30308 [Follower]" leader= +31:58.373 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 127.0.0.1 +31:58.374 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d 127.0.0.1 +2021-10-05T16:31:58.374Z [INFO] agent.router: Initializing LAN area manager +31:58.374 [INFO] Agent1: Started DNS server: address=127.0.0.1:30303 network=udp +31:58.374 [INFO] Agent1.server: Adding LAN server: server="Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d (Addr: tcp/127.0.0.1:30308) (DC: dc1)" +31:58.374 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d.dc1 area=wan +31:58.374 [INFO] Agent1: Started DNS server: address=127.0.0.1:30303 network=tcp +31:58.375 [INFO] Agent1: Started HTTP server: address=127.0.0.1:30304 network=tcp +31:58.375 [INFO] Agent1: started state syncer +31:58.375 [INFO] Agent1: Started gRPC server: address=127.0.0.1:30344 network=tcp +31:58.434 [WARN] Agent1.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.434 [INFO] Agent1.server.raft: entering candidate state: node="Node at 127.0.0.1:30308 [Candidate]" term=2 +31:58.456 [DEBUG] Agent1.server.raft: votes: needed=1 +31:58.456 [DEBUG] Agent1.server.raft: vote granted: from=e16b1ca7-ac90-850e-14c8-07e1e08b8f4d term=2 tally=1 +31:58.456 [INFO] Agent1.server.raft: election won: tally=1 +31:58.456 [INFO] Agent1.server.raft: entering leader state: leader="Node at 127.0.0.1:30308 [Leader]" +31:58.456 [INFO] Agent1.server: cluster leadership acquired +31:58.456 [INFO] Agent1.server: New leader elected: payload=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:58.471 [DEBUG] Agent1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30308 +31:58.505 [INFO] Agent1: Synced node info +31:58.505 [DEBUG] Agent1: Node info in sync +31:58.521 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.547 [DEBUG] Agent1: Node info in sync +31:58.551 [INFO] Agent1.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.551 [INFO] Agent1.leader: started routine: routine="federation state anti-entropy" +31:58.551 [INFO] Agent1.leader: started routine: routine="federation state pruning" +31:58.551 [INFO] Agent1.leader: started routine: routine="intermediate cert renew watch" +31:58.551 [INFO] Agent1.leader: started routine: routine="CA root pruning" +31:58.551 [DEBUG] Agent1.server: Skipping self join check for node since the cluster is too small: node=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:58.551 [INFO] Agent1.server: member joined, marking health alive: member=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:58.581 [INFO] Agent1.server: federation state anti-entropy synced +31:59.310 [INFO] Agent1: (LAN) joining: lan_addresses=[127.0.0.1:30327] +31:59.317 [DEBUG] Agent1.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:30327 +31:59.318 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-af776d73-a83f-1d79-6f88-a8045824d8e4 127.0.0.1 +31:59.318 [INFO] Agent1: (LAN) joined: number_of_nodes=1 +31:59.318 [DEBUG] Agent1: systemd notify failed: error="No socket" +31:59.319 [INFO] Agent1.server: Adding LAN server: server="Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +31:59.319 [ERROR] Agent1.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 other=Node-e16b1ca7-ac90-850e-14c8-07e1e08b8f4d +31:59.319 [INFO] Agent1.server: member joined, marking health alive: member=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +31:59.319 [DEBUG] Agent1.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:33316 +31:59.319 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 127.0.0.1 +31:59.320 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 area=wan +31:59.875 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (timeout reached) +32:00.376 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-af776d73-a83f-1d79-6f88-a8045824d8e4 has failed, no acks received +32:00.482 [WARN] Agent1: error getting server health from server: server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:30343: connect: connection refused" +32:01.485 [WARN] Agent1: error getting server health from server: server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 error="context deadline exceeded" +32:01.876 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (timeout reached) +32:02.484 [WARN] Agent1: error getting server health from server: server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:30343: connect: connection refused" +32:03.375 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-af776d73-a83f-1d79-6f88-a8045824d8e4 has failed, no acks received +32:03.484 [WARN] Agent1: error getting server health from server: server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 error="context deadline exceeded" +32:04.376 [INFO] Agent1.server.memberlist.lan: memberlist: Marking Node-af776d73-a83f-1d79-6f88-a8045824d8e4 as failed, suspect timeout reached (0 peer confirmations) +32:04.376 [INFO] Agent1.server.serf.lan: serf: EventMemberFailed: Node-af776d73-a83f-1d79-6f88-a8045824d8e4 127.0.0.1 +32:04.376 [INFO] Agent1.server: Removing LAN server: server="Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +32:04.376 [INFO] Agent1.server: member failed, marking health critical: member=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +32:04.388 [INFO] Agent1: Force leaving node: node=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 +32:04.388 [INFO] Agent1.server.serf.lan: serf: EventMemberLeave (forced): Node-af776d73-a83f-1d79-6f88-a8045824d8e4 127.0.0.1 +32:04.388 [INFO] Agent1.server.serf.lan: serf: EventMemberReap (forced): Node-af776d73-a83f-1d79-6f88-a8045824d8e4 127.0.0.1 +32:04.388 [INFO] Agent1.server: Removing LAN server: server="Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +32:04.388 [INFO] Agent1.server: Removing LAN server: server="Node-af776d73-a83f-1d79-6f88-a8045824d8e4 (Addr: tcp/127.0.0.1:30343) (DC: dc1)" +32:04.393 [INFO] Agent1.server: deregistering member: member=Node-af776d73-a83f-1d79-6f88-a8045824d8e4 reason=left +32:06.374 [DEBUG] Agent1.server.memberlist.wan: memberlist: Failed ping: Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 (timeout reached) +32:08.374 [INFO] Agent1.server.memberlist.wan: memberlist: Suspect Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 has failed, no acks received +32:12.389 [INFO] Agent1.server.serf.wan: serf: EventMemberReap (forced): Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 127.0.0.1 +32:12.389 [INFO] Agent1: Requesting shutdown +32:12.389 [INFO] Agent1.server: shutting down server +32:12.389 [DEBUG] Agent1.leader: stopping routine: routine="CA root pruning" +32:12.389 [DEBUG] Agent1.leader: stopping routine: routine="federation state anti-entropy" +32:12.389 [DEBUG] Agent1.leader: stopping routine: routine="federation state pruning" +32:12.389 [DEBUG] Agent1.leader: stopping routine: routine="intermediate cert renew watch" +32:12.389 [WARN] Agent1.server.serf.lan: serf: Shutdown without a Leave +32:12.389 [INFO] Agent1.server: Handled event for server in area: event=member-reap server=Node-af776d73-a83f-1d79-6f88-a8045824d8e4.dc1 area=wan +32:12.389 [ERROR] Agent1.server: error performing anti-entropy sync of federation state: error="context canceled" +32:12.389 [DEBUG] Agent1.leader: stopped routine: routine="federation state anti-entropy" +32:12.389 [DEBUG] Agent1.leader: stopped routine: routine="CA root pruning" +32:12.389 [DEBUG] Agent1.leader: stopped routine: routine="federation state pruning" +32:12.389 [DEBUG] Agent1.leader: stopped routine: routine="intermediate cert renew watch" +32:12.404 [WARN] Agent1.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:12.408Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:12.408Z [INFO] agent.router.manager: shutting down +32:12.408 [INFO] Agent1: consul server down +32:12.408 [INFO] Agent1: shutdown complete +32:12.408 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:30303 network=tcp +32:12.408 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:30303 network=udp +32:12.408 [INFO] Agent1: Stopping server: protocol=HTTP address=127.0.0.1:30304 network=tcp +32:12.909 [INFO] Agent1: Waiting for endpoints to shut down +32:12.909 [INFO] Agent1: Endpoints down +--- PASS: TestAgent_ForceLeavePrune (14.64s) === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{7_7_7_7_6_7_7_5_7_7_-5} -2022-11-07T22:46:25.772Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.781Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.814 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d23410b0-c39d-4a66-0880-793aa55935e6 Address:127.0.0.1:11450}]" -46:25.814 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11450 [Follower]" leader= -46:25.815 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d23410b0-c39d-4a66-0880-793aa55935e6.dc1 127.0.0.1 -46:25.815 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d23410b0-c39d-4a66-0880-793aa55935e6 127.0.0.1 -2022-11-07T22:46:25.815Z [INFO] agent.router: Initializing LAN area manager -46:25.815 [INFO] TestAgent.server: Adding LAN server: server="Node-d23410b0-c39d-4a66-0880-793aa55935e6 (Addr: tcp/127.0.0.1:11450) (DC: dc1)" -46:25.815 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d23410b0-c39d-4a66-0880-793aa55935e6.dc1 area=wan -46:25.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11347 network=udp -46:25.815 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11347 network=tcp -46:25.816 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11348 network=tcp -46:25.816 [INFO] TestAgent: started state syncer -46:25.816 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11493 network=tcp -46:25.853 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.853 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11450 [Candidate]" term=2 -46:25.862 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.862 [DEBUG] TestAgent.server.raft: vote granted: from=d23410b0-c39d-4a66-0880-793aa55935e6 term=2 tally=1 -46:25.862 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.862 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11450 [Leader]" -46:25.862 [INFO] TestAgent.server: cluster leadership acquired -46:25.862 [INFO] TestAgent.server: New leader elected: payload=Node-d23410b0-c39d-4a66-0880-793aa55935e6 -46:25.868 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11450 -46:25.883 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.917 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.917 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.917 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.917 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.917 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.917 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d23410b0-c39d-4a66-0880-793aa55935e6 -46:25.917 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d23410b0-c39d-4a66-0880-793aa55935e6 -46:25.941 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.061 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.064 [INFO] TestAgent: Synced node info -46:26.064 [DEBUG] TestAgent: Node info in sync -46:26.096 [INFO] TestAgent: Requesting shutdown -46:26.096 [INFO] TestAgent.server: shutting down server -46:26.096 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.096 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.096 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.096 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.096 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.096 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.096 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.096 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.096 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.096 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.099 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.104Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.104Z [INFO] agent.router.manager: shutting down -46:26.104 [INFO] TestAgent: consul server down -46:26.104 [INFO] TestAgent: shutdown complete -46:26.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11347 network=tcp -46:26.105 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11347 network=udp -46:26.105 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11348 network=tcp -46:26.605 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.605 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_ExternalConnectProxy (0.84s) +[WARN] freeport: 4 out of 18 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:11.058Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:11.075Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:11.161 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46936632-ada5-d417-9778-644d40cef753 Address:127.0.0.1:29685}]" +32:11.162 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:11.162 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29685 [Follower]" leader= +32:11.162 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:11.162Z [INFO] agent.router: Initializing LAN area manager +32:11.163 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29685) (DC: dc1)" +32:11.163 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:11.163 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=udp +32:11.163 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29666 network=tcp +32:11.163 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29667 network=tcp +32:11.163 [INFO] TestAgent: started state syncer +32:11.163 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29686 network=tcp +32:11.222 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:11.222 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29685 [Candidate]" term=2 +32:11.245 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:11.245 [DEBUG] TestAgent.server.raft: vote granted: from=46936632-ada5-d417-9778-644d40cef753 term=2 tally=1 +32:11.245 [INFO] TestAgent.server.raft: election won: tally=1 +32:11.245 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29685 [Leader]" +32:11.246 [INFO] TestAgent.server: cluster leadership acquired +32:11.246 [INFO] TestAgent.server: New leader elected: payload=test-node +32:11.272 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29685 +32:11.321 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:11.384 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:11.384 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:11.384 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:11.384 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:11.384 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:11.384 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:11.384 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:11.422 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:11.426 [INFO] TestAgent.server: federation state anti-entropy synced +32:11.443 [INFO] TestAgent: Synced node info +32:12.637 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.346548ms client=127.0.0.1:41746 client_network=udp +32:12.639 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.16473ms client=127.0.0.1:36221 client_network=udp +32:12.644 [DEBUG] TestAgent.dns: request served from client: name=c1a8195a-a211-a1ca-01c0-5ab3e7fd1dd1.query.consul. type=A class=IN latency=5.134808ms client=127.0.0.1:43533 client_network=udp +32:12.644 [INFO] TestAgent: Requesting shutdown +32:12.644 [INFO] TestAgent.server: shutting down server +32:12.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:12.644 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:12.644 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:12.644 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:12.644 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:12.644 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:12.644 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:12.644 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:12.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:12.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:12.652 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:12.664Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:12.664Z [INFO] agent.router.manager: shutting down +32:12.664 [INFO] TestAgent: consul server down +32:12.664 [INFO] TestAgent: shutdown complete +32:12.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=tcp +32:12.664 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29666 network=udp +32:12.664 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29667 network=tcp +32:13.168 [INFO] TestAgent: Waiting for endpoints to shut down +32:13.168 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{7_7_7_7_6_7_7_5_7_7_-5} -2022-11-07T22:46:25.769Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.783Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.821 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:511d37e9-05e8-f534-2695-02510c888cf7 Address:127.0.0.1:11324}]" -46:25.821 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11324 [Follower]" leader= -46:25.822 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-511d37e9-05e8-f534-2695-02510c888cf7.dc1 127.0.0.1 -46:25.823 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-511d37e9-05e8-f534-2695-02510c888cf7 127.0.0.1 -2022-11-07T22:46:25.823Z [INFO] agent.router: Initializing LAN area manager -46:25.823 [INFO] TestAgent.server: Adding LAN server: server="Node-511d37e9-05e8-f534-2695-02510c888cf7 (Addr: tcp/127.0.0.1:11324) (DC: dc1)" -46:25.823 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-511d37e9-05e8-f534-2695-02510c888cf7.dc1 area=wan -46:25.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11382 network=udp -46:25.823 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11382 network=tcp -46:25.823 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11383 network=tcp -46:25.823 [INFO] TestAgent: started state syncer -46:25.823 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11346 network=tcp -46:25.869 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.869 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11324 [Candidate]" term=2 -46:25.882 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.882 [DEBUG] TestAgent.server.raft: vote granted: from=511d37e9-05e8-f534-2695-02510c888cf7 term=2 tally=1 -46:25.882 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.882 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11324 [Leader]" -46:25.882 [INFO] TestAgent.server: cluster leadership acquired -46:25.882 [INFO] TestAgent.server: New leader elected: payload=Node-511d37e9-05e8-f534-2695-02510c888cf7 -46:25.890 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11324 -46:25.915 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.954 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.954 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.954 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.954 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.954 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.954 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-511d37e9-05e8-f534-2695-02510c888cf7 -46:25.954 [INFO] TestAgent.server: member joined, marking health alive: member=Node-511d37e9-05e8-f534-2695-02510c888cf7 -46:25.967 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.121 [INFO] TestAgent: Requesting shutdown -46:26.121 [INFO] TestAgent.server: shutting down server -46:26.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.121 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.121 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.121 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.121 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.121 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.121 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.121 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.121 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:26.121 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.121 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.125 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.129Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.129Z [INFO] agent.router.manager: shutting down -46:26.130 [INFO] TestAgent: consul server down -46:26.130 [INFO] TestAgent: shutdown complete -46:26.130 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11382 network=tcp -46:26.130 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11382 network=udp -46:26.130 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11383 network=tcp -46:26.630 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.630 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Services_Sidecar (0.88s) +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:11.763Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:11.774Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:11.811 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22e9c10e-851c-c653-92f0-34186264c685 Address:127.0.0.1:29908}]" +32:11.812 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:11.813 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:11.813Z [INFO] agent.router: Initializing LAN area manager +32:11.813 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29908 [Follower]" leader= +32:11.813 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29908) (DC: dc1)" +32:11.813 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:11.813 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=tcp +32:11.814 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29659 network=udp +32:11.814 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29877 network=tcp +32:11.814 [INFO] TestAgent: started state syncer +32:11.814 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29909 network=tcp +32:11.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:11.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29908 [Candidate]" term=2 +32:11.920 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:11.920 [DEBUG] TestAgent.server.raft: vote granted: from=22e9c10e-851c-c653-92f0-34186264c685 term=2 tally=1 +32:11.920 [INFO] TestAgent.server.raft: election won: tally=1 +32:11.920 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29908 [Leader]" +32:11.920 [INFO] TestAgent.server: cluster leadership acquired +32:11.920 [INFO] TestAgent.server: New leader elected: payload=test-node +32:11.937 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29908 +32:11.959 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.005 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.005 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.005 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.005 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.005 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.005 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.005 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.030 [INFO] TestAgent.server: federation state anti-entropy synced +32:12.296 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:12.321 [INFO] TestAgent: Synced node info +32:13.462 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.663146ms client=127.0.0.1:46822 client_network=udp +32:13.464 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.615623ms client=127.0.0.1:54420 client_network=udp +32:13.466 [DEBUG] TestAgent.dns: request served from client: name=2f84cda9-d847-47c6-48ae-0494c02f597d.query.consul. type=ANY class=IN latency=1.774814ms client=127.0.0.1:58431 client_network=udp +32:13.468 [INFO] TestAgent: Requesting shutdown +32:13.468 [INFO] TestAgent.server: shutting down server +32:13.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:13.468 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:13.468 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:13.468 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:13.468 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:13.468 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:13.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:13.468 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:13.468 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:13.468 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:13.477 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:13.485Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:13.485Z [INFO] agent.router.manager: shutting down +32:13.485 [INFO] TestAgent: consul server down +32:13.485 [INFO] TestAgent: shutdown complete +32:13.485 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=tcp +32:13.485 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29659 network=udp +32:13.485 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29877 network=tcp +32:13.988 [INFO] TestAgent: Waiting for endpoints to shut down +32:13.988 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{6_6_6_6_6_6_6_5_6_6_-5} -2022-11-07T22:46:25.784Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.792Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.832 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9f54024d-4d14-dad2-6be0-d05db8e047e6 Address:127.0.0.1:11471}]" -46:25.832 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11471 [Follower]" leader= -46:25.832 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9f54024d-4d14-dad2-6be0-d05db8e047e6.dc1 127.0.0.1 -46:25.833 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9f54024d-4d14-dad2-6be0-d05db8e047e6 127.0.0.1 -2022-11-07T22:46:25.833Z [INFO] agent.router: Initializing LAN area manager -46:25.833 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11494 network=udp -46:25.833 [INFO] TestAgent.server: Adding LAN server: server="Node-9f54024d-4d14-dad2-6be0-d05db8e047e6 (Addr: tcp/127.0.0.1:11471) (DC: dc1)" -46:25.833 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9f54024d-4d14-dad2-6be0-d05db8e047e6.dc1 area=wan -46:25.833 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11494 network=tcp -46:25.833 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11495 network=tcp -46:25.833 [INFO] TestAgent: started state syncer -46:25.833 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11423 network=tcp -46:25.886 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.886 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11471 [Candidate]" term=2 -46:25.899 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:25.899 [DEBUG] TestAgent.server.raft: vote granted: from=9f54024d-4d14-dad2-6be0-d05db8e047e6 term=2 tally=1 -46:25.899 [INFO] TestAgent.server.raft: election won: tally=1 -46:25.899 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11471 [Leader]" -46:25.899 [INFO] TestAgent.server: cluster leadership acquired -46:25.899 [INFO] TestAgent.server: New leader elected: payload=Node-9f54024d-4d14-dad2-6be0-d05db8e047e6 -46:25.907 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11471 -46:25.937 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:25.958 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:25.958 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:25.958 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:25.958 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:25.958 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:25.958 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9f54024d-4d14-dad2-6be0-d05db8e047e6 -46:25.958 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9f54024d-4d14-dad2-6be0-d05db8e047e6 -46:25.970 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.030 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.033 [INFO] TestAgent: Synced node info -46:26.033 [DEBUG] TestAgent: Node info in sync -46:26.157 [INFO] TestAgent: Requesting shutdown -46:26.157 [INFO] TestAgent.server: shutting down server -46:26.157 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.157 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.157 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.157 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.157 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.157 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.161 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.164Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.164Z [INFO] agent.router.manager: shutting down -46:26.164 [INFO] TestAgent: consul server down -46:26.164 [INFO] TestAgent: shutdown complete -46:26.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11494 network=tcp -46:26.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11494 network=udp -46:26.164 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11495 network=tcp -46:26.664 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.664 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServicesFiltered (0.89s) +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:12.185Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.202Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d156f2f7-912e-0ac6-b1f1-15dd9a9411f6 Address:127.0.0.1:29846}]" +32:12.253 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29846 [Follower]" leader= +32:12.256 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.257 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.257Z [INFO] agent.router: Initializing LAN area manager +32:12.257 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=udp +32:12.257 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29846) (DC: dc1)" +32:12.257 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.258 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29876 network=tcp +32:12.258 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29842 network=tcp +32:12.258 [INFO] TestAgent: started state syncer +32:12.258 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29847 network=tcp +32:12.311 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.311 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29846 [Candidate]" term=2 +32:12.330 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.330 [DEBUG] TestAgent.server.raft: vote granted: from=d156f2f7-912e-0ac6-b1f1-15dd9a9411f6 term=2 tally=1 +32:12.330 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.330 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29846 [Leader]" +32:12.331 [INFO] TestAgent.server: cluster leadership acquired +32:12.331 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.344 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29846 +32:12.380 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.448 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.448 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.448 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.448 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.448 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.448 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.448 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.509 [INFO] TestAgent.server: federation state anti-entropy synced +32:12.716 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:12.752 [INFO] TestAgent: Synced node info +32:12.752 [DEBUG] TestAgent: Node info in sync +32:13.989 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.1817ms client=127.0.0.1:45325 client_network=udp +32:13.990 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=790.349µs client=127.0.0.1:48856 client_network=udp +32:13.991 [DEBUG] TestAgent.dns: request served from client: name=efb32b00-d931-2761-fb05-1a8aa48f0b3b.query.consul. type=A class=IN latency=771.724µs client=127.0.0.1:45621 client_network=udp +32:13.991 [INFO] TestAgent: Requesting shutdown +32:13.991 [INFO] TestAgent.server: shutting down server +32:13.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:13.991 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:13.991 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:13.991 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:13.991 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:13.991 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:13.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:13.991 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:13.991 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:13.991 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.004 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.017Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.017Z [INFO] agent.router.manager: shutting down +32:14.017 [INFO] TestAgent: consul server down +32:14.017 [INFO] TestAgent: shutdown complete +32:14.017 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=tcp +32:14.017 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29876 network=udp +32:14.017 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29842 network=tcp +32:14.520 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.520 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{6_6_6_6_6_6_6_5_6_6_-5} -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:20.519Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.532Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.563 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f74b3b20-a66a-587b-d68e-a5fb873c2c55 Address:127.0.0.1:11913}]" -46:20.563 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11913 [Follower]" leader= -46:20.564 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 127.0.0.1 -46:20.565 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 127.0.0.1 -2022-11-07T22:46:20.565Z [INFO] agent.router: Initializing LAN area manager -46:20.565 [INFO] TestAgent.server: Adding LAN server: server="Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (Addr: tcp/127.0.0.1:11913) (DC: dc1)" -46:20.565 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 area=wan -46:20.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11880 network=tcp -46:20.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11880 network=udp -46:20.565 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11909 network=tcp -46:20.565 [INFO] TestAgent: started state syncer -46:20.566 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11914 network=tcp -46:20.603 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.603 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11913 [Candidate]" term=2 -46:20.615 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.615 [DEBUG] TestAgent.server.raft: vote granted: from=f74b3b20-a66a-587b-d68e-a5fb873c2c55 term=2 tally=1 -46:20.615 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.615 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11913 [Leader]" -46:20.615 [INFO] TestAgent.server: cluster leadership acquired -46:20.616 [INFO] TestAgent.server: New leader elected: payload=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:20.623 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11913 -46:20.639 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.656 [INFO] TestAgent: Synced node info -46:20.671 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.671 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.671 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.671 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.671 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.671 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:20.671 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:20.694 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.740 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:59260 -46:20.740 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 127.0.0.1 -46:20.740 [INFO] TestAgent.server: Adding LAN server: server="Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 (Addr: tcp/127.0.0.1:11892) (DC: dc1)" -46:20.740 [INFO] TestAgent.server: New leader elected: payload=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.740 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 other=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:20.740 [INFO] TestAgent.server: member joined, marking health alive: member=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.740 [INFO] TestAgent: Requesting shutdown -46:20.740 [INFO] TestAgent.server: shutting down server -46:20.740 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.740 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.740 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.740 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.740 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.741 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.741 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.741 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.741 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 127.0.0.1:11891 -46:20.741 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:58052 -46:20.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 127.0.0.1 -46:20.742 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 area=wan -46:20.742 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 address=127.0.0.1:11891 -46:20.750 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.760Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.760Z [INFO] agent.router.manager: shutting down -46:20.760 [INFO] TestAgent: consul server down -46:20.760 [INFO] TestAgent: shutdown complete -46:20.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11880 network=tcp -46:20.760 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11880 network=udp -46:20.760 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11909 network=tcp -46:21.260 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.260 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.295Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.305Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.351 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 Address:127.0.0.1:11892}]" -46:20.351 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11892 [Follower]" leader= -46:20.352 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 127.0.0.1 -46:20.352 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 127.0.0.1 -2022-11-07T22:46:20.352Z [INFO] agent.router: Initializing LAN area manager -46:20.352 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11908 network=udp -46:20.352 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954.dc1 area=wan -46:20.352 [INFO] TestAgent.server: Adding LAN server: server="Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 (Addr: tcp/127.0.0.1:11892) (DC: dc1)" -46:20.353 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11908 network=tcp -46:20.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11888 network=tcp -46:20.353 [INFO] TestAgent: started state syncer -46:20.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11886 network=tcp -46:20.392 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.392 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11892 [Candidate]" term=2 -46:20.401 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.401 [DEBUG] TestAgent.server.raft: vote granted: from=49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 term=2 tally=1 -46:20.401 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.401 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11892 [Leader]" -46:20.401 [INFO] TestAgent.server: cluster leadership acquired -46:20.402 [INFO] TestAgent.server: New leader elected: payload=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.410 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11892 -46:20.433 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.465 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.465 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.465 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.465 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.465 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.465 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.465 [INFO] TestAgent.server: member joined, marking health alive: member=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.472 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.568 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.573 [INFO] TestAgent: Synced node info -46:20.739 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:11911] -46:20.739 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11911 -46:20.740 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 127.0.0.1 -46:20.740 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:20.740 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:20.741 [INFO] TestAgent.server: Adding LAN server: server="Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (Addr: tcp/127.0.0.1:11913) (DC: dc1)" -46:20.741 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 other=Node-49e4cfd0-4ebd-ad29-5f8d-f3ae63f2a954 -46:20.741 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:20.741 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:58590 -46:20.741 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 127.0.0.1:11912 -46:20.741 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 127.0.0.1 -46:20.742 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 area=wan -46:20.742 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55.dc1 address=127.0.0.1:11912 -46:21.861 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (timeout reached) -46:22.314 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:22.314 [DEBUG] TestAgent: Node info in sync -46:22.314 [DEBUG] TestAgent: Node info in sync -46:22.359 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 has failed, no acks received -46:22.416 [WARN] TestAgent: error getting server health from server: server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:11913: connect: connection refused" -46:23.415 [WARN] TestAgent: error getting server health from server: server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 error="context deadline exceeded" -46:23.853 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (timeout reached) -46:24.416 [WARN] TestAgent: error getting server health from server: server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:11913: connect: connection refused" -46:25.353 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 has failed, no acks received -46:25.416 [WARN] TestAgent: error getting server health from server: server=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 error="context deadline exceeded" -46:25.853 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (timeout reached) -46:26.359 [INFO] TestAgent.server.memberlist.lan: memberlist: Marking Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 as failed, suspect timeout reached (0 peer confirmations) -46:26.359 [INFO] TestAgent.server.serf.lan: serf: EventMemberFailed: Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 127.0.0.1 -46:26.359 [INFO] TestAgent.server: Removing LAN server: server="Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (Addr: tcp/127.0.0.1:11913) (DC: dc1)" -46:26.359 [INFO] TestAgent.server: member failed, marking health critical: member=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:26.394 [INFO] TestAgent: Force leaving node: node=Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 -46:26.394 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave (forced): Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 127.0.0.1 -46:26.394 [INFO] TestAgent: Requesting shutdown -46:26.394 [INFO] TestAgent.server: shutting down server -46:26.394 [INFO] TestAgent.server: Removing LAN server: server="Node-f74b3b20-a66a-587b-d68e-a5fb873c2c55 (Addr: tcp/127.0.0.1:11913) (DC: dc1)" -46:26.394 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.394 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.394 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.394 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.394 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.394 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.394 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.394 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.394 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.394 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.400 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.407Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.407Z [INFO] agent.router.manager: shutting down -46:26.407 [INFO] TestAgent: consul server down -46:26.407 [INFO] TestAgent: shutdown complete -46:26.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11908 network=tcp -46:26.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11908 network=udp -46:26.407 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11888 network=tcp -46:26.907 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.907 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ForceLeave (6.63s) -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{6_6_6_6_6_6_6_5_6_6_-5} -2022-11-07T22:46:26.158Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:26.158Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.168Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:26.168Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.193 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec Address:127.0.0.1:11485}]" -46:26.193 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11485 [Follower]" leader= -46:26.194 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec.dc1 127.0.0.1 -46:26.194 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec 127.0.0.1 -2022-11-07T22:46:26.194Z [INFO] agent.router: Initializing LAN area manager -46:26.194 [INFO] TestAgent.server: Adding LAN server: server="Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec (Addr: tcp/127.0.0.1:11485) (DC: dc1)" -46:26.194 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec.dc1 area=wan -46:26.194 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11466 network=udp -46:26.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11466 network=tcp -46:26.195 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11467 network=tcp -46:26.195 [INFO] TestAgent: started state syncer -46:26.195 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11472 network=tcp -46:26.245 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:26.259 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.259 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11485 [Candidate]" term=2 -46:26.275 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.275 [DEBUG] TestAgent.server.raft: vote granted: from=83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec term=2 tally=1 -46:26.275 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.275 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11485 [Leader]" -46:26.275 [INFO] TestAgent.server: cluster leadership acquired -46:26.275 [INFO] TestAgent.server: New leader elected: payload=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.285 [INFO] TestAgent.server: initializing acls -46:26.289 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:26.289 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:26.295 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:26.307 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:26.308 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:26.308 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:26.308 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.308 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec.dc1 -46:26.308 [INFO] TestAgent.server: Updating LAN server: server="Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec (Addr: tcp/127.0.0.1:11485) (DC: dc1)" -46:26.308 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec.dc1 area=wan -46:26.317 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.334 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.334 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.334 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.334 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.334 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.334 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.334 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.338 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.345 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.530 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.530 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-83c84ab7-e9c8-2ab4-c6cb-b63a548cdeec -46:26.543 [INFO] TestAgent: Requesting shutdown -46:26.543 [INFO] TestAgent.server: shutting down server -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.543 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.543 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.543 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.543 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:26.543 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.552 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.556Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.556Z [INFO] agent.router.manager: shutting down -46:26.556 [INFO] TestAgent: consul server down -46:26.556 [INFO] TestAgent: shutdown complete -46:26.556 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11466 network=tcp -46:26.556 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11466 network=udp -46:26.556 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11467 network=tcp -46:27.057 [INFO] TestAgent: Waiting for endpoints to shut down -46:27.057 [INFO] TestAgent: Endpoints down ---- PASS: TestACL_Legacy_Get (1.62s) - --- PASS: TestACL_Legacy_Get/wrong_id (0.71s) - --- PASS: TestACL_Legacy_Get/right_id (0.91s) +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:12.214Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.248Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.322 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a96080b8-b8a7-dc99-76d2-827d344e157c Address:127.0.0.1:29860}]" +32:12.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.323 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.323Z [INFO] agent.router: Initializing LAN area manager +32:12.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=udp +32:12.323 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29860 [Follower]" leader= +32:12.323 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29860) (DC: dc1)" +32:12.324 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.324 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29848 network=tcp +32:12.324 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29863 network=tcp +32:12.324 [INFO] TestAgent: started state syncer +32:12.324 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29861 network=tcp +32:12.379 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.379 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29860 [Candidate]" term=2 +32:12.399 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.399 [DEBUG] TestAgent.server.raft: vote granted: from=a96080b8-b8a7-dc99-76d2-827d344e157c term=2 tally=1 +32:12.400 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.400 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29860 [Leader]" +32:12.400 [INFO] TestAgent.server: cluster leadership acquired +32:12.400 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.425 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29860 +32:12.497 [INFO] TestAgent: Synced node info +32:12.497 [DEBUG] TestAgent: Node info in sync +32:12.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.612 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.612 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.612 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.612 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.612 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.612 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.612 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.635 [INFO] TestAgent.server: federation state anti-entropy synced +32:12.814 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:12.814 [DEBUG] TestAgent: Node info in sync +32:14.093 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.367755ms client=127.0.0.1:59925 client_network=udp +32:14.094 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=976.731µs client=127.0.0.1:50797 client_network=udp +32:14.103 [DEBUG] TestAgent.dns: request served from client: name=51c88945-2d99-b943-48be-978d1be197ac.query.consul. type=ANY class=IN latency=8.38332ms client=127.0.0.1:39380 client_network=udp +32:14.104 [INFO] TestAgent: Requesting shutdown +32:14.104 [INFO] TestAgent.server: shutting down server +32:14.104 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.104 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.104 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.104 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.104 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.104 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.104 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.104 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.104 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.104 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.124 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.132Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.132Z [INFO] agent.router.manager: shutting down +32:14.132 [INFO] TestAgent: consul server down +32:14.132 [INFO] TestAgent: shutdown complete +32:14.132 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=tcp +32:14.132 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29848 network=udp +32:14.132 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29863 network=tcp +32:14.635 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.635 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{3_3_3_3_3_3_3_3_3_3_3} +2021-10-05T16:32:12.165Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.202Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.304 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f900f1d7-b0ed-9f01-6b98-db0d4a671bff Address:127.0.0.1:29874}]" +32:12.304 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29874 [Follower]" leader= +32:12.305 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.305 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.305Z [INFO] agent.router: Initializing LAN area manager +32:12.305 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=udp +32:12.306 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29874) (DC: dc1)" +32:12.306 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29862 network=tcp +32:12.306 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29849 network=tcp +32:12.306 [INFO] TestAgent: started state syncer +32:12.307 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29875 network=tcp +32:12.358 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.358 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29874 [Candidate]" term=2 +32:12.396 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.396 [DEBUG] TestAgent.server.raft: vote granted: from=f900f1d7-b0ed-9f01-6b98-db0d4a671bff term=2 tally=1 +32:12.396 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.396 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29874 [Leader]" +32:12.396 [INFO] TestAgent.server: cluster leadership acquired +32:12.396 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.432 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29874 +32:12.485 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.553 [INFO] TestAgent: Synced node info +32:12.553 [DEBUG] TestAgent: Node info in sync +32:12.583 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.583 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.583 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.583 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.583 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.584 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.584 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.644 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.149 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.248728ms client=127.0.0.1:40241 client_network=udp +32:14.154 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=3.375777ms client=127.0.0.1:54675 client_network=udp +32:14.156 [DEBUG] TestAgent.dns: request served from client: name=40ce07fa-b11b-908e-aae8-6c5afe6fa609.query.consul. type=AAAA class=IN latency=1.140205ms client=127.0.0.1:47397 client_network=udp +32:14.160 [INFO] TestAgent: Requesting shutdown +32:14.160 [INFO] TestAgent.server: shutting down server +32:14.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.160 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.160 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.160 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.160 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.160 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.160 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.160 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.160 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.168 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.184Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.184Z [INFO] agent.router.manager: shutting down +32:14.184 [INFO] TestAgent: consul server down +32:14.184 [INFO] TestAgent: shutdown complete +32:14.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=tcp +32:14.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29862 network=udp +32:14.184 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29849 network=tcp +32:14.685 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.685 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{5_5_5_5_5_5_5_5_5_5_5} -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:25.885Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.895Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:25.940 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e12a5012-4cb1-5a3a-b7f6-106b40a4a607 Address:127.0.0.1:11331}]" -46:25.940 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11331 [Follower]" leader= -46:25.941 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:25.941 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:25.941Z [INFO] agent.router: Initializing LAN area manager -46:25.941 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11331) (DC: dc1)" -46:25.941 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:25.941 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11452 network=udp -46:25.941 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11452 network=tcp -46:25.942 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11453 network=tcp -46:25.942 [INFO] TestAgent: started state syncer -46:25.942 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11465 network=tcp -46:25.992 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:25.992 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11331 [Candidate]" term=2 -46:26.004 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.004 [DEBUG] TestAgent.server.raft: vote granted: from=e12a5012-4cb1-5a3a-b7f6-106b40a4a607 term=2 tally=1 -46:26.004 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.004 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11331 [Leader]" -46:26.004 [INFO] TestAgent.server: cluster leadership acquired -46:26.004 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.012 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11331 -46:26.027 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.049 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.049 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.049 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.049 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.049 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.049 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.049 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.061 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.331 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.336 [INFO] TestAgent: Synced node info -46:26.336 [DEBUG] TestAgent: Node info in sync -46:27.120 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.899609ms client=127.0.0.1:56031 client_network=udp -46:27.122 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.253284ms client=127.0.0.1:34555 client_network=udp -46:27.123 [DEBUG] TestAgent.dns: request served from client: name=e86981f9-28a1-b838-ed56-320e26879499.query.consul. type=A class=IN latency=1.294561ms client=127.0.0.1:53284 client_network=udp -46:27.124 [INFO] TestAgent: Requesting shutdown -46:27.124 [INFO] TestAgent.server: shutting down server -46:27.124 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.124 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.124 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.124 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.124 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.124 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.124 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.124 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.124 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.124 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.133 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.147Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.147Z [INFO] agent.router.manager: shutting down -46:27.147 [INFO] TestAgent: consul server down -46:27.147 [INFO] TestAgent: shutdown complete -46:27.147 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11452 network=tcp -46:27.147 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11452 network=udp -46:27.147 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11453 network=tcp -46:27.647 [INFO] TestAgent: Waiting for endpoints to shut down -46:27.647 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:12.277Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.331Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.404 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:812ad947-98f4-8cf9-0486-35c089118cc9 Address:127.0.0.1:29929}]" +32:12.405 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.405 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.405Z [INFO] agent.router: Initializing LAN area manager +32:12.406 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29929 [Follower]" leader= +32:12.406 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29929) (DC: dc1)" +32:12.406 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.406 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29910 network=tcp +32:12.406 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29910 network=udp +32:12.407 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29904 network=tcp +32:12.407 [INFO] TestAgent: started state syncer +32:12.407 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29930 network=tcp +32:12.448 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.448 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29929 [Candidate]" term=2 +32:12.514 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.514 [DEBUG] TestAgent.server.raft: vote granted: from=812ad947-98f4-8cf9-0486-35c089118cc9 term=2 tally=1 +32:12.514 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.514 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29929 [Leader]" +32:12.515 [INFO] TestAgent.server: cluster leadership acquired +32:12.515 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.547 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29929 +32:12.550 [INFO] TestAgent: Synced node info +32:12.625 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.672 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.672 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.672 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.672 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.672 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.672 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.672 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.717 [INFO] TestAgent.server: federation state anti-entropy synced +32:12.830 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:12.830 [DEBUG] TestAgent: Node info in sync +32:12.830 [DEBUG] TestAgent: Node info in sync +32:14.151 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.090703ms client=127.0.0.1:56342 client_network=udp +32:14.161 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=8.618013ms client=127.0.0.1:47250 client_network=udp +32:14.165 [DEBUG] TestAgent.dns: request served from client: name=0b03ec22-b255-5c41-eb34-d8839b98faa2.query.consul. type=AAAA class=IN latency=1.157133ms client=127.0.0.1:53681 client_network=udp +32:14.165 [INFO] TestAgent: Requesting shutdown +32:14.165 [INFO] TestAgent.server: shutting down server +32:14.165 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.165 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.165 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.165 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.165 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.165 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.165 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.165 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.165 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.165 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.180 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.192Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.192Z [INFO] agent.router.manager: shutting down +32:14.192 [INFO] TestAgent: consul server down +32:14.192 [INFO] TestAgent: shutdown complete +32:14.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29910 network=tcp +32:14.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29910 network=udp +32:14.192 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29904 network=tcp +32:14.692 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.692 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{5_5_5_5_5_5_5_5_5_5_5} -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:26.237Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.249Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.284 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7e1489a0-4500-3d17-7bae-666ce837841a Address:127.0.0.1:11506}]" -46:26.285 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11506 [Follower]" leader= -46:26.285 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.285 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.285Z [INFO] agent.router: Initializing LAN area manager -46:26.285 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.285 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11506) (DC: dc1)" -46:26.286 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11473 network=udp -46:26.286 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11473 network=tcp -46:26.286 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11474 network=tcp -46:26.286 [INFO] TestAgent: started state syncer -46:26.286 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11437 network=tcp -46:26.334 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.334 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11506 [Candidate]" term=2 -46:26.347 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.347 [DEBUG] TestAgent.server.raft: vote granted: from=7e1489a0-4500-3d17-7bae-666ce837841a term=2 tally=1 -46:26.347 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.347 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11506 [Leader]" -46:26.348 [INFO] TestAgent.server: cluster leadership acquired -46:26.348 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.354 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11506 -46:26.364 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.395 [INFO] TestAgent: Synced node info -46:26.395 [DEBUG] TestAgent: Node info in sync -46:26.395 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.395 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.395 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.395 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.395 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.395 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.395 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.404 [INFO] TestAgent.server: federation state anti-entropy synced -46:27.010 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:27.010 [DEBUG] TestAgent: Node info in sync -46:27.343 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=992.902µs client=127.0.0.1:52320 client_network=udp -46:27.343 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=654.406µs client=127.0.0.1:48840 client_network=udp -46:27.344 [DEBUG] TestAgent.dns: request served from client: name=812ace5e-6d5f-06cd-adcd-a33106585e77.query.consul. type=AAAA class=IN latency=650.793µs client=127.0.0.1:49151 client_network=udp -46:27.344 [INFO] TestAgent: Requesting shutdown -46:27.344 [INFO] TestAgent.server: shutting down server -46:27.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.344 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.344 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.344 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.344 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.344 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.344 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.344 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.344 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.353 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.366Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.366Z [INFO] agent.router.manager: shutting down -46:27.366 [INFO] TestAgent: consul server down -46:27.366 [INFO] TestAgent: shutdown complete -46:27.366 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11473 network=tcp -46:27.366 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11473 network=udp -46:27.366 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11474 network=tcp -46:27.867 [INFO] TestAgent: Waiting for endpoints to shut down -46:27.867 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:12.337Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.353Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.415 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2dccd0b7-5b1f-22d0-0649-4f2057510ab2 Address:127.0.0.1:29901}]" +32:12.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.416 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29901 [Follower]" leader= +32:12.417 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.417Z [INFO] agent.router: Initializing LAN area manager +32:12.417 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29892 network=udp +32:12.418 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29901) (DC: dc1)" +32:12.418 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.418 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29892 network=tcp +32:12.418 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29918 network=tcp +32:12.418 [INFO] TestAgent: started state syncer +32:12.418 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29902 network=tcp +32:12.476 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.476 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29901 [Candidate]" term=2 +32:12.548 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.548 [DEBUG] TestAgent.server.raft: vote granted: from=2dccd0b7-5b1f-22d0-0649-4f2057510ab2 term=2 tally=1 +32:12.548 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.548 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29901 [Leader]" +32:12.554 [INFO] TestAgent.server: cluster leadership acquired +32:12.555 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.570 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29901 +32:12.613 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.672 [INFO] TestAgent: Synced node info +32:12.672 [DEBUG] TestAgent: Node info in sync +32:12.700 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.700 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.700 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.700 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.700 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.700 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.700 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.733 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.201 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.531299ms client=127.0.0.1:40045 client_network=udp +32:14.205 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.503931ms client=127.0.0.1:60527 client_network=udp +32:14.214 [DEBUG] TestAgent.dns: request served from client: name=4351701f-0b0d-6a24-fc67-b6da30ef485e.query.consul. type=ANY class=IN latency=1.553124ms client=127.0.0.1:42120 client_network=udp +32:14.220 [INFO] TestAgent: Requesting shutdown +32:14.220 [INFO] TestAgent.server: shutting down server +32:14.220 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.220 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.220 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.220 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.220 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.220 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.220 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.220 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.228 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.238Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.238Z [INFO] agent.router.manager: shutting down +32:14.238 [INFO] TestAgent: consul server down +32:14.238 [INFO] TestAgent: shutdown complete +32:14.238 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29892 network=tcp +32:14.238 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29892 network=udp +32:14.239 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29918 network=tcp +32:14.741 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.741 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{5_5_5_5_5_5_5_5_5_5_5} -2022-11-07T22:46:26.256Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.271Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.305 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:417c5a01-0a39-c470-5f17-25704a26c838 Address:127.0.0.1:11499}]" -46:26.305 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11499 [Follower]" leader= -46:26.305 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.306 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.306Z [INFO] agent.router: Initializing LAN area manager -46:26.306 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11499) (DC: dc1)" -46:26.306 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11438 network=udp -46:26.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11438 network=tcp -46:26.307 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11439 network=tcp -46:26.307 [INFO] TestAgent: started state syncer -46:26.307 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11520 network=tcp -46:26.365 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.365 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11499 [Candidate]" term=2 -46:26.377 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.377 [DEBUG] TestAgent.server.raft: vote granted: from=417c5a01-0a39-c470-5f17-25704a26c838 term=2 tally=1 -46:26.377 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.377 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11499 [Leader]" -46:26.377 [INFO] TestAgent.server: cluster leadership acquired -46:26.378 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.386 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11499 -46:26.405 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.434 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.434 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.434 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.434 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.434 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.434 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.434 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.449 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.615 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.633 [INFO] TestAgent: Synced node info -46:27.416 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.018225ms client=127.0.0.1:51663 client_network=udp -46:27.417 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=708.663µs client=127.0.0.1:57629 client_network=udp -46:27.418 [DEBUG] TestAgent.dns: request served from client: name=60980c4f-fe46-8782-a3cd-61c21aa1e3ad.query.consul. type=A class=IN latency=901.027µs client=127.0.0.1:33005 client_network=udp -46:27.418 [INFO] TestAgent: Requesting shutdown -46:27.418 [INFO] TestAgent.server: shutting down server -46:27.418 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.418 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.418 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.418 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.418 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.418 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.418 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.418 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.418 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.418 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.427 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.439Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.439Z [INFO] agent.router.manager: shutting down -46:27.439 [INFO] TestAgent: consul server down -46:27.439 [INFO] TestAgent: shutdown complete -46:27.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11438 network=tcp -46:27.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11438 network=udp -46:27.439 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11439 network=tcp -46:27.940 [INFO] TestAgent: Waiting for endpoints to shut down -46:27.940 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:12.356Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.419Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.549 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f176c4ef-6c47-d4a7-cabe-fca688767be5 Address:127.0.0.1:29922}]" +32:12.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:12.552 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:12.552Z [INFO] agent.router: Initializing LAN area manager +32:12.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29917 network=udp +32:12.553 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29917 network=tcp +32:12.554 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29897 network=tcp +32:12.554 [INFO] TestAgent: started state syncer +32:12.554 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29923 network=tcp +32:12.555 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29922 [Follower]" leader= +32:12.555 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:12.555 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29922) (DC: dc1)" +32:12.592 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.592 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29922 [Candidate]" term=2 +32:12.640 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.640 [DEBUG] TestAgent.server.raft: vote granted: from=f176c4ef-6c47-d4a7-cabe-fca688767be5 term=2 tally=1 +32:12.640 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.640 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29922 [Leader]" +32:12.641 [INFO] TestAgent.server: cluster leadership acquired +32:12.641 [INFO] TestAgent.server: New leader elected: payload=test-node +32:12.680 [INFO] TestAgent: Synced node info +32:12.680 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29922 +32:12.756 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.815 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.815 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.815 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.815 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.815 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.815 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:12.815 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:12.847 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.088 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:14.088 [DEBUG] TestAgent: Node info in sync +32:14.088 [DEBUG] TestAgent: Node info in sync +32:14.196 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.111388ms client=127.0.0.1:55895 client_network=udp +32:14.205 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.454029ms client=127.0.0.1:48970 client_network=udp +32:14.213 [DEBUG] TestAgent.dns: request served from client: name=b6af46ea-6f30-fcc3-03d6-fcf21948717e.query.consul. type=A class=IN latency=1.175914ms client=127.0.0.1:60791 client_network=udp +32:14.222 [INFO] TestAgent: Requesting shutdown +32:14.222 [INFO] TestAgent.server: shutting down server +32:14.222 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.222 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.222 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.222 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.222 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.222 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.222 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.222 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.222 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.222 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.229 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.242Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.242Z [INFO] agent.router.manager: shutting down +32:14.242 [INFO] TestAgent: consul server down +32:14.242 [INFO] TestAgent: shutdown complete +32:14.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29917 network=tcp +32:14.242 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29917 network=udp +32:14.242 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29897 network=tcp +32:14.744 [INFO] TestAgent: Waiting for endpoints to shut down +32:14.744 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{4_4_4_4_4_4_4_4_4_4_4} -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:26.273Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.282Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.311 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1d59cf45-3456-7630-7427-fcc6f02ade08 Address:127.0.0.1:11464}]" -46:26.311 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11464 [Follower]" leader= -46:26.311 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.312Z [INFO] agent.router: Initializing LAN area manager -46:26.312 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11464) (DC: dc1)" -46:26.312 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11521 network=udp -46:26.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11521 network=tcp -46:26.312 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11522 network=tcp -46:26.313 [INFO] TestAgent: started state syncer -46:26.313 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11500 network=tcp -46:26.377 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.377 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11464 [Candidate]" term=2 -46:26.392 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.392 [DEBUG] TestAgent.server.raft: vote granted: from=1d59cf45-3456-7630-7427-fcc6f02ade08 term=2 tally=1 -46:26.392 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.392 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11464 [Leader]" -46:26.392 [INFO] TestAgent.server: cluster leadership acquired -46:26.392 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.401 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11464 -46:26.425 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.455 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.455 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.455 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.455 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.455 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.455 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.455 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.469 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.643 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.647 [INFO] TestAgent: Synced node info -46:26.909 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.909 [DEBUG] TestAgent: Node info in sync -46:26.909 [DEBUG] TestAgent: Node info in sync -46:27.573 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.368921ms client=127.0.0.1:45996 client_network=udp -46:27.574 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=873.001µs client=127.0.0.1:33411 client_network=udp -46:27.575 [DEBUG] TestAgent.dns: request served from client: name=362bcb16-c880-13a8-6957-a76e75a96e60.query.consul. type=ANY class=IN latency=783.589µs client=127.0.0.1:39724 client_network=udp -46:27.575 [INFO] TestAgent: Requesting shutdown -46:27.575 [INFO] TestAgent.server: shutting down server -46:27.575 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.575 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.575 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.575 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.575 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.575 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.575 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.575 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.575 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.575 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.586 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.597Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.597Z [INFO] agent.router.manager: shutting down -46:27.597 [INFO] TestAgent: consul server down -46:27.597 [INFO] TestAgent: shutdown complete -46:27.597 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11521 network=tcp -46:27.597 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11521 network=udp -46:27.597 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11522 network=tcp -46:28.098 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.098 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 39 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:12.930Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.965Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:13.038 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fc27fb0a-754f-a611-5fb9-829c50000f0e Address:127.0.0.1:29915}]" +32:13.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:13.039 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:13.039Z [INFO] agent.router: Initializing LAN area manager +32:13.040 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29915 [Follower]" leader= +32:13.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29931 network=udp +32:13.040 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:13.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29931 network=tcp +32:13.040 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29915) (DC: dc1)" +32:13.041 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29932 network=tcp +32:13.041 [INFO] TestAgent: started state syncer +32:13.041 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29916 network=tcp +32:13.099 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:13.099 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29915 [Candidate]" term=2 +32:13.128 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:13.128 [DEBUG] TestAgent.server.raft: vote granted: from=fc27fb0a-754f-a611-5fb9-829c50000f0e term=2 tally=1 +32:13.128 [INFO] TestAgent.server.raft: election won: tally=1 +32:13.128 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29915 [Leader]" +32:13.129 [INFO] TestAgent.server: cluster leadership acquired +32:13.129 [INFO] TestAgent.server: New leader elected: payload=test-node +32:13.152 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29915 +32:13.195 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:13.219 [INFO] TestAgent: Synced node info +32:13.219 [DEBUG] TestAgent: Node info in sync +32:13.252 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:13.252 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:13.252 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:13.252 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:13.252 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:13.252 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:13.252 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:13.292 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.566 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.761884ms client=127.0.0.1:49828 client_network=udp +32:14.569 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=2.063275ms client=127.0.0.1:42799 client_network=udp +32:14.574 [DEBUG] TestAgent.dns: request served from client: name=14769b1b-ebb4-0ff2-1e73-46ec6ad4c0d7.query.consul. type=AAAA class=IN latency=1.451783ms client=127.0.0.1:57728 client_network=udp +32:14.574 [INFO] TestAgent: Requesting shutdown +32:14.574 [INFO] TestAgent.server: shutting down server +32:14.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.574 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.574 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.574 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.574 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.574 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.574 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.588 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.600Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.600Z [INFO] agent.router.manager: shutting down +32:14.600 [INFO] TestAgent: consul server down +32:14.600 [INFO] TestAgent: shutdown complete +32:14.600 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29931 network=tcp +32:14.600 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29931 network=udp +32:14.600 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29932 network=tcp +32:15.103 [INFO] TestAgent: Waiting for endpoints to shut down +32:15.103 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{4_4_4_4_4_4_4_4_4_4_4} -2022-11-07T22:46:19.754Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.764Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.796 [INFO] dc2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b Address:127.0.0.1:11843}]" -46:19.796 [INFO] dc2.server.raft: entering follower state: follower="Node at 127.0.0.1:11843 [Follower]" leader= -46:19.797 [INFO] dc2.server.serf.wan: serf: EventMemberJoin: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 127.0.0.1 -46:19.797 [INFO] dc2.server.serf.lan: serf: EventMemberJoin: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b 127.0.0.1 -2022-11-07T22:46:19.797Z [INFO] agent.router: Initializing LAN area manager -46:19.798 [INFO] dc2: Started DNS server: address=127.0.0.1:11836 network=udp -46:19.798 [INFO] dc2.server: Adding LAN server: server="Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b (Addr: tcp/127.0.0.1:11843) (DC: dc2)" -46:19.798 [INFO] dc2: Started DNS server: address=127.0.0.1:11836 network=tcp -46:19.798 [INFO] dc2.server: Handled event for server in area: event=member-join server=Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 area=wan -46:19.798 [INFO] dc2: Started HTTP server: address=127.0.0.1:11853 network=tcp -46:19.798 [INFO] dc2: started state syncer -46:19.798 [INFO] dc2: Started gRPC server: address=127.0.0.1:11770 network=tcp -46:19.840 [WARN] dc2.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.840 [INFO] dc2.server.raft: entering candidate state: node="Node at 127.0.0.1:11843 [Candidate]" term=2 -46:19.853 [DEBUG] dc2.server.raft: votes: needed=1 -46:19.853 [DEBUG] dc2.server.raft: vote granted: from=a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b term=2 tally=1 -46:19.853 [INFO] dc2.server.raft: election won: tally=1 -46:19.853 [INFO] dc2.server.raft: entering leader state: leader="Node at 127.0.0.1:11843 [Leader]" -46:19.853 [INFO] dc2.server: cluster leadership acquired -46:19.853 [INFO] dc2.server: New leader elected: payload=Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b -46:19.861 [DEBUG] dc2.server: Cannot upgrade to new ACLs, servers in acl datacenter are not yet upgraded: ACLDatacenter=dc1 mode=1 found=false -46:19.869 [WARN] dc2.server.connect: primary datacenter is configured but unreachable - deferring initialization of the secondary datacenter CA -46:19.869 [INFO] dc2.leader: started routine: routine="config entry replication" -46:19.869 [INFO] dc2.leader: started routine: routine="federation state replication" -46:19.869 [INFO] dc2.leader: started routine: routine="federation state anti-entropy" -46:19.869 [INFO] dc2.leader: started routine: routine="secondary CA roots watch" -46:19.869 [INFO] dc2.leader: started routine: routine="intention replication" -46:19.869 [INFO] dc2.leader: started routine: routine="intermediate cert renew watch" -46:19.869 [INFO] dc2.leader: started routine: routine="CA root pruning" -46:19.869 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConfigEntry.ListAll -46:19.869 [DEBUG] dc2.server.connect: starting Connect CA root replication from primary datacenter: primary=dc1 -46:19.869 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=ConnectCA.Roots -46:19.869 [ERROR] dc2.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: No path to datacenter" -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: federation states are not enabled in the primary dc -46:19.869 [DEBUG] dc2.server: Skipping self join check for node since the cluster is too small: node=Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b -46:19.869 [INFO] dc2.server: member joined, marking health alive: member=Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b -46:19.869 [DEBUG] dc2.server.connect: starting Connect intention replication from primary datacenter: primary=dc1 -46:19.869 [WARN] dc2.server.rpc: RPC request for DC is currently failing as no path was found: datacenter=dc1 method=Intention.List -46:19.869 [ERROR] dc2.server.connect: error replicating intentions: routine="intention replication" error="No path to datacenter" -46:20.080 [INFO] dc2: (WAN) joining: wan_addresses=[127.0.0.1:11593] -46:20.081 [DEBUG] dc2.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:11593 -46:20.081 [INFO] dc2.server.serf.wan: serf: EventMemberJoin: Node-109cf0b0-96f7-3027-317d-82cab9aebbc6.dc1 127.0.0.1 -46:20.081 [INFO] dc2: (WAN) joined: number_of_nodes=1 -46:20.081 [INFO] dc2.server: Handled event for server in area: event=member-join server=Node-109cf0b0-96f7-3027-317d-82cab9aebbc6.dc1 area=wan -46:20.140 [DEBUG] dc2: Skipping remote check since it is managed automatically: check=serfHealth -46:20.150 [INFO] dc2: Synced node info -46:20.256 [DEBUG] dc2: Skipping remote check since it is managed automatically: check=serfHealth -46:20.256 [DEBUG] dc2: Node info in sync -46:20.256 [DEBUG] dc2: Node info in sync -46:20.352 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:20.852 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:20.904 [DEBUG] dc2.server.replication.config_entry: finished fetching config entries: amount=0 -46:20.904 [DEBUG] dc2.server.replication.config_entry: Config Entry replication: local=0 remote=0 -46:20.904 [DEBUG] dc2.server.replication.config_entry: Config Entry replication: deletions=0 updates=0 -46:20.904 [DEBUG] dc2.server.replication.config_entry: replication completed through remote index: index=1 -46:20.963 [DEBUG] dc2.server: finished fetching remote objects: amount=1 -46:20.963 [DEBUG] dc2.server: diffing replication state: local_amount=0 remote_amount=1 -46:20.963 [DEBUG] dc2.server: diffed replication state: deletions=0 updates=1 -46:20.963 [DEBUG] dc2.server: performing updates: updates=1 -46:20.969 [DEBUG] dc2.server: finished updates -46:20.969 [DEBUG] dc2.replication.federation_state: replication completed through remote index: index=10 -46:21.355 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:21.851 [DEBUG] dc2.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:21.874 [DEBUG] connect.ca.consul: consul CA provider configured: id=ad:4a:c6:ab:ef:63:c9:60:1a:51:7f:19:62:e3:e9:d9:0e:76:55:10:6e:74:24:69:28:a1:6c:b8:b9:8f:fd:89 is_primary=false -46:21.894 [INFO] dc2.server.connect: received new intermediate certificate from primary datacenter -46:21.900 [INFO] dc2.server.connect: updated root certificates from primary datacenter -46:21.925 [DEBUG] dc2: Node info in sync -46:21.931 [INFO] dc2: Synced service: service=foo -46:21.931 [DEBUG] dc2: Check in sync: check=service:foo -46:21.932 [DEBUG] dc2: Node info in sync -46:21.932 [DEBUG] dc2: Service in sync: service=foo -46:21.932 [DEBUG] dc2: Check in sync: check=service:foo -46:24.875 [DEBUG] dc2.server: finished fetching remote objects: amount=2 -46:24.875 [DEBUG] dc2.server: diffing replication state: local_amount=1 remote_amount=2 -46:24.875 [DEBUG] dc2.server: diffed replication state: deletions=0 updates=1 -46:24.875 [DEBUG] dc2.server: performing updates: updates=1 -46:24.875 [INFO] dc2.server: federation state anti-entropy synced -46:24.879 [DEBUG] dc2.server: finished updates -46:24.879 [DEBUG] dc2.replication.federation_state: replication completed through remote index: index=19 -46:27.055 [INFO] dc2.server.connect: received new intermediate certificate from primary datacenter -46:27.065 [INFO] dc2.server.connect: updated root certificates from primary datacenter -46:27.089 [INFO] dc2.server.connect: updated root certificates from primary datacenter -46:27.105 [INFO] dc2: Requesting shutdown -46:27.105 [INFO] dc2.server: shutting down server -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="federation state anti-entropy" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="secondary CA roots watch" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="intention replication" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="intermediate cert renew watch" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="CA root pruning" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="config entry replication" -46:27.105 [DEBUG] dc2.leader: stopping routine: routine="federation state replication" -46:27.105 [WARN] dc2.server.serf.lan: serf: Shutdown without a Leave -46:27.105 [DEBUG] dc2.leader: stopped routine: routine="federation state anti-entropy" -46:27.105 [DEBUG] dc2.leader: stopped routine: routine="intermediate cert renew watch" -46:27.105 [DEBUG] dc2.leader: stopped routine: routine="CA root pruning" -46:27.114 [WARN] dc2.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.119Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.119Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.119Z [INFO] agent.router.manager: shutting down -46:27.120 [INFO] dc2: consul server down -46:27.120 [INFO] dc2: shutdown complete -46:27.120 [INFO] dc2: Stopping server: protocol=DNS address=127.0.0.1:11836 network=tcp -46:27.120 [INFO] dc2: Stopping server: protocol=DNS address=127.0.0.1:11836 network=udp -46:27.120 [INFO] dc2: Stopping server: protocol=HTTP address=127.0.0.1:11853 network=tcp -46:27.120 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:11594 datacenter=dc1 method=ConfigEntry.ListAll error="rpc error making call: EOF" -46:27.120 [INFO] dc2.server.replication.config_entry: stopped replication -46:27.120 [DEBUG] dc2.leader: stopped routine: routine="config entry replication" -46:27.120 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:11594 datacenter=dc1 method=ConnectCA.Roots error="rpc error making call: EOF" -46:27.120 [ERROR] dc2.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Error retrieving the primary datacenter's roots: rpc error making call: EOF" -46:27.120 [DEBUG] dc2.leader: stopped routine: routine="secondary CA roots watch" -46:27.120 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:11594 datacenter=dc1 method=Intention.List error="rpc error making call: EOF" -46:27.120 [ERROR] dc2.server.connect: error replicating intentions: routine="intention replication" error="rpc error making call: EOF" -46:27.120 [DEBUG] dc2.leader: stopped routine: routine="intention replication" -46:27.120 [ERROR] dc2.server.rpc: RPC failed to server in DC: server=127.0.0.1:11594 datacenter=dc1 method=FederationState.List error="rpc error making call: EOF" -46:27.120 [WARN] dc2.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: EOF" -46:27.120 [INFO] dc2.replication.federation_state: stopped replication -46:27.120 [DEBUG] dc2.leader: stopped routine: routine="federation state replication" -46:27.620 [INFO] dc2: Waiting for endpoints to shut down -46:27.620 [INFO] dc2: Endpoints down -2022-11-07T22:46:19.310Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.323Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.350 [INFO] dc1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:109cf0b0-96f7-3027-317d-82cab9aebbc6 Address:127.0.0.1:11594}]" -46:19.350 [INFO] dc1.server.raft: entering follower state: follower="Node at 127.0.0.1:11594 [Follower]" leader= -46:19.350 [INFO] dc1.server.serf.wan: serf: EventMemberJoin: Node-109cf0b0-96f7-3027-317d-82cab9aebbc6.dc1 127.0.0.1 -46:19.351 [INFO] dc1.server.serf.lan: serf: EventMemberJoin: Node-109cf0b0-96f7-3027-317d-82cab9aebbc6 127.0.0.1 -2022-11-07T22:46:19.351Z [INFO] agent.router: Initializing LAN area manager -46:19.351 [INFO] dc1.server: Adding LAN server: server="Node-109cf0b0-96f7-3027-317d-82cab9aebbc6 (Addr: tcp/127.0.0.1:11594) (DC: dc1)" -46:19.351 [INFO] dc1.server: Handled event for server in area: event=member-join server=Node-109cf0b0-96f7-3027-317d-82cab9aebbc6.dc1 area=wan -46:19.351 [INFO] dc1: Started DNS server: address=127.0.0.1:11547 network=udp -46:19.351 [INFO] dc1: Started DNS server: address=127.0.0.1:11547 network=tcp -46:19.352 [INFO] dc1: Started HTTP server: address=127.0.0.1:11590 network=tcp -46:19.352 [INFO] dc1: started state syncer -46:19.352 [INFO] dc1: Started gRPC server: address=127.0.0.1:11763 network=tcp -46:19.403 [WARN] dc1.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.403 [INFO] dc1.server.raft: entering candidate state: node="Node at 127.0.0.1:11594 [Candidate]" term=2 -46:19.414 [DEBUG] dc1.server.raft: votes: needed=1 -46:19.414 [DEBUG] dc1.server.raft: vote granted: from=109cf0b0-96f7-3027-317d-82cab9aebbc6 term=2 tally=1 -46:19.414 [INFO] dc1.server.raft: election won: tally=1 -46:19.414 [INFO] dc1.server.raft: entering leader state: leader="Node at 127.0.0.1:11594 [Leader]" -46:19.414 [INFO] dc1.server: cluster leadership acquired -46:19.415 [INFO] dc1.server: New leader elected: payload=Node-109cf0b0-96f7-3027-317d-82cab9aebbc6 -46:19.421 [DEBUG] dc1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11594 -46:19.437 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.462 [INFO] dc1.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.462 [INFO] dc1.leader: started routine: routine="federation state anti-entropy" -46:19.462 [INFO] dc1.leader: started routine: routine="federation state pruning" -46:19.462 [INFO] dc1.leader: started routine: routine="intermediate cert renew watch" -46:19.462 [INFO] dc1.leader: started routine: routine="CA root pruning" -46:19.462 [DEBUG] dc1.server: Skipping self join check for node since the cluster is too small: node=Node-109cf0b0-96f7-3027-317d-82cab9aebbc6 -46:19.462 [INFO] dc1.server: member joined, marking health alive: member=Node-109cf0b0-96f7-3027-317d-82cab9aebbc6 -46:19.477 [INFO] dc1.server: federation state anti-entropy synced -46:19.666 [DEBUG] dc1: Skipping remote check since it is managed automatically: check=serfHealth -46:19.669 [INFO] dc1: Synced node info -46:19.669 [DEBUG] dc1: Node info in sync -46:20.081 [DEBUG] dc1.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:38572 -46:20.081 [INFO] dc1.server.serf.wan: serf: EventMemberJoin: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 127.0.0.1 -46:20.081 [INFO] dc1.server: Handled event for server in area: event=member-join server=Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 area=wan -46:20.086 [DEBUG] connect.ca.consul: consul CA provider configured: id=62:77:2b:52:97:0b:fc:2d:df:1a:5d:ef:60:85:8d:fe:14:71:b7:86:a3:d5:8c:91:67:e7:1d:e1:9d:a8:8f:e6 is_primary=true -46:20.107 [INFO] dc1.server.connect: CA rotated to new root under provider: provider=consul -46:20.298 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:20.798 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:21.297 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:21.797 [DEBUG] dc1.server.serf.wan: serf: messageJoinType: Node-a385efbf-aed7-eed2-a09b-c7a2fa8b4e9b.dc2 -46:21.812 [DEBUG] dc1: Skipping remote check since it is managed automatically: check=serfHealth -46:21.812 [DEBUG] dc1: Node info in sync -46:26.977 [DEBUG] connect.ca.consul: consul CA provider configured: id=f4:86:f3:c1:a7:82:d7:cc:90:5b:a9:28:a0:e5:d0:8f:cf:c7:1f:89:51:67:66:9c:99:9b:01:42:99:7a:30:8c is_primary=true -46:27.023 [INFO] dc1.server.connect: CA rotated to new root under provider: provider=consul -46:27.620 [INFO] dc1: Requesting shutdown -46:27.620 [INFO] dc1.server: shutting down server -46:27.620 [DEBUG] dc1.leader: stopping routine: routine="intermediate cert renew watch" -46:27.620 [DEBUG] dc1.leader: stopping routine: routine="CA root pruning" -46:27.620 [DEBUG] dc1.leader: stopping routine: routine="federation state anti-entropy" -46:27.620 [DEBUG] dc1.leader: stopping routine: routine="federation state pruning" -46:27.620 [WARN] dc1.server.serf.lan: serf: Shutdown without a Leave -46:27.620 [ERROR] dc1.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.620 [DEBUG] dc1.leader: stopped routine: routine="federation state anti-entropy" -46:27.620 [DEBUG] dc1.leader: stopped routine: routine="intermediate cert renew watch" -46:27.620 [DEBUG] dc1.leader: stopped routine: routine="CA root pruning" -46:27.620 [DEBUG] dc1.leader: stopped routine: routine="federation state pruning" -46:27.627 [WARN] dc1.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.641Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.641Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.641Z [INFO] agent.router.manager: shutting down -46:27.641 [INFO] dc1: consul server down -46:27.641 [INFO] dc1: shutdown complete -46:27.641 [INFO] dc1: Stopping server: protocol=DNS address=127.0.0.1:11547 network=tcp -46:27.641 [INFO] dc1: Stopping server: protocol=DNS address=127.0.0.1:11547 network=udp -46:27.641 [INFO] dc1: Stopping server: protocol=HTTP address=127.0.0.1:11590 network=tcp -46:28.142 [INFO] dc1: Waiting for endpoints to shut down -46:28.142 [INFO] dc1: Endpoints down ---- PASS: TestAgentConnectCALeafCert_secondaryDC_good (8.84s) +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:13.182Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:13.193Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:13.267 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:595b2589-0f2d-9905-983e-d7fce6ad807c Address:127.0.0.1:29776}]" +32:13.267 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29776 [Follower]" leader= +32:13.268 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:13.270 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:13.270Z [INFO] agent.router: Initializing LAN area manager +32:13.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29903 network=udp +32:13.270 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29776) (DC: dc1)" +32:13.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29903 network=tcp +32:13.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:13.271 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29856 network=tcp +32:13.271 [INFO] TestAgent: started state syncer +32:13.271 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29777 network=tcp +32:13.308 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:13.308 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29776 [Candidate]" term=2 +32:13.338 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:13.338 [DEBUG] TestAgent.server.raft: vote granted: from=595b2589-0f2d-9905-983e-d7fce6ad807c term=2 tally=1 +32:13.338 [INFO] TestAgent.server.raft: election won: tally=1 +32:13.338 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29776 [Leader]" +32:13.338 [INFO] TestAgent.server: cluster leadership acquired +32:13.338 [INFO] TestAgent.server: New leader elected: payload=test-node +32:13.350 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29776 +32:13.382 [INFO] TestAgent: Synced node info +32:13.429 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:13.468 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:13.468 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:13.468 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:13.468 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:13.468 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:13.468 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:13.468 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:13.501 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.785 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.418159ms client=127.0.0.1:49112 client_network=udp +32:14.797 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.341018ms client=127.0.0.1:38483 client_network=udp +32:14.804 [DEBUG] TestAgent.dns: request served from client: name=145a9a47-aeb0-6cee-058a-33b71ef1a236.query.consul. type=A class=IN latency=1.559004ms client=127.0.0.1:55122 client_network=udp +32:14.804 [INFO] TestAgent: Requesting shutdown +32:14.804 [INFO] TestAgent.server: shutting down server +32:14.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:14.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:14.804 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:14.804 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:14.804 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:14.804 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:14.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:14.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:14.804 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:14.804 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:14.816 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:14.828Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:14.828Z [INFO] agent.router.manager: shutting down +32:14.828 [INFO] TestAgent: consul server down +32:14.828 [INFO] TestAgent: shutdown complete +32:14.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29903 network=tcp +32:14.828 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29903 network=udp +32:14.828 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29856 network=tcp +32:15.330 [INFO] TestAgent: Waiting for endpoints to shut down +32:15.330 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{4_4_4_4_4_4_4_4_4_4_4} -2022-11-07T22:46:26.366Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.377Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.415 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1c4c36b0-8c81-0d89-319f-bfab9bbca7c4 Address:127.0.0.1:11282}]" -46:26.415 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11282 [Follower]" leader= -46:26.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.416 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.416Z [INFO] agent.router: Initializing LAN area manager -46:26.416 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11282) (DC: dc1)" -46:26.416 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.416 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11501 network=tcp -46:26.416 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11501 network=udp -46:26.417 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11502 network=tcp -46:26.417 [INFO] TestAgent: started state syncer -46:26.417 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11374 network=tcp -46:26.480 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.480 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11282 [Candidate]" term=2 -46:26.494 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.494 [DEBUG] TestAgent.server.raft: vote granted: from=1c4c36b0-8c81-0d89-319f-bfab9bbca7c4 term=2 tally=1 -46:26.494 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.494 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11282 [Leader]" -46:26.494 [INFO] TestAgent.server: cluster leadership acquired -46:26.494 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.510 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11282 -46:26.530 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.562 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.562 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.562 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.562 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.562 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.562 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.562 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.583 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.611 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.617 [INFO] TestAgent: Synced node info -46:27.654 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.152989ms client=127.0.0.1:49595 client_network=udp -46:27.655 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=999.16µs client=127.0.0.1:59749 client_network=udp -46:27.656 [DEBUG] TestAgent.dns: request served from client: name=ad83ea6b-c7f2-e085-c8bc-a3f4efb5826c.query.consul. type=AAAA class=IN latency=781.495µs client=127.0.0.1:59014 client_network=udp -46:27.656 [INFO] TestAgent: Requesting shutdown -46:27.656 [INFO] TestAgent.server: shutting down server -46:27.656 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.656 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.656 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.656 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.656 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.656 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.656 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.656 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.664 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.675Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.675Z [INFO] agent.router.manager: shutting down -46:27.675 [INFO] TestAgent: consul server down -46:27.675 [INFO] TestAgent: shutdown complete -46:27.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11501 network=tcp -46:27.675 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11501 network=udp -46:27.675 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11502 network=tcp -46:28.176 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.176 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:14.015Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.053Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.150 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4d736316-38e1-97c6-ae4f-891f660a617f Address:127.0.0.1:29853}]" +32:14.151 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.151 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.151Z [INFO] agent.router: Initializing LAN area manager +32:14.152 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29853 [Follower]" leader= +32:14.152 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.152 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29853) (DC: dc1)" +32:14.152 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=udp +32:14.152 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29771 network=tcp +32:14.153 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29772 network=tcp +32:14.153 [INFO] TestAgent: started state syncer +32:14.153 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29854 network=tcp +32:14.194 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.194 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29853 [Candidate]" term=2 +32:14.237 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.237 [DEBUG] TestAgent.server.raft: vote granted: from=4d736316-38e1-97c6-ae4f-891f660a617f term=2 tally=1 +32:14.237 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.237 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29853 [Leader]" +32:14.238 [INFO] TestAgent.server: cluster leadership acquired +32:14.238 [INFO] TestAgent.server: New leader elected: payload=test-node +32:14.255 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29853 +32:14.287 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:14.321 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:14.321 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:14.321 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:14.321 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:14.321 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:14.321 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:14.321 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:14.355 [INFO] TestAgent.server: federation state anti-entropy synced +32:14.423 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:14.432 [INFO] TestAgent: Synced node info +32:14.432 [DEBUG] TestAgent: Node info in sync +32:15.375 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:15.375 [DEBUG] TestAgent: Node info in sync +32:15.759 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.724372ms client=127.0.0.1:57722 client_network=udp +32:15.761 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.411364ms client=127.0.0.1:37651 client_network=udp +32:15.764 [INFO] TestAgent: Requesting shutdown +32:15.764 [INFO] TestAgent.server: shutting down server +32:15.764 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:15.764 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:15.764 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:15.764 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:15.764 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:15.764 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:15.764 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:15.764 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:15.764 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:15.764 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:15.768 [DEBUG] TestAgent.dns: request served from client: name=30719926-fd3f-1c03-2b0c-4c6730d98947.query.consul. type=ANY class=IN latency=6.138561ms client=127.0.0.1:49862 client_network=udp +32:15.776 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:15.792Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:15.792Z [INFO] agent.router.manager: shutting down +32:15.792 [INFO] TestAgent: consul server down +32:15.792 [INFO] TestAgent: shutdown complete +32:15.792 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=tcp +32:15.792 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29771 network=udp +32:15.792 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29772 network=tcp +32:16.292 [INFO] TestAgent: Waiting for endpoints to shut down +32:16.292 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{3_3_3_3_3_3_3_3_3_3_3} -2022-11-07T22:46:26.444Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.454Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.488 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e793bd4b-b55a-3352-9b32-1f0cb3732b35 Address:127.0.0.1:11394}]" -46:26.489 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.489 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11394 [Follower]" leader= -46:26.489 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.489Z [INFO] agent.router: Initializing LAN area manager -46:26.490 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11394) (DC: dc1)" -46:26.490 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.490 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11375 network=udp -46:26.490 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11375 network=tcp -46:26.490 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11376 network=tcp -46:26.490 [INFO] TestAgent: started state syncer -46:26.490 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11269 network=tcp -46:26.528 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.528 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11394 [Candidate]" term=2 -46:26.541 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.541 [DEBUG] TestAgent.server.raft: vote granted: from=e793bd4b-b55a-3352-9b32-1f0cb3732b35 term=2 tally=1 -46:26.541 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.541 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11394 [Leader]" -46:26.541 [INFO] TestAgent.server: cluster leadership acquired -46:26.541 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.557 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11394 -46:26.575 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.610 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.610 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.610 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.610 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.610 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.610 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.610 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.629 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.917 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.924 [INFO] TestAgent: Synced node info -46:26.924 [DEBUG] TestAgent: Node info in sync -46:27.697 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=715.929µs client=127.0.0.1:46693 client_network=udp -46:27.698 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=924.535µs client=127.0.0.1:47861 client_network=udp -46:27.699 [DEBUG] TestAgent.dns: request served from client: name=cea53f18-cca8-e23d-a007-b619149071ff.query.consul. type=A class=IN latency=985.53µs client=127.0.0.1:36616 client_network=udp -46:27.699 [INFO] TestAgent: Requesting shutdown -46:27.699 [INFO] TestAgent.server: shutting down server -46:27.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.699 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.699 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.699 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.699 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.699 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.699 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.707 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.720Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.720Z [INFO] agent.router.manager: shutting down -46:27.721 [INFO] TestAgent: consul server down -46:27.721 [INFO] TestAgent: shutdown complete -46:27.721 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11375 network=tcp -46:27.721 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11375 network=udp -46:27.721 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11376 network=tcp -46:28.221 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.221 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:14.542Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.561Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.607 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:87611214-fcf5-0929-67d3-32cc7d0e96f8 Address:127.0.0.1:29950}]" +32:14.608 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29950 [Follower]" leader= +32:14.608 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.608 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.608Z [INFO] agent.router: Initializing LAN area manager +32:14.609 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=udp +32:14.609 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29950) (DC: dc1)" +32:14.609 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.609 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29883 network=tcp +32:14.609 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29946 network=tcp +32:14.617 [INFO] TestAgent: started state syncer +32:14.631 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29951 network=tcp +32:14.656 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.656 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29950 [Candidate]" term=2 +32:14.686 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.686 [DEBUG] TestAgent.server.raft: vote granted: from=87611214-fcf5-0929-67d3-32cc7d0e96f8 term=2 tally=1 +32:14.686 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.686 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29950 [Leader]" +32:14.692 [INFO] TestAgent.server: cluster leadership acquired +32:14.692 [INFO] TestAgent.server: New leader elected: payload=test-node +32:14.708 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29950 +32:14.784 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:14.840 [INFO] TestAgent: Synced node info +32:14.888 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:14.888 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:14.888 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:14.888 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:14.888 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:14.888 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:14.888 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:14.903 [INFO] TestAgent.server: federation state anti-entropy synced +32:15.380 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:15.380 [DEBUG] TestAgent: Node info in sync +32:15.380 [DEBUG] TestAgent: Node info in sync +32:16.255 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.066848ms client=127.0.0.1:36720 client_network=udp +32:16.256 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=874.752µs client=127.0.0.1:54590 client_network=udp +32:16.265 [DEBUG] TestAgent.dns: request served from client: name=115efb03-010d-3d05-781f-48e53b1e4154.query.consul. type=AAAA class=IN latency=1.275155ms client=127.0.0.1:54996 client_network=udp +32:16.268 [INFO] TestAgent: Requesting shutdown +32:16.268 [INFO] TestAgent.server: shutting down server +32:16.268 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.268 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.268 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.268 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.268 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.268 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.268 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.268 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.268 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.268 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.280 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.296Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.296Z [INFO] agent.router.manager: shutting down +32:16.296 [INFO] TestAgent: consul server down +32:16.296 [INFO] TestAgent: shutdown complete +32:16.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=tcp +32:16.296 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29883 network=udp +32:16.296 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29946 network=tcp +32:16.798 [INFO] TestAgent: Waiting for endpoints to shut down +32:16.799 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{3_3_3_3_3_3_3_3_3_3_3} -2022-11-07T22:46:26.601Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.615Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.655 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:56ae8fb8-0978-1a20-d0af-5a6c69137bb8 Address:127.0.0.1:11526}]" -46:26.655 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11526 [Follower]" leader= -46:26.655 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.656 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.656Z [INFO] agent.router: Initializing LAN area manager -46:26.656 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11526) (DC: dc1)" -46:26.656 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.656 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11480 network=udp -46:26.656 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11480 network=tcp -46:26.656 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11481 network=tcp -46:26.656 [INFO] TestAgent: started state syncer -46:26.656 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11444 network=tcp -46:26.714 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.714 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11526 [Candidate]" term=2 -46:26.729 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.729 [DEBUG] TestAgent.server.raft: vote granted: from=56ae8fb8-0978-1a20-d0af-5a6c69137bb8 term=2 tally=1 -46:26.729 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.729 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11526 [Leader]" -46:26.729 [INFO] TestAgent.server: cluster leadership acquired -46:26.729 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.742 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11526 -46:26.765 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.813 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.813 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.813 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.813 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.813 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.813 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.813 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.839 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.846 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:26.862 [INFO] TestAgent: Synced node info -46:27.719 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=857.546µs client=127.0.0.1:43126 client_network=udp -46:27.719 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=540.868µs client=127.0.0.1:55032 client_network=udp -46:27.720 [DEBUG] TestAgent.dns: request served from client: name=254973ba-a936-74f0-c12c-7d85c654ff7b.query.consul. type=AAAA class=IN latency=576.739µs client=127.0.0.1:34223 client_network=udp -46:27.720 [INFO] TestAgent: Requesting shutdown -46:27.720 [INFO] TestAgent.server: shutting down server -46:27.720 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.720 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.720 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.720 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.720 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.720 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.720 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.720 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.720 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.720 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.732 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.741Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.741Z [INFO] agent.router.manager: shutting down -46:27.741 [INFO] TestAgent: consul server down -46:27.741 [INFO] TestAgent: shutdown complete -46:27.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11480 network=tcp -46:27.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11480 network=udp -46:27.741 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11481 network=tcp -46:28.242 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.242 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{3_3_3_3_3_3_3_3_3_3_3} -2022-11-07T22:46:26.677Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.689Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.733 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b72a2493-dcfd-f86d-1a46-6a2df23690f3 Address:127.0.0.1:11589}]" -46:26.733 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11589 [Follower]" leader= -46:26.733 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.734 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.734Z [INFO] agent.router: Initializing LAN area manager -46:26.734 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.734 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11589) (DC: dc1)" -46:26.734 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11528 network=tcp -46:26.734 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11528 network=udp -46:26.735 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11529 network=tcp -46:26.735 [INFO] TestAgent: started state syncer -46:26.735 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11562 network=tcp -46:26.768 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.768 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11589 [Candidate]" term=2 -46:26.780 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.780 [DEBUG] TestAgent.server.raft: vote granted: from=b72a2493-dcfd-f86d-1a46-6a2df23690f3 term=2 tally=1 -46:26.780 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.780 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11589 [Leader]" -46:26.780 [INFO] TestAgent.server: cluster leadership acquired -46:26.780 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.793 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11589 -46:26.820 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.862 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.862 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.862 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.862 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.862 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.862 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.862 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.877 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.985 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:27.020 [INFO] TestAgent: Synced node info -46:27.815 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=766.774µs client=127.0.0.1:58256 client_network=udp -46:27.816 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=757.835µs client=127.0.0.1:35586 client_network=udp -46:27.817 [DEBUG] TestAgent.dns: request served from client: name=152fd7bb-085c-6339-dee2-3badaaae63a6.query.consul. type=AAAA class=IN latency=631.32µs client=127.0.0.1:34499 client_network=udp -46:27.817 [INFO] TestAgent: Requesting shutdown -46:27.817 [INFO] TestAgent.server: shutting down server -46:27.817 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.817 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.817 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.817 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.817 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.817 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.817 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.817 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.817 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.817 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.823 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.832Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.832Z [INFO] agent.router.manager: shutting down -46:27.832 [INFO] TestAgent: consul server down -46:27.832 [INFO] TestAgent: shutdown complete -46:27.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11528 network=tcp -46:27.832 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11528 network=udp -46:27.832 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11529 network=tcp -46:28.332 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.332 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:14.711Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.768Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.836 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:276bb40c-4842-d9f4-b7db-a31ac0161835 Address:127.0.0.1:29999}]" +32:14.837 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.837 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.837Z [INFO] agent.router: Initializing LAN area manager +32:14.838 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29987 network=udp +32:14.838 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29999 [Follower]" leader= +32:14.838 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.838 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29999) (DC: dc1)" +32:14.838 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29987 network=tcp +32:14.839 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29974 network=tcp +32:14.839 [INFO] TestAgent: started state syncer +32:14.839 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30000 network=tcp +32:14.888 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.888 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29999 [Candidate]" term=2 +32:14.917 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.917 [DEBUG] TestAgent.server.raft: vote granted: from=276bb40c-4842-d9f4-b7db-a31ac0161835 term=2 tally=1 +32:14.917 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.917 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29999 [Leader]" +32:14.918 [INFO] TestAgent.server: cluster leadership acquired +32:14.918 [INFO] TestAgent.server: New leader elected: payload=test-node +32:14.927 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29999 +32:14.943 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:14.988 [INFO] TestAgent: Synced node info +32:14.988 [DEBUG] TestAgent: Node info in sync +32:15.007 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.007 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.007 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.007 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.007 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.007 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.007 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.031 [INFO] TestAgent.server: federation state anti-entropy synced +32:16.455 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.083823ms client=127.0.0.1:41193 client_network=udp +32:16.457 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.104075ms client=127.0.0.1:60812 client_network=udp +32:16.458 [DEBUG] TestAgent.dns: request served from client: name=2697a9d8-278e-9174-7fc5-c5535ea650a5.query.consul. type=AAAA class=IN latency=962.206µs client=127.0.0.1:35781 client_network=udp +32:16.458 [INFO] TestAgent: Requesting shutdown +32:16.459 [INFO] TestAgent.server: shutting down server +32:16.459 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.459 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.459 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.459 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.459 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.459 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.459 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.459 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.459 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.459 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.468 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.480Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.480Z [INFO] agent.router.manager: shutting down +32:16.480 [INFO] TestAgent: consul server down +32:16.480 [INFO] TestAgent: shutdown complete +32:16.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29987 network=tcp +32:16.480 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29987 network=udp +32:16.480 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29974 network=tcp +32:16.980 [INFO] TestAgent: Waiting for endpoints to shut down +32:16.980 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{1_1_1_1_1_1_1_1_1_1_1} +2021-10-05T16:32:14.797Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.827Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.912 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58cfc978-1b4f-8bcd-b175-a71b0f45517f Address:127.0.0.1:29867}]" +32:14.913 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.913 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.913Z [INFO] agent.router: Initializing LAN area manager +32:14.913 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29980 network=udp +32:14.913 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29867 [Follower]" leader= +32:14.914 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29867) (DC: dc1)" +32:14.914 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.914 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29980 network=tcp +32:14.914 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29870 network=tcp +32:14.914 [INFO] TestAgent: started state syncer +32:14.914 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29868 network=tcp +32:14.964 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.964 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29867 [Candidate]" term=2 +32:15.011 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:15.011 [DEBUG] TestAgent.server.raft: vote granted: from=58cfc978-1b4f-8bcd-b175-a71b0f45517f term=2 tally=1 +32:15.011 [INFO] TestAgent.server.raft: election won: tally=1 +32:15.011 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29867 [Leader]" +32:15.012 [INFO] TestAgent.server: cluster leadership acquired +32:15.012 [INFO] TestAgent.server: New leader elected: payload=test-node +32:15.029 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29867 +32:15.072 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.125 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.125 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.125 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.125 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.125 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.125 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.125 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.148 [INFO] TestAgent: Synced node info +32:15.148 [INFO] TestAgent.server: federation state anti-entropy synced +32:16.469 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.055423ms client=127.0.0.1:56284 client_network=udp +32:16.470 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=923.538µs client=127.0.0.1:34679 client_network=udp +32:16.471 [DEBUG] TestAgent.dns: request served from client: name=b6ac13bc-ae5e-a421-451a-29b84e555dde.query.consul. type=A class=IN latency=1.145076ms client=127.0.0.1:34717 client_network=udp +32:16.471 [INFO] TestAgent: Requesting shutdown +32:16.471 [INFO] TestAgent.server: shutting down server +32:16.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.471 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.471 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.471 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.472 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.472 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.472 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.472 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.472 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.499 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.512Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.512Z [INFO] agent.router.manager: shutting down +32:16.516 [INFO] TestAgent: consul server down +32:16.516 [INFO] TestAgent: shutdown complete +32:16.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29980 network=tcp +32:16.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29980 network=udp +32:16.516 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29870 network=tcp +32:17.016 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.016 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{2_2_2_2_2_2_2_2_2_2_2} -[WARN] freeport: 10 out of 59 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:26.616Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.625Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.665 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ddf9b9d-6ee7-0aab-a67c-26127fd75412 Address:127.0.0.1:11457}]" -46:26.665 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11457 [Follower]" leader= -46:26.666 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.667 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.667Z [INFO] agent.router: Initializing LAN area manager -46:26.667 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.667 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11457) (DC: dc1)" -46:26.667 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11445 network=udp -46:26.667 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11445 network=tcp -46:26.668 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11446 network=tcp -46:26.668 [INFO] TestAgent: started state syncer -46:26.668 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11416 network=tcp -46:26.709 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.709 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11457 [Candidate]" term=2 -46:26.731 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.731 [DEBUG] TestAgent.server.raft: vote granted: from=0ddf9b9d-6ee7-0aab-a67c-26127fd75412 term=2 tally=1 -46:26.731 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.731 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11457 [Leader]" -46:26.731 [INFO] TestAgent.server: cluster leadership acquired -46:26.731 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.743 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11457 -46:26.768 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.797 [INFO] TestAgent: Synced node info -46:26.797 [DEBUG] TestAgent: Node info in sync -46:26.820 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.820 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.820 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.820 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.820 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.820 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.820 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.839 [INFO] TestAgent.server: federation state anti-entropy synced -46:27.884 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=699.074µs client=127.0.0.1:60853 client_network=udp -46:27.884 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=631.467µs client=127.0.0.1:59540 client_network=udp -46:27.885 [DEBUG] TestAgent.dns: request served from client: name=31e74419-a35d-3155-90fc-6d2c2fd795d0.query.consul. type=A class=IN latency=636.097µs client=127.0.0.1:53347 client_network=udp -46:27.885 [INFO] TestAgent: Requesting shutdown -46:27.885 [INFO] TestAgent.server: shutting down server -46:27.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.885 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.885 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.885 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.885 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.885 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.885 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.896 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.908Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.908Z [INFO] agent.router.manager: shutting down -46:27.908 [INFO] TestAgent: consul server down -46:27.908 [INFO] TestAgent: shutdown complete -46:27.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11445 network=tcp -46:27.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11445 network=udp -46:27.908 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11446 network=tcp -46:28.408 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.408 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:14.809Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.829Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.893 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:118c1451-7e1a-5d03-e923-5817562f8476 Address:127.0.0.1:29936}]" +32:14.893 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29936 [Follower]" leader= +32:14.894 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.894 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.894Z [INFO] agent.router: Initializing LAN area manager +32:14.894 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29938 network=udp +32:14.894 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29936) (DC: dc1)" +32:14.895 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.895 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29938 network=tcp +32:14.895 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29911 network=tcp +32:14.895 [INFO] TestAgent: started state syncer +32:14.895 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29937 network=tcp +32:14.937 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.937 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29936 [Candidate]" term=2 +32:14.974 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.974 [DEBUG] TestAgent.server.raft: vote granted: from=118c1451-7e1a-5d03-e923-5817562f8476 term=2 tally=1 +32:14.974 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.974 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29936 [Leader]" +32:14.974 [INFO] TestAgent.server: cluster leadership acquired +32:14.974 [INFO] TestAgent.server: New leader elected: payload=test-node +32:15.000 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29936 +32:15.048 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.100 [INFO] TestAgent: Synced node info +32:15.130 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.130 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.130 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.130 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.130 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.130 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.130 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.169 [INFO] TestAgent.server: federation state anti-entropy synced +32:15.343 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:15.343 [DEBUG] TestAgent: Node info in sync +32:15.344 [DEBUG] TestAgent: Node info in sync +32:16.482 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.337747ms client=127.0.0.1:47543 client_network=udp +32:16.490 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=7.516085ms client=127.0.0.1:35653 client_network=udp +32:16.494 [DEBUG] TestAgent.dns: request served from client: name=4897ab98-4081-6c3a-ad8d-1e8cbaf238b1.query.consul. type=ANY class=IN latency=1.632019ms client=127.0.0.1:35542 client_network=udp +32:16.494 [INFO] TestAgent: Requesting shutdown +32:16.494 [INFO] TestAgent.server: shutting down server +32:16.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.494 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.494 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.494 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.494 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.494 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.494 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.520 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.533Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.533Z [INFO] agent.router.manager: shutting down +32:16.533 [INFO] TestAgent: consul server down +32:16.533 [INFO] TestAgent: shutdown complete +32:16.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29938 network=tcp +32:16.533 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29938 network=udp +32:16.533 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29911 network=tcp +32:17.036 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.036 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{2_2_2_2_2_2_2_2_2_2_2} -2022-11-07T22:46:26.574Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.584Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.628 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97ce66cc-d5e8-9ad7-10bd-06ea00e47e07 Address:127.0.0.1:11519}]" -46:26.628 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11519 [Follower]" leader= -46:26.629 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.629 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.629Z [INFO] agent.router: Initializing LAN area manager -46:26.629 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.629 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11519) (DC: dc1)" -46:26.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11270 network=udp -46:26.629 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11270 network=tcp -46:26.630 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11271 network=tcp -46:26.630 [INFO] TestAgent: started state syncer -46:26.630 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11479 network=tcp -46:26.686 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.686 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11519 [Candidate]" term=2 -46:26.705 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.705 [DEBUG] TestAgent.server.raft: vote granted: from=97ce66cc-d5e8-9ad7-10bd-06ea00e47e07 term=2 tally=1 -46:26.705 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.705 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11519 [Leader]" -46:26.706 [INFO] TestAgent.server: cluster leadership acquired -46:26.706 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.718 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11519 -46:26.744 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.791 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.791 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.791 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.791 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.791 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.791 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.791 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.822 [INFO] TestAgent.server: federation state anti-entropy synced -46:27.120 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:27.151 [INFO] TestAgent: Synced node info -46:27.152 [DEBUG] TestAgent: Node info in sync -46:27.895 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=804.717µs client=127.0.0.1:45351 client_network=udp -46:27.896 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=699.215µs client=127.0.0.1:43535 client_network=udp -46:27.897 [DEBUG] TestAgent.dns: request served from client: name=8b89a6c0-fd85-1df3-d166-7c39bee648eb.query.consul. type=ANY class=IN latency=1.118078ms client=127.0.0.1:55652 client_network=udp -46:27.897 [INFO] TestAgent: Requesting shutdown -46:27.898 [INFO] TestAgent.server: shutting down server -46:27.898 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.898 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.898 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.898 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.898 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.898 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.898 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.898 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.898 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.898 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.908 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.925Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.925Z [INFO] agent.router.manager: shutting down -46:27.926 [INFO] TestAgent: consul server down -46:27.926 [INFO] TestAgent: shutdown complete -46:27.926 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11270 network=tcp -46:27.926 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11270 network=udp -46:27.926 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11271 network=tcp -46:28.426 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.426 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{9_9_8_8_6_8_8_5_8_8_-5} -2022-11-07T22:46:26.641Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.651Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.695 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f184c4ea-74e2-ba98-3cfd-5181f8b68e10 Address:127.0.0.1:11533}]" -46:26.695 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11533 [Follower]" leader= -46:26.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.696 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.696Z [INFO] agent.router: Initializing LAN area manager -46:26.696 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11533) (DC: dc1)" -46:26.697 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11417 network=udp -46:26.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11417 network=tcp -46:26.697 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11418 network=tcp -46:26.697 [INFO] TestAgent: started state syncer -46:26.697 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11527 network=tcp -46:26.741 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.741 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11533 [Candidate]" term=2 -46:26.760 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.760 [DEBUG] TestAgent.server.raft: vote granted: from=f184c4ea-74e2-ba98-3cfd-5181f8b68e10 term=2 tally=1 -46:26.760 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.760 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11533 [Leader]" -46:26.760 [INFO] TestAgent.server: cluster leadership acquired -46:26.760 [INFO] TestAgent.server: New leader elected: payload=test-node -46:26.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11533 -46:26.801 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.832 [INFO] TestAgent: Synced node info -46:26.832 [DEBUG] TestAgent: Node info in sync -46:26.845 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.845 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.845 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.845 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.845 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.845 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:26.845 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:26.866 [INFO] TestAgent.server: federation state anti-entropy synced -46:27.929 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.311937ms client=127.0.0.1:43361 client_network=udp -46:27.930 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.152411ms client=127.0.0.1:50197 client_network=udp -46:27.931 [DEBUG] TestAgent.dns: request served from client: name=71fbffd7-3102-e05a-5cf1-cd00d130f550.query.consul. type=ANY class=IN latency=1.258576ms client=127.0.0.1:60655 client_network=udp -46:27.931 [INFO] TestAgent: Requesting shutdown -46:27.931 [INFO] TestAgent.server: shutting down server -46:27.931 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:27.931 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:27.931 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:27.931 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:27.931 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:27.932 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:27.932 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:27.942 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:27.953Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:27.953Z [INFO] agent.router.manager: shutting down -46:27.953 [INFO] TestAgent: consul server down -46:27.953 [INFO] TestAgent: shutdown complete -46:27.953 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11417 network=tcp -46:27.953 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11417 network=udp -46:27.953 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11418 network=tcp -46:28.453 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.453 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 25 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:14.737Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.801Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.888 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:33b2fc2e-f3c7-fca2-3e17-12d467f25935 Address:127.0.0.1:30020}]" +32:14.888 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30020 [Follower]" leader= +32:14.889 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.889 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.889Z [INFO] agent.router: Initializing LAN area manager +32:14.890 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29973 network=udp +32:14.890 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30020) (DC: dc1)" +32:14.890 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.890 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29973 network=tcp +32:14.891 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29995 network=tcp +32:14.891 [INFO] TestAgent: started state syncer +32:14.891 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30021 network=tcp +32:14.957 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.957 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30020 [Candidate]" term=2 +32:14.994 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.994 [DEBUG] TestAgent.server.raft: vote granted: from=33b2fc2e-f3c7-fca2-3e17-12d467f25935 term=2 tally=1 +32:14.994 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.994 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30020 [Leader]" +32:14.995 [INFO] TestAgent.server: cluster leadership acquired +32:14.995 [INFO] TestAgent.server: New leader elected: payload=test-node +32:15.016 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30020 +32:15.060 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.115 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.115 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.115 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.115 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.115 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.115 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.115 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.133 [INFO] TestAgent.server: federation state anti-entropy synced +32:15.235 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:15.267 [INFO] TestAgent: Synced node info +32:15.267 [DEBUG] TestAgent: Node info in sync +32:16.546 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=2.185742ms client=127.0.0.1:43828 client_network=udp +32:16.548 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.854749ms client=127.0.0.1:37316 client_network=udp +32:16.550 [DEBUG] TestAgent.dns: request served from client: name=3433b252-39d9-957d-6881-be39d0d8b3f0.query.consul. type=ANY class=IN latency=2.307615ms client=127.0.0.1:43135 client_network=udp +32:16.568 [INFO] TestAgent: Requesting shutdown +32:16.568 [INFO] TestAgent.server: shutting down server +32:16.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.568 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.568 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.568 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.568 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.568 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.568 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.581 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.589Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.589Z [INFO] agent.router.manager: shutting down +32:16.589 [INFO] TestAgent: consul server down +32:16.589 [INFO] TestAgent: shutdown complete +32:16.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29973 network=tcp +32:16.589 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29973 network=udp +32:16.589 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29995 network=tcp +32:17.092 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.092 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:14.687Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:14.737Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:14.820 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:410cd9ac-392f-c83d-9526-c0fe3ac02176 Address:127.0.0.1:29992}]" +32:14.821 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:14.821 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:14.821Z [INFO] agent.router: Initializing LAN area manager +32:14.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30001 network=udp +32:14.822 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29992 [Follower]" leader= +32:14.822 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29992) (DC: dc1)" +32:14.822 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:14.822 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30001 network=tcp +32:14.823 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29988 network=tcp +32:14.823 [INFO] TestAgent: started state syncer +32:14.823 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29993 network=tcp +32:14.868 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:14.868 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29992 [Candidate]" term=2 +32:14.916 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:14.916 [DEBUG] TestAgent.server.raft: vote granted: from=410cd9ac-392f-c83d-9526-c0fe3ac02176 term=2 tally=1 +32:14.916 [INFO] TestAgent.server.raft: election won: tally=1 +32:14.916 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29992 [Leader]" +32:14.916 [INFO] TestAgent.server: cluster leadership acquired +32:14.916 [INFO] TestAgent.server: New leader elected: payload=test-node +32:14.927 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29992 +32:14.968 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.017 [INFO] TestAgent: Synced node info +32:15.017 [DEBUG] TestAgent: Node info in sync +32:15.036 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.036 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.036 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.036 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.036 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.036 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.036 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.053 [INFO] TestAgent.server: federation state anti-entropy synced +32:16.550 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.716814ms client=127.0.0.1:44797 client_network=udp +32:16.570 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.706194ms client=127.0.0.1:55587 client_network=udp +32:16.574 [DEBUG] TestAgent.dns: request served from client: name=a0391276-e198-5a0b-c074-564559fedf7c.query.consul. type=A class=IN latency=1.715594ms client=127.0.0.1:44274 client_network=udp +32:16.574 [INFO] TestAgent: Requesting shutdown +32:16.574 [INFO] TestAgent.server: shutting down server +32:16.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.574 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.574 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.574 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.574 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.574 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.574 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.574 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.574 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.587 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.595Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.595Z [INFO] agent.router.manager: shutting down +32:16.595 [INFO] TestAgent: consul server down +32:16.595 [INFO] TestAgent: shutdown complete +32:16.595 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30001 network=tcp +32:16.595 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30001 network=udp +32:16.595 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29988 network=tcp +32:17.096 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.096 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{2_2_2_2_2_2_2_2_2_2_2} -2022-11-07T22:46:26.920Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:26.930Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.989 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef6f1aa1-7610-5995-4c45-234b0829ebaf Address:127.0.0.1:11429}]" -46:26.990 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11429 [Follower]" leader= -46:26.990 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:26.991 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:26.991Z [INFO] agent.router: Initializing LAN area manager -46:26.991 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11429) (DC: dc1)" -46:26.991 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:26.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11563 network=udp -46:26.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11563 network=tcp -46:26.991 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11564 network=tcp -46:26.991 [INFO] TestAgent: started state syncer -46:26.992 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11486 network=tcp -46:27.033 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.033 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11429 [Candidate]" term=2 -46:27.058 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.058 [DEBUG] TestAgent.server.raft: vote granted: from=ef6f1aa1-7610-5995-4c45-234b0829ebaf term=2 tally=1 -46:27.058 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.058 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11429 [Leader]" -46:27.058 [INFO] TestAgent.server: cluster leadership acquired -46:27.058 [INFO] TestAgent.server: New leader elected: payload=test-node -46:27.068 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11429 -46:27.091 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:27.121 [INFO] TestAgent: Synced node info -46:27.121 [DEBUG] TestAgent: Node info in sync -46:27.154 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:27.154 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:27.154 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:27.154 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:27.154 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:27.154 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:27.154 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:27.183 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.057 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=749.053µs client=127.0.0.1:54794 client_network=udp -46:28.058 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=622.794µs client=127.0.0.1:44793 client_network=udp -46:28.059 [DEBUG] TestAgent.dns: request served from client: name=f082a113-5d72-149f-1b14-1253af649601.query.consul. type=A class=IN latency=671.705µs client=127.0.0.1:33552 client_network=udp -46:28.059 [INFO] TestAgent: Requesting shutdown -46:28.059 [INFO] TestAgent.server: shutting down server -46:28.059 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.059 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.059 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.059 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.059 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.059 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.059 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.059 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.059 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.059 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.070 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.075Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.075Z [INFO] agent.router.manager: shutting down -46:28.075 [INFO] TestAgent: consul server down -46:28.075 [INFO] TestAgent: shutdown complete -46:28.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11563 network=tcp -46:28.075 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11563 network=udp -46:28.075 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11564 network=tcp -46:28.576 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.576 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{1_1_1_1_1_1_1_1_1_1_1} -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:27.067Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.076Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.143 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e93fe83-4368-c019-d249-568e708f38b1 Address:127.0.0.1:11540}]" -46:27.144 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11540 [Follower]" leader= -46:27.144 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:27.144 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:27.144Z [INFO] agent.router: Initializing LAN area manager -46:27.145 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:27.145 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11540) (DC: dc1)" -46:27.145 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11487 network=udp -46:27.145 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11487 network=tcp -46:27.145 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11488 network=tcp -46:27.145 [INFO] TestAgent: started state syncer -46:27.145 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11548 network=tcp -46:27.198 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.198 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11540 [Candidate]" term=2 -46:27.222 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.222 [DEBUG] TestAgent.server.raft: vote granted: from=2e93fe83-4368-c019-d249-568e708f38b1 term=2 tally=1 -46:27.222 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.222 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11540 [Leader]" -46:27.222 [INFO] TestAgent.server: cluster leadership acquired -46:27.222 [INFO] TestAgent.server: New leader elected: payload=test-node -46:27.238 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11540 -46:27.265 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:27.312 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:27.312 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:27.312 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:27.312 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:27.312 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:27.312 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:27.312 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:27.338 [INFO] TestAgent.server: federation state anti-entropy synced -46:27.530 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:27.543 [INFO] TestAgent: Synced node info -46:27.543 [DEBUG] TestAgent: Node info in sync -46:28.155 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.047237ms client=127.0.0.1:46819 client_network=udp -46:28.156 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=823.929µs client=127.0.0.1:48234 client_network=udp -46:28.157 [DEBUG] TestAgent.dns: request served from client: name=7295842d-498e-877e-1c8a-c674bf996d2c.query.consul. type=ANY class=IN latency=830.212µs client=127.0.0.1:42130 client_network=udp -46:28.157 [INFO] TestAgent: Requesting shutdown -46:28.157 [INFO] TestAgent.server: shutting down server -46:28.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.157 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.157 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.157 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.157 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.158 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.158 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.164 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.171Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.171Z [INFO] agent.router.manager: shutting down -46:28.171 [INFO] TestAgent: consul server down -46:28.171 [INFO] TestAgent: shutdown complete -46:28.171 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11487 network=tcp -46:28.171 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11487 network=udp -46:28.171 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11488 network=tcp -46:28.672 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.672 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{1_1_1_1_1_1_1_1_1_1_1} -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:27.657Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.667Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.731 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:333652b2-d079-8364-f115-5324d81b1682 Address:127.0.0.1:11561}]" -46:27.731 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11561 [Follower]" leader= -46:27.732 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:27.732 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:27.732Z [INFO] agent.router: Initializing LAN area manager -46:27.732 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11549 network=udp -46:27.733 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11561) (DC: dc1)" -46:27.733 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11549 network=tcp -46:27.733 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:27.733 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11550 network=tcp -46:27.733 [INFO] TestAgent: started state syncer -46:27.733 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11639 network=tcp -46:27.767 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.767 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11561 [Candidate]" term=2 -46:27.779 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.779 [DEBUG] TestAgent.server.raft: vote granted: from=333652b2-d079-8364-f115-5324d81b1682 term=2 tally=1 -46:27.779 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.779 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11561 [Leader]" -46:27.779 [INFO] TestAgent.server: cluster leadership acquired -46:27.779 [INFO] TestAgent.server: New leader elected: payload=test-node -46:27.792 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11561 -46:27.823 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:27.869 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:27.869 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:27.869 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:27.869 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:27.869 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:27.869 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:27.869 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:27.890 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.165 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.177 [INFO] TestAgent: Synced node info -46:28.177 [DEBUG] TestAgent: Node info in sync -46:28.679 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.679 [DEBUG] TestAgent: Node info in sync -46:28.800 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=692.082µs client=127.0.0.1:33004 client_network=udp -46:28.801 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=563.106µs client=127.0.0.1:36610 client_network=udp -46:28.802 [DEBUG] TestAgent.dns: request served from client: name=49eac9a4-8759-b910-e5be-971915f114b0.query.consul. type=AAAA class=IN latency=588.654µs client=127.0.0.1:51642 client_network=udp -46:28.802 [INFO] TestAgent: Requesting shutdown -46:28.802 [INFO] TestAgent.server: shutting down server -46:28.802 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.802 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.802 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.802 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.802 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.802 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.802 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.802 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.802 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.802 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.813 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.827Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.827Z [INFO] agent.router.manager: shutting down -46:28.827 [INFO] TestAgent: consul server down -46:28.827 [INFO] TestAgent: shutdown complete -46:28.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11549 network=tcp -46:28.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11549 network=udp -46:28.827 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11550 network=tcp -46:29.327 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.327 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{1_1_1_1_1_1_1_1_1_1_1} -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:27.877Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.886Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.957 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06ad8d1c-14b1-03ec-1fb6-b57023959344 Address:127.0.0.1:11443}]" -46:27.957 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11443 [Follower]" leader= -46:27.958 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:27.958 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:27.958Z [INFO] agent.router: Initializing LAN area manager -46:27.958 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11443) (DC: dc1)" -46:27.958 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:27.959 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11640 network=tcp -46:27.959 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11640 network=udp -46:27.959 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11641 network=tcp -46:27.959 [INFO] TestAgent: started state syncer -46:27.959 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11430 network=tcp -46:28.022 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.022 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11443 [Candidate]" term=2 -46:28.041 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.041 [DEBUG] TestAgent.server.raft: vote granted: from=06ad8d1c-14b1-03ec-1fb6-b57023959344 term=2 tally=1 -46:28.041 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.041 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11443 [Leader]" -46:28.042 [INFO] TestAgent.server: cluster leadership acquired -46:28.042 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.050 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11443 -46:28.073 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.103 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.103 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.103 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.103 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.103 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.103 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.103 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.119 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.233 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.242 [INFO] TestAgent: Synced node info -46:28.468 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.468 [DEBUG] TestAgent: Node info in sync -46:28.468 [DEBUG] TestAgent: Node info in sync -46:29.071 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.031807ms client=127.0.0.1:56997 client_network=udp -46:29.072 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.030557ms client=127.0.0.1:34670 client_network=udp -46:29.073 [DEBUG] TestAgent.dns: request served from client: name=4fa65be8-99b7-c576-a12c-4349dcb33e1b.query.consul. type=A class=IN latency=871.599µs client=127.0.0.1:53869 client_network=udp -46:29.073 [INFO] TestAgent: Requesting shutdown -46:29.073 [INFO] TestAgent.server: shutting down server -46:29.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.073 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.073 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.073 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.073 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.073 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.073 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.081 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.097Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.097Z [INFO] agent.router.manager: shutting down -46:29.097 [INFO] TestAgent: consul server down -46:29.097 [INFO] TestAgent: shutdown complete -46:29.097 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11640 network=tcp -46:29.097 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11640 network=udp -46:29.097 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11641 network=tcp -46:29.597 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.597 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:15.358Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:15.381Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:15.462 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:010a10cc-efa6-a3c0-e74c-40ad6bfa87bd Address:127.0.0.1:29881}]" +32:15.462 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29881 [Follower]" leader= +32:15.463 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:15.463 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:15.463Z [INFO] agent.router: Initializing LAN area manager +32:15.464 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29945 network=udp +32:15.464 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29881) (DC: dc1)" +32:15.464 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:15.464 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29945 network=tcp +32:15.464 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29925 network=tcp +32:15.464 [INFO] TestAgent: started state syncer +32:15.465 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29882 network=tcp +32:15.524 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:15.524 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29881 [Candidate]" term=2 +32:15.555 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:15.555 [DEBUG] TestAgent.server.raft: vote granted: from=010a10cc-efa6-a3c0-e74c-40ad6bfa87bd term=2 tally=1 +32:15.555 [INFO] TestAgent.server.raft: election won: tally=1 +32:15.555 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29881 [Leader]" +32:15.555 [INFO] TestAgent.server: cluster leadership acquired +32:15.556 [INFO] TestAgent.server: New leader elected: payload=test-node +32:15.572 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29881 +32:15.612 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.651 [INFO] TestAgent: Synced node info +32:15.690 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.690 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.690 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.690 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.690 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.691 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.691 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.738 [INFO] TestAgent.server: federation state anti-entropy synced +32:16.779 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.315861ms client=127.0.0.1:56141 client_network=udp +32:16.780 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.211664ms client=127.0.0.1:42143 client_network=udp +32:16.786 [DEBUG] TestAgent.dns: request served from client: name=47724686-9c5e-c11d-69bf-f959de081f14.query.consul. type=A class=IN latency=1.427524ms client=127.0.0.1:35158 client_network=udp +32:16.787 [INFO] TestAgent: Requesting shutdown +32:16.787 [INFO] TestAgent.server: shutting down server +32:16.787 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.787 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.787 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.787 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.787 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.787 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.787 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.787 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.787 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.787 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.791 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.794Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.794Z [INFO] agent.router.manager: shutting down +32:16.794 [INFO] TestAgent: consul server down +32:16.794 [INFO] TestAgent: shutdown complete +32:16.794 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29945 network=tcp +32:16.795 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29945 network=udp +32:16.795 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29925 network=tcp +32:17.296 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.296 [INFO] TestAgent: Endpoints down +=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{1_1_1_1_1_1_1_1_1_1_1} +[WARN] freeport: 4 out of 25 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:15.146Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:15.179Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:15.270 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bdc5c2bc-9d4d-d1ad-fc8c-db0bf1f170b0 Address:127.0.0.1:29943}]" +32:15.271 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:15.271 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:15.271Z [INFO] agent.router: Initializing LAN area manager +32:15.271 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29924 network=udp +32:15.271 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29943 [Follower]" leader= +32:15.272 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29943) (DC: dc1)" +32:15.272 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:15.272 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29924 network=tcp +32:15.273 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29939 network=tcp +32:15.273 [INFO] TestAgent: started state syncer +32:15.273 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29944 network=tcp +32:15.340 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:15.340 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29943 [Candidate]" term=2 +32:15.361 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:15.361 [DEBUG] TestAgent.server.raft: vote granted: from=bdc5c2bc-9d4d-d1ad-fc8c-db0bf1f170b0 term=2 tally=1 +32:15.361 [INFO] TestAgent.server.raft: election won: tally=1 +32:15.361 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29943 [Leader]" +32:15.361 [INFO] TestAgent.server: cluster leadership acquired +32:15.362 [INFO] TestAgent.server: New leader elected: payload=test-node +32:15.377 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29943 +32:15.406 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:15.472 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:15.472 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:15.472 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:15.472 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:15.472 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:15.472 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:15.472 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:15.511 [INFO] TestAgent: Synced node info +32:15.511 [DEBUG] TestAgent: Node info in sync +32:15.523 [INFO] TestAgent.server: federation state anti-entropy synced +32:16.921 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.32712ms client=127.0.0.1:57659 client_network=udp +32:16.929 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.027317ms client=127.0.0.1:59629 client_network=udp +32:16.930 [DEBUG] TestAgent.dns: request served from client: name=e35d44d1-66be-5e56-a084-c328dd6414ff.query.consul. type=AAAA class=IN latency=935.832µs client=127.0.0.1:49190 client_network=udp +32:16.930 [INFO] TestAgent: Requesting shutdown +32:16.930 [INFO] TestAgent.server: shutting down server +32:16.930 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:16.930 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:16.930 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:16.930 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:16.930 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:16.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:16.930 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:16.930 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:16.930 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:16.930 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:16.935 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:16.940Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:16.940Z [INFO] agent.router.manager: shutting down +32:16.940 [INFO] TestAgent: consul server down +32:16.940 [INFO] TestAgent: shutdown complete +32:16.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29924 network=tcp +32:16.940 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29924 network=udp +32:16.940 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29939 network=tcp +32:17.440 [INFO] TestAgent: Waiting for endpoints to shut down +32:17.440 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{0_0_0_0_0_0_0_0_0_0_0} -2022-11-07T22:46:27.956Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.971Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.023 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d62b79b9-c811-d438-905c-a007ace56a73 Address:127.0.0.1:11638}]" -46:28.023 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11638 [Follower]" leader= -46:28.024 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.024 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.024Z [INFO] agent.router: Initializing LAN area manager -46:28.026 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11638) (DC: dc1)" -46:28.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11431 network=udp -46:28.033 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11431 network=tcp -46:28.034 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11432 network=tcp -46:28.034 [INFO] TestAgent: started state syncer -46:28.036 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11555 network=tcp -46:28.085 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.085 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11638 [Candidate]" term=2 -46:28.103 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.103 [DEBUG] TestAgent.server.raft: vote granted: from=d62b79b9-c811-d438-905c-a007ace56a73 term=2 tally=1 -46:28.103 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.103 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11638 [Leader]" -46:28.103 [INFO] TestAgent.server: cluster leadership acquired -46:28.103 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.111 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11638 -46:28.130 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.160 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.160 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.160 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.160 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.160 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.160 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.160 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.166 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.312 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.333 [INFO] TestAgent: Synced node info -46:28.333 [DEBUG] TestAgent: Node info in sync -46:29.081 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=846.914µs client=127.0.0.1:58654 client_network=udp -46:29.082 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=804.798µs client=127.0.0.1:34964 client_network=udp -46:29.083 [DEBUG] TestAgent.dns: request served from client: name=1a1956ee-b07e-0340-8cac-8dbd515ff6be.query.consul. type=ANY class=IN latency=828.385µs client=127.0.0.1:45471 client_network=udp -46:29.083 [INFO] TestAgent: Requesting shutdown -46:29.083 [INFO] TestAgent.server: shutting down server -46:29.083 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.083 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.083 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.083 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.083 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.083 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.083 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.083 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.083 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.083 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.097 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.104Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.104Z [INFO] agent.router.manager: shutting down -46:29.104 [INFO] TestAgent: consul server down -46:29.104 [INFO] TestAgent: shutdown complete -46:29.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11431 network=tcp -46:29.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11431 network=udp -46:29.104 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11432 network=tcp -46:29.604 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.604 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:16.312Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:16.328Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:16.403 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f893a5a-4b01-d4b6-bc9e-16cd90086cca Address:127.0.0.1:29978}]" +32:16.403 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29978 [Follower]" leader= +32:16.404 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:16.404 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:16.404Z [INFO] agent.router: Initializing LAN area manager +32:16.405 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=udp +32:16.405 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29978) (DC: dc1)" +32:16.405 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:16.405 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29869 network=tcp +32:16.405 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30002 network=tcp +32:16.405 [INFO] TestAgent: started state syncer +32:16.405 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29979 network=tcp +32:16.461 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:16.461 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29978 [Candidate]" term=2 +32:16.482 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:16.482 [DEBUG] TestAgent.server.raft: vote granted: from=4f893a5a-4b01-d4b6-bc9e-16cd90086cca term=2 tally=1 +32:16.482 [INFO] TestAgent.server.raft: election won: tally=1 +32:16.482 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29978 [Leader]" +32:16.484 [INFO] TestAgent.server: cluster leadership acquired +32:16.484 [INFO] TestAgent.server: New leader elected: payload=test-node +32:16.501 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29978 +32:16.576 [INFO] TestAgent: Synced node info +32:16.576 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:16.652 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:16.652 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:16.652 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:16.652 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:16.652 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:16.652 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:16.652 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:16.692 [INFO] TestAgent.server: federation state anti-entropy synced +32:18.177 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.511462ms client=127.0.0.1:58664 client_network=udp +32:18.182 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.666034ms client=127.0.0.1:52285 client_network=udp +32:18.189 [DEBUG] TestAgent.dns: request served from client: name=c1e0c2eb-61df-c90b-7711-82b82069e7ff.query.consul. type=ANY class=IN latency=6.131846ms client=127.0.0.1:43768 client_network=udp +32:18.192 [INFO] TestAgent: Requesting shutdown +32:18.192 [INFO] TestAgent.server: shutting down server +32:18.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:18.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:18.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:18.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:18.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:18.192 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:18.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:18.192 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:18.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:18.192 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:18.204 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:18.208Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:18.208Z [INFO] agent.router.manager: shutting down +32:18.208 [INFO] TestAgent: consul server down +32:18.208 [INFO] TestAgent: shutdown complete +32:18.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=tcp +32:18.208 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29869 network=udp +32:18.208 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30002 network=tcp +32:18.712 [INFO] TestAgent: Waiting for endpoints to shut down +32:18.712 [INFO] TestAgent: Endpoints down === CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{0_0_0_0_0_0_0_0_0_0_0} -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:28.153Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.162Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.192 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:73559aa2-01ac-a481-9c7f-bb22e0e6a9ad Address:127.0.0.1:11568}]" -46:28.192 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11568 [Follower]" leader= -46:28.192 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.193 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.193Z [INFO] agent.router: Initializing LAN area manager -46:28.193 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11568) (DC: dc1)" -46:28.193 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11535 network=udp -46:28.193 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11535 network=tcp -46:28.193 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11536 network=tcp -46:28.193 [INFO] TestAgent: started state syncer -46:28.193 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11604 network=tcp -46:28.233 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.233 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11568 [Candidate]" term=2 -46:28.251 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.251 [DEBUG] TestAgent.server.raft: vote granted: from=73559aa2-01ac-a481-9c7f-bb22e0e6a9ad term=2 tally=1 -46:28.251 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.251 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11568 [Leader]" -46:28.251 [INFO] TestAgent.server: cluster leadership acquired -46:28.251 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.263 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11568 -46:28.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.325 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.325 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.325 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.325 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.325 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.325 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.325 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.342 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.549 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.565 [INFO] TestAgent: Synced node info -46:28.565 [DEBUG] TestAgent: Node info in sync -46:29.396 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=831.997µs client=127.0.0.1:46537 client_network=udp -46:29.397 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=728.252µs client=127.0.0.1:34085 client_network=udp -46:29.398 [DEBUG] TestAgent.dns: request served from client: name=ce384539-6e98-4547-ba3b-87b92bbc2e30.query.consul. type=A class=IN latency=827.769µs client=127.0.0.1:60633 client_network=udp -46:29.398 [INFO] TestAgent: Requesting shutdown -46:29.398 [INFO] TestAgent.server: shutting down server -46:29.398 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.398 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.398 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.398 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.398 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.398 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.398 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.398 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.398 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.398 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.409 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.419Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.419Z [INFO] agent.router.manager: shutting down -46:29.419 [INFO] TestAgent: consul server down -46:29.419 [INFO] TestAgent: shutdown complete -46:29.419 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11535 network=tcp -46:29.419 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11535 network=udp -46:29.419 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11536 network=tcp -46:29.920 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.920 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} -2022-11-07T22:46:28.185Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.194Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.247 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef4075b6-1f4d-2c0a-f067-e6c44e769c4f Address:127.0.0.1:11617}]" -46:28.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11617 [Follower]" leader= -46:28.248 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.249Z [INFO] agent.router: Initializing LAN area manager -46:28.249 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.249 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11617) (DC: dc1)" -46:28.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11605 network=tcp -46:28.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11605 network=udp -46:28.249 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11606 network=tcp -46:28.249 [INFO] TestAgent: started state syncer -46:28.249 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11597 network=tcp -46:28.286 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.286 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11617 [Candidate]" term=2 -46:28.304 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.304 [DEBUG] TestAgent.server.raft: vote granted: from=ef4075b6-1f4d-2c0a-f067-e6c44e769c4f term=2 tally=1 -46:28.304 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.304 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11617 [Leader]" -46:28.304 [INFO] TestAgent.server: cluster leadership acquired -46:28.304 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11617 -46:28.338 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.364 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.364 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.364 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.364 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.364 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.365 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.365 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.375 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.471 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.483 [INFO] TestAgent: Synced node info -46:29.410 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.292139ms client=127.0.0.1:56874 client_network=udp -46:29.411 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.212461ms client=127.0.0.1:37282 client_network=udp -46:29.412 [DEBUG] TestAgent.dns: request served from client: name=1074ddbe-cc82-83a0-af58-821e246de990.query.consul. type=ANY class=IN latency=815.785µs client=127.0.0.1:41614 client_network=udp -46:29.412 [INFO] TestAgent: Requesting shutdown -46:29.412 [INFO] TestAgent.server: shutting down server -46:29.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.412 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.412 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.412 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.412 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.412 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.412 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.412 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.412 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.419 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.431Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.431Z [INFO] agent.router.manager: shutting down -46:29.431 [INFO] TestAgent: consul server down -46:29.431 [INFO] TestAgent: shutdown complete -46:29.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11605 network=tcp -46:29.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11605 network=udp -46:29.431 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11606 network=tcp -46:29.931 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.931 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} -2022-11-07T22:46:28.252Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.262Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.311 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e76549ed-3833-0aff-18db-0ddf4ac90809 Address:127.0.0.1:11575}]" -46:28.311 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11575 [Follower]" leader= -46:28.311 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.312 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.312Z [INFO] agent.router: Initializing LAN area manager -46:28.312 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.312 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11575) (DC: dc1)" -46:28.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11619 network=udp -46:28.312 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11619 network=tcp -46:28.312 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11620 network=tcp -46:28.313 [INFO] TestAgent: started state syncer -46:28.313 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11611 network=tcp -46:28.353 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.353 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11575 [Candidate]" term=2 -46:28.368 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.368 [DEBUG] TestAgent.server.raft: vote granted: from=e76549ed-3833-0aff-18db-0ddf4ac90809 term=2 tally=1 -46:28.368 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.368 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11575 [Leader]" -46:28.368 [INFO] TestAgent.server: cluster leadership acquired -46:28.368 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.375 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11575 -46:28.394 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.422 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.422 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.422 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.422 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.422 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.422 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.422 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.439 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.662 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.684 [INFO] TestAgent: Synced node info -46:29.418 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=783.648µs client=127.0.0.1:48357 client_network=udp -46:29.419 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=939.674µs client=127.0.0.1:59379 client_network=udp -46:29.421 [DEBUG] TestAgent.dns: request served from client: name=affee6c3-bebb-7919-af5d-1e16d91f81a6.query.consul. type=A class=IN latency=1.098888ms client=127.0.0.1:47237 client_network=udp -46:29.421 [INFO] TestAgent: Requesting shutdown -46:29.421 [INFO] TestAgent.server: shutting down server -46:29.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.421 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.421 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.421 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.421 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.421 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.421 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.431 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.439Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.439Z [INFO] agent.router.manager: shutting down -46:29.439 [INFO] TestAgent: consul server down -46:29.439 [INFO] TestAgent: shutdown complete -46:29.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11619 network=tcp -46:29.439 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11619 network=udp -46:29.439 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11620 network=tcp -46:29.939 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.939 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} -2022-11-07T22:46:28.109Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.118Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.163 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3ccfbbd-0bb9-fe9a-e183-b9b07e3d9cf7 Address:127.0.0.1:11401}]" -46:28.163 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11401 [Follower]" leader= -46:28.163 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.164 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.164Z [INFO] agent.router: Initializing LAN area manager -46:28.164 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11401) (DC: dc1)" -46:28.164 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.164 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11556 network=udp -46:28.164 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11556 network=tcp -46:28.164 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11557 network=tcp -46:28.164 [INFO] TestAgent: started state syncer -46:28.164 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11534 network=tcp -46:28.215 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.215 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11401 [Candidate]" term=2 -46:28.229 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.229 [DEBUG] TestAgent.server.raft: vote granted: from=f3ccfbbd-0bb9-fe9a-e183-b9b07e3d9cf7 term=2 tally=1 -46:28.229 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.229 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11401 [Leader]" -46:28.229 [INFO] TestAgent.server: cluster leadership acquired -46:28.229 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.239 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11401 -46:28.264 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.304 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.304 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.304 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.304 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.304 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.304 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.304 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.313 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.593 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.629 [INFO] TestAgent: Synced node info -46:28.629 [DEBUG] TestAgent: Node info in sync -46:29.030 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:29.030 [DEBUG] TestAgent: Node info in sync -46:29.464 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.102625ms client=127.0.0.1:48533 client_network=udp -46:29.465 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=797.942µs client=127.0.0.1:52701 client_network=udp -46:29.466 [DEBUG] TestAgent.dns: request served from client: name=df4b0f06-8d83-c551-be91-bd6de9d32a7a.query.consul. type=AAAA class=IN latency=688.404µs client=127.0.0.1:58232 client_network=udp -46:29.466 [INFO] TestAgent: Requesting shutdown -46:29.466 [INFO] TestAgent.server: shutting down server -46:29.466 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.466 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.466 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.466 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.466 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.466 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.466 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.466 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.466 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.466 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.474 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.483Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.483Z [INFO] agent.router.manager: shutting down -46:29.484 [INFO] TestAgent: consul server down -46:29.484 [INFO] TestAgent: shutdown complete -46:29.484 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11556 network=tcp -46:29.484 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11556 network=udp -46:29.484 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11557 network=tcp -46:29.984 [INFO] TestAgent: Waiting for endpoints to shut down -46:29.984 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} -[WARN] freeport: 10 out of 38 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:28.418Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.429Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.464 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ccf268ec-f15f-23dc-ea62-6fdb3c340e78 Address:127.0.0.1:11673}]" -46:28.464 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11673 [Follower]" leader= -46:28.464 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.465 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.465Z [INFO] agent.router: Initializing LAN area manager -46:28.465 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11673) (DC: dc1)" -46:28.465 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11654 network=tcp -46:28.465 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11654 network=udp -46:28.466 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11655 network=tcp -46:28.466 [INFO] TestAgent: started state syncer -46:28.466 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11625 network=tcp -46:28.530 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.530 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11673 [Candidate]" term=2 -46:28.551 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.551 [DEBUG] TestAgent.server.raft: vote granted: from=ccf268ec-f15f-23dc-ea62-6fdb3c340e78 term=2 tally=1 -46:28.551 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.551 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11673 [Leader]" -46:28.551 [INFO] TestAgent.server: cluster leadership acquired -46:28.552 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.562 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11673 -46:28.589 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.639 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.639 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.639 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.639 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.639 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.639 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.639 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.656 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.686 [INFO] TestAgent: Synced node info -46:28.686 [DEBUG] TestAgent: Node info in sync -46:29.011 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:29.011 [DEBUG] TestAgent: Node info in sync -46:29.617 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=932.285µs client=127.0.0.1:45221 client_network=udp -46:29.618 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=875.78µs client=127.0.0.1:46211 client_network=udp -46:29.619 [DEBUG] TestAgent.dns: request served from client: name=48e4a143-6286-d37c-6d79-b71ea79cac37.query.consul. type=AAAA class=IN latency=695.698µs client=127.0.0.1:48836 client_network=udp -46:29.619 [INFO] TestAgent: Requesting shutdown -46:29.619 [INFO] TestAgent.server: shutting down server -46:29.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.619 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.619 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.619 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.619 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.619 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.619 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.629 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.635Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.635Z [INFO] agent.router.manager: shutting down -46:29.635 [INFO] TestAgent: consul server down -46:29.635 [INFO] TestAgent: shutdown complete -46:29.635 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11654 network=tcp -46:29.635 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11654 network=udp -46:29.635 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11655 network=tcp -46:30.135 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.135 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} -2022-11-07T22:46:28.437Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.447Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.494 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d34c59d4-edb3-8dfe-2171-3020f6dfbf62 Address:127.0.0.1:11645}]" -46:28.494 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11645 [Follower]" leader= -46:28.495 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.495 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.495Z [INFO] agent.router: Initializing LAN area manager -46:28.495 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11645) (DC: dc1)" -46:28.495 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.496 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11626 network=udp -46:28.496 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11626 network=tcp -46:28.496 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11627 network=tcp -46:28.496 [INFO] TestAgent: started state syncer -46:28.496 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11576 network=tcp -46:28.536 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.536 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11645 [Candidate]" term=2 -46:28.558 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.559 [DEBUG] TestAgent.server.raft: vote granted: from=d34c59d4-edb3-8dfe-2171-3020f6dfbf62 term=2 tally=1 -46:28.559 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.559 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11645 [Leader]" -46:28.559 [INFO] TestAgent.server: cluster leadership acquired -46:28.559 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.576 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11645 -46:28.608 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.655 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.655 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.655 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.655 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.655 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.655 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.655 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.680 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.746 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.773 [INFO] TestAgent: Synced node info -46:28.773 [DEBUG] TestAgent: Node info in sync -46:29.627 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.373729ms client=127.0.0.1:53016 client_network=udp -46:29.629 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.222532ms client=127.0.0.1:36233 client_network=udp -46:29.630 [DEBUG] TestAgent.dns: request served from client: name=5afff9c3-1946-d765-62e2-1e5588be5ac2.query.consul. type=ANY class=IN latency=942.336µs client=127.0.0.1:49080 client_network=udp -46:29.630 [INFO] TestAgent: Requesting shutdown -46:29.630 [INFO] TestAgent.server: shutting down server -46:29.630 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.630 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.630 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.630 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.630 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.630 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.630 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.630 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.630 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.630 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.635 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.644Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.644Z [INFO] agent.router.manager: shutting down -46:29.644 [INFO] TestAgent: consul server down -46:29.644 [INFO] TestAgent: shutdown complete -46:29.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11626 network=tcp -46:29.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11626 network=udp -46:29.644 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11627 network=tcp -46:30.144 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.144 [INFO] TestAgent: Endpoints down -=== CONT TestDNS_ServiceLookup_AnswerLimits/A_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} -2022-11-07T22:46:28.469Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.479Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.538 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:574514bc-eb7f-e943-d54a-22d96c9ed711 Address:127.0.0.1:11680}]" -46:28.538 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11680 [Follower]" leader= -46:28.538 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.539 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.539Z [INFO] agent.router: Initializing LAN area manager -46:28.539 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11680) (DC: dc1)" -46:28.539 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.539 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11577 network=tcp -46:28.539 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11577 network=udp -46:28.540 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11578 network=tcp -46:28.540 [INFO] TestAgent: started state syncer -46:28.540 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11674 network=tcp -46:28.594 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.594 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11680 [Candidate]" term=2 -46:28.616 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.616 [DEBUG] TestAgent.server.raft: vote granted: from=574514bc-eb7f-e943-d54a-22d96c9ed711 term=2 tally=1 -46:28.616 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.616 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11680 [Leader]" -46:28.616 [INFO] TestAgent.server: cluster leadership acquired -46:28.616 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.636 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11680 -46:28.670 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.720 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.720 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.720 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.720 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.720 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.720 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.720 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.756 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.869 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.887 [INFO] TestAgent: Synced node info -46:29.636 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=726.047µs client=127.0.0.1:36091 client_network=udp -46:29.637 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=593.297µs client=127.0.0.1:56361 client_network=udp -46:29.638 [DEBUG] TestAgent.dns: request served from client: name=1a2a1c0d-0ce7-7a68-7401-4d3fb09a4ea2.query.consul. type=A class=IN latency=686.986µs client=127.0.0.1:41576 client_network=udp -46:29.638 [INFO] TestAgent: Requesting shutdown -46:29.638 [INFO] TestAgent.server: shutting down server -46:29.638 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.638 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.638 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.638 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.638 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.638 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.638 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.638 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.638 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.638 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.644 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.659Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.659Z [INFO] agent.router.manager: shutting down -46:29.659 [INFO] TestAgent: consul server down -46:29.659 [INFO] TestAgent: shutdown complete -46:29.659 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11577 network=tcp -46:29.659 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11577 network=udp -46:29.659 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11578 network=tcp -46:30.159 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.159 [INFO] TestAgent: Endpoints down +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:16.850Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:16.877Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:16.930 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2effb4fb-fa6e-d23b-8a17-ece4b408f992 Address:127.0.0.1:29971}]" +32:16.931 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:16.931 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29971 [Follower]" leader= +32:16.932 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:16.932Z [INFO] agent.router: Initializing LAN area manager +32:16.932 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:16.932 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29971) (DC: dc1)" +32:16.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29959 network=udp +32:16.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29959 network=tcp +32:16.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29953 network=tcp +32:16.932 [INFO] TestAgent: started state syncer +32:16.933 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29972 network=tcp +32:17.001 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.001 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29971 [Candidate]" term=2 +32:17.016 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.016 [DEBUG] TestAgent.server.raft: vote granted: from=2effb4fb-fa6e-d23b-8a17-ece4b408f992 term=2 tally=1 +32:17.016 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.016 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29971 [Leader]" +32:17.016 [INFO] TestAgent.server: cluster leadership acquired +32:17.016 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.039 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29971 +32:17.055 [INFO] TestAgent: Synced node info +32:17.086 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.172 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.172 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.172 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.172 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.172 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.172 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.172 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.233 [INFO] TestAgent.server: federation state anti-entropy synced +32:18.953 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.282069ms client=127.0.0.1:34558 client_network=udp +32:18.954 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.144549ms client=127.0.0.1:41578 client_network=udp +32:18.957 [DEBUG] TestAgent.dns: request served from client: name=41fc2a52-3793-c6b9-1349-afa9a0eedf01.query.consul. type=AAAA class=IN latency=1.3141ms client=127.0.0.1:32801 client_network=udp +32:18.964 [INFO] TestAgent: Requesting shutdown +32:18.964 [INFO] TestAgent.server: shutting down server +32:18.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:18.964 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:18.964 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:18.964 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:18.964 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:18.964 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:18.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:18.964 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:18.964 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:18.964 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:18.993 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:18.998Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:18.998Z [INFO] agent.router.manager: shutting down +32:18.998 [INFO] TestAgent: consul server down +32:18.998 [INFO] TestAgent: shutdown complete +32:18.998 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29959 network=tcp +32:18.998 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29959 network=udp +32:18.998 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29953 network=tcp +32:19.499 [INFO] TestAgent: Waiting for endpoints to shut down +32:19.516 [INFO] TestAgent: Endpoints down === CONT TestUiServices/No_Filter -2022-11-07T22:46:05.971Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:05.981Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:06.030 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9b93921b-2eef-38c3-4d60-75fbdab957db Address:127.0.0.1:12040}]" -46:06.030 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12040 [Follower]" leader= -46:06.030 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9b93921b-2eef-38c3-4d60-75fbdab957db.dc1 127.0.0.1 -46:06.031 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9b93921b-2eef-38c3-4d60-75fbdab957db 127.0.0.1 -2022-11-07T22:46:06.031Z [INFO] agent.router: Initializing LAN area manager -46:06.031 [INFO] TestAgent.server: Adding LAN server: server="Node-9b93921b-2eef-38c3-4d60-75fbdab957db (Addr: tcp/127.0.0.1:12040) (DC: dc1)" -46:06.031 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9b93921b-2eef-38c3-4d60-75fbdab957db.dc1 area=wan -46:06.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12035 network=tcp -46:06.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12035 network=udp -46:06.032 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12036 network=tcp -46:06.032 [INFO] TestAgent: started state syncer -46:06.032 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12041 network=tcp -46:06.097 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:06.097 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12040 [Candidate]" term=2 -46:06.113 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:06.113 [DEBUG] TestAgent.server.raft: vote granted: from=9b93921b-2eef-38c3-4d60-75fbdab957db term=2 tally=1 -46:06.113 [INFO] TestAgent.server.raft: election won: tally=1 -46:06.113 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12040 [Leader]" -46:06.113 [INFO] TestAgent.server: cluster leadership acquired -46:06.113 [INFO] TestAgent.server: New leader elected: payload=Node-9b93921b-2eef-38c3-4d60-75fbdab957db -46:06.122 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12040 -46:06.137 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:06.171 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:06.171 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:06.171 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:06.171 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:06.171 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:06.171 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9b93921b-2eef-38c3-4d60-75fbdab957db -46:06.171 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9b93921b-2eef-38c3-4d60-75fbdab957db -46:06.182 [INFO] TestAgent.server: federation state anti-entropy synced -46:06.277 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:06.285 [INFO] TestAgent: Synced node info -46:06.291 [INFO] TestAgent: Requesting shutdown -46:06.291 [INFO] TestAgent.server: shutting down server -46:06.291 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.291 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.291 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.291 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.291 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.291 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.291 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.291 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.291 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.291 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.295 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.300Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.300Z [INFO] agent.router.manager: shutting down -46:06.301 [INFO] TestAgent: consul server down -46:06.301 [INFO] TestAgent: shutdown complete -46:06.301 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12035 network=tcp -46:06.301 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12035 network=udp -46:06.301 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12036 network=tcp -46:06.801 [INFO] TestAgent: Waiting for endpoints to shut down -46:06.801 [INFO] TestAgent: Endpoints down ---- PASS: TestUiServices (0.84s) +2021-10-05T16:31:31.281Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:31.336Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:31.400 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 Address:127.0.0.1:29734}]" +31:31.401 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6.dc1 127.0.0.1 +31:31.401 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 127.0.0.1 +2021-10-05T16:31:31.401Z [INFO] agent.router: Initializing LAN area manager +31:31.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29722 network=udp +31:31.402 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29734 [Follower]" leader= +31:31.402 [INFO] TestAgent.server: Adding LAN server: server="Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 (Addr: tcp/127.0.0.1:29734) (DC: dc1)" +31:31.402 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6.dc1 area=wan +31:31.402 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29722 network=tcp +31:31.403 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29723 network=tcp +31:31.403 [INFO] TestAgent: started state syncer +31:31.403 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29735 network=tcp +31:31.456 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:31.456 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29734 [Candidate]" term=2 +31:31.486 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:31.486 [DEBUG] TestAgent.server.raft: vote granted: from=df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 term=2 tally=1 +31:31.486 [INFO] TestAgent.server.raft: election won: tally=1 +31:31.486 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29734 [Leader]" +31:31.486 [INFO] TestAgent.server: cluster leadership acquired +31:31.486 [INFO] TestAgent.server: New leader elected: payload=Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 +31:31.495 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29734 +31:31.528 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:31.568 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:31.568 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:31.568 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:31.568 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:31.568 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:31.568 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 +31:31.568 [INFO] TestAgent.server: member joined, marking health alive: member=Node-df28d22f-2ab1-b0b3-c13d-4e34e45f0de6 +31:31.589 [INFO] TestAgent.server: federation state anti-entropy synced +31:31.677 [INFO] TestAgent: Requesting shutdown +31:31.677 [INFO] TestAgent.server: shutting down server +31:31.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:31.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:31.677 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:31.677 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:31.677 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:31.677 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:31.677 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:31.677 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:31.677 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:31.677 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:31.677 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:31.685 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:31.689Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:31.689Z [INFO] agent.router.manager: shutting down +31:31.689 [INFO] TestAgent: consul server down +31:31.689 [INFO] TestAgent: shutdown complete +31:31.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29722 network=tcp +31:31.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29722 network=udp +31:31.689 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29723 network=tcp +31:32.191 [INFO] TestAgent: Waiting for endpoints to shut down +31:32.191 [INFO] TestAgent: Endpoints down +--- PASS: TestUiServices (0.95s) --- PASS: TestUiServices/Filtered (0.00s) --- PASS: TestUiServices/No_Filter (0.00s) -2022-11-07T22:46:28.231Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.241Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.291 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a022541f-1bf1-11d9-ff98-26bbd5dc97e0 Address:127.0.0.1:11624}]" -46:28.291 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11624 [Follower]" leader= -46:28.291 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.292 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.292Z [INFO] agent.router: Initializing LAN area manager -46:28.292 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.292 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11598 network=udp -46:28.292 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11624) (DC: dc1)" -46:28.292 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11598 network=tcp -46:28.293 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11599 network=tcp -46:28.293 [INFO] TestAgent: started state syncer -46:28.293 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11618 network=tcp -46:28.333 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.333 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11624 [Candidate]" term=2 -46:28.348 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.348 [DEBUG] TestAgent.server.raft: vote granted: from=a022541f-1bf1-11d9-ff98-26bbd5dc97e0 term=2 tally=1 -46:28.348 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.348 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11624 [Leader]" -46:28.348 [INFO] TestAgent.server: cluster leadership acquired -46:28.348 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.360 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11624 -46:28.381 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.412 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.412 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.412 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.412 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.412 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.413 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.413 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.426 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.605 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.610 [INFO] TestAgent: Synced node info -46:28.610 [DEBUG] TestAgent: Node info in sync -46:29.649 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=727.916µs client=127.0.0.1:39331 client_network=udp -46:29.650 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=612.536µs client=127.0.0.1:44366 client_network=udp -46:29.650 [DEBUG] TestAgent.dns: request served from client: name=49105931-53a3-f7d7-9a07-f0d5525dc7ba.query.consul. type=AAAA class=IN latency=720.056µs client=127.0.0.1:37115 client_network=udp -46:29.650 [INFO] TestAgent: Requesting shutdown -46:29.651 [INFO] TestAgent.server: shutting down server -46:29.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.651 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.651 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.651 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.651 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.651 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.651 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.651 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.651 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.659 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.666Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.666Z [INFO] agent.router.manager: shutting down -46:29.667 [INFO] TestAgent: consul server down -46:29.667 [INFO] TestAgent: shutdown complete -46:29.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11598 network=tcp -46:29.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11598 network=udp -46:29.667 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11599 network=tcp -46:30.167 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.167 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:28.343Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.354Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.398 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d3ab474d-b2e7-2a1b-8142-7f7629d95a7f Address:127.0.0.1:11610}]" -46:28.398 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11610 [Follower]" leader= -46:28.398 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.399 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.399Z [INFO] agent.router: Initializing LAN area manager -46:28.399 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11610) (DC: dc1)" -46:28.399 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.399 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11612 network=udp -46:28.399 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11612 network=tcp -46:28.400 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11613 network=tcp -46:28.400 [INFO] TestAgent: started state syncer -46:28.400 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11653 network=tcp -46:28.464 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.464 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11610 [Candidate]" term=2 -46:28.483 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.483 [DEBUG] TestAgent.server.raft: vote granted: from=d3ab474d-b2e7-2a1b-8142-7f7629d95a7f term=2 tally=1 -46:28.483 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.483 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11610 [Leader]" -46:28.483 [INFO] TestAgent.server: cluster leadership acquired -46:28.483 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.496 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11610 -46:28.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.574 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.574 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.574 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.574 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.574 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.574 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.574 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.600 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.674 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.680 [INFO] TestAgent: Synced node info -46:29.690 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.32448ms client=127.0.0.1:37660 client_network=udp -46:29.691 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.125658ms client=127.0.0.1:35120 client_network=udp -46:29.692 [DEBUG] TestAgent.dns: request served from client: name=ab650392-0c37-6733-b9df-c6ad462a2d29.query.consul. type=ANY class=IN latency=1.199674ms client=127.0.0.1:36266 client_network=udp -46:29.692 [INFO] TestAgent: Requesting shutdown -46:29.692 [INFO] TestAgent.server: shutting down server -46:29.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.693 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.693 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.693 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.693 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.693 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.693 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.693 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.697 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.701Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.701Z [INFO] agent.router.manager: shutting down -46:29.701 [INFO] TestAgent: consul server down -46:29.701 [INFO] TestAgent: shutdown complete -46:29.701 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11612 network=tcp -46:29.701 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11612 network=udp -46:29.701 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11613 network=tcp -46:30.201 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.201 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:28.586Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.595Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.661 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:29056b6a-5b6b-9b8b-4df2-b2ab782f21b5 Address:127.0.0.1:11666}]" -46:28.662 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11666 [Follower]" leader= -46:28.662 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.663 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.663Z [INFO] agent.router: Initializing LAN area manager -46:28.663 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11675 network=udp -46:28.663 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11666) (DC: dc1)" -46:28.663 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.663 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11675 network=tcp -46:28.664 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11676 network=tcp -46:28.664 [INFO] TestAgent: started state syncer -46:28.664 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11667 network=tcp -46:28.731 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.731 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11666 [Candidate]" term=2 -46:28.751 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.751 [DEBUG] TestAgent.server.raft: vote granted: from=29056b6a-5b6b-9b8b-4df2-b2ab782f21b5 term=2 tally=1 -46:28.751 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.751 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11666 [Leader]" -46:28.751 [INFO] TestAgent.server: cluster leadership acquired -46:28.751 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.769 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11666 -46:28.805 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.857 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.857 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.857 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.857 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.857 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.857 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.857 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.870 [INFO] TestAgent.server: federation state anti-entropy synced -46:29.043 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:29.070 [INFO] TestAgent: Synced node info -46:29.070 [DEBUG] TestAgent: Node info in sync -46:29.723 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=983.594µs client=127.0.0.1:35437 client_network=udp -46:29.724 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=827.577µs client=127.0.0.1:33294 client_network=udp -46:29.725 [DEBUG] TestAgent.dns: request served from client: name=5d4dc875-a86d-9200-c69a-c673062d1bc2.query.consul. type=ANY class=IN latency=817.361µs client=127.0.0.1:35897 client_network=udp -46:29.725 [INFO] TestAgent: Requesting shutdown -46:29.725 [INFO] TestAgent.server: shutting down server -46:29.725 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.725 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.725 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.725 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.725 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.725 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.725 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.725 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.725 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.725 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.730 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.737Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.737Z [INFO] agent.router.manager: shutting down -46:29.737 [INFO] TestAgent: consul server down -46:29.737 [INFO] TestAgent: shutdown complete -46:29.737 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11675 network=tcp -46:29.737 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11675 network=udp -46:29.737 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11676 network=tcp -46:30.238 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.238 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:28.685Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:28.701Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:28.751 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8b185b1-cf6a-af30-efc2-057a733721d7 Address:127.0.0.1:11631}]" -46:28.751 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11631 [Follower]" leader= -46:28.752 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:28.752 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:28.752Z [INFO] agent.router: Initializing LAN area manager -46:28.753 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11631) (DC: dc1)" -46:28.753 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:28.753 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11668 network=tcp -46:28.753 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11668 network=udp -46:28.753 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11669 network=tcp -46:28.753 [INFO] TestAgent: started state syncer -46:28.753 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11569 network=tcp -46:28.801 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:28.801 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11631 [Candidate]" term=2 -46:28.819 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:28.819 [DEBUG] TestAgent.server.raft: vote granted: from=b8b185b1-cf6a-af30-efc2-057a733721d7 term=2 tally=1 -46:28.819 [INFO] TestAgent.server.raft: election won: tally=1 -46:28.819 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11631 [Leader]" -46:28.819 [INFO] TestAgent.server: cluster leadership acquired -46:28.820 [INFO] TestAgent.server: New leader elected: payload=test-node -46:28.837 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11631 -46:28.870 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.920 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.920 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.920 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.920 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.920 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.920 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:28.920 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:28.933 [INFO] TestAgent.server: federation state anti-entropy synced -46:29.161 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:29.175 [INFO] TestAgent: Synced node info -46:29.175 [DEBUG] TestAgent: Node info in sync -46:29.824 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=886.291µs client=127.0.0.1:37070 client_network=udp -46:29.825 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=768.493µs client=127.0.0.1:46418 client_network=udp -46:29.827 [DEBUG] TestAgent.dns: request served from client: name=855860bc-1524-544c-dd78-b7ab096792df.query.consul. type=AAAA class=IN latency=1.117995ms client=127.0.0.1:55405 client_network=udp -46:29.827 [INFO] TestAgent: Requesting shutdown -46:29.827 [INFO] TestAgent.server: shutting down server -46:29.827 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.827 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.827 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.827 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.827 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.827 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.827 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.827 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.827 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.827 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.832 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.838Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.838Z [INFO] agent.router.manager: shutting down -46:29.838 [INFO] TestAgent: consul server down -46:29.838 [INFO] TestAgent: shutdown complete -46:29.838 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11668 network=tcp -46:29.838 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11668 network=udp -46:29.838 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11669 network=tcp -46:30.338 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.338 [INFO] TestAgent: Endpoints down -[WARN] freeport: 10 out of 66 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:18.583Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:18.594Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:18.635 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:adf6485a-e093-356f-6072-b641a50b6b87 Address:127.0.0.1:11636}]" -46:18.635 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11636 [Follower]" leader= -46:18.635 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-adf6485a-e093-356f-6072-b641a50b6b87.dc1 127.0.0.1 -46:18.636 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-adf6485a-e093-356f-6072-b641a50b6b87 127.0.0.1 -2022-11-07T22:46:18.636Z [INFO] agent.router: Initializing LAN area manager -46:18.636 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-adf6485a-e093-356f-6072-b641a50b6b87.dc1 area=wan -46:18.636 [INFO] TestAgent.server: Adding LAN server: server="Node-adf6485a-e093-356f-6072-b641a50b6b87 (Addr: tcp/127.0.0.1:11636) (DC: dc1)" -46:18.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11603 network=udp -46:18.636 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11603 network=tcp -46:18.636 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11632 network=tcp -46:18.636 [INFO] TestAgent: started state syncer -46:18.636 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11637 network=tcp -46:18.683 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:18.683 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11636 [Candidate]" term=2 -46:18.698 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:18.698 [DEBUG] TestAgent.server.raft: vote granted: from=adf6485a-e093-356f-6072-b641a50b6b87 term=2 tally=1 -46:18.698 [INFO] TestAgent.server.raft: election won: tally=1 -46:18.698 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11636 [Leader]" -46:18.698 [INFO] TestAgent.server: cluster leadership acquired -46:18.698 [INFO] TestAgent.server: New leader elected: payload=Node-adf6485a-e093-356f-6072-b641a50b6b87 -46:18.709 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11636 -46:18.729 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:18.765 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:18.765 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:18.765 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:18.765 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:18.765 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:18.765 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-adf6485a-e093-356f-6072-b641a50b6b87 -46:18.765 [INFO] TestAgent.server: member joined, marking health alive: member=Node-adf6485a-e093-356f-6072-b641a50b6b87 -46:18.784 [INFO] TestAgent.server: federation state anti-entropy synced -46:18.962 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:18.972 [INFO] TestAgent: Synced node info -46:18.972 [DEBUG] TestAgent: Node info in sync -46:19.052 [INFO] TestAgent.server: server starting leave -46:19.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-adf6485a-e093-356f-6072-b641a50b6b87.dc1 127.0.0.1 -46:19.053 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-adf6485a-e093-356f-6072-b641a50b6b87.dc1 area=wan -2022-11-07T22:46:19.053Z [INFO] agent.router.manager: shutting down -46:20.255 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.255 [DEBUG] TestAgent: Node info in sync -46:22.053 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-adf6485a-e093-356f-6072-b641a50b6b87 127.0.0.1 -46:22.053 [INFO] TestAgent.server: Removing LAN server: server="Node-adf6485a-e093-356f-6072-b641a50b6b87 (Addr: tcp/127.0.0.1:11636) (DC: dc1)" -46:22.053 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-adf6485a-e093-356f-6072-b641a50b6b87 -46:22.716 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:24.715 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:25.053 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s -46:26.715 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:28.716 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:28.716 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" -46:30.054 [INFO] TestAgent: Requesting shutdown -46:30.054 [INFO] TestAgent.server: shutting down server -46:30.054 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:30.054 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:30.054 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:30.054 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:30.054 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:30.054 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:30.054 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:30.054 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:30.054 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:46:30.070Z [INFO] agent.router.manager: shutting down -46:30.070 [INFO] TestAgent: consul server down -46:30.070 [INFO] TestAgent: shutdown complete -46:30.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11603 network=tcp -46:30.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11603 network=udp -46:30.070 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11632 network=tcp -46:30.571 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.571 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_StartStop (12.00s) -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:29.345Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.361Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.418 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b396aac0-5e2c-7b80-382f-01953edac792 Address:127.0.0.1:11694}]" -46:29.418 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11694 [Follower]" leader= -46:29.418 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:29.419 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:29.419Z [INFO] agent.router: Initializing LAN area manager -46:29.419 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11694) (DC: dc1)" -46:29.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11570 network=udp -46:29.419 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:29.419 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11570 network=tcp -46:29.419 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11571 network=tcp -46:29.419 [INFO] TestAgent: started state syncer -46:29.419 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11695 network=tcp -46:29.476 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:29.476 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11694 [Candidate]" term=2 -46:29.491 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:29.491 [DEBUG] TestAgent.server.raft: vote granted: from=b396aac0-5e2c-7b80-382f-01953edac792 term=2 tally=1 -46:29.491 [INFO] TestAgent.server.raft: election won: tally=1 -46:29.491 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11694 [Leader]" -46:29.491 [INFO] TestAgent.server: cluster leadership acquired -46:29.491 [INFO] TestAgent.server: New leader elected: payload=test-node -46:29.505 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11694 -46:29.528 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:29.569 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:29.569 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:29.569 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:29.569 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:29.569 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:29.569 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:29.569 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:29.580 [INFO] TestAgent.server: federation state anti-entropy synced -46:29.760 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:29.766 [INFO] TestAgent: Synced node info -46:29.766 [DEBUG] TestAgent: Node info in sync -46:30.326 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=813.325µs client=127.0.0.1:36641 client_network=udp -46:30.327 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=726.838µs client=127.0.0.1:34544 client_network=udp -46:30.328 [DEBUG] TestAgent.dns: request served from client: name=836482be-b8c3-2d0f-a2cc-9e6cb4bca445.query.consul. type=A class=IN latency=939.677µs client=127.0.0.1:37325 client_network=udp -46:30.328 [INFO] TestAgent: Requesting shutdown -46:30.328 [INFO] TestAgent.server: shutting down server -46:30.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:30.328 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:30.328 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:30.328 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:30.328 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:30.328 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:30.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:30.328 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:30.328 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:30.328 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:30.337 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:30.347Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:30.347Z [INFO] agent.router.manager: shutting down -46:30.347 [INFO] TestAgent: consul server down -46:30.347 [INFO] TestAgent: shutdown complete -46:30.347 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11570 network=tcp -46:30.347 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11570 network=udp -46:30.347 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11571 network=tcp -46:30.848 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.848 [INFO] TestAgent: Endpoints down -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:29.620Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.635Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.683 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0a86f2a-0e89-93f9-473d-d3e5dd9f5450 Address:127.0.0.1:11659}]" -46:29.683 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11659 [Follower]" leader= -46:29.683 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:29.684 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:29.684Z [INFO] agent.router: Initializing LAN area manager -46:29.684 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:29.684 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11659) (DC: dc1)" -46:29.684 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11689 network=tcp -46:29.684 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11689 network=udp -46:29.684 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11690 network=tcp -46:29.684 [INFO] TestAgent: started state syncer -46:29.685 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11681 network=tcp -46:29.733 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:29.733 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11659 [Candidate]" term=2 -46:29.746 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:29.746 [DEBUG] TestAgent.server.raft: vote granted: from=b0a86f2a-0e89-93f9-473d-d3e5dd9f5450 term=2 tally=1 -46:29.746 [INFO] TestAgent.server.raft: election won: tally=1 -46:29.746 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11659 [Leader]" -46:29.746 [INFO] TestAgent.server: cluster leadership acquired -46:29.746 [INFO] TestAgent.server: New leader elected: payload=test-node -46:29.755 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11659 -46:29.773 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:29.808 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:29.808 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:29.808 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:29.808 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:29.808 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:29.808 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:29.808 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:29.825 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.043 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.048 [INFO] TestAgent: Synced node info -46:30.110 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.110 [DEBUG] TestAgent: Node info in sync -46:30.110 [DEBUG] TestAgent: Node info in sync -46:30.881 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=854.253µs client=127.0.0.1:51601 client_network=udp -46:30.882 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=687.189µs client=127.0.0.1:33206 client_network=udp -46:30.883 [DEBUG] TestAgent.dns: request served from client: name=288df7e5-4176-215b-3c3d-0c9442bbdd95.query.consul. type=AAAA class=IN latency=718.268µs client=127.0.0.1:53887 client_network=udp -46:30.883 [INFO] TestAgent: Requesting shutdown -46:30.883 [INFO] TestAgent.server: shutting down server -46:30.883 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:30.883 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:30.883 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:30.883 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:30.883 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:30.883 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:30.883 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:30.883 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:30.883 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:30.883 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:30.895 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:30.905Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:30.905Z [INFO] agent.router.manager: shutting down -46:30.905 [INFO] TestAgent: consul server down -46:30.905 [INFO] TestAgent: shutdown complete -46:30.905 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11689 network=tcp -46:30.905 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11689 network=udp -46:30.905 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11690 network=tcp -46:31.406 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.406 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:29.608Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.617Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.664 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e7b721ff-0f74-7490-e846-022e4870ea3d Address:127.0.0.1:11715}]" -46:29.664 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11715 [Follower]" leader= -46:29.665 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:29.665 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:29.665Z [INFO] agent.router: Initializing LAN area manager -46:29.665 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11715) (DC: dc1)" -46:29.665 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:29.665 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11696 network=tcp -46:29.665 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11696 network=udp -46:29.666 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11697 network=tcp -46:29.666 [INFO] TestAgent: started state syncer -46:29.666 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11688 network=tcp -46:29.701 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:29.701 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11715 [Candidate]" term=2 -46:29.711 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:29.711 [DEBUG] TestAgent.server.raft: vote granted: from=e7b721ff-0f74-7490-e846-022e4870ea3d term=2 tally=1 -46:29.711 [INFO] TestAgent.server.raft: election won: tally=1 -46:29.711 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11715 [Leader]" -46:29.711 [INFO] TestAgent.server: cluster leadership acquired -46:29.712 [INFO] TestAgent.server: New leader elected: payload=test-node -46:29.723 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11715 -46:29.740 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:29.774 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:29.774 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:29.774 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:29.774 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:29.774 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:29.774 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:29.774 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:29.800 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.068 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.078 [INFO] TestAgent: Synced node info -46:30.888 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=877.575µs client=127.0.0.1:51172 client_network=udp -46:30.890 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.320986ms client=127.0.0.1:42737 client_network=udp -46:30.891 [DEBUG] TestAgent.dns: request served from client: name=7ae77f37-21b8-ff17-85cb-9a8d25cc406e.query.consul. type=ANY class=IN latency=1.294989ms client=127.0.0.1:45072 client_network=udp -46:30.891 [INFO] TestAgent: Requesting shutdown -46:30.891 [INFO] TestAgent.server: shutting down server -46:30.891 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:30.891 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:30.891 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:30.891 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:30.891 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:30.892 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:30.892 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:30.892 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:30.892 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:30.892 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:30.902 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:30.908Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:30.908Z [INFO] agent.router.manager: shutting down -46:30.908 [INFO] TestAgent: consul server down -46:30.908 [INFO] TestAgent: shutdown complete -46:30.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11696 network=tcp -46:30.908 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11696 network=udp -46:30.908 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11697 network=tcp -46:31.409 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.409 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:29.954Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.964Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:30.001 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:09e38328-3ced-d4ad-f841-ec3103eca453 Address:127.0.0.1:11750}]" -46:30.001 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11750 [Follower]" leader= -46:30.002 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:30.002 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:30.002Z [INFO] agent.router: Initializing LAN area manager -46:30.002 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11750) (DC: dc1)" -46:30.002 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:30.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11717 network=udp -46:30.002 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11717 network=tcp -46:30.003 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11718 network=tcp -46:30.003 [INFO] TestAgent: started state syncer -46:30.003 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11793 network=tcp -46:30.037 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.037 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11750 [Candidate]" term=2 -46:30.047 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.047 [DEBUG] TestAgent.server.raft: vote granted: from=09e38328-3ced-d4ad-f841-ec3103eca453 term=2 tally=1 -46:30.047 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.047 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11750 [Leader]" -46:30.048 [INFO] TestAgent.server: cluster leadership acquired -46:30.048 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.056 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11750 -46:30.084 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.118 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.118 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.118 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.118 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.118 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.118 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.118 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.133 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.435 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.447 [INFO] TestAgent: Synced node info -46:30.447 [DEBUG] TestAgent: Node info in sync -46:30.980 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=927.699µs client=127.0.0.1:47155 client_network=udp -46:30.980 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=663.919µs client=127.0.0.1:38700 client_network=udp -46:30.981 [DEBUG] TestAgent.dns: request served from client: name=e747b20b-fd07-a497-6709-4f8ac5ea3f0c.query.consul. type=A class=IN latency=643.622µs client=127.0.0.1:52771 client_network=udp -46:30.981 [INFO] TestAgent: Requesting shutdown -46:30.981 [INFO] TestAgent.server: shutting down server -46:30.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:30.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:30.981 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:30.981 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:30.981 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:30.981 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:30.981 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:30.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:30.981 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:30.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:30.986 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:30.993Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:30.993Z [INFO] agent.router.manager: shutting down -46:30.993 [INFO] TestAgent: consul server down -46:30.993 [INFO] TestAgent: shutdown complete -46:30.993 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11717 network=tcp -46:30.993 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11717 network=udp -46:30.993 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11718 network=tcp -46:31.494 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.494 [INFO] TestAgent: Endpoints down -[WARN] freeport: 10 out of 31 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:29.943Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.952Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.985 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1185acbe-6e6d-27f1-965c-a1fe22b973e1 Address:127.0.0.1:11701}]" -46:29.985 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11701 [Follower]" leader= -46:29.985 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:29.986 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:29.986Z [INFO] agent.router: Initializing LAN area manager -46:29.986 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11701) (DC: dc1)" -46:29.986 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:29.986 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11661 network=udp -46:29.986 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11661 network=tcp -46:29.986 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11662 network=tcp -46:29.986 [INFO] TestAgent: started state syncer -46:29.986 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11716 network=tcp -46:30.053 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.053 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11701 [Candidate]" term=2 -46:30.075 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.075 [DEBUG] TestAgent.server.raft: vote granted: from=1185acbe-6e6d-27f1-965c-a1fe22b973e1 term=2 tally=1 -46:30.075 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.075 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11701 [Leader]" -46:30.075 [INFO] TestAgent.server: cluster leadership acquired -46:30.075 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.086 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11701 -46:30.112 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.152 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.152 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.152 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.152 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.152 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.152 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.152 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.164 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.251 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.257 [INFO] TestAgent: Synced node info -46:31.103 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=893.02µs client=127.0.0.1:49135 client_network=udp -46:31.104 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=792.905µs client=127.0.0.1:58894 client_network=udp -46:31.105 [DEBUG] TestAgent.dns: request served from client: name=4141c0f1-ae78-cdbb-4b4c-c40ea9bff1d6.query.consul. type=ANY class=IN latency=768.42µs client=127.0.0.1:34262 client_network=udp -46:31.105 [INFO] TestAgent: Requesting shutdown -46:31.105 [INFO] TestAgent.server: shutting down server -46:31.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.105 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.105 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.105 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.105 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.105 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.105 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.105 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.105 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.112 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.128Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.128Z [INFO] agent.router.manager: shutting down -46:31.129 [INFO] TestAgent: consul server down -46:31.129 [INFO] TestAgent: shutdown complete -46:31.129 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11661 network=tcp -46:31.129 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11661 network=udp -46:31.129 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11662 network=tcp -46:31.629 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.629 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:29.931Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.940Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.967 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c5c96809-b217-dc81-ac8f-d322576ecb74 Address:127.0.0.1:11652}]" -46:29.967 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11652 [Follower]" leader= -46:29.967 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:29.968 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:29.968Z [INFO] agent.router: Initializing LAN area manager -46:29.968 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11652) (DC: dc1)" -46:29.968 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:29.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11682 network=tcp -46:29.968 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11682 network=udp -46:29.968 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11683 network=tcp -46:29.968 [INFO] TestAgent: started state syncer -46:29.969 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11660 network=tcp -46:30.027 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.027 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11652 [Candidate]" term=2 -46:30.040 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.040 [DEBUG] TestAgent.server.raft: vote granted: from=c5c96809-b217-dc81-ac8f-d322576ecb74 term=2 tally=1 -46:30.040 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.040 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11652 [Leader]" -46:30.040 [INFO] TestAgent.server: cluster leadership acquired -46:30.041 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.050 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11652 -46:30.070 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.107 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.107 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.107 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.107 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.107 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.107 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.107 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.138 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.313 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.323 [INFO] TestAgent: Synced node info -46:31.155 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=773.921µs client=127.0.0.1:56790 client_network=udp -46:31.156 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=617.59µs client=127.0.0.1:47675 client_network=udp -46:31.156 [DEBUG] TestAgent.dns: request served from client: name=44c337c9-67b3-1f72-4217-ebc174c0f3ae.query.consul. type=AAAA class=IN latency=610.37µs client=127.0.0.1:36084 client_network=udp -46:31.156 [INFO] TestAgent: Requesting shutdown -46:31.157 [INFO] TestAgent.server: shutting down server -46:31.157 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.157 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.157 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.157 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.157 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.157 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.157 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.157 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.160 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.166Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.166Z [INFO] agent.router.manager: shutting down -46:31.166 [INFO] TestAgent: consul server down -46:31.166 [INFO] TestAgent: shutdown complete -46:31.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11682 network=tcp -46:31.166 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11682 network=udp -46:31.166 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11683 network=tcp -46:31.667 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.667 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:29.995Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:30.005Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:30.034 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7c9d1140-5b1c-dacf-4ac2-5615935199f5 Address:127.0.0.1:11582}]" -46:30.035 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:30.035 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11582 [Follower]" leader= -46:30.035 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:30.035Z [INFO] agent.router: Initializing LAN area manager -46:30.036 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11794 network=udp -46:30.036 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11582) (DC: dc1)" -46:30.036 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:30.036 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11794 network=tcp -46:30.036 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11795 network=tcp -46:30.036 [INFO] TestAgent: started state syncer -46:30.036 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11583 network=tcp -46:30.099 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.099 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11582 [Candidate]" term=2 -46:30.115 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.115 [DEBUG] TestAgent.server.raft: vote granted: from=7c9d1140-5b1c-dacf-4ac2-5615935199f5 term=2 tally=1 -46:30.115 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.115 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11582 [Leader]" -46:30.115 [INFO] TestAgent.server: cluster leadership acquired -46:30.115 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.128 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11582 -46:30.147 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.182 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.182 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.182 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.182 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.182 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.182 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.182 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.193 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.240 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.246 [INFO] TestAgent: Synced node info -46:30.246 [DEBUG] TestAgent: Node info in sync -46:31.160 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=858.232µs client=127.0.0.1:55413 client_network=udp -46:31.161 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=687.48µs client=127.0.0.1:39806 client_network=udp -46:31.162 [DEBUG] TestAgent.dns: request served from client: name=59e2801b-f9f7-420e-2277-bbc547074b8e.query.consul. type=ANY class=IN latency=773.52µs client=127.0.0.1:59415 client_network=udp -46:31.162 [INFO] TestAgent: Requesting shutdown -46:31.162 [INFO] TestAgent.server: shutting down server -46:31.162 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.162 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.162 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.162 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.162 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.162 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.162 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.162 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.162 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.162 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.171 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.179Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.179Z [INFO] agent.router.manager: shutting down -46:31.179 [INFO] TestAgent: consul server down -46:31.179 [INFO] TestAgent: shutdown complete -46:31.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11794 network=tcp -46:31.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11794 network=udp -46:31.179 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11795 network=tcp -46:31.680 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.680 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:30.146Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:30.156Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:30.193 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fad3f53-d6a8-c3c9-565b-b0d59995e453 Address:127.0.0.1:11687}]" -46:30.193 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11687 [Follower]" leader= -46:30.194 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:30.194 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:30.194Z [INFO] agent.router: Initializing LAN area manager -46:30.195 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:30.195 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11687) (DC: dc1)" -46:30.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11584 network=udp -46:30.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11584 network=tcp -46:30.195 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11585 network=tcp -46:30.195 [INFO] TestAgent: started state syncer -46:30.195 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11723 network=tcp -46:30.240 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.240 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11687 [Candidate]" term=2 -46:30.255 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.255 [DEBUG] TestAgent.server.raft: vote granted: from=2fad3f53-d6a8-c3c9-565b-b0d59995e453 term=2 tally=1 -46:30.255 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.255 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11687 [Leader]" -46:30.255 [INFO] TestAgent.server: cluster leadership acquired -46:30.256 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.266 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11687 -46:30.286 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.331 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.331 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.331 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.331 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.331 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.331 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.332 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.364 [INFO] TestAgent.server: federation state anti-entropy synced -46:30.471 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.478 [INFO] TestAgent: Synced node info -46:30.596 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:30.596 [DEBUG] TestAgent: Node info in sync -46:30.596 [DEBUG] TestAgent: Node info in sync -46:31.225 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.131276ms client=127.0.0.1:56582 client_network=udp -46:31.226 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=993.719µs client=127.0.0.1:46861 client_network=udp -46:31.227 [DEBUG] TestAgent.dns: request served from client: name=450a7cf8-c712-db64-f669-f7faa718e265.query.consul. type=AAAA class=IN latency=978.564µs client=127.0.0.1:51663 client_network=udp -46:31.228 [INFO] TestAgent: Requesting shutdown -46:31.228 [INFO] TestAgent.server: shutting down server -46:31.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.228 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.228 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.228 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.228 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.228 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.228 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.228 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.228 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.243 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.254Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.254Z [INFO] agent.router.manager: shutting down -46:31.254 [INFO] TestAgent: consul server down -46:31.254 [INFO] TestAgent: shutdown complete -46:31.254 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11584 network=tcp -46:31.254 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11584 network=udp -46:31.254 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11585 network=tcp -46:31.754 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.754 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:30.156Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:30.164Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:30.201 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:acecc2dc-4f53-c4fa-0ed9-9173515cee41 Address:127.0.0.1:11708}]" -46:30.201 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11708 [Follower]" leader= -46:30.202 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -46:30.203 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:46:30.203Z [INFO] agent.router: Initializing LAN area manager -46:30.203 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:11708) (DC: dc1)" -46:30.203 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -46:30.203 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11724 network=udp -46:30.203 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11724 network=tcp -46:30.203 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11725 network=tcp -46:30.203 [INFO] TestAgent: started state syncer -46:30.203 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11730 network=tcp -46:30.267 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:30.267 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11708 [Candidate]" term=2 -46:30.286 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:30.286 [DEBUG] TestAgent.server.raft: vote granted: from=acecc2dc-4f53-c4fa-0ed9-9173515cee41 term=2 tally=1 -46:30.286 [INFO] TestAgent.server.raft: election won: tally=1 -46:30.286 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11708 [Leader]" -46:30.286 [INFO] TestAgent.server: cluster leadership acquired -46:30.286 [INFO] TestAgent.server: New leader elected: payload=test-node -46:30.298 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11708 -46:30.320 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:30.344 [INFO] TestAgent: Synced node info -46:30.344 [DEBUG] TestAgent: Node info in sync -46:30.372 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:30.372 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:30.372 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:30.372 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:30.372 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:30.372 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -46:30.372 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -46:30.383 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.305 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=809.031µs client=127.0.0.1:58975 client_network=udp -46:31.306 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=645.782µs client=127.0.0.1:36780 client_network=udp -46:31.307 [DEBUG] TestAgent.dns: request served from client: name=34bd9dc3-ec44-836e-1ebd-60b6c27165fb.query.consul. type=A class=IN latency=641.023µs client=127.0.0.1:53342 client_network=udp -46:31.307 [INFO] TestAgent: Requesting shutdown -46:31.307 [INFO] TestAgent.server: shutting down server -46:31.307 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.307 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.307 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.307 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.307 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.307 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.307 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.307 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.307 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.307 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.313 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.322Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.322Z [INFO] agent.router.manager: shutting down -46:31.323 [INFO] TestAgent: consul server down -46:31.323 [INFO] TestAgent: shutdown complete -46:31.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11724 network=tcp -46:31.323 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11724 network=udp -46:31.323 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11725 network=tcp -46:31.823 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.823 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_ServiceLookup_AnswerLimits (0.00s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} (1.77s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (1.64s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (1.70s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (1.84s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (1.82s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (1.79s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (1.65s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (1.67s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (1.80s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (1.86s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (1.82s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (1.67s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{5_5_5_5_5_5_5_5_5_5_5} (1.62s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{5_5_5_5_5_5_5_5_5_5_5} (1.68s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{5_5_5_5_5_5_5_5_5_5_5} (1.73s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{4_4_4_4_4_4_4_4_4_4_4} (1.66s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{4_4_4_4_4_4_4_4_4_4_4} (1.78s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{3_3_3_3_3_3_3_3_3_3_3} (1.76s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{3_3_3_3_3_3_3_3_3_3_3} (1.70s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{4_4_4_4_4_4_4_4_4_4_4} (1.89s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{2_2_2_2_2_2_2_2_2_2_2} (1.73s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (1.72s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{2_2_2_2_2_2_2_2_2_2_2} (1.71s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{3_3_3_3_3_3_3_3_3_3_3} (1.95s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{2_2_2_2_2_2_2_2_2_2_2} (1.87s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{1_1_1_1_1_1_1_1_1_1_1} (1.66s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{1_1_1_1_1_1_1_1_1_1_1} (1.67s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{1_1_1_1_1_1_1_1_1_1_1} (1.52s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{0_0_0_0_0_0_0_0_0_0_0} (1.80s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{0_0_0_0_0_0_0_0_0_0_0} (1.81s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (1.55s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (1.70s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (1.75s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (1.70s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (1.62s) - --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (1.68s) -[WARN] freeport: 10 out of 45 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:20.562Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.562Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.571Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:46:20.571Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.603 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 Address:127.0.0.1:11948}]" -46:20.603 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11948 [Follower]" leader= -46:20.603 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 127.0.0.1 -46:20.604 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 127.0.0.1 -2022-11-07T22:46:20.604Z [INFO] agent.router: Initializing LAN area manager -46:20.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11943 network=udp -46:20.604 [INFO] TestAgent.server: Adding LAN server: server="Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 (Addr: tcp/127.0.0.1:11948) (DC: dc1)" -46:20.604 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 area=wan -46:20.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11943 network=tcp -46:20.605 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11944 network=tcp -46:20.605 [INFO] TestAgent: started state syncer -46:20.605 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11900 network=tcp -46:20.655 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:20.666 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.666 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11948 [Candidate]" term=2 -46:20.685 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.685 [DEBUG] TestAgent.server.raft: vote granted: from=9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 term=2 tally=1 -46:20.685 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.685 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11948 [Leader]" -46:20.685 [INFO] TestAgent.server: cluster leadership acquired -46:20.685 [INFO] TestAgent.server: New leader elected: payload=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.700 [INFO] TestAgent.server: initializing acls -46:20.704 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:20.704 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:20.713 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:20.740 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:20.740 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:20.740 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:20.740 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.740 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 -46:20.741 [INFO] TestAgent.server: Updating LAN server: server="Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 (Addr: tcp/127.0.0.1:11948) (DC: dc1)" -46:20.741 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 area=wan -46:20.756 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.773 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.773 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.773 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.773 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.773 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.773 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.773 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.777 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.786 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.789 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:20.802 [INFO] TestAgent.server: server starting leave -46:20.802 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 127.0.0.1 -46:20.802 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56.dc1 area=wan -2022-11-07T22:46:20.802Z [INFO] agent.router.manager: shutting down -46:20.887 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.893 [INFO] TestAgent: Synced node info -46:23.803 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 127.0.0.1 -46:23.803 [INFO] TestAgent.server: Removing LAN server: server="Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 (Addr: tcp/127.0.0.1:11948) (DC: dc1)" -46:23.803 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-9c2a1bd7-aaa6-b2f2-9ca9-bf197fe38c56 -46:23.814 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:23.814 [DEBUG] TestAgent: Node info in sync -46:23.814 [DEBUG] TestAgent: Node info in sync -46:24.747 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:26.747 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:26.803 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s -46:28.747 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:30.747 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:30.747 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" -46:31.804 [INFO] TestAgent: Requesting shutdown -46:31.804 [INFO] TestAgent.server: shutting down server -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.804 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.804 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.804 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:46:31.809Z [INFO] agent.router.manager: shutting down -46:31.809 [INFO] TestAgent: consul server down -46:31.809 [INFO] TestAgent: shutdown complete -46:31.809 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11943 network=tcp -46:31.809 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11943 network=udp -46:31.809 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11944 network=tcp -46:32.309 [INFO] TestAgent: Waiting for endpoints to shut down -46:32.309 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_Leave_ACLDeny (11.76s) - --- PASS: TestAgent_Leave_ACLDeny/no_token (0.00s) - --- PASS: TestAgent_Leave_ACLDeny/read-only_token (0.01s) - --- PASS: TestAgent_Leave_ACLDeny/agent_master_token (11.01s) +[WARN] freeport: 4 out of 18 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:16.998Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.021Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.142 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c6519e0c-db94-909f-3a96-ad4092add244 Address:127.0.0.1:30006}]" +32:17.143 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.144 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.144Z [INFO] agent.router: Initializing LAN area manager +32:17.144 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30006 [Follower]" leader= +32:17.144 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.144 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30006) (DC: dc1)" +32:17.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30029 network=udp +32:17.144 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30029 network=tcp +32:17.145 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30009 network=tcp +32:17.145 [INFO] TestAgent: started state syncer +32:17.145 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30007 network=tcp +32:17.183 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.184 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30006 [Candidate]" term=2 +32:17.244 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.244 [DEBUG] TestAgent.server.raft: vote granted: from=c6519e0c-db94-909f-3a96-ad4092add244 term=2 tally=1 +32:17.244 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.244 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30006 [Leader]" +32:17.248 [INFO] TestAgent.server: cluster leadership acquired +32:17.248 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.280 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30006 +32:17.316 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.340 [INFO] TestAgent: Synced node info +32:17.340 [DEBUG] TestAgent: Node info in sync +32:17.372 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.372 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.372 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.372 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.372 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.372 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.372 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.440 [INFO] TestAgent.server: federation state anti-entropy synced +32:19.145 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=4.687512ms client=127.0.0.1:49482 client_network=udp +32:19.167 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=21.803328ms client=127.0.0.1:52549 client_network=udp +32:19.192 [DEBUG] TestAgent.dns: request served from client: name=27485fa2-054f-40dc-3553-0eb7c8bd5c40.query.consul. type=ANY class=IN latency=20.796117ms client=127.0.0.1:43761 client_network=udp +32:19.192 [INFO] TestAgent: Requesting shutdown +32:19.192 [INFO] TestAgent.server: shutting down server +32:19.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.193 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.193 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.193 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.193 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.193 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.204Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.204Z [INFO] agent.router.manager: shutting down +32:19.210 [INFO] TestAgent: consul server down +32:19.210 [INFO] TestAgent: shutdown complete +32:19.210 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30029 network=tcp +32:19.228 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30029 network=udp +32:19.228 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30009 network=tcp +32:19.729 [INFO] TestAgent: Waiting for endpoints to shut down +32:19.729 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:17.144Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.176Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.303 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:89c7d317-b8e5-bd56-2e5c-001e07774d6b Address:127.0.0.1:29964}]" +32:17.304 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29964 [Follower]" leader= +32:17.304 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.305 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.305Z [INFO] agent.router: Initializing LAN area manager +32:17.305 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30057 network=udp +32:17.305 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29964) (DC: dc1)" +32:17.305 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.305 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30057 network=tcp +32:17.306 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29960 network=tcp +32:17.306 [INFO] TestAgent: started state syncer +32:17.307 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29965 network=tcp +32:17.355 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.355 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29964 [Candidate]" term=2 +32:17.383 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.383 [DEBUG] TestAgent.server.raft: vote granted: from=89c7d317-b8e5-bd56-2e5c-001e07774d6b term=2 tally=1 +32:17.383 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.383 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29964 [Leader]" +32:17.384 [INFO] TestAgent.server: cluster leadership acquired +32:17.384 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.404 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29964 +32:17.452 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.513 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.513 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.513 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.513 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.513 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.514 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.514 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.542 [INFO] TestAgent: Synced node info +32:17.542 [DEBUG] TestAgent: Node info in sync +32:17.573 [INFO] TestAgent.server: federation state anti-entropy synced +32:18.461 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:18.480 [INFO] TestAgent: Synced node info +32:19.204 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=12.082695ms client=127.0.0.1:41349 client_network=udp +32:19.207 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=2.282381ms client=127.0.0.1:54592 client_network=udp +32:19.231 [DEBUG] TestAgent.dns: request served from client: name=2f5cde29-5ddf-e1b5-72e8-0d1808ca3f5b.query.consul. type=A class=IN latency=2.966356ms client=127.0.0.1:42484 client_network=udp +32:19.231 [INFO] TestAgent: Requesting shutdown +32:19.231 [INFO] TestAgent.server: shutting down server +32:19.231 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.231 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.231 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.231 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.231 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.231 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.231 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.231 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.231 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.231 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.238 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.243Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.243Z [INFO] agent.router.manager: shutting down +32:19.243 [INFO] TestAgent: consul server down +32:19.243 [INFO] TestAgent: shutdown complete +32:19.243 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30057 network=tcp +32:19.243 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30057 network=udp +32:19.243 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29960 network=tcp +32:19.744 [INFO] TestAgent: Waiting for endpoints to shut down +32:19.744 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:17.041Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.065Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.264 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:60fa77bc-50f3-7123-faf1-873dcb403079 Address:127.0.0.1:30048}]" +32:17.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30048 [Follower]" leader= +32:17.265 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.265 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.265Z [INFO] agent.router: Initializing LAN area manager +32:17.265 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29966 network=udp +32:17.266 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30048) (DC: dc1)" +32:17.266 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.266 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29966 network=tcp +32:17.266 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30044 network=tcp +32:17.266 [INFO] TestAgent: started state syncer +32:17.267 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30049 network=tcp +32:17.328 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.328 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30048 [Candidate]" term=2 +32:17.361 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.361 [DEBUG] TestAgent.server.raft: vote granted: from=60fa77bc-50f3-7123-faf1-873dcb403079 term=2 tally=1 +32:17.361 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.361 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30048 [Leader]" +32:17.361 [INFO] TestAgent.server: cluster leadership acquired +32:17.361 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.377 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30048 +32:17.438 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.495 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.495 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.495 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.495 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.495 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.495 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.495 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.524 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.540 [INFO] TestAgent: Synced node info +32:17.540 [DEBUG] TestAgent: Node info in sync +32:17.540 [INFO] TestAgent.server: federation state anti-entropy synced +32:17.851 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.851 [DEBUG] TestAgent: Node info in sync +32:19.264 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.699529ms client=127.0.0.1:49249 client_network=udp +32:19.266 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.671977ms client=127.0.0.1:52060 client_network=udp +32:19.274 [DEBUG] TestAgent.dns: request served from client: name=71e7a5b7-3da9-1e66-fe92-39cacce8b755.query.consul. type=ANY class=IN latency=1.99708ms client=127.0.0.1:40391 client_network=udp +32:19.274 [INFO] TestAgent: Requesting shutdown +32:19.274 [INFO] TestAgent.server: shutting down server +32:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.274 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.274 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.274 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.274 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.278 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.278Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.278Z [INFO] agent.router.manager: shutting down +32:19.279 [INFO] TestAgent: consul server down +32:19.279 [INFO] TestAgent: shutdown complete +32:19.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29966 network=tcp +32:19.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29966 network=udp +32:19.281 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30044 network=tcp +32:19.783 [INFO] TestAgent: Waiting for endpoints to shut down +32:19.783 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 25 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false -[WARN] freeport: 10 out of 17 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:17.067Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.110Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.278 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bbbb429f-4be1-5787-cfb9-903949548f92 Address:127.0.0.1:30055}]" +32:17.279 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30055 [Follower]" leader= +32:17.279 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.280 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.280Z [INFO] agent.router: Initializing LAN area manager +32:17.280 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30015 network=udp +32:17.280 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30055) (DC: dc1)" +32:17.280 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30015 network=tcp +32:17.280 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.281 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30051 network=tcp +32:17.281 [INFO] TestAgent: started state syncer +32:17.281 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30056 network=tcp +32:17.338 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.338 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30055 [Candidate]" term=2 +32:17.371 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.371 [DEBUG] TestAgent.server.raft: vote granted: from=bbbb429f-4be1-5787-cfb9-903949548f92 term=2 tally=1 +32:17.372 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.372 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30055 [Leader]" +32:17.372 [INFO] TestAgent.server: cluster leadership acquired +32:17.372 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.396 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30055 +32:17.440 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.514 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.514 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.514 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.514 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.514 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.515 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.515 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.542 [INFO] TestAgent.server: federation state anti-entropy synced +32:17.694 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.744 [INFO] TestAgent: Synced node info +32:18.980 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:18.980 [DEBUG] TestAgent: Node info in sync +32:18.980 [DEBUG] TestAgent: Node info in sync +32:19.388 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.248775ms client=127.0.0.1:36959 client_network=udp +32:19.397 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.275797ms client=127.0.0.1:57024 client_network=udp +32:19.399 [DEBUG] TestAgent.dns: request served from client: name=feef02ef-05e4-5443-9fe4-03cbbeb7be29.query.consul. type=AAAA class=IN latency=1.247746ms client=127.0.0.1:36856 client_network=udp +32:19.400 [INFO] TestAgent: Requesting shutdown +32:19.400 [INFO] TestAgent.server: shutting down server +32:19.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.400 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.400 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.400 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.400 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.400 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.400 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.400 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.400 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.412 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.425Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.425Z [INFO] agent.router.manager: shutting down +32:19.425 [INFO] TestAgent: consul server down +32:19.425 [INFO] TestAgent: shutdown complete +32:19.425 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30015 network=tcp +32:19.425 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30015 network=udp +32:19.425 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30051 network=tcp +32:19.928 [INFO] TestAgent: Waiting for endpoints to shut down +32:19.928 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:17.324Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.349Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.442 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e36b3928-4af1-77b1-183b-1875fd9a91ff Address:127.0.0.1:30041}]" +32:17.442 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30041 [Follower]" leader= +32:17.444 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.445 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.445Z [INFO] agent.router: Initializing LAN area manager +32:17.450 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30041) (DC: dc1)" +32:17.450 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.450 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30008 network=tcp +32:17.450 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30008 network=udp +32:17.451 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30037 network=tcp +32:17.451 [INFO] TestAgent: started state syncer +32:17.451 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30042 network=tcp +32:17.510 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.510 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30041 [Candidate]" term=2 +32:17.542 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.542 [DEBUG] TestAgent.server.raft: vote granted: from=e36b3928-4af1-77b1-183b-1875fd9a91ff term=2 tally=1 +32:17.542 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.542 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30041 [Leader]" +32:17.542 [INFO] TestAgent.server: cluster leadership acquired +32:17.543 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30041 +32:17.614 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.690 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.690 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.690 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.690 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.690 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.690 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.690 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.728 [INFO] TestAgent.server: federation state anti-entropy synced +32:17.808 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.831 [INFO] TestAgent: Synced node info +32:19.490 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.720012ms client=127.0.0.1:46187 client_network=udp +32:19.513 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.502079ms client=127.0.0.1:33572 client_network=udp +32:19.519 [DEBUG] TestAgent.dns: request served from client: name=b4b448b1-3c8f-6c26-a2c9-bbbe777b8378.query.consul. type=AAAA class=IN latency=1.017815ms client=127.0.0.1:33452 client_network=udp +32:19.519 [INFO] TestAgent: Requesting shutdown +32:19.519 [INFO] TestAgent.server: shutting down server +32:19.519 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.519 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.519 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.519 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.519 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.519 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.519 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.519 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.519 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.519 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.524 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.525Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.525Z [INFO] agent.router.manager: shutting down +32:19.525 [INFO] TestAgent: consul server down +32:19.525 [INFO] TestAgent: shutdown complete +32:19.525 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30008 network=tcp +32:19.525 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30008 network=udp +32:19.525 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30037 network=tcp +32:20.028 [INFO] TestAgent: Waiting for endpoints to shut down +32:20.028 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:17.154Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.177Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.298 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5988b08e-d295-db62-b010-8ea76cdc1a9b Address:127.0.0.1:29957}]" +32:17.299 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.300 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29957 [Follower]" leader= +32:17.300 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.300Z [INFO] agent.router: Initializing LAN area manager +32:17.300 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:29957) (DC: dc1)" +32:17.300 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.300 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29994 network=udp +32:17.300 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29994 network=tcp +32:17.301 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30016 network=tcp +32:17.301 [INFO] TestAgent: started state syncer +32:17.301 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29958 network=tcp +32:17.366 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.366 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29957 [Candidate]" term=2 +32:17.400 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.400 [DEBUG] TestAgent.server.raft: vote granted: from=5988b08e-d295-db62-b010-8ea76cdc1a9b term=2 tally=1 +32:17.400 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.400 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29957 [Leader]" +32:17.400 [INFO] TestAgent.server: cluster leadership acquired +32:17.400 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29957 +32:17.456 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.528 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.528 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.528 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.528 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.528 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.528 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.528 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.576 [INFO] TestAgent.server: federation state anti-entropy synced +32:17.719 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.729 [INFO] TestAgent: Synced node info +32:17.729 [DEBUG] TestAgent: Node info in sync +32:19.487 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=A class=IN latency=1.835848ms client=127.0.0.1:37166 client_network=udp +32:19.489 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=A class=IN latency=1.625015ms client=127.0.0.1:60668 client_network=udp +32:19.513 [DEBUG] TestAgent.dns: request served from client: name=c4d378a3-9519-1ba6-9559-ee40902cdfdf.query.consul. type=A class=IN latency=1.27861ms client=127.0.0.1:39343 client_network=udp +32:19.516 [INFO] TestAgent: Requesting shutdown +32:19.516 [INFO] TestAgent.server: shutting down server +32:19.516 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.516 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.516 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.516 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.516 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.516 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.516 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.516 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.516 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.516 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.524 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.524Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.524Z [INFO] agent.router.manager: shutting down +32:19.524 [INFO] TestAgent: consul server down +32:19.524 [INFO] TestAgent: shutdown complete +32:19.524 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29994 network=tcp +32:19.524 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29994 network=udp +32:19.525 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30016 network=tcp +32:20.028 [INFO] TestAgent: Waiting for endpoints to shut down +32:20.028 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 25 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:17.461Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:17.476Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:17.566 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b13fd73b-7130-602a-6674-b21fcde3eb54 Address:127.0.0.1:30111}]" +32:17.566 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30111 [Follower]" leader= +32:17.567 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:17.567 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:17.568Z [INFO] agent.router: Initializing LAN area manager +32:17.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30064 network=udp +32:17.568 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30111) (DC: dc1)" +32:17.568 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:17.569 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30064 network=tcp +32:17.569 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30030 network=tcp +32:17.569 [INFO] TestAgent: started state syncer +32:17.571 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30112 network=tcp +32:17.603 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:17.603 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30111 [Candidate]" term=2 +32:17.624 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:17.624 [DEBUG] TestAgent.server.raft: vote granted: from=b13fd73b-7130-602a-6674-b21fcde3eb54 term=2 tally=1 +32:17.624 [INFO] TestAgent.server.raft: election won: tally=1 +32:17.624 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30111 [Leader]" +32:17.624 [INFO] TestAgent.server: cluster leadership acquired +32:17.624 [INFO] TestAgent.server: New leader elected: payload=test-node +32:17.640 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30111 +32:17.681 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:17.741 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:17.741 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:17.741 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:17.741 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:17.741 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:17.741 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:17.741 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:17.765 [INFO] TestAgent.server: federation state anti-entropy synced +32:17.805 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:17.819 [INFO] TestAgent: Synced node info +32:19.519 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:19.519 [DEBUG] TestAgent: Node info in sync +32:19.519 [DEBUG] TestAgent: Node info in sync +32:19.613 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=ANY class=IN latency=1.42809ms client=127.0.0.1:50090 client_network=udp +32:19.615 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=ANY class=IN latency=1.53822ms client=127.0.0.1:34273 client_network=udp +32:19.625 [DEBUG] TestAgent.dns: request served from client: name=b3427783-0e0f-ac1d-41aa-ed9b390439ee.query.consul. type=ANY class=IN latency=9.527276ms client=127.0.0.1:43266 client_network=udp +32:19.625 [INFO] TestAgent: Requesting shutdown +32:19.625 [INFO] TestAgent.server: shutting down server +32:19.625 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:19.625 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:19.625 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:19.625 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:19.625 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:19.625 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:19.625 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:19.625 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:19.625 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:19.625 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:19.649 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:19.649Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:19.649Z [INFO] agent.router.manager: shutting down +32:19.649 [INFO] TestAgent: consul server down +32:19.649 [INFO] TestAgent: shutdown complete +32:19.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30064 network=tcp +32:19.649 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30064 network=udp +32:19.649 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30030 network=tcp +32:20.152 [INFO] TestAgent: Waiting for endpoints to shut down +32:20.152 [INFO] TestAgent: Endpoints down +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:32:18.772Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:18.825Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:18.952 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:16b2223b-391b-301a-9803-ba9a32b1df8b Address:127.0.0.1:30062}]" +32:18.952 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +32:18.953 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:32:18.953Z [INFO] agent.router: Initializing LAN area manager +32:18.953 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30036 network=udp +32:18.953 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30062 [Follower]" leader= +32:18.953 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:30062) (DC: dc1)" +32:18.953 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +32:18.953 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30036 network=tcp +32:18.954 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30023 network=tcp +32:18.954 [INFO] TestAgent: started state syncer +32:18.954 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30063 network=tcp +32:19.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:19.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30062 [Candidate]" term=2 +32:19.055 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:19.055 [DEBUG] TestAgent.server.raft: vote granted: from=16b2223b-391b-301a-9803-ba9a32b1df8b term=2 tally=1 +32:19.055 [INFO] TestAgent.server.raft: election won: tally=1 +32:19.055 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30062 [Leader]" +32:19.055 [INFO] TestAgent.server: cluster leadership acquired +32:19.056 [INFO] TestAgent.server: New leader elected: payload=test-node +32:19.085 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30062 +32:19.143 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:19.209 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:19.209 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:19.209 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:19.209 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:19.209 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:19.228 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +32:19.228 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +32:19.238 [INFO] TestAgent: Synced node info +32:19.238 [DEBUG] TestAgent: Node info in sync +32:19.264 [INFO] TestAgent.server: federation state anti-entropy synced +32:20.492 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:20.508 [INFO] TestAgent: Synced node info +32:20.529 [DEBUG] TestAgent.dns: request served from client: name=api-tier.service.consul. type=AAAA class=IN latency=1.293161ms client=127.0.0.1:53607 client_network=udp +32:20.537 [DEBUG] TestAgent.dns: request served from client: name=api-tier.query.consul. type=AAAA class=IN latency=1.338775ms client=127.0.0.1:37492 client_network=udp +32:20.541 [DEBUG] TestAgent.dns: request served from client: name=7d4989bb-8b66-7fa1-9bf3-19dfa12c4af0.query.consul. type=AAAA class=IN latency=1.312653ms client=127.0.0.1:49820 client_network=udp +32:20.541 [INFO] TestAgent: Requesting shutdown +32:20.541 [INFO] TestAgent.server: shutting down server +32:20.541 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:20.541 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:20.542 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:20.542 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:20.542 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:20.542 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:20.542 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:20.542 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:20.542 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:20.542 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:20.545 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:20.546Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:20.546Z [INFO] agent.router.manager: shutting down +32:20.546 [INFO] TestAgent: consul server down +32:20.546 [INFO] TestAgent: shutdown complete +32:20.546 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30036 network=tcp +32:20.546 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30036 network=udp +32:20.546 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30023 network=tcp +32:21.048 [INFO] TestAgent: Waiting for endpoints to shut down +32:21.048 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_ServiceLookup_AnswerLimits (0.00s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{0_0_0_0_0_0_0_0_0_0_0} (2.17s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (2.28s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (2.13s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (2.29s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (2.39s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{30_30_8_8_6_8_8_5_8_-5_-5} (2.46s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (2.52s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{20_20_8_8_6_8_8_5_8_-5_-5} (2.14s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (2.24s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (2.35s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (2.44s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{9_9_8_8_6_8_8_5_8_8_-5} (2.55s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (2.45s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (2.43s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{8_8_8_8_6_8_8_5_8_8_-5} (2.43s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (2.19s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{7_7_7_7_6_7_7_5_7_7_-5} (2.16s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (2.30s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{6_6_6_6_6_6_6_5_6_6_-5} (2.29s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{5_5_5_5_5_5_5_5_5_5_5} (2.29s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{5_5_5_5_5_5_5_5_5_5_5} (2.28s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{4_4_4_4_4_4_4_4_4_4_4} (2.29s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{3_3_3_3_3_3_3_3_3_3_3} (2.48s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{5_5_5_5_5_5_5_5_5_5_5} (2.43s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{4_4_4_4_4_4_4_4_4_4_4} (1.97s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{4_4_4_4_4_4_4_4_4_4_4} (2.34s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{3_3_3_3_3_3_3_3_3_3_3} (2.43s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{3_3_3_3_3_3_3_3_3_3_3} (2.70s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{1_1_1_1_1_1_1_1_1_1_1} (2.75s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{2_2_2_2_2_2_2_2_2_2_2} (2.63s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{2_2_2_2_2_2_2_2_2_2_2} (2.77s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{2_2_2_2_2_2_2_2_2_2_2} (2.89s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{1_1_1_1_1_1_1_1_1_1_1} (2.73s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/A_lookup_{1_1_1_1_1_1_1_1_1_1_1} (2.92s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/ANY_lookup_{0_0_0_0_0_0_0_0_0_0_0} (2.71s) + --- PASS: TestDNS_ServiceLookup_AnswerLimits/AAAA_lookup_{0_0_0_0_0_0_0_0_0_0_0} (2.33s) +[WARN] freeport: 4 out of 11 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=false +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=false +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=false +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=true +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=false +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=true -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=true -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=false -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=false +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=true +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=false === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=false#01 === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true#01 -2022-11-07T22:46:22.333Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.347Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.380 [INFO] Agent2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c1d8fc88-44bf-59f7-65b3-6dee02c8af73 Address:127.0.0.1:11152}]" -46:22.380 [INFO] Agent2.server.raft: entering follower state: follower="Node at 127.0.0.1:11152 [Follower]" leader= -46:22.381 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 127.0.0.1 -46:22.381 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 127.0.0.1 -2022-11-07T22:46:22.381Z [INFO] agent.router: Initializing LAN area manager -46:22.382 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 area=wan -46:22.382 [INFO] Agent2.server: Adding LAN server: server="Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (Addr: tcp/127.0.0.1:11152) (DC: dc1)" -46:22.382 [INFO] Agent2: Started DNS server: address=127.0.0.1:11084 network=udp -46:22.382 [INFO] Agent2: Started DNS server: address=127.0.0.1:11084 network=tcp -46:22.382 [INFO] Agent2: Started HTTP server: address=127.0.0.1:11057 network=tcp -46:22.382 [INFO] Agent2: started state syncer -46:22.382 [INFO] Agent2: Started gRPC server: address=127.0.0.1:11153 network=tcp -46:22.422 [WARN] Agent2.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.422 [INFO] Agent2.server.raft: entering candidate state: node="Node at 127.0.0.1:11152 [Candidate]" term=2 -46:22.435 [DEBUG] Agent2.server.raft: votes: needed=1 -46:22.435 [DEBUG] Agent2.server.raft: vote granted: from=c1d8fc88-44bf-59f7-65b3-6dee02c8af73 term=2 tally=1 -46:22.435 [INFO] Agent2.server.raft: election won: tally=1 -46:22.435 [INFO] Agent2.server.raft: entering leader state: leader="Node at 127.0.0.1:11152 [Leader]" -46:22.435 [INFO] Agent2.server: cluster leadership acquired -46:22.436 [INFO] Agent2.server: New leader elected: payload=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:22.448 [DEBUG] Agent2.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11152 -46:22.469 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:22.490 [INFO] Agent2: Synced node info -46:22.490 [DEBUG] Agent2: Node info in sync -46:22.513 [INFO] Agent2.server.connect: initialized primary datacenter CA with provider: provider=consul -46:22.513 [INFO] Agent2.leader: started routine: routine="federation state anti-entropy" -46:22.513 [INFO] Agent2.leader: started routine: routine="federation state pruning" -46:22.513 [INFO] Agent2.leader: started routine: routine="intermediate cert renew watch" -46:22.513 [INFO] Agent2.leader: started routine: routine="CA root pruning" -46:22.513 [DEBUG] Agent2.server: Skipping self join check for node since the cluster is too small: node=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:22.513 [INFO] Agent2.server: member joined, marking health alive: member=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:22.525 [INFO] Agent2.server: federation state anti-entropy synced -46:22.800 [DEBUG] Agent2.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:57152 -46:22.800 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c 127.0.0.1 -46:22.800 [INFO] Agent2.server: Adding LAN server: server="Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c (Addr: tcp/127.0.0.1:11047) (DC: dc1)" -46:22.800 [INFO] Agent2.server: New leader elected: payload=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:22.800 [ERROR] Agent2.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c other=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:22.800 [INFO] Agent2.server: member joined, marking health alive: member=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:22.801 [DEBUG] Agent2.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 127.0.0.1:11046 -46:22.801 [INFO] Agent2: Requesting shutdown -46:22.801 [INFO] Agent2.server: shutting down server -46:22.801 [DEBUG] Agent2.leader: stopping routine: routine="federation state anti-entropy" -46:22.801 [DEBUG] Agent2.leader: stopping routine: routine="federation state pruning" -46:22.801 [DEBUG] Agent2.leader: stopping routine: routine="intermediate cert renew watch" -46:22.801 [DEBUG] Agent2.leader: stopping routine: routine="CA root pruning" -46:22.801 [WARN] Agent2.server.serf.lan: serf: Shutdown without a Leave -46:22.802 [ERROR] Agent2.server: error performing anti-entropy sync of federation state: error="context canceled" -46:22.802 [DEBUG] Agent2.leader: stopped routine: routine="federation state anti-entropy" -46:22.802 [DEBUG] Agent2.leader: stopped routine: routine="federation state pruning" -46:22.802 [DEBUG] Agent2.leader: stopped routine: routine="intermediate cert renew watch" -46:22.802 [DEBUG] Agent2.leader: stopped routine: routine="CA root pruning" -46:22.802 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 127.0.0.1 -46:22.802 [DEBUG] Agent2.server: Successfully performed flood-join for server at address: server=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 address=127.0.0.1:11046 -46:22.802 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 area=wan -46:22.806 [WARN] Agent2.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:22.809Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:22.809Z [INFO] agent.router.manager: shutting down -46:22.809 [INFO] Agent2: consul server down -46:22.809 [INFO] Agent2: shutdown complete -46:22.809 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11084 network=tcp -46:22.809 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:11084 network=udp -46:22.809 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:11057 network=tcp -46:23.310 [INFO] Agent2: Waiting for endpoints to shut down -46:23.310 [INFO] Agent2: Endpoints down -2022-11-07T22:46:21.840Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:21.855Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:21.879 [INFO] Agent1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ad365249-86b4-7cdd-eb59-1c6a3297b55c Address:127.0.0.1:11047}]" -46:21.879 [INFO] Agent1.server.raft: entering follower state: follower="Node at 127.0.0.1:11047 [Follower]" leader= -46:21.879 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 127.0.0.1 -46:21.880 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c 127.0.0.1 -2022-11-07T22:46:21.880Z [INFO] agent.router: Initializing LAN area manager -46:21.880 [INFO] Agent1.server: Adding LAN server: server="Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c (Addr: tcp/127.0.0.1:11047) (DC: dc1)" -46:21.880 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c.dc1 area=wan -46:21.880 [INFO] Agent1: Started DNS server: address=127.0.0.1:11042 network=udp -46:21.880 [INFO] Agent1: Started DNS server: address=127.0.0.1:11042 network=tcp -46:21.881 [INFO] Agent1: Started HTTP server: address=127.0.0.1:11162 network=tcp -46:21.881 [INFO] Agent1: started state syncer -46:21.881 [INFO] Agent1: Started gRPC server: address=127.0.0.1:11048 network=tcp -46:21.920 [WARN] Agent1.server.raft: heartbeat timeout reached, starting election: last-leader= -46:21.920 [INFO] Agent1.server.raft: entering candidate state: node="Node at 127.0.0.1:11047 [Candidate]" term=2 -46:21.931 [DEBUG] Agent1.server.raft: votes: needed=1 -46:21.931 [DEBUG] Agent1.server.raft: vote granted: from=ad365249-86b4-7cdd-eb59-1c6a3297b55c term=2 tally=1 -46:21.931 [INFO] Agent1.server.raft: election won: tally=1 -46:21.931 [INFO] Agent1.server.raft: entering leader state: leader="Node at 127.0.0.1:11047 [Leader]" -46:21.931 [INFO] Agent1.server: cluster leadership acquired -46:21.932 [INFO] Agent1.server: New leader elected: payload=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:21.938 [DEBUG] Agent1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11047 -46:21.952 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:21.971 [INFO] Agent1.server.connect: initialized primary datacenter CA with provider: provider=consul -46:21.971 [INFO] Agent1.leader: started routine: routine="federation state anti-entropy" -46:21.971 [INFO] Agent1.leader: started routine: routine="federation state pruning" -46:21.971 [INFO] Agent1.leader: started routine: routine="intermediate cert renew watch" -46:21.971 [INFO] Agent1.leader: started routine: routine="CA root pruning" -46:21.971 [DEBUG] Agent1.server: Skipping self join check for node since the cluster is too small: node=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:21.971 [INFO] Agent1.server: member joined, marking health alive: member=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:21.982 [INFO] Agent1.server: federation state anti-entropy synced -46:22.033 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth -46:22.036 [INFO] Agent1: Synced node info -46:22.036 [DEBUG] Agent1: Node info in sync -46:22.799 [INFO] Agent1: (LAN) joining: lan_addresses=[127.0.0.1:11150] -46:22.800 [DEBUG] Agent1.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:11150 -46:22.801 [DEBUG] Agent1.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:59248 -46:22.801 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 127.0.0.1 -46:22.801 [INFO] Agent1: (LAN) joined: number_of_nodes=1 -46:22.801 [DEBUG] Agent1: systemd notify failed: error="No socket" -46:22.801 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 127.0.0.1 -46:22.801 [INFO] Agent1.server: Adding LAN server: server="Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (Addr: tcp/127.0.0.1:11152) (DC: dc1)" -46:22.801 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 area=wan -46:22.802 [ERROR] Agent1.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 other=Node-ad365249-86b4-7cdd-eb59-1c6a3297b55c -46:22.802 [INFO] Agent1.server: member joined, marking health alive: member=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:23.381 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (timeout reached) -46:23.881 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 has failed, no acks received -46:23.942 [WARN] Agent1: error getting server health from server: server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:11152: connect: connection refused" -46:24.599 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth -46:24.599 [DEBUG] Agent1: Node info in sync -46:24.942 [WARN] Agent1: error getting server health from server: server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 error="context deadline exceeded" -46:25.381 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (timeout reached) -46:25.942 [WARN] Agent1: error getting server health from server: server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:11152: connect: connection refused" -46:26.881 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 has failed, no acks received -46:26.942 [WARN] Agent1: error getting server health from server: server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 error="context deadline exceeded" -46:27.881 [INFO] Agent1.server.memberlist.lan: memberlist: Marking Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 as failed, suspect timeout reached (0 peer confirmations) -46:27.881 [INFO] Agent1.server.serf.lan: serf: EventMemberFailed: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 127.0.0.1 -46:27.881 [INFO] Agent1.server: Removing LAN server: server="Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (Addr: tcp/127.0.0.1:11152) (DC: dc1)" -46:27.881 [INFO] Agent1.server: member failed, marking health critical: member=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:27.889 [INFO] Agent1: Force leaving node: node=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 -46:27.889 [INFO] Agent1.server.serf.lan: serf: EventMemberLeave (forced): Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 127.0.0.1 -46:27.889 [INFO] Agent1.server.serf.lan: serf: EventMemberReap (forced): Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 127.0.0.1 -46:27.889 [INFO] Agent1.server: Removing LAN server: server="Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (Addr: tcp/127.0.0.1:11152) (DC: dc1)" -46:27.889 [INFO] Agent1.server: Removing LAN server: server="Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (Addr: tcp/127.0.0.1:11152) (DC: dc1)" -46:27.889 [INFO] Agent1.server: deregistering member: member=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 reason=left -46:28.381 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 (timeout reached) -46:29.880 [DEBUG] Agent1.server.memberlist.wan: memberlist: Failed ping: Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 (timeout reached) -46:30.880 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73 has failed, no acks received -46:31.881 [INFO] Agent1.server.memberlist.wan: memberlist: Suspect Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 has failed, no acks received -46:35.890 [INFO] Agent1.server.serf.wan: serf: EventMemberReap (forced): Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 127.0.0.1 -46:35.890 [INFO] Agent1: Requesting shutdown -46:35.890 [INFO] Agent1.server: shutting down server -46:35.890 [DEBUG] Agent1.leader: stopping routine: routine="federation state anti-entropy" -46:35.890 [DEBUG] Agent1.leader: stopping routine: routine="federation state pruning" -46:35.890 [DEBUG] Agent1.leader: stopping routine: routine="intermediate cert renew watch" -46:35.890 [DEBUG] Agent1.leader: stopping routine: routine="CA root pruning" -46:35.890 [INFO] Agent1.server: Handled event for server in area: event=member-reap server=Node-c1d8fc88-44bf-59f7-65b3-6dee02c8af73.dc1 area=wan -46:35.890 [WARN] Agent1.server.serf.lan: serf: Shutdown without a Leave -46:35.890 [ERROR] Agent1.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.890 [DEBUG] Agent1.leader: stopped routine: routine="federation state anti-entropy" -46:35.890 [DEBUG] Agent1.leader: stopped routine: routine="federation state pruning" -46:35.890 [DEBUG] Agent1.leader: stopped routine: routine="intermediate cert renew watch" -46:35.890 [DEBUG] Agent1.leader: stopped routine: routine="CA root pruning" -46:35.895 [WARN] Agent1.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.899Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.899Z [INFO] agent.router.manager: shutting down -46:35.899 [INFO] Agent1: consul server down -46:35.899 [INFO] Agent1: shutdown complete -46:35.899 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:11042 network=tcp -46:35.899 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:11042 network=udp -46:35.899 [INFO] Agent1: Stopping server: protocol=HTTP address=127.0.0.1:11162 network=tcp -46:36.402 [INFO] Agent1: Waiting for endpoints to shut down -46:36.402 [INFO] Agent1: Endpoints down ---- PASS: TestAgent_ForceLeavePrune (14.58s) -[WARN] freeport: 10 out of 24 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=false#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! === RUN TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=true#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=true#01 -=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=false#01 -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -[WARN] freeport: 10 out of 10 pending ports are still in use; something probably didn't wait around for the port to be closed! -2022-11-07T22:46:04.878Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.889Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.910 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:413bd3be-7fe5-39ce-2c1c-03655583d9a3 Address:127.0.0.1:11865}]" -46:04.910 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11865 [Follower]" leader= -46:04.910 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3.dc1 127.0.0.1 -46:04.911 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3 127.0.0.1 -2022-11-07T22:46:04.911Z [INFO] agent.router: Initializing LAN area manager -46:04.911 [INFO] TestAgent.server: Adding LAN server: server="Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3 (Addr: tcp/127.0.0.1:11865) (DC: dc1)" -46:04.911 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3.dc1 area=wan -46:04.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11860 network=udp -46:04.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11860 network=tcp -46:04.911 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11861 network=tcp -46:04.911 [INFO] TestAgent: started state syncer -46:04.911 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11866 network=tcp -46:04.978 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.978 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11865 [Candidate]" term=2 -46:04.993 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.993 [DEBUG] TestAgent.server.raft: vote granted: from=413bd3be-7fe5-39ce-2c1c-03655583d9a3 term=2 tally=1 -46:04.993 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.993 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11865 [Leader]" -46:04.993 [INFO] TestAgent.server: cluster leadership acquired -46:04.993 [INFO] TestAgent.server: New leader elected: payload=Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3 -46:05.002 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11865 -46:05.033 [INFO] TestAgent: Synced node info -46:05.033 [DEBUG] TestAgent: Node info in sync -46:05.033 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:05.051 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:05.051 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:05.051 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:05.051 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:05.051 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:05.051 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3 -46:05.051 [INFO] TestAgent.server: member joined, marking health alive: member=Node-413bd3be-7fe5-39ce-2c1c-03655583d9a3 -46:05.065 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.286 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=125.15µs client=127.0.0.1:50762 client_network=tcp -46:05.286 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=103.188µs client=127.0.0.1:50764 client_network=tcp -46:05.287 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=108.238µs client=127.0.0.1:47373 client_network=udp -46:05.287 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=100.801µs client=127.0.0.1:39161 client_network=udp -46:05.288 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=130.316µs client=127.0.0.1:50766 client_network=tcp -46:05.288 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=121.783µs client=127.0.0.1:50768 client_network=tcp -46:05.288 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=84.88µs client=127.0.0.1:57144 client_network=udp -46:05.289 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=92.506µs client=127.0.0.1:58329 client_network=udp -46:05.289 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=120.232µs client=127.0.0.1:50770 client_network=tcp -46:05.289 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=98.898µs client=127.0.0.1:50772 client_network=tcp -46:05.290 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=81.715µs client=127.0.0.1:41295 client_network=udp -46:05.290 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=104.039µs client=127.0.0.1:46987 client_network=udp -46:05.290 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=107.448µs client=127.0.0.1:50774 client_network=tcp -46:05.291 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=86.415µs client=127.0.0.1:50776 client_network=tcp -46:05.291 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=114.184µs client=127.0.0.1:57390 client_network=udp -46:05.291 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=114.372µs client=127.0.0.1:58289 client_network=udp -46:05.292 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=102.593µs client=127.0.0.1:50778 client_network=tcp -46:05.292 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=111.205µs client=127.0.0.1:50780 client_network=tcp -46:05.292 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=104.9µs client=127.0.0.1:46360 client_network=udp -46:05.293 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=100.25µs client=127.0.0.1:52350 client_network=udp -46:05.293 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=93.25µs client=127.0.0.1:50782 client_network=tcp -46:05.293 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=97.37µs client=127.0.0.1:50784 client_network=tcp -46:05.294 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=97.014µs client=127.0.0.1:56029 client_network=udp -46:05.294 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=111.234µs client=127.0.0.1:43934 client_network=udp -46:05.294 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=106.537µs client=127.0.0.1:50786 client_network=tcp -46:05.295 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=91.064µs client=127.0.0.1:50788 client_network=tcp -46:05.295 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=105.466µs client=127.0.0.1:46359 client_network=udp -46:05.295 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=93.822µs client=127.0.0.1:36957 client_network=udp -46:05.296 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=125.322µs client=127.0.0.1:50790 client_network=tcp -46:05.296 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=98.623µs client=127.0.0.1:50792 client_network=tcp -46:05.297 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=87.773µs client=127.0.0.1:57427 client_network=udp -46:05.297 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=69.417µs client=127.0.0.1:36559 client_network=udp -46:05.297 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=115.043µs client=127.0.0.1:50794 client_network=tcp -46:05.298 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=114.605µs client=127.0.0.1:50796 client_network=tcp -46:05.298 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=114.347µs client=127.0.0.1:38685 client_network=udp -46:05.298 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=90.166µs client=127.0.0.1:46289 client_network=udp -46:05.299 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=111.567µs client=127.0.0.1:50798 client_network=tcp -46:05.299 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=81.235µs client=127.0.0.1:50800 client_network=tcp -46:05.299 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=70.943µs client=127.0.0.1:56492 client_network=udp -46:05.300 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=103.718µs client=127.0.0.1:45284 client_network=udp -46:05.300 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=111.12µs client=127.0.0.1:50802 client_network=tcp -46:05.300 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=110.837µs client=127.0.0.1:50804 client_network=tcp -46:05.301 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=126.062µs client=127.0.0.1:55388 client_network=udp -46:05.301 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=102.269µs client=127.0.0.1:37770 client_network=udp -46:05.301 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=121.766µs client=127.0.0.1:50806 client_network=tcp -46:05.302 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=116.537µs client=127.0.0.1:50808 client_network=tcp -46:05.302 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=113.806µs client=127.0.0.1:58110 client_network=udp -46:05.302 [DEBUG] TestAgent.dns: request served from client: name=4989a072-22f1-e705-1464-d0742ba7670b.query.consul. type=ANY class=IN latency=112.372µs client=127.0.0.1:47285 client_network=udp -46:07.321 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:07.321 [DEBUG] TestAgent: Node info in sync -46:32.135 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:32.137 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=59.577756ms client=127.0.0.1:51110 client_network=tcp -46:32.359 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:32.360 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=44.927376ms client=127.0.0.1:51114 client_network=tcp -46:32.554 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=54.707104ms client=127.0.0.1:48184 client_network=udp -46:32.601 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=43.699547ms client=127.0.0.1:46974 client_network=udp -46:32.652 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:32.654 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=47.958588ms client=127.0.0.1:51116 client_network=tcp -46:32.931 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:32.933 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=75.36011ms client=127.0.0.1:51118 client_network=tcp -46:33.170 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=52.237009ms client=127.0.0.1:55728 client_network=udp -46:33.220 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=44.521714ms client=127.0.0.1:50689 client_network=udp -46:33.277 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:33.278 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=47.897926ms client=127.0.0.1:51120 client_network=tcp -46:33.446 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:33.447 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=46.085241ms client=127.0.0.1:51122 client_network=tcp -46:33.642 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=42.535766ms client=127.0.0.1:59716 client_network=udp -46:33.717 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=64.783559ms client=127.0.0.1:40268 client_network=udp -46:33.769 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:33.771 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=50.35908ms client=127.0.0.1:51126 client_network=tcp -46:34.026 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:34.028 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=69.777652ms client=127.0.0.1:51128 client_network=tcp -46:34.258 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=46.437436ms client=127.0.0.1:57523 client_network=udp -46:34.308 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=44.573043ms client=127.0.0.1:45743 client_network=udp -46:34.363 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:34.364 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=46.756189ms client=127.0.0.1:51130 client_network=tcp -46:34.545 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:34.546 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=43.33992ms client=127.0.0.1:51132 client_network=tcp -46:34.712 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=40.594497ms client=127.0.0.1:46636 client_network=udp -46:34.756 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=40.886474ms client=127.0.0.1:43047 client_network=udp -46:34.806 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:34.808 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=48.42859ms client=127.0.0.1:51134 client_network=tcp -46:35.040 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:35.042 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=51.004165ms client=127.0.0.1:51136 client_network=tcp -46:35.310 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=76.79287ms client=127.0.0.1:47168 client_network=udp -46:35.360 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=44.842282ms client=127.0.0.1:38403 client_network=udp -46:35.413 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:35.415 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=48.626769ms client=127.0.0.1:51178 client_network=tcp -46:35.601 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:35.602 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=44.214338ms client=127.0.0.1:51194 client_network=tcp -46:35.779 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=47.198178ms client=127.0.0.1:37611 client_network=udp -46:35.829 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=47.279011ms client=127.0.0.1:52078 client_network=udp -46:35.886 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:35.889 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=55.616112ms client=127.0.0.1:51196 client_network=tcp -46:36.119 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:36.121 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=49.66809ms client=127.0.0.1:51198 client_network=tcp -46:36.338 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=45.105867ms client=127.0.0.1:41688 client_network=udp -46:36.385 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=42.498031ms client=127.0.0.1:49093 client_network=udp -46:36.454 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:36.456 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=66.105078ms client=127.0.0.1:51200 client_network=tcp -46:36.647 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:36.648 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=46.260976ms client=127.0.0.1:51202 client_network=tcp -46:36.828 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=52.145489ms client=127.0.0.1:59206 client_network=udp -46:36.883 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=51.942841ms client=127.0.0.1:33467 client_network=udp -46:36.936 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:36.938 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=51.015012ms client=127.0.0.1:51204 client_network=tcp -46:37.176 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:37.178 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=57.686154ms client=127.0.0.1:51212 client_network=tcp -46:37.408 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=48.437146ms client=127.0.0.1:36491 client_network=udp -46:37.458 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=45.198956ms client=127.0.0.1:35853 client_network=udp -46:37.509 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:37.510 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=44.577897ms client=127.0.0.1:51244 client_network=tcp -46:37.679 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 -46:37.680 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=44.392935ms client=127.0.0.1:51246 client_network=tcp -46:37.850 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=44.758762ms client=127.0.0.1:50266 client_network=udp -46:37.896 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=42.120802ms client=127.0.0.1:34353 client_network=udp -46:37.953 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:37.955 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=54.789519ms client=127.0.0.1:51248 client_network=tcp -46:38.180 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. 255 1}"] records=4091/5001 size=65523/80083 -46:38.182 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=53.10345ms client=127.0.0.1:51250 client_network=tcp -46:38.428 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=59.361517ms client=127.0.0.1:34709 client_network=udp -46:38.479 [DEBUG] TestAgent.dns: request served from client: name=641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul. type=ANY class=IN latency=44.719746ms client=127.0.0.1:54818 client_network=udp -46:38.485 [INFO] TestAgent: Requesting shutdown -46:38.485 [INFO] TestAgent.server: shutting down server -46:38.485 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:38.485 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:38.485 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:38.485 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:38.485 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:38.485 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:38.485 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:38.485 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:38.485 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:38.485 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:38.490 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:38.491Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:38.491Z [INFO] agent.router.manager: shutting down -46:38.491 [INFO] TestAgent: consul server down -46:38.491 [INFO] TestAgent: shutdown complete -46:38.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11860 network=tcp -46:38.491 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11860 network=udp -46:38.491 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11861 network=tcp -46:38.991 [INFO] TestAgent: Waiting for endpoints to shut down -46:38.991 [INFO] TestAgent: Endpoints down ---- PASS: TestDNS_TCP_and_UDP_Truncate (34.13s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=true (0.00s) +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=true#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=true#01 +=== RUN TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=false#01 +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +[WARN] freeport: 4 out of 4 pending ports are still in use; something probably didn't wait around for the port to be closed! +2021-10-05T16:31:20.090Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.107Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.210 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f5033a9-434a-0580-7404-3dcc33e58ca5 Address:127.0.0.1:30092}]" +31:20.211 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f5033a9-434a-0580-7404-3dcc33e58ca5.dc1 127.0.0.1 +31:20.212 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 127.0.0.1 +2021-10-05T16:31:20.212Z [INFO] agent.router: Initializing LAN area manager +31:20.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30087 network=udp +31:20.212 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30092 [Follower]" leader= +31:20.212 [INFO] TestAgent.server: Adding LAN server: server="Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 (Addr: tcp/127.0.0.1:30092) (DC: dc1)" +31:20.212 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f5033a9-434a-0580-7404-3dcc33e58ca5.dc1 area=wan +31:20.212 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30087 network=tcp +31:20.213 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30088 network=tcp +31:20.213 [INFO] TestAgent: started state syncer +31:20.213 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30079 network=tcp +31:20.249 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.249 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30092 [Candidate]" term=2 +31:20.275 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.276 [DEBUG] TestAgent.server.raft: vote granted: from=8f5033a9-434a-0580-7404-3dcc33e58ca5 term=2 tally=1 +31:20.276 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.276 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30092 [Leader]" +31:20.276 [INFO] TestAgent.server: cluster leadership acquired +31:20.276 [INFO] TestAgent.server: New leader elected: payload=Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 +31:20.285 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30092 +31:20.336 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.367 [INFO] TestAgent: Synced node info +31:20.410 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.410 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.410 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.410 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.410 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 +31:20.410 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 +31:20.443 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.444 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=210.137µs client=127.0.0.1:53956 client_network=tcp +31:20.449 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=155.539µs client=127.0.0.1:53958 client_network=tcp +31:20.460 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=429.997µs client=127.0.0.1:37915 client_network=udp +31:20.467 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=532.058µs client=127.0.0.1:35033 client_network=udp +31:20.467 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=305.255µs client=127.0.0.1:53962 client_network=tcp +31:20.468 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=164.205µs client=127.0.0.1:53964 client_network=tcp +31:20.468 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=116.234µs client=127.0.0.1:46482 client_network=udp +31:20.469 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=112.025µs client=127.0.0.1:52598 client_network=udp +31:20.469 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=103.751µs client=127.0.0.1:53966 client_network=tcp +31:20.470 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=100.301µs client=127.0.0.1:53968 client_network=tcp +31:20.470 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=92.734µs client=127.0.0.1:40444 client_network=udp +31:20.470 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=88.471µs client=127.0.0.1:60955 client_network=udp +31:20.471 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=107.018µs client=127.0.0.1:53970 client_network=tcp +31:20.471 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=103.461µs client=127.0.0.1:53972 client_network=tcp +31:20.471 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=111.538µs client=127.0.0.1:55309 client_network=udp +31:20.486 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=290.684µs client=127.0.0.1:33634 client_network=udp +31:20.488 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=139.42µs client=127.0.0.1:53976 client_network=tcp +31:20.489 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=139.936µs client=127.0.0.1:53978 client_network=tcp +31:20.489 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=190.049µs client=127.0.0.1:54861 client_network=udp +31:20.496 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=150.753µs client=127.0.0.1:54085 client_network=udp +31:20.497 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=182.66µs client=127.0.0.1:53980 client_network=tcp +31:20.498 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=130.899µs client=127.0.0.1:53982 client_network=tcp +31:20.498 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=136.05µs client=127.0.0.1:59071 client_network=udp +31:20.500 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=143.895µs client=127.0.0.1:56881 client_network=udp +31:20.500 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=125.216µs client=127.0.0.1:53986 client_network=tcp +31:20.501 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=86.808µs client=127.0.0.1:53988 client_network=tcp +31:20.501 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=100.909µs client=127.0.0.1:41776 client_network=udp +31:20.502 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=133.357µs client=127.0.0.1:52428 client_network=udp +31:20.502 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=128.359µs client=127.0.0.1:53990 client_network=tcp +31:20.503 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=173.619µs client=127.0.0.1:53992 client_network=tcp +31:20.508 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=186.853µs client=127.0.0.1:50868 client_network=udp +31:20.512 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=168.579µs client=127.0.0.1:47782 client_network=udp +31:20.515 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=132.509µs client=127.0.0.1:53998 client_network=tcp +31:20.516 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=111.294µs client=127.0.0.1:54000 client_network=tcp +31:20.516 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=196.469µs client=127.0.0.1:46271 client_network=udp +31:20.517 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=126.079µs client=127.0.0.1:55678 client_network=udp +31:20.517 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=160.394µs client=127.0.0.1:54002 client_network=tcp +31:20.518 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=124.313µs client=127.0.0.1:54004 client_network=tcp +31:20.518 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=126.934µs client=127.0.0.1:40742 client_network=udp +31:20.519 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=139.764µs client=127.0.0.1:34022 client_network=udp +31:20.519 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=191.324µs client=127.0.0.1:54006 client_network=tcp +31:20.520 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=141.503µs client=127.0.0.1:54008 client_network=tcp +31:20.521 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=361.978µs client=127.0.0.1:36843 client_network=udp +31:20.521 [DEBUG] TestAgent.dns: request served from client: name=normal.service.consul. type=ANY class=IN latency=143.28µs client=127.0.0.1:36930 client_network=udp +31:20.522 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=125.841µs client=127.0.0.1:54010 client_network=tcp +31:20.522 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=147.626µs client=127.0.0.1:54012 client_network=tcp +31:20.523 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=178.926µs client=127.0.0.1:58586 client_network=udp +31:20.523 [DEBUG] TestAgent.dns: request served from client: name=526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul. type=ANY class=IN latency=130.728µs client=127.0.0.1:44075 client_network=udp +31:20.938 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.938 [DEBUG] TestAgent: Node info in sync +31:20.938 [DEBUG] TestAgent: Node info in sync +32:19.022 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:19.024 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=223.018331ms client=127.0.0.1:54444 client_network=tcp +32:19.947 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:19.949 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=101.540185ms client=127.0.0.1:54450 client_network=tcp +32:20.296 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f5033a9-434a-0580-7404-3dcc33e58ca5 +32:20.598 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=133.263212ms client=127.0.0.1:55980 client_network=udp +32:20.736 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=131.879975ms client=127.0.0.1:60841 client_network=udp +32:20.853 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:20.856 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=109.533671ms client=127.0.0.1:54456 client_network=tcp +32:21.639 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:21.644 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=109.561722ms client=127.0.0.1:54458 client_network=tcp +32:22.321 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=123.128157ms client=127.0.0.1:49547 client_network=udp +32:22.502 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=161.855059ms client=127.0.0.1:55677 client_network=udp +32:22.615 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:22.617 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=104.861577ms client=127.0.0.1:54470 client_network=tcp +32:23.240 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:23.242 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=121.704976ms client=127.0.0.1:54522 client_network=tcp +32:23.834 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=157.040166ms client=127.0.0.1:55816 client_network=udp +32:24.002 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=152.619225ms client=127.0.0.1:50831 client_network=udp +32:24.158 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:24.162 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=150.216982ms client=127.0.0.1:54524 client_network=tcp +32:25.034 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:25.037 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=189.72402ms client=127.0.0.1:54534 client_network=tcp +32:25.803 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=162.788064ms client=127.0.0.1:33266 client_network=udp +32:25.890 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=79.383486ms client=127.0.0.1:47975 client_network=udp +32:26.085 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:26.087 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=178.317686ms client=127.0.0.1:54552 client_network=tcp +32:26.780 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:26.782 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=130.941066ms client=127.0.0.1:54556 client_network=tcp +32:27.333 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=117.395961ms client=127.0.0.1:50396 client_network=udp +32:27.427 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=86.327732ms client=127.0.0.1:36130 client_network=udp +32:27.519 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:27.522 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=89.341631ms client=127.0.0.1:54562 client_network=tcp +32:28.338 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:28.342 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=94.642739ms client=127.0.0.1:54580 client_network=tcp +32:29.114 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=118.451143ms client=127.0.0.1:59491 client_network=udp +32:29.195 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=69.909546ms client=127.0.0.1:46525 client_network=udp +32:29.360 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:29.362 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=138.475093ms client=127.0.0.1:54586 client_network=tcp +32:30.133 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:30.135 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=170.20205ms client=127.0.0.1:54588 client_network=tcp +32:30.682 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=164.976963ms client=127.0.0.1:39960 client_network=udp +32:30.897 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=201.486543ms client=127.0.0.1:52555 client_network=udp +32:31.125 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:31.137 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=233.145726ms client=127.0.0.1:54594 client_network=tcp +32:32.202 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:32.206 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=136.870167ms client=127.0.0.1:54600 client_network=tcp +32:33.066 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=171.280599ms client=127.0.0.1:37523 client_network=udp +32:33.202 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=124.700023ms client=127.0.0.1:57220 client_network=udp +32:33.336 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:33.338 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=126.200489ms client=127.0.0.1:54606 client_network=tcp +32:34.060 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:34.062 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=87.661437ms client=127.0.0.1:54612 client_network=tcp +32:34.624 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=91.949239ms client=127.0.0.1:45340 client_network=udp +32:34.708 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=77.660738ms client=127.0.0.1:48218 client_network=udp +32:34.872 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:34.874 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=128.900651ms client=127.0.0.1:54622 client_network=tcp +32:35.723 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:35.726 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=129.855074ms client=127.0.0.1:54632 client_network=tcp +32:36.743 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=203.286658ms client=127.0.0.1:44606 client_network=udp +32:36.857 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=89.301692ms client=127.0.0.1:43739 client_network=udp +32:37.046 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:37.052 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=180.146407ms client=127.0.0.1:54638 client_network=tcp +32:37.592 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{truncated.service.consul. 255 1}"] records=4092/5001 size=65514/80058 +32:37.593 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=94.21182ms client=127.0.0.1:54640 client_network=tcp +32:38.019 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=79.455242ms client=127.0.0.1:45495 client_network=udp +32:38.106 [DEBUG] TestAgent.dns: request served from client: name=truncated.service.consul. type=ANY class=IN latency=80.30551ms client=127.0.0.1:51999 client_network=udp +32:38.252 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:38.264 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=149.211572ms client=127.0.0.1:54652 client_network=tcp +32:39.002 [DEBUG] TestAgent.dns: TCP answer to question too large, truncated: question=["{b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. 255 1}"] records=4091/5001 size=65523/80083 +32:39.010 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=185.798852ms client=127.0.0.1:54654 client_network=tcp +32:39.740 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=108.147394ms client=127.0.0.1:39974 client_network=udp +32:39.923 [DEBUG] TestAgent.dns: request served from client: name=b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul. type=ANY class=IN latency=165.386807ms client=127.0.0.1:37637 client_network=udp +32:39.933 [INFO] TestAgent: Requesting shutdown +32:39.933 [INFO] TestAgent.server: shutting down server +32:39.933 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:39.933 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:39.933 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:39.933 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:39.933 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:39.933 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:39.933 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:39.933 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:39.933 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:39.933 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:39.942 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:39.944Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:39.944Z [INFO] agent.router.manager: shutting down +32:39.944 [INFO] TestAgent: consul server down +32:39.944 [INFO] TestAgent: shutdown complete +32:39.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30087 network=tcp +32:39.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30087 network=udp +32:39.944 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30088 network=tcp +32:40.448 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.448 [INFO] TestAgent: Endpoints down +--- PASS: TestDNS_TCP_and_UDP_Truncate (80.41s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=true (0.01s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=true (0.01s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=false (0.01s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=false (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=true (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=false (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=true (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=false (0.01s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=true (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=true (0.01s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=false (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=true (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=false (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=true (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=false (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=255)_compressed=false#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=255)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=255)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=255)_compressed=false#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=255)_compressed=false#01 (0.01s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=255)_compressed=false#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=1)_compressed=false#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=1)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=1)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=1)_compressed=false#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=1)_compressed=false#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=1)_compressed=false#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._tcp_(qType:=33)_compressed=false#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=true#01 (0.00s) --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_normal.service.consul._udp_(qType:=33)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._tcp_(qType:=33)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=true#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_4989a072-22f1-e705-1464-d0742ba7670b.query.consul._udp_(qType:=33)_compressed=false#01 (0.00s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false (0.18s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true (0.06s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=true (0.25s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=false (0.26s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=true (0.06s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=false (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false (0.20s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false (0.07s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=true (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=false (0.25s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=true (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=false (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true (0.19s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true (0.04s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false (0.04s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=true (0.23s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=false (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=true (0.08s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=false (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true#01 (0.19s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false#01 (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=true#01 (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=255)_compressed=false#01 (0.22s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=true#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=255)_compressed=false#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true#01 (0.21s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false#01 (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true#01 (0.06s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=true#01 (0.23s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=1)_compressed=false#01 (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=true#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=1)_compressed=false#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true#01 (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false#01 (0.17s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false#01 (0.05s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=true#01 (0.23s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._tcp_(qType:=33)_compressed=false#01 (0.24s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=true#01 (0.07s) - --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_641d7046-7fd2-3a31-d838-3d4d9750317b.query.consul._udp_(qType:=33)_compressed=false#01 (0.05s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._tcp_(qType:=33)_compressed=false#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=true#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_526110bb-2f07-1e0f-db4c-a2c0a2adbdd5.query.consul._udp_(qType:=33)_compressed=false#01 (0.00s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true (1.05s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false (0.62s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true (0.14s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false (0.14s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=true (0.79s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=false (0.66s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=true (0.14s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=false (0.17s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true (0.61s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false (0.56s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true (0.17s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false (0.16s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=true (0.82s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=false (0.81s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=true (0.17s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=false (0.10s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true (0.74s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false (0.56s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true (0.13s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false (0.09s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=true (0.81s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=false (0.75s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=true (0.13s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=false (0.10s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=true#01 (0.74s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=255)_compressed=false#01 (0.55s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=true#01 (0.18s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=255)_compressed=false#01 (0.21s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=true#01 (1.17s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=255)_compressed=false#01 (0.83s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=true#01 (0.18s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=255)_compressed=false#01 (0.13s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=true#01 (0.76s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=1)_compressed=false#01 (0.56s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=true#01 (0.10s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=1)_compressed=false#01 (0.12s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=true#01 (0.85s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=1)_compressed=false#01 (0.94s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=true#01 (0.23s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=1)_compressed=false#01 (0.10s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=true#01 (0.63s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._tcp_(qType:=33)_compressed=false#01 (0.44s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=true#01 (0.09s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_truncated.service.consul._udp_(qType:=33)_compressed=false#01 (0.09s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=true#01 (0.71s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._tcp_(qType:=33)_compressed=false#01 (0.81s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=true#01 (0.13s) + --- PASS: TestDNS_TCP_and_UDP_Truncate/lookup_b1b158c9-ee73-abee-dab6-7c15aa0da9f6.query.consul._udp_(qType:=33)_compressed=false#01 (0.18s) PASS -ok github.com/hashicorp/consul/agent 123.969s +ok github.com/hashicorp/consul/agent 186.596s === RUN TestAE_scaleFactor === PAUSE TestAE_scaleFactor === RUN TestAE_Pause_nestedPauseResume @@ -56311,7 +56542,7 @@ === RUN TestAE_FSM/fullSyncState === RUN TestAE_FSM/fullSyncState/Paused_->_retryFullSyncState === RUN TestAE_FSM/fullSyncState/SyncFull()_error_->_retryFullSyncState -2022-11-07T22:44:28.512Z [ERROR] anti_entropy: failed to sync remote state: error=boom +2021-10-05T16:29:18.392Z [ERROR] anti_entropy: failed to sync remote state: error=boom === RUN TestAE_FSM/fullSyncState/SyncFull()_OK_->_partialSyncState === RUN TestAE_FSM/retryFullSyncState === RUN TestAE_FSM/retryFullSyncState/shutdownEvent_->_doneState @@ -56324,11 +56555,11 @@ === RUN TestAE_FSM/partialSyncState/syncFullTimerEvent_->_fullSyncState === RUN TestAE_FSM/partialSyncState/syncChangesEvent+Paused_->_partialSyncState === RUN TestAE_FSM/partialSyncState/syncChangesEvent+SyncChanges()_error_->_partialSyncState -2022-11-07T22:44:28.512Z [ERROR] anti_entropy: failed to sync changes: error=boom +2021-10-05T16:29:18.397Z [ERROR] anti_entropy: failed to sync changes: error=boom === RUN TestAE_FSM/partialSyncState/syncChangesEvent+SyncChanges()_OK_->_partialSyncState === RUN TestAE_FSM/partialSyncState/invalid_event_->_panic_ === RUN TestAE_FSM/invalid_state_->_panic_ ---- PASS: TestAE_FSM (0.00s) +--- PASS: TestAE_FSM (0.01s) --- PASS: TestAE_FSM/fullSyncState (0.00s) --- PASS: TestAE_FSM/fullSyncState/Paused_->_retryFullSyncState (0.00s) --- PASS: TestAE_FSM/fullSyncState/SyncFull()_error_->_retryFullSyncState (0.00s) @@ -56372,67 +56603,67 @@ === CONT TestAE_scaleFactor === RUN TestAE_scaleFactor/100_nodes === RUN TestAE_scaleFactor/200_nodes -=== CONT TestAE_Pause_nestedPauseResume === RUN TestAE_scaleFactor/1000_nodes === RUN TestAE_scaleFactor/10000_nodes ---- PASS: TestAE_Pause_nestedPauseResume (0.00s) --- PASS: TestAE_scaleFactor (0.00s) --- PASS: TestAE_scaleFactor/100_nodes (0.00s) --- PASS: TestAE_scaleFactor/200_nodes (0.00s) --- PASS: TestAE_scaleFactor/1000_nodes (0.00s) --- PASS: TestAE_scaleFactor/10000_nodes (0.00s) +=== CONT TestAE_Pause_nestedPauseResume +--- PASS: TestAE_Pause_nestedPauseResume (0.00s) PASS -ok github.com/hashicorp/consul/agent/ae 2.254s +ok github.com/hashicorp/consul/agent/ae 2.357s === RUN TestNew -=== RUN TestNew/no-direct-rpc -=== RUN TestNew/no-config-loader -=== RUN TestNew/no-cache === RUN TestNew/no-tls-configurator === RUN TestNew/no-tokens === RUN TestNew/ok +=== RUN TestNew/no-direct-rpc +=== RUN TestNew/no-config-loader +=== RUN TestNew/no-cache --- PASS: TestNew (0.00s) - --- PASS: TestNew/no-direct-rpc (0.00s) - --- PASS: TestNew/no-config-loader (0.00s) - --- PASS: TestNew/no-cache (0.00s) --- PASS: TestNew/no-tls-configurator (0.00s) --- PASS: TestNew/no-tokens (0.00s) --- PASS: TestNew/ok (0.00s) + --- PASS: TestNew/no-direct-rpc (0.00s) + --- PASS: TestNew/no-config-loader (0.00s) + --- PASS: TestNew/no-cache (0.00s) === RUN TestReadConfig --- PASS: TestReadConfig (0.00s) === RUN TestInitialConfiguration_disabled ---- PASS: TestInitialConfiguration_disabled (0.01s) +--- PASS: TestInitialConfiguration_disabled (0.03s) === RUN TestInitialConfiguration_cancelled -2022-11-07T22:44:32.754Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_cancelled-auto-config492994209/auto-config.json -2022-11-07T22:44:32.754Z [INFO] TestInitialConfiguration_cancelled.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:32.756Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 -2022-11-07T22:44:32.756Z [ERROR] TestInitialConfiguration_cancelled.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:8300 error="injected error" -2022-11-07T22:44:32.756Z [ERROR] TestInitialConfiguration_cancelled.auto_config: No server successfully responded to the auto-config request -2022-11-07T22:44:32.756Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 -2022-11-07T22:44:32.756Z [ERROR] TestInitialConfiguration_cancelled.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:8300 error="injected error" -2022-11-07T22:44:32.756Z [ERROR] TestInitialConfiguration_cancelled.auto_config: No server successfully responded to the auto-config request -2022-11-07T22:44:32.848Z [INFO] TestInitialConfiguration_cancelled.auto_config: interrupted during initial auto configuration: err="context deadline exceeded" +2021-10-05T16:29:27.889Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_cancelled-auto-config000186405/auto-config.json +2021-10-05T16:29:27.889Z [INFO] TestInitialConfiguration_cancelled.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:27.891Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 +2021-10-05T16:29:27.891Z [ERROR] TestInitialConfiguration_cancelled.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:8300 error="injected error" +2021-10-05T16:29:27.891Z [ERROR] TestInitialConfiguration_cancelled.auto_config: No server successfully responded to the auto-config request +2021-10-05T16:29:27.891Z [DEBUG] TestInitialConfiguration_cancelled.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 +2021-10-05T16:29:27.891Z [ERROR] TestInitialConfiguration_cancelled.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:8300 error="injected error" +2021-10-05T16:29:27.891Z [ERROR] TestInitialConfiguration_cancelled.auto_config: No server successfully responded to the auto-config request +2021-10-05T16:29:27.980Z [INFO] TestInitialConfiguration_cancelled.auto_config: interrupted during initial auto configuration: err="context deadline exceeded" mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,string) mock_test.go:242: PASS: FindLANServer() ---- PASS: TestInitialConfiguration_cancelled (0.10s) +--- PASS: TestInitialConfiguration_cancelled (0.11s) === RUN TestInitialConfiguration_restored -2022-11-07T22:44:32.867Z [DEBUG] TestInitialConfiguration_restored.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_restored-auto-config827132812/auto-config.json -2022-11-07T22:44:32.868Z [INFO] TestInitialConfiguration_restored.auto_config: read persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_restored-auto-config827132812/auto-config.json -2022-11-07T22:44:32.868Z [DEBUG] TestInitialConfiguration_restored.auto_config: updating auto-config settings -2022-11-07T22:44:32.879Z [DEBUG] TestInitialConfiguration_restored.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.046Z [DEBUG] TestInitialConfiguration_restored.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_restored-auto-config886542144/auto-config.json +2021-10-05T16:29:28.053Z [INFO] TestInitialConfiguration_restored.auto_config: read persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_restored-auto-config886542144/auto-config.json +2021-10-05T16:29:28.053Z [DEBUG] TestInitialConfiguration_restored.auto_config: updating auto-config settings +2021-10-05T16:29:28.070Z [DEBUG] TestInitialConfiguration_restored.auto_config: auto-config settings were persisted to disk mock_test.go:243: PASS: Prepopulate(string,cache.FetchResult,string,string,string) mock_test.go:243: PASS: Prepopulate(string,cache.FetchResult,string,string,string) mock_test.go:244: PASS: AgentToken() mock_test.go:244: PASS: UpdateAgentToken(string,token.TokenSource) mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) ---- PASS: TestInitialConfiguration_restored (0.04s) +--- PASS: TestInitialConfiguration_restored (0.09s) === RUN TestInitialConfiguration_success -2022-11-07T22:44:32.899Z [DEBUG] TestInitialConfiguration_success.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_success-auto-config426294651/auto-config.json -2022-11-07T22:44:32.899Z [INFO] TestInitialConfiguration_success.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:32.900Z [DEBUG] TestInitialConfiguration_success.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 -2022-11-07T22:44:32.900Z [DEBUG] TestInitialConfiguration_success.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:32.900Z [DEBUG] TestInitialConfiguration_success.auto_config: updating auto-config settings -2022-11-07T22:44:32.909Z [DEBUG] TestInitialConfiguration_success.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.104Z [DEBUG] TestInitialConfiguration_success.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_success-auto-config331186079/auto-config.json +2021-10-05T16:29:28.104Z [INFO] TestInitialConfiguration_success.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.107Z [DEBUG] TestInitialConfiguration_success.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:8300 +2021-10-05T16:29:28.107Z [DEBUG] TestInitialConfiguration_success.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.107Z [DEBUG] TestInitialConfiguration_success.auto_config: updating auto-config settings +2021-10-05T16:29:28.130Z [DEBUG] TestInitialConfiguration_success.auto_config: auto-config settings were persisted to disk mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Prepopulate(string,cache.FetchResult,string,string,string) @@ -56441,29 +56672,29 @@ mock_test.go:244: PASS: UpdateAgentToken(string,token.TokenSource) mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) ---- PASS: TestInitialConfiguration_success (0.03s) +--- PASS: TestInitialConfiguration_success (0.06s) === RUN TestInitialConfiguration_retries -2022-11-07T22:44:32.929Z [DEBUG] TestInitialConfiguration_retries.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_retries-auto-config649954462/auto-config.json -2022-11-07T22:44:32.929Z [INFO] TestInitialConfiguration_retries.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.1:8300 error="injected failure" -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.2:8398 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.2:8398 error="injected failure" -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.3:8399 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.3:8399 error="injected failure" -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:1234 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:1234 error="injected failure" -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: No server successfully responded to the auto-config request -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.1:8300 error="injected failure" -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.2:8398 -2022-11-07T22:44:32.930Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.2:8398 error="injected failure" -2022-11-07T22:44:32.930Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.3:8399 -2022-11-07T22:44:32.931Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.3:8399 error="injected failure" -2022-11-07T22:44:32.931Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:1234 -2022-11-07T22:44:32.931Z [DEBUG] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:32.931Z [DEBUG] TestInitialConfiguration_retries.auto_config: updating auto-config settings -2022-11-07T22:44:32.939Z [DEBUG] TestInitialConfiguration_retries.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.166Z [DEBUG] TestInitialConfiguration_retries.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestInitialConfiguration_retries-auto-config879675506/auto-config.json +2021-10-05T16:29:28.166Z [INFO] TestInitialConfiguration_retries.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.169Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.169Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.1:8300 error="injected failure" +2021-10-05T16:29:28.169Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.2:8398 +2021-10-05T16:29:28.169Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.2:8398 error="injected failure" +2021-10-05T16:29:28.169Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.3:8399 +2021-10-05T16:29:28.169Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.3:8399 error="injected failure" +2021-10-05T16:29:28.169Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:1234 +2021-10-05T16:29:28.169Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=127.0.0.1:1234 error="injected failure" +2021-10-05T16:29:28.169Z [ERROR] TestInitialConfiguration_retries.auto_config: No server successfully responded to the auto-config request +2021-10-05T16:29:28.170Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.170Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.1:8300 error="injected failure" +2021-10-05T16:29:28.170Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.2:8398 +2021-10-05T16:29:28.170Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.2:8398 error="injected failure" +2021-10-05T16:29:28.170Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.3:8399 +2021-10-05T16:29:28.170Z [ERROR] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC failed: addr=198.18.0.3:8399 error="injected failure" +2021-10-05T16:29:28.170Z [DEBUG] TestInitialConfiguration_retries.auto_config: making AutoConfig.InitialConfiguration RPC: addr=127.0.0.1:1234 +2021-10-05T16:29:28.171Z [DEBUG] TestInitialConfiguration_retries.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.171Z [DEBUG] TestInitialConfiguration_retries.auto_config: updating auto-config settings +2021-10-05T16:29:28.188Z [DEBUG] TestInitialConfiguration_retries.auto_config: auto-config settings were persisted to disk mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) @@ -56476,33 +56707,33 @@ mock_test.go:244: PASS: UpdateAgentToken(string,token.TokenSource) mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) ---- PASS: TestInitialConfiguration_retries (0.03s) +--- PASS: TestInitialConfiguration_retries (0.08s) === RUN TestGoRoutineManagement -2022-11-07T22:44:32.956Z [INFO] TestGoRoutineManagement.auto_config: auto-config started -2022-11-07T22:44:32.967Z [DEBUG] TestGoRoutineManagement.auto_config: stopping auto-config -2022-11-07T22:44:32.967Z [DEBUG] TestGoRoutineManagement.auto_config: auto-config has been stopped -2022-11-07T22:44:32.967Z [INFO] TestGoRoutineManagement.auto_config: auto-config started -2022-11-07T22:44:32.967Z [DEBUG] TestGoRoutineManagement.auto_config: stopping auto-config -2022-11-07T22:44:32.967Z [DEBUG] TestGoRoutineManagement.auto_config: auto-config has been stopped +2021-10-05T16:29:28.241Z [INFO] TestGoRoutineManagement.auto_config: auto-config started +2021-10-05T16:29:28.251Z [DEBUG] TestGoRoutineManagement.auto_config: stopping auto-config +2021-10-05T16:29:28.251Z [DEBUG] TestGoRoutineManagement.auto_config: auto-config has been stopped +2021-10-05T16:29:28.252Z [INFO] TestGoRoutineManagement.auto_config: auto-config started +2021-10-05T16:29:28.252Z [DEBUG] TestGoRoutineManagement.auto_config: stopping auto-config +2021-10-05T16:29:28.252Z [DEBUG] TestGoRoutineManagement.auto_config: auto-config has been stopped mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) mock_test.go:243: PASS: Notify(string,string,*cachetype.ConnectCALeafRequest,string,string) mock_test.go:244: PASS: AgentToken() mock_test.go:244: PASS: Notify(token.TokenKind) mock_test.go:244: PASS: StopNotify(token.Notifier) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestGoRoutineManagement (0.02s) +--- PASS: TestGoRoutineManagement (0.03s) === RUN TestTokenUpdate -2022-11-07T22:44:32.982Z [DEBUG] TestTokenUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestTokenUpdate-auto-config117349760/auto-config.json -2022-11-07T22:44:32.982Z [INFO] TestTokenUpdate.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:32.984Z [DEBUG] TestTokenUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:32.984Z [DEBUG] TestTokenUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:32.984Z [DEBUG] TestTokenUpdate.auto_config: updating auto-config settings -2022-11-07T22:44:32.992Z [DEBUG] TestTokenUpdate.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.003Z [INFO] TestTokenUpdate.auto_config: auto-config started -2022-11-07T22:44:33.003Z [DEBUG] TestTokenUpdate.auto_config: handling a token update event -2022-11-07T22:44:33.003Z [DEBUG] TestTokenUpdate.auto_config: Agent token updated - resetting watches -2022-11-07T22:44:33.003Z [DEBUG] TestTokenUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.003Z [DEBUG] TestTokenUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.285Z [DEBUG] TestTokenUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestTokenUpdate-auto-config412817012/auto-config.json +2021-10-05T16:29:28.285Z [INFO] TestTokenUpdate.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.287Z [DEBUG] TestTokenUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.287Z [DEBUG] TestTokenUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.287Z [DEBUG] TestTokenUpdate.auto_config: updating auto-config settings +2021-10-05T16:29:28.307Z [DEBUG] TestTokenUpdate.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.326Z [INFO] TestTokenUpdate.auto_config: auto-config started +2021-10-05T16:29:28.326Z [DEBUG] TestTokenUpdate.auto_config: handling a token update event +2021-10-05T16:29:28.326Z [DEBUG] TestTokenUpdate.auto_config: Agent token updated - resetting watches +2021-10-05T16:29:28.327Z [DEBUG] TestTokenUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.327Z [DEBUG] TestTokenUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56519,24 +56750,22 @@ mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestTokenUpdate (0.04s) +--- PASS: TestTokenUpdate (0.08s) === RUN TestRootsUpdate -2022-11-07T22:44:33.025Z [DEBUG] TestRootsUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config179547359/auto-config.json -2022-11-07T22:44:33.025Z [INFO] TestRootsUpdate.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:33.028Z [DEBUG] TestRootsUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.028Z [DEBUG] TestRootsUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:33.028Z [DEBUG] TestRootsUpdate.auto_config: updating auto-config settings -2022-11-07T22:44:33.040Z [DEBUG] TestRootsUpdate.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.047Z [INFO] TestRootsUpdate.auto_config: auto-config started -2022-11-07T22:44:33.049Z [DEBUG] TestRootsUpdate.auto_config: handling a cache update event: correlation_id=roots -2022-11-07T22:44:33.049Z [DEBUG] TestRootsUpdate.auto_config: roots watch fired - updating CA certificates -2022-11-07T22:44:33.050Z [DEBUG] TestRootsUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config179547359/auto-config.json -2022-11-07T22:44:33.052Z [INFO] TestRootsUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config179547359/auto-config.json -2022-11-07T22:44:33.053Z [DEBUG] TestRootsUpdate.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.078Z [DEBUG] TestRootsUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config179547359/auto-config.json -2022-11-07T22:44:33.079Z [INFO] TestRootsUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config179547359/auto-config.json -2022-11-07T22:44:33.079Z [DEBUG] TestRootsUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.079Z [DEBUG] TestRootsUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.347Z [DEBUG] TestRootsUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config231585603/auto-config.json +2021-10-05T16:29:28.347Z [INFO] TestRootsUpdate.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.349Z [DEBUG] TestRootsUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.350Z [DEBUG] TestRootsUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.350Z [DEBUG] TestRootsUpdate.auto_config: updating auto-config settings +2021-10-05T16:29:28.370Z [DEBUG] TestRootsUpdate.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.380Z [INFO] TestRootsUpdate.auto_config: auto-config started +2021-10-05T16:29:28.385Z [DEBUG] TestRootsUpdate.auto_config: handling a cache update event: correlation_id=roots +2021-10-05T16:29:28.385Z [DEBUG] TestRootsUpdate.auto_config: roots watch fired - updating CA certificates +2021-10-05T16:29:28.387Z [DEBUG] TestRootsUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config231585603/auto-config.json +2021-10-05T16:29:28.387Z [DEBUG] TestRootsUpdate.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.390Z [INFO] TestRootsUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestRootsUpdate-auto-config231585603/auto-config.json +2021-10-05T16:29:28.390Z [DEBUG] TestRootsUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.390Z [DEBUG] TestRootsUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56552,24 +56781,22 @@ mock_test.go:245: PASS: AutoEncryptCertNotAfter() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestRootsUpdate (0.08s) +--- PASS: TestRootsUpdate (0.06s) === RUN TestCertUpdate -2022-11-07T22:44:33.092Z [DEBUG] TestCertUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config911819442/auto-config.json -2022-11-07T22:44:33.092Z [INFO] TestCertUpdate.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:33.093Z [DEBUG] TestCertUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.094Z [DEBUG] TestCertUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:33.094Z [DEBUG] TestCertUpdate.auto_config: updating auto-config settings -2022-11-07T22:44:33.103Z [DEBUG] TestCertUpdate.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.110Z [INFO] TestCertUpdate.auto_config: auto-config started -2022-11-07T22:44:33.112Z [DEBUG] TestCertUpdate.auto_config: handling a cache update event: correlation_id=leaf -2022-11-07T22:44:33.112Z [DEBUG] TestCertUpdate.auto_config: leaf certificate watch fired - updating TLS certificate -2022-11-07T22:44:33.112Z [DEBUG] TestCertUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config911819442/auto-config.json -2022-11-07T22:44:33.114Z [DEBUG] TestCertUpdate.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.115Z [INFO] TestCertUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config911819442/auto-config.json -2022-11-07T22:44:33.140Z [DEBUG] TestCertUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config911819442/auto-config.json -2022-11-07T22:44:33.142Z [INFO] TestCertUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config911819442/auto-config.json -2022-11-07T22:44:33.142Z [DEBUG] TestCertUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.142Z [DEBUG] TestCertUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.415Z [DEBUG] TestCertUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config343521734/auto-config.json +2021-10-05T16:29:28.415Z [INFO] TestCertUpdate.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.417Z [DEBUG] TestCertUpdate.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.418Z [DEBUG] TestCertUpdate.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.418Z [DEBUG] TestCertUpdate.auto_config: updating auto-config settings +2021-10-05T16:29:28.430Z [DEBUG] TestCertUpdate.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.445Z [INFO] TestCertUpdate.auto_config: auto-config started +2021-10-05T16:29:28.449Z [DEBUG] TestCertUpdate.auto_config: handling a cache update event: correlation_id=leaf +2021-10-05T16:29:28.449Z [DEBUG] TestCertUpdate.auto_config: leaf certificate watch fired - updating TLS certificate +2021-10-05T16:29:28.451Z [DEBUG] TestCertUpdate.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.451Z [DEBUG] TestCertUpdate.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config343521734/auto-config.json +2021-10-05T16:29:28.452Z [INFO] TestCertUpdate.auto_config: read persisted configuration: path=/tmp/consul-test/TestCertUpdate-auto-config343521734/auto-config.json +2021-10-05T16:29:28.452Z [DEBUG] TestCertUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.452Z [DEBUG] TestCertUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56587,26 +56814,26 @@ mock_test.go:245: PASS: AutoEncryptCertNotAfter() --- PASS: TestCertUpdate (0.06s) === RUN TestFallback -2022-11-07T22:44:33.155Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config986474601/auto-config.json -2022-11-07T22:44:33.155Z [INFO] TestFallback.auto_config: retrieving initial agent auto configuration remotely -2022-11-07T22:44:33.156Z [DEBUG] TestFallback.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.156Z [DEBUG] TestFallback.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:33.156Z [DEBUG] TestFallback.auto_config: updating auto-config settings -2022-11-07T22:44:33.164Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.171Z [INFO] TestFallback.auto_config: auto-config started -2022-11-07T22:44:33.177Z [DEBUG] TestFallback.auto_config: handling a cache update event: correlation_id=leaf -2022-11-07T22:44:33.177Z [DEBUG] TestFallback.auto_config: leaf certificate watch fired - updating TLS certificate -2022-11-07T22:44:33.178Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.178Z [WARN] TestFallback.auto_config: agent's client certificate has expired -2022-11-07T22:44:33.180Z [DEBUG] TestFallback.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.23.2:8300 -2022-11-07T22:44:33.180Z [DEBUG] TestFallback.auto_config: AutoConfig.InitialConfiguration RPC was successful -2022-11-07T22:44:33.181Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config986474601/auto-config.json -2022-11-07T22:44:33.181Z [INFO] TestFallback.auto_config: read persisted configuration: path=/tmp/consul-test/TestFallback-auto-config986474601/auto-config.json -2022-11-07T22:44:33.188Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk -2022-11-07T22:44:33.207Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config986474601/auto-config.json -2022-11-07T22:44:33.209Z [INFO] TestFallback.auto_config: read persisted configuration: path=/tmp/consul-test/TestFallback-auto-config986474601/auto-config.json -2022-11-07T22:44:33.211Z [DEBUG] TestFallback.auto_config: stopping auto-config -2022-11-07T22:44:33.212Z [DEBUG] TestFallback.auto_config: auto-config has been stopped +2021-10-05T16:29:28.471Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config942139501/auto-config.json +2021-10-05T16:29:28.471Z [INFO] TestFallback.auto_config: retrieving initial agent auto configuration remotely +2021-10-05T16:29:28.474Z [DEBUG] TestFallback.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.474Z [DEBUG] TestFallback.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.474Z [DEBUG] TestFallback.auto_config: updating auto-config settings +2021-10-05T16:29:28.522Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.533Z [INFO] TestFallback.auto_config: auto-config started +2021-10-05T16:29:28.548Z [DEBUG] TestFallback.auto_config: handling a cache update event: correlation_id=leaf +2021-10-05T16:29:28.548Z [DEBUG] TestFallback.auto_config: leaf certificate watch fired - updating TLS certificate +2021-10-05T16:29:28.550Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.550Z [WARN] TestFallback.auto_config: agent's client certificate has expired +2021-10-05T16:29:28.556Z [DEBUG] TestFallback.auto_config: making AutoConfig.InitialConfiguration RPC: addr=198.18.23.2:8300 +2021-10-05T16:29:28.558Z [DEBUG] TestFallback.auto_config: AutoConfig.InitialConfiguration RPC was successful +2021-10-05T16:29:28.566Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config942139501/auto-config.json +2021-10-05T16:29:28.568Z [INFO] TestFallback.auto_config: read persisted configuration: path=/tmp/consul-test/TestFallback-auto-config942139501/auto-config.json +2021-10-05T16:29:28.582Z [DEBUG] TestFallback.auto_config: auto-config settings were persisted to disk +2021-10-05T16:29:28.600Z [DEBUG] TestFallback.auto_config: attempting to restore any persisted configuration: path=/tmp/consul-test/TestFallback-auto-config942139501/auto-config.json +2021-10-05T16:29:28.602Z [INFO] TestFallback.auto_config: read persisted configuration: path=/tmp/consul-test/TestFallback-auto-config942139501/auto-config.json +2021-10-05T16:29:28.602Z [DEBUG] TestFallback.auto_config: stopping auto-config +2021-10-05T16:29:28.602Z [DEBUG] TestFallback.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*pbautoconf.AutoConfigRequest,*pbautoconf.AutoConfigResponse) mock_test.go:242: PASS: FindLANServer() @@ -56631,7 +56858,7 @@ mock_test.go:245: PASS: AutoEncryptCertExpired() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestFallback (0.07s) +--- PASS: TestFallback (0.15s) === RUN TestIntroToken === RUN TestIntroToken/config === RUN TestIntroToken/file @@ -56643,25 +56870,25 @@ === RUN TestAutoEncrypt_generateCSR === RUN TestAutoEncrypt_generateCSR/ip-sans === RUN TestAutoEncrypt_generateCSR/dns-sans ---- PASS: TestAutoEncrypt_generateCSR (0.00s) +--- PASS: TestAutoEncrypt_generateCSR (0.01s) --- PASS: TestAutoEncrypt_generateCSR/ip-sans (0.00s) --- PASS: TestAutoEncrypt_generateCSR/dns-sans (0.00s) === RUN TestAutoEncrypt_hosts === RUN TestAutoEncrypt_hosts/router-override === RUN TestAutoEncrypt_hosts/various-addresses === RUN TestAutoEncrypt_hosts/split-host-port-error -2022-11-07T22:44:33.218Z [WARN] TestAutoEncrypt_hosts/split-host-port-error: error splitting host address into IP and port: address=this-is-not:a:ip:and_port error="address this-is-not:a:ip:and_port: too many colons in address" +2021-10-05T16:29:28.611Z [WARN] TestAutoEncrypt_hosts/split-host-port-error: error splitting host address into IP and port: address=this-is-not:a:ip:and_port error="address this-is-not:a:ip:and_port: too many colons in address" --- PASS: TestAutoEncrypt_hosts (0.00s) --- PASS: TestAutoEncrypt_hosts/router-override (0.00s) --- PASS: TestAutoEncrypt_hosts/various-addresses (0.00s) --- PASS: TestAutoEncrypt_hosts/split-host-port-error (0.00s) === RUN TestAutoEncrypt_InitialCerts -2022-11-07T22:44:33.223Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.223Z [ERROR] TestAutoEncrypt_InitialCerts: AutoEncrypt.Sign RPC failed: addr=198.18.0.1:8300 error="injected error" -2022-11-07T22:44:33.223Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.2:8300 -2022-11-07T22:44:33.223Z [ERROR] TestAutoEncrypt_InitialCerts: AutoEncrypt.Sign RPC failed: addr=198.18.0.2:8300 error="injected error" -2022-11-07T22:44:33.223Z [ERROR] TestAutoEncrypt_InitialCerts: No servers successfully responded to the auto-encrypt request -2022-11-07T22:44:33.223Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.619Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.619Z [ERROR] TestAutoEncrypt_InitialCerts: AutoEncrypt.Sign RPC failed: addr=198.18.0.1:8300 error="injected error" +2021-10-05T16:29:28.619Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.2:8300 +2021-10-05T16:29:28.619Z [ERROR] TestAutoEncrypt_InitialCerts: AutoEncrypt.Sign RPC failed: addr=198.18.0.2:8300 error="injected error" +2021-10-05T16:29:28.619Z [ERROR] TestAutoEncrypt_InitialCerts: No servers successfully responded to the auto-encrypt request +2021-10-05T16:29:28.619Z [DEBUG] TestAutoEncrypt_InitialCerts: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) @@ -56669,8 +56896,8 @@ mock_test.go:244: PASS: AgentToken() --- PASS: TestAutoEncrypt_InitialCerts (0.01s) === RUN TestAutoEncrypt_InitialConfiguration -2022-11-07T22:44:33.235Z [DEBUG] TestAutoEncrypt_InitialConfiguration.auto_config: making AutoEncrypt.Sign RPC: addr=127.0.0.1:8300 -2022-11-07T22:44:33.236Z [INFO] TestAutoEncrypt_InitialConfiguration.auto_config: automatically upgraded to TLS +2021-10-05T16:29:28.651Z [DEBUG] TestAutoEncrypt_InitialConfiguration.auto_config: making AutoEncrypt.Sign RPC: addr=127.0.0.1:8300 +2021-10-05T16:29:28.652Z [INFO] TestAutoEncrypt_InitialConfiguration.auto_config: automatically upgraded to TLS mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Prepopulate(string,cache.FetchResult,string,string,string) @@ -56678,15 +56905,15 @@ mock_test.go:244: PASS: AgentToken() mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) ---- PASS: TestAutoEncrypt_InitialConfiguration (0.01s) +--- PASS: TestAutoEncrypt_InitialConfiguration (0.03s) === RUN TestAutoEncrypt_TokenUpdate -2022-11-07T22:44:33.258Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.258Z [INFO] TestAutoEncrypt_TokenUpdate.auto_config: automatically upgraded to TLS -2022-11-07T22:44:33.259Z [INFO] TestAutoEncrypt_TokenUpdate.auto_config: auto-config started -2022-11-07T22:44:33.259Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: handling a token update event -2022-11-07T22:44:33.259Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: Agent token updated - resetting watches -2022-11-07T22:44:33.259Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.260Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.676Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.676Z [INFO] TestAutoEncrypt_TokenUpdate.auto_config: automatically upgraded to TLS +2021-10-05T16:29:28.677Z [INFO] TestAutoEncrypt_TokenUpdate.auto_config: auto-config started +2021-10-05T16:29:28.677Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: handling a token update event +2021-10-05T16:29:28.677Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: Agent token updated - resetting watches +2021-10-05T16:29:28.677Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.677Z [DEBUG] TestAutoEncrypt_TokenUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56703,15 +56930,15 @@ mock_test.go:244: PASS: AgentToken() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestAutoEncrypt_TokenUpdate (0.02s) +--- PASS: TestAutoEncrypt_TokenUpdate (0.03s) === RUN TestAutoEncrypt_RootsUpdate -2022-11-07T22:44:33.273Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.274Z [INFO] TestAutoEncrypt_RootsUpdate.auto_config: automatically upgraded to TLS -2022-11-07T22:44:33.274Z [INFO] TestAutoEncrypt_RootsUpdate.auto_config: auto-config started -2022-11-07T22:44:33.277Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: handling a cache update event: correlation_id=roots -2022-11-07T22:44:33.277Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: roots watch fired - updating CA certificates -2022-11-07T22:44:33.277Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.277Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.697Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.698Z [INFO] TestAutoEncrypt_RootsUpdate.auto_config: automatically upgraded to TLS +2021-10-05T16:29:28.698Z [INFO] TestAutoEncrypt_RootsUpdate.auto_config: auto-config started +2021-10-05T16:29:28.711Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: handling a cache update event: correlation_id=roots +2021-10-05T16:29:28.711Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: roots watch fired - updating CA certificates +2021-10-05T16:29:28.712Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.716Z [DEBUG] TestAutoEncrypt_RootsUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56727,15 +56954,15 @@ mock_test.go:245: PASS: AutoEncryptCertNotAfter() mock_test.go:245: PASS: UpdateAutoTLSCA([]string) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestAutoEncrypt_RootsUpdate (0.02s) +--- PASS: TestAutoEncrypt_RootsUpdate (0.04s) === RUN TestAutoEncrypt_CertUpdate -2022-11-07T22:44:33.289Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.290Z [INFO] TestAutoEncrypt_CertUpdate.auto_config: automatically upgraded to TLS -2022-11-07T22:44:33.290Z [INFO] TestAutoEncrypt_CertUpdate.auto_config: auto-config started -2022-11-07T22:44:33.294Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: handling a cache update event: correlation_id=leaf -2022-11-07T22:44:33.294Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: leaf certificate watch fired - updating TLS certificate -2022-11-07T22:44:33.294Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: stopping auto-config -2022-11-07T22:44:33.294Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: auto-config has been stopped +2021-10-05T16:29:28.744Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.745Z [INFO] TestAutoEncrypt_CertUpdate.auto_config: automatically upgraded to TLS +2021-10-05T16:29:28.745Z [INFO] TestAutoEncrypt_CertUpdate.auto_config: auto-config started +2021-10-05T16:29:28.750Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: handling a cache update event: correlation_id=leaf +2021-10-05T16:29:28.750Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: leaf certificate watch fired - updating TLS certificate +2021-10-05T16:29:28.750Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: stopping auto-config +2021-10-05T16:29:28.750Z [DEBUG] TestAutoEncrypt_CertUpdate.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:242: PASS: FindLANServer() mock_test.go:243: PASS: Notify(string,string,*structs.DCSpecificRequest,string,string) @@ -56751,17 +56978,17 @@ mock_test.go:245: PASS: AutoEncryptCertNotAfter() mock_test.go:245: PASS: UpdateAutoTLSCert(string,string) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestAutoEncrypt_CertUpdate (0.02s) +--- PASS: TestAutoEncrypt_CertUpdate (0.03s) === RUN TestAutoEncrypt_Fallback -2022-11-07T22:44:33.306Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 -2022-11-07T22:44:33.306Z [INFO] TestAutoEncrypt_Fallback.auto_config: automatically upgraded to TLS -2022-11-07T22:44:33.307Z [INFO] TestAutoEncrypt_Fallback.auto_config: auto-config started -2022-11-07T22:44:33.313Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: handling a cache update event: correlation_id=leaf -2022-11-07T22:44:33.313Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: leaf certificate watch fired - updating TLS certificate -2022-11-07T22:44:33.313Z [WARN] TestAutoEncrypt_Fallback.auto_config: agent's client certificate has expired -2022-11-07T22:44:33.314Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.23.2:8300 -2022-11-07T22:44:33.315Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: stopping auto-config -2022-11-07T22:44:33.315Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: auto-config has been stopped +2021-10-05T16:29:28.787Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.0.1:8300 +2021-10-05T16:29:28.789Z [INFO] TestAutoEncrypt_Fallback.auto_config: automatically upgraded to TLS +2021-10-05T16:29:28.789Z [INFO] TestAutoEncrypt_Fallback.auto_config: auto-config started +2021-10-05T16:29:28.801Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: handling a cache update event: correlation_id=leaf +2021-10-05T16:29:28.802Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: leaf certificate watch fired - updating TLS certificate +2021-10-05T16:29:28.802Z [WARN] TestAutoEncrypt_Fallback.auto_config: agent's client certificate has expired +2021-10-05T16:29:28.804Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: making AutoEncrypt.Sign RPC: addr=198.18.23.2:8300 +2021-10-05T16:29:28.805Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: stopping auto-config +2021-10-05T16:29:28.805Z [DEBUG] TestAutoEncrypt_Fallback.auto_config: auto-config has been stopped mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:241: PASS: RPC(string,string,*net.TCPAddr,string,*structs.CASignRequest,*structs.SignedResponse) mock_test.go:242: PASS: FindLANServer() @@ -56786,37 +57013,37 @@ mock_test.go:245: PASS: AutoEncryptCertExpired() mock_test.go:245: PASS: UpdateAutoTLS([]string,[]string,string,string,bool) mock_test.go:245: PASS: AutoEncryptCertNotAfter() ---- PASS: TestAutoEncrypt_Fallback (0.02s) +--- PASS: TestAutoEncrypt_Fallback (0.06s) === RUN TestTranslateConfig --- PASS: TestTranslateConfig (0.00s) === RUN TestCArootsTranslation ---- PASS: TestCArootsTranslation (0.00s) +--- PASS: TestCArootsTranslation (0.01s) PASS -ok github.com/hashicorp/consul/agent/auto-config 0.598s +ok github.com/hashicorp/consul/agent/auto-config 1.044s === RUN TestBuildAndValidate_HTTPMaxConnsPerClientExceedsRLimit === PAUSE TestBuildAndValidate_HTTPMaxConnsPerClientExceedsRLimit === RUN TestBuilder_validateEnterpriseConfigKeys +=== RUN TestBuilder_validateEnterpriseConfigKeys/non_voting_server +=== RUN TestBuilder_validateEnterpriseConfigKeys/segments === RUN TestBuilder_validateEnterpriseConfigKeys/autopilot.redundancy_zone_tag === RUN TestBuilder_validateEnterpriseConfigKeys/autopilot.upgrade_version_tag +=== RUN TestBuilder_validateEnterpriseConfigKeys/dns_config.prefer_namespace === RUN TestBuilder_validateEnterpriseConfigKeys/acl.msp_disable_bootstrap -=== RUN TestBuilder_validateEnterpriseConfigKeys/non_voting_server === RUN TestBuilder_validateEnterpriseConfigKeys/segment -=== RUN TestBuilder_validateEnterpriseConfigKeys/segments -=== RUN TestBuilder_validateEnterpriseConfigKeys/multi === RUN TestBuilder_validateEnterpriseConfigKeys/autopilot.disable_upgrade_migration -=== RUN TestBuilder_validateEnterpriseConfigKeys/dns_config.prefer_namespace === RUN TestBuilder_validateEnterpriseConfigKeys/acl.tokens.managed_service_provider +=== RUN TestBuilder_validateEnterpriseConfigKeys/multi --- PASS: TestBuilder_validateEnterpriseConfigKeys (0.00s) + --- PASS: TestBuilder_validateEnterpriseConfigKeys/non_voting_server (0.00s) + --- PASS: TestBuilder_validateEnterpriseConfigKeys/segments (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/autopilot.redundancy_zone_tag (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/autopilot.upgrade_version_tag (0.00s) + --- PASS: TestBuilder_validateEnterpriseConfigKeys/dns_config.prefer_namespace (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/acl.msp_disable_bootstrap (0.00s) - --- PASS: TestBuilder_validateEnterpriseConfigKeys/non_voting_server (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/segment (0.00s) - --- PASS: TestBuilder_validateEnterpriseConfigKeys/segments (0.00s) - --- PASS: TestBuilder_validateEnterpriseConfigKeys/multi (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/autopilot.disable_upgrade_migration (0.00s) - --- PASS: TestBuilder_validateEnterpriseConfigKeys/dns_config.prefer_namespace (0.00s) --- PASS: TestBuilder_validateEnterpriseConfigKeys/acl.tokens.managed_service_provider (0.00s) + --- PASS: TestBuilder_validateEnterpriseConfigKeys/multi (0.00s) === RUN TestLoad --- PASS: TestLoad (0.02s) === RUN TestShouldParseFile @@ -56839,9 +57066,9 @@ === RUN TestBuilder_BuildAndValidate_NodeName/invalid_character_-_unicode === RUN TestBuilder_BuildAndValidate_NodeName/invalid_character_-_slash === RUN TestBuilder_BuildAndValidate_NodeName/too_long ---- PASS: TestBuilder_BuildAndValidate_NodeName (0.02s) +--- PASS: TestBuilder_BuildAndValidate_NodeName (0.04s) --- PASS: TestBuilder_BuildAndValidate_NodeName/invalid_character_-_unicode (0.01s) - --- PASS: TestBuilder_BuildAndValidate_NodeName/invalid_character_-_slash (0.01s) + --- PASS: TestBuilder_BuildAndValidate_NodeName/invalid_character_-_slash (0.02s) --- PASS: TestBuilder_BuildAndValidate_NodeName/too_long (0.01s) === RUN TestAddFlags_WithParse === RUN TestAddFlags_WithParse/#00 @@ -56883,8 +57110,8 @@ --- PASS: TestAddFlags_WithParse/-primary-gateway_foo.local_-primary-gateway_bar.local (0.00s) === RUN TestMerge === RUN TestMerge/top_level_fields ---- PASS: TestMerge (0.00s) - --- PASS: TestMerge/top_level_fields (0.00s) +--- PASS: TestMerge (0.01s) + --- PASS: TestMerge/top_level_fields (0.01s) === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise-wan @@ -57222,343 +57449,343 @@ === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_claim_assertion === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_ok === RUN TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_ok ---- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases (4.44s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise_and_-advertise-wan (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-bind (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-bootstrap (0.01s) +--- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases (8.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise-wan (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-advertise_and_-advertise-wan (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-bind (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-bootstrap (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-bootstrap-expect (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-client (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-dir (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-file_json (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-file_hcl_and_json (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-client (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-dir (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-file_json (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-file_hcl_and_json (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-data-dir_empty (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-data-dir_non-directory (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-datacenter (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-datacenter_empty (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-datacenter (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-datacenter_empty (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-dev (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-disable-host-node-id (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-disable-keyring-file (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-dns-port (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-domain (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can't_be_prefixed_by_DC (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-disable-host-node-id (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-disable-keyring-file (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-dns-port (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-domain (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can't_be_prefixed_by_DC (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can't_be_prefixed_by_service (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can_be_prefixed_by_non-keywords (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can_be_prefixed_by_non-keywords (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-alt-domain_can't_be_prefixed_by_DC#01 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-enable-script-checks (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-encrypt (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-format_disabled,_skip_unknown_files (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-enable-script-checks (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-encrypt (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-format_disabled,_skip_unknown_files (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-format=json (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-format=hcl (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-config-format=hcl (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-http-port (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-join (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-join (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-join-wan (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-log-level (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-log-level (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-log-json (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:-log-rotate-max-files (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:-log-rotate-max-files (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node-id (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node-meta (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:-log-rotate-max-files (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node-id (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-node-meta (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-non-voting-server (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-pid-file (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-pid-file (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:-primary-gateway (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:-primary-gateway (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-protocol (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-protocol (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-raft-protocol (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-recursor (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-rejoin (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-interval (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-interval-wan (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-recursor (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-rejoin (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-interval (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-interval-wan (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-join (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-join-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-max (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-max-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-lan-bind (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-lan-port (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-wan-bind (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-wan-port (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-server (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-server-port (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-join-wan (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-max (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-retry-max-wan (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-lan-bind (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-lan-port (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-wan-bind (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-serf-wan-port (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-server (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-server-port (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-syslog (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-ui (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-ui (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-ui-dir (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-ui-content-path (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_v4 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_v4 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_v6 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_v6 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_and_advertise_set_should_not_detect (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_and_advertise_set_should_not_detect (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_==_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_==_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_<_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_<_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_>_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_>_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports_==_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports_==_0 (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports_<_0 (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports_<_0 (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_template_and_ports (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_template_and_ports (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client,_address_template_and_ports (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client,_address_template_and_ports (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_lan_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_lan_template (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_wan_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_wan_template (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/-ui-content-path (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_v4 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_v4 (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_v6 (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_v6 (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_any_and_advertise_set_should_not_detect (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_any_and_advertise_set_should_not_detect (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_==_0 (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_==_0 (0.06s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_<_0 (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_<_0 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr_and_ports_>_0 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr_and_ports_>_0 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports_==_0 (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports_==_0 (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports_<_0 (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports_<_0 (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_addr,_addresses_and_ports (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_addr,_addresses_and_ports (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_template_and_ports (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_template_and_ports (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client,_address_template_and_ports (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client,_address_template_and_ports (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_lan_template (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_lan_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_wan_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_wan_template (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_lan_with_ports (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_lan_with_ports (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_wan_with_ports (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_wan_with_ports (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:allow_disabling_serf_wan_port (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_lan_with_ports (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_wan_with_ports (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_wan_with_ports (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:allow_disabling_serf_wan_port (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:allow_disabling_serf_wan_port (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:serf_bind_address_lan_template (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:serf_bind_address_lan_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:serf_bind_address_wan_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:serf_bind_address_wan_template (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:serf_bind_address_lan_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:serf_bind_address_lan_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:serf_bind_address_wan_template (0.06s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:serf_bind_address_wan_template (0.04s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_recursor_templates_with_deduplication (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_recursor_templates_with_deduplication (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:start_join_address_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:start_join_address_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:start_join_wan_address_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:start_join_wan_address_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:retry_join_address_template (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:retry_join_address_template (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_recursor_templates_with_deduplication (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:start_join_address_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:start_join_address_template (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:start_join_wan_address_template (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:start_join_wan_address_template (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:retry_join_address_template (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:retry_join_address_template (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:retry_join_wan_address_template (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:retry_join_wan_address_template (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:min/max_ports_for_dynamic_exposed_listeners (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:min/max_ports_for_dynamic_exposed_listeners (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/defaults_for_dynamic_exposed_listeners (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:retry_join_wan_address_template (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:min/max_ports_for_dynamic_exposed_listeners (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:min/max_ports_for_dynamic_exposed_listeners (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/defaults_for_dynamic_exposed_listeners (0.05s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:precedence:_merge_order (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:precedence:_merge_order (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:precedence:_flag_before_file (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:precedence:_flag_before_file (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:raft_performance_scaling (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:raft_performance_scaling (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/Serf_Allowed_CIDRS_LAN,_multiple_values_from_flags (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Serf_Allowed_CIDRS_LAN/WAN,_multiple_values_from_HCL/JSON (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Serf_Allowed_CIDRS_LAN/WAN,_multiple_values_from_HCL/JSON (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/Serf_Allowed_CIDRS_WAN,_multiple_values_from_flags (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:invalid_input (0.00s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:precedence:_merge_order (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:precedence:_flag_before_file (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:precedence:_flag_before_file (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:raft_performance_scaling (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:raft_performance_scaling (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/Serf_Allowed_CIDRS_LAN,_multiple_values_from_flags (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Serf_Allowed_CIDRS_LAN/WAN,_multiple_values_from_HCL/JSON (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Serf_Allowed_CIDRS_LAN/WAN,_multiple_values_from_HCL/JSON (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/Serf_Allowed_CIDRS_WAN,_multiple_values_from_flags (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:invalid_input (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:invalid_input (0.00s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:datacenter_is_lower-cased (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:datacenter_is_lower-cased (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_datacenter_is_lower-cased (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_datacenter_is_lower-cased (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_replication_token_enables_acl_replication (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_replication_token_enables_acl_replication (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_enforce_version_8_is_deprecated (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:datacenter_is_lower-cased (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:datacenter_is_lower-cased (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_datacenter_is_lower-cased (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_datacenter_is_lower-cased (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_replication_token_enables_acl_replication (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_replication_token_enables_acl_replication (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_enforce_version_8_is_deprecated (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_enforce_version_8_is_deprecated (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_fails_v4 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_fails_v4 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_none_v4 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_none_v4 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_multiple_v4 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_multiple_v4 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_multiple_v4 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_multiple_v4 (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_fails_v6 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_fails_v6 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_none_v6 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_none_v6 (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_none_v6 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_address_detect_multiple_v6 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_multiple_v6 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_address_detect_multiple_v6 (0.04s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/ae_interval_invalid_==_0 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/ae_interval_invalid_<_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_datacenter_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_datacenter_invalid (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:acl_datacenter_invalid (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:acl_datacenter_invalid (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:autopilot.max_trailing_logs_invalid (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:autopilot.max_trailing_logs_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_cannot_be_empty (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_cannot_be_empty (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_cannot_be_empty (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_cannot_be_empty (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_does_not_allow_multiple_addresses (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_does_not_allow_multiple_addresses (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_cannot_be_a_unix_socket (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_cannot_be_a_unix_socket (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bind_addr_cannot_be_a_unix_socket (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bind_addr_cannot_be_a_unix_socket (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap_without_server (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap_without_server (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_without_server (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap_without_server (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_without_server (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_without_server (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_and_dev_mode (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_invalid (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_and_dev_mode (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_and_dev_mode (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_and_bootstrap (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_and_bootstrap (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect=1_equals_bootstrap (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect=1_equals_bootstrap (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect=2_warning (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect=2_warning (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_>_2_but_even_warning (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_and_bootstrap (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect=1_equals_bootstrap (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect=1_equals_bootstrap (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect=2_warning (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect=2_warning (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:bootstrap-expect_>_2_but_even_warning (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:bootstrap-expect_>_2_but_even_warning (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_mode_sets_LeaveOnTerm_and_SkipLeaveOnInt_correctly (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_mode_sets_LeaveOnTerm_and_SkipLeaveOnInt_correctly (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_mode_sets_LeaveOnTerm_and_SkipLeaveOnInt_correctly (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_mode_sets_LeaveOnTerm_and_SkipLeaveOnInt_correctly (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:client_does_not_allow_socket (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_does_not_allow_socket (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:client_does_not_allow_socket (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:datacenter_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:datacenter_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_does_not_allow_socket (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_does_not_allow_socket (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ui_and_ui_dir (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ui_and_ui_dir (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_addr_any (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_addr_any (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:datacenter_invalid (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_does_not_allow_socket (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_does_not_allow_socket (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ui_and_ui_dir (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ui_and_ui_dir (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_addr_any (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_addr_any (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:advertise_addr_wan_any (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_addr_wan_any (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:recursors_any (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:recursors_any (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_config.udp_answer_limit_invalid (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:advertise_addr_wan_any (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:recursors_any (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:recursors_any (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_config.udp_answer_limit_invalid (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_config.udp_answer_limit_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_config.a_record_limit_invalid (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:dns_config.a_record_limit_invalid (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:dns_config.a_record_limit_invalid (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:performance.raft_multiplier_<_0 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:performance.raft_multiplier_<_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:performance.raft_multiplier_==_0 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:performance.raft_multiplier_==_0 (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:performance.raft_multiplier_==_0 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:performance.raft_multiplier_>_10 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:performance.raft_multiplier_>_10 (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:performance.raft_multiplier_>_10 (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/node_name_invalid (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:node_meta_key_too_long (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:node_meta_key_too_long (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:node_meta_key_too_long (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:node_meta_value_too_long (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:node_meta_value_too_long (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:node_meta_value_too_long (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:node_meta_too_many_keys (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:node_meta_too_many_keys (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_listeners_dns_vs_http (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_listeners_dns_vs_http (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_listeners_dns_vs_http (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_listeners_dns_vs_https (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_listeners_dns_vs_https (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_listeners_http_vs_https (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_listeners_dns_vs_https (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_listeners_http_vs_https (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_listeners_http_vs_https (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_advertise_addresses_HTTP_vs_RPC (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_HTTP_vs_RPC (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_advertise_addresses_RPC_vs_Serf_LAN (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_RPC_vs_Serf_LAN (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_advertise_addresses_RPC_vs_Serf_WAN (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_RPC_vs_Serf_WAN (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_defaults_to_true (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:http_use_cache_defaults_to_true (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_is_enabled_when_true (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:http_use_cache_is_enabled_when_true (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_is_disabled_when_false (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_HTTP_vs_RPC (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_advertise_addresses_RPC_vs_Serf_LAN (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_RPC_vs_Serf_LAN (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:unique_advertise_addresses_RPC_vs_Serf_WAN (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:unique_advertise_addresses_RPC_vs_Serf_WAN (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_defaults_to_true (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:http_use_cache_defaults_to_true (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_is_enabled_when_true (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:http_use_cache_is_enabled_when_true (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:http_use_cache_is_disabled_when_false (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:http_use_cache_is_disabled_when_false (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:sidecar_service_can't_have_ID (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:sidecar_service_can't_have_ID (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:sidecar_service_can't_have_ID (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:sidecar_service_can't_have_nested_sidecar (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:sidecar_service_can't_have_nested_sidecar (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:sidecar_service_can't_have_nested_sidecar (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:telemetry.prefix_filter_cannot_be_empty (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:telemetry.prefix_filter_cannot_be_empty (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:telemetry.prefix_filter_cannot_be_empty (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:telemetry.prefix_filter_must_start_with_+_or_- (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:telemetry.prefix_filter_must_start_with_+_or_- (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:encrypt_has_invalid_key (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:telemetry.prefix_filter_must_start_with_+_or_- (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:telemetry.prefix_filter_must_start_with_+_or_- (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:encrypt_has_invalid_key (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:encrypt_has_invalid_key (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:multiple_check_files (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:multiple_check_files (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:multiple_check_files (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:grpc_check (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:grpc_check (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:grpc_check (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:alias_check_with_no_node (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:alias_check_with_no_node (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:multiple_service_files (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:multiple_service_files (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:service_with_wrong_meta:_too_long_key (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:alias_check_with_no_node (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:multiple_service_files (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:multiple_service_files (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:service_with_wrong_meta:_too_long_key (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:service_with_wrong_meta:_too_long_key (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:service_with_wrong_meta:_too_long_value (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:service_with_wrong_meta:_too_long_value (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:service_with_wrong_meta:_too_many_meta (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:service_with_wrong_meta:_too_many_meta (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:translated_keys (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:translated_keys (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ignore_snapshot_agent_sub-object (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:translated_keys (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ignore_snapshot_agent_sub-object (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ignore_snapshot_agent_sub-object (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:service.connectsidecar_service_with_checks_and_upstreams (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:service.connectsidecar_service_with_checks_and_upstreams (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:services.connect.sidecar_service_with_checks_and_upstreams (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:services.connect.sidecar_service_with_checks_and_upstreams (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:services.connect.sidecar_service_with_checks_and_upstreams (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:verify_server_hostname_implies_verify_outgoing (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:verify_server_hostname_implies_verify_outgoing (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:verify_server_hostname_implies_verify_outgoing (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_works_implies_connect (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_works_implies_connect (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_works_implies_connect (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_works_with_verify_incoming (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_works_with_verify_incoming (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_works_with_verify_incoming_rpc (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_works_with_verify_incoming_rpc (0.03s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_warns_without_verify_incoming_or_verify_incoming_rpc (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_warns_without_verify_incoming_or_verify_incoming_rpc (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_works_with_verify_incoming_rpc (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_warns_without_verify_incoming_or_verify_incoming_rpc (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_warns_without_verify_incoming_or_verify_incoming_rpc (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.allow_tls_errors_in_client_mode (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_errors_in_client_mode (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.allow_tls_errors_in_client_mode (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_encrypt.tls_errors_in_server_mode (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_encrypt.tls_errors_in_server_mode (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:test_connect_vault_provider_configuration (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:test_connect_vault_provider_configuration (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:test_connect_vault_provider_configuration (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Connect_AWS_CA_provider_configuration (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Connect_AWS_CA_provider_configuration (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Connect_AWS_CA_provider_configuration (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Connect_AWS_CA_provider_configuration (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Connect_AWS_CA_provider_TTL_validation (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Connect_AWS_CA_provider_TTL_validation (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Connect_AWS_CA_provider_EC_key_length_validation (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Connect_AWS_CA_provider_EC_key_length_validation (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:Connect_AWS_CA_provider_EC_key_length_validation (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:Connect_AWS_CA_provider_EC_key_length_validation (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_requires_connect.enabled (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_requires_connect.enabled (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_requires_connect.enabled (0.04s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_cannot_use_-join-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_cannot_use_-join-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_cannot_use_-retry-join-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_cannot_use_-retry-join-wan (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_requires_server_mode (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_cannot_use_-join-wan (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_cannot_use_-retry-join-wan (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_cannot_use_-retry-join-wan (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_requires_server_mode (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_requires_server_mode (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_requires_no_slashes_in_node_names (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_requires_no_slashes_in_node_names (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:primary_gateways_requires_server_mode (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:primary_gateways_requires_server_mode (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:primary_gateways_only_works_in_a_secondary_datacenter (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:primary_gateways_only_works_in_a_secondary_datacenter (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:primary_gateways_only_works_in_a_secondary_datacenter (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:connect.enable_mesh_gateway_wan_federation_in_secondary_with_primary_gateways_configured (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:connect.enable_mesh_gateway_wan_federation_in_secondary_with_primary_gateways_configured (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_doesn't_parse (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_doesn't_parse (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_doesn't_parse (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_unknown_kind (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_unknown_kind (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_invalid_service-defaults (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_invalid_service-defaults (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_invalid_service-defaults (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_proxy-defaults_(snake-case) (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_proxy-defaults_(snake-case) (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_proxy-defaults_(snake-case) (0.03s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_proxy-defaults_(camel-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_proxy-defaults_(camel-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_service-defaults_(snake-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-defaults_(snake-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_service-defaults_(camel-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-defaults_(camel-case) (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_proxy-defaults_(camel-case) (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_proxy-defaults_(camel-case) (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_service-defaults_(snake-case) (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-defaults_(snake-case) (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_service-defaults_(camel-case) (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-defaults_(camel-case) (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:ConfigEntry_bootstrap_service-router_(snake-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-router_(snake-case) (0.02s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/default_limits (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_and_auto_encrypt_error (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_and_auto_encrypt_error (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:ConfigEntry_bootstrap_service-router_(snake-case) (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/default_limits (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_and_auto_encrypt_error (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_and_auto_encrypt_error (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_not_allowed_for_servers (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_not_allowed_for_servers (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_tls_not_enabled (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_tls_not_enabled (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_server_tls_not_enabled (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_server_tls_not_enabled (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_no_intro_token (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_no_intro_token (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_no_server_addresses (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_no_server_addresses (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_client (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_client (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_client_not_allowed (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_not_allowed_for_servers (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_tls_not_enabled (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_tls_not_enabled (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_server_tls_not_enabled (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_server_tls_not_enabled (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_no_intro_token (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_no_intro_token (0.04s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_no_server_addresses (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_no_server_addresses (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_client (0.07s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_client (0.05s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_client_not_allowed (0.02s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_client_not_allowed (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_config (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_config (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_config (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_config_2 (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_config_2 (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_config_2 (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_config_2 (0.03s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_require_token_replication_in_secondary (0.01s) --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_require_token_replication_in_secondary (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_claim_assertion (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_claim_assertion (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_ok (0.01s) - --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_ok (0.01s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_invalid_claim_assertion (0.03s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_invalid_claim_assertion (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/json:auto_config_authorizer_ok (0.02s) + --- PASS: TestBuilder_BuildAndValide_ConfigFlagsAndEdgecases/hcl:auto_config_authorizer_ok (0.03s) === RUN TestNewBuilder_InvalidConfigFormat --- PASS: TestNewBuilder_InvalidConfigFormat (0.00s) === RUN TestFullConfig @@ -57643,30 +57870,30 @@ === RUN Test_UIPathBuilder --- PASS: Test_UIPathBuilder (0.00s) === RUN TestConnectCAConfiguration -=== RUN TestConnectCAConfiguration/defaults -=== RUN TestConnectCAConfiguration/cluster-id-override === RUN TestConnectCAConfiguration/cluster-id-non-uuid === RUN TestConnectCAConfiguration/provider-override === RUN TestConnectCAConfiguration/other-config +=== RUN TestConnectCAConfiguration/defaults +=== RUN TestConnectCAConfiguration/cluster-id-override --- PASS: TestConnectCAConfiguration (0.00s) - --- PASS: TestConnectCAConfiguration/defaults (0.00s) - --- PASS: TestConnectCAConfiguration/cluster-id-override (0.00s) --- PASS: TestConnectCAConfiguration/cluster-id-non-uuid (0.00s) --- PASS: TestConnectCAConfiguration/provider-override (0.00s) --- PASS: TestConnectCAConfiguration/other-config (0.00s) + --- PASS: TestConnectCAConfiguration/defaults (0.00s) + --- PASS: TestConnectCAConfiguration/cluster-id-override (0.00s) === RUN TestSegments segment_oss_test.go:13: DM-skipped --- SKIP: TestSegments (0.00s) === CONT TestBuildAndValidate_HTTPMaxConnsPerClientExceedsRLimit === CONT TestConfigDecodeBytes --- PASS: TestConfigDecodeBytes (0.00s) ---- PASS: TestBuildAndValidate_HTTPMaxConnsPerClientExceedsRLimit (0.01s) +--- PASS: TestBuildAndValidate_HTTPMaxConnsPerClientExceedsRLimit (0.02s) PASS -ok github.com/hashicorp/consul/agent/config 4.526s +ok github.com/hashicorp/consul/agent/config 8.195s === RUN TestCollectHostInfo ---- PASS: TestCollectHostInfo (0.01s) +--- PASS: TestCollectHostInfo (0.00s) PASS -ok github.com/hashicorp/consul/agent/debug 0.013s +ok github.com/hashicorp/consul/agent/debug 0.021s ? github.com/hashicorp/consul/agent/dns [no test files] ? github.com/hashicorp/consul/agent/exec [no test files] === RUN TestAgentAntiEntropy_Services @@ -57713,1031 +57940,1012 @@ === RUN TestAliasNotifications_local === PAUSE TestAliasNotifications_local === CONT TestAgentAntiEntropy_Services -=== CONT TestAgentAntiEntropy_NodeInfo -=== CONT TestAgentAntiEntropy_Checks [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestAgentAntiEntropy_NodeInfo === CONT TestAliasNotifications_local +=== CONT TestState_Notify +--- PASS: TestState_Notify (0.00s) +=== CONT TestAgent_sendCoordinate +=== CONT TestAgent_AliasCheck_ServiceNotification +=== CONT TestAgentAntiEntropy_Checks === CONT TestAgentAntiEntropy_Check_DeferSync -=== CONT TestAgentAntiEntropy_EnableTagOverride +=== CONT TestAgent_UpdateCheck_DiscardOutput === CONT TestAgentAntiEntropy_Checks_ACLDeny -=== CONT TestAgent_ServiceTokens -=== CONT TestState_Notify +=== CONT TestAgentAntiEntropy_RemovingServiceAndCheck [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +--- PASS: TestAgent_AliasCheck_ServiceNotification (0.03s) +=== CONT TestAgentAntiEntropy_EnableTagOverride === CONT TestAgent_sendCoordinate -=== CONT TestAgent_AliasCheck_ServiceNotification + state_test.go:1985: 128 5 100ms +2021-10-05T16:29:38.837Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:38.865Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:38.996 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 Address:127.0.0.1:27363}]" +29:38.997 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +29:38.999 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0.dc1 127.0.0.1 +29:39.062 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.062 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +29:39.064 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 127.0.0.1 +2021-10-05T16:29:39.064Z [INFO] agent.router: Initializing LAN area manager +29:39.065 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +29:39.065 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0.dc1 area=wan +29:39.065 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +29:39.066 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +29:39.066 [INFO] TestAgent: started state syncer +29:39.066 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +29:39.067 [WARN] TestAgent.server.raft: unable to get address for sever, using fallback address: id=ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 fallback=127.0.0.1:27363 error="Could not find address for server id ebc6538d-f99c-1a2f-66d3-18e3c9906bf0" +29:39.090 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.090 [DEBUG] TestAgent.server.raft: vote granted: from=ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 term=2 tally=1 +29:39.090 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.090 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +29:39.092 [INFO] TestAgent.server: cluster leadership acquired +29:39.097 [INFO] TestAgent.server: Adding LAN server: server="Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +29:39.097 [INFO] TestAgent.server: New leader elected: payload=Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 +29:39.149 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +29:39.193 [INFO] TestAgent: Synced node info +29:39.245 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.290 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.290 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.290 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.290 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.290 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.290 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 +29:39.291 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 +29:39.304 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebc6538d-f99c-1a2f-66d3-18e3c9906bf0 +29:39.323 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.369 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.379 [INFO] TestAgent: Synced node info +29:39.387 [INFO] TestAgent: Synced service: service=redis +29:39.402 [INFO] TestAgent: Synced service: service=web +29:39.416 [INFO] TestAgent: Synced service: service=api +29:39.431 [INFO] TestAgent: Synced service: service=cache +29:39.436 [INFO] TestAgent: Deregistered service: service=lb +29:39.436 [DEBUG] TestAgent: Service in sync: service=mysql +29:39.437 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.459 [INFO] TestAgent: Synced node info +29:39.459 [DEBUG] TestAgent: Service in sync: service=mysql +29:39.460 [DEBUG] TestAgent: Service in sync: service=redis +29:39.460 [DEBUG] TestAgent: Service in sync: service=web +29:39.476 [INFO] TestAgent: Deregistered service: service=api +29:39.476 [DEBUG] TestAgent: Service in sync: service=cache +29:39.476 [INFO] TestAgent: Requesting shutdown +29:39.476 [INFO] TestAgent.server: shutting down server +29:39.476 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.476 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.476 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.476 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.476 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.476 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.476 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.476 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.476 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.476 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.492 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.508Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.508Z [INFO] agent.router.manager: shutting down +29:39.508 [INFO] TestAgent: consul server down +29:39.508 [INFO] TestAgent: shutdown complete +29:39.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +29:39.508 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +29:39.508 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +29:40.012 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.012 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Services (1.30s) +=== CONT TestAgentAntiEntropy_Services_ACLDeny +2021-10-05T16:29:38.980Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.073Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.233 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40e4a748-2192-161a-0510-9bf59fe950b5 Address:127.0.0.1:27370}]" +29:39.234 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27370 [Follower]" leader= +29:39.235 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-103ad76c-d4d9-640a-3e30-3d5acce29852.dc1 127.0.0.1 +29:39.236 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-103ad76c-d4d9-640a-3e30-3d5acce29852 127.0.0.1 +2021-10-05T16:29:39.236Z [INFO] agent.router: Initializing LAN area manager +29:39.237 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=udp +29:39.237 [INFO] TestAgent.server: Adding LAN server: server="Node-103ad76c-d4d9-640a-3e30-3d5acce29852 (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +29:39.237 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-103ad76c-d4d9-640a-3e30-3d5acce29852.dc1 area=wan +29:39.237 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=tcp +29:39.244 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27366 network=tcp +29:39.244 [INFO] TestAgent: started state syncer +29:39.244 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27371 network=tcp +29:39.278 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.278 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27370 [Candidate]" term=2 +29:39.298 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.298 [DEBUG] TestAgent.server.raft: vote granted: from=40e4a748-2192-161a-0510-9bf59fe950b5 term=2 tally=1 +29:39.298 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.299 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27370 [Leader]" +29:39.299 [INFO] TestAgent.server: cluster leadership acquired +29:39.299 [INFO] TestAgent.server: New leader elected: payload=Node-103ad76c-d4d9-640a-3e30-3d5acce29852 +29:39.316 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27370 +29:39.365 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.420 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.420 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.420 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.420 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.420 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.420 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-103ad76c-d4d9-640a-3e30-3d5acce29852 +29:39.420 [INFO] TestAgent.server: member joined, marking health alive: member=Node-103ad76c-d4d9-640a-3e30-3d5acce29852 +29:39.457 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.471 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.477 [INFO] TestAgent: Synced node info +29:39.483 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.497 [INFO] TestAgent: Synced node info +29:39.497 [INFO] TestAgent: Requesting shutdown +29:39.497 [INFO] TestAgent.server: shutting down server +29:39.497 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.497 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.497 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.497 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.497 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.497 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.497 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.497 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:39.497 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.497 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.497 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.504 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.515Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.515Z [INFO] agent.router.manager: shutting down +29:39.515 [INFO] TestAgent: consul server down +29:39.515 [INFO] TestAgent: shutdown complete +29:39.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=tcp +29:39.515 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=udp +29:39.515 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27366 network=tcp +29:40.016 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.016 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_NodeInfo (1.31s) +=== CONT TestAgentAntiEntropy_Services_WithChecks +2021-10-05T16:29:38.991Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.157Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.274 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f95e355f-d096-9617-bdbf-eb57b4b21358 Address:127.0.0.1:27419}]" +29:39.275 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27419 [Follower]" leader= +29:39.275 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f95e355f-d096-9617-bdbf-eb57b4b21358.dc1 127.0.0.1 +29:39.276 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f95e355f-d096-9617-bdbf-eb57b4b21358 127.0.0.1 +2021-10-05T16:29:39.276Z [INFO] agent.router: Initializing LAN area manager +29:39.276 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27414 network=udp +29:39.276 [INFO] TestAgent.server: Adding LAN server: server="Node-f95e355f-d096-9617-bdbf-eb57b4b21358 (Addr: tcp/127.0.0.1:27419) (DC: dc1)" +29:39.277 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f95e355f-d096-9617-bdbf-eb57b4b21358.dc1 area=wan +29:39.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27414 network=tcp +29:39.277 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27415 network=tcp +29:39.277 [INFO] TestAgent: started state syncer +29:39.277 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27420 network=tcp +29:39.324 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.324 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27419 [Candidate]" term=2 +29:39.361 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.361 [DEBUG] TestAgent.server.raft: vote granted: from=f95e355f-d096-9617-bdbf-eb57b4b21358 term=2 tally=1 +29:39.361 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.361 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27419 [Leader]" +29:39.362 [INFO] TestAgent.server: cluster leadership acquired +29:39.362 [INFO] TestAgent.server: New leader elected: payload=Node-f95e355f-d096-9617-bdbf-eb57b4b21358 +29:39.371 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27419 +29:39.399 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.453 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.453 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.453 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.453 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.453 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.453 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f95e355f-d096-9617-bdbf-eb57b4b21358 +29:39.453 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f95e355f-d096-9617-bdbf-eb57b4b21358 +29:39.484 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.517 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.524 [INFO] TestAgent: Synced node info +29:39.530 [INFO] TestAgent: Deregistered service: service=deleted-check-service +29:39.539 [INFO] TestAgent: Deregistered check: check=lb +29:39.539 [INFO] TestAgent: Requesting shutdown +29:39.539 [INFO] TestAgent.server: shutting down server +29:39.539 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.539 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.539 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.539 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.539 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.539 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.539 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.539 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:39.539 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.539 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.539 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.547 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.559Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.559Z [INFO] agent.router.manager: shutting down +29:39.559 [INFO] TestAgent: consul server down +29:39.559 [INFO] TestAgent: shutdown complete +29:39.559 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27414 network=tcp +29:39.559 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27414 network=udp +29:39.559 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27415 network=tcp +29:40.059 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.059 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_RemovingServiceAndCheck (1.35s) === CONT TestAgent_ServiceWatchCh +2021-10-05T16:29:39.108Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.205Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.281 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2b3b1821-2caf-aae5-365b-23b3a6bd1906 Address:127.0.0.1:27405}]" +29:39.281 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27405 [Follower]" leader= +29:39.282 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906.dc1 127.0.0.1 +29:39.283 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906 127.0.0.1 +2021-10-05T16:29:39.283Z [INFO] agent.router: Initializing LAN area manager +29:39.284 [INFO] TestAgent.server: Adding LAN server: server="Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906 (Addr: tcp/127.0.0.1:27405) (DC: dc1)" +29:39.284 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906.dc1 area=wan +29:39.284 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27400 network=udp +29:39.284 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27400 network=tcp +29:39.284 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27401 network=tcp +29:39.284 [INFO] TestAgent: started state syncer +29:39.284 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27406 network=tcp +29:39.335 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.335 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27405 [Candidate]" term=2 +29:39.384 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.384 [DEBUG] TestAgent.server.raft: vote granted: from=2b3b1821-2caf-aae5-365b-23b3a6bd1906 term=2 tally=1 +29:39.384 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.384 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27405 [Leader]" +29:39.384 [INFO] TestAgent.server: cluster leadership acquired +29:39.384 [INFO] TestAgent.server: New leader elected: payload=Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906 +29:39.412 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27405 +29:39.471 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.523 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.523 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.523 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.523 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.524 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.524 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906 +29:39.524 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2b3b1821-2caf-aae5-365b-23b3a6bd1906 +29:39.524 [INFO] TestAgent: Synced node info +29:39.524 [DEBUG] TestAgent: Node info in sync +29:39.557 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.557 [DEBUG] TestAgent: Node info in sync +29:39.564 [INFO] TestAgent: Synced check: check=web +29:39.564 [INFO] TestAgent: Requesting shutdown +29:39.564 [INFO] TestAgent.server: shutting down server +29:39.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.564 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.564 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.564 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.564 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.564 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.564 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.564 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.564 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.575 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.584Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.584Z [INFO] agent.router.manager: shutting down +29:39.584 [INFO] TestAgent: consul server down +29:39.584 [INFO] TestAgent: shutdown complete +29:39.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27400 network=tcp +29:39.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27400 network=udp +29:39.584 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27401 network=tcp +29:40.084 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.084 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_UpdateCheck_DiscardOutput (1.37s) +=== CONT TestAgentAntiEntropy_Services_ConnectProxy +2021-10-05T16:29:38.893Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:38.983Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.156 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 Address:127.0.0.1:27426}]" +29:39.157 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27426 [Follower]" leader= +29:39.158 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724.dc1 127.0.0.1 +29:39.194 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 127.0.0.1 +2021-10-05T16:29:39.194Z [INFO] agent.router: Initializing LAN area manager +29:39.194 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27421 network=udp +29:39.195 [INFO] TestAgent.server: Adding LAN server: server="Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 (Addr: tcp/127.0.0.1:27426) (DC: dc1)" +29:39.195 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724.dc1 area=wan +29:39.196 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27421 network=tcp +29:39.200 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.200 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27426 [Candidate]" term=2 +29:39.206 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27422 network=tcp +29:39.206 [INFO] TestAgent: started state syncer +29:39.207 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27427 network=tcp +29:39.253 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.253 [DEBUG] TestAgent.server.raft: vote granted: from=1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 term=2 tally=1 +29:39.253 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.253 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27426 [Leader]" +29:39.256 [INFO] TestAgent.server: cluster leadership acquired +29:39.256 [INFO] TestAgent.server: New leader elected: payload=Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 +29:39.274 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27426 +29:39.275 [INFO] TestAgent: Synced node info +29:39.316 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.374 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.374 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.374 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.374 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.374 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.374 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 +29:39.374 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e52c60b-e67a-1ba5-f8d4-7be3ddb64724 +29:39.404 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.440 [DEBUG] TestAgent: Node info in sync +29:39.456 [INFO] TestAgent: Synced service: service=svc_id1 +29:39.472 [INFO] TestAgent: Synced service: service=svc_id2 +29:39.505 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.530 [INFO] TestAgent: Synced node info +29:39.542 [INFO] TestAgent: Synced service: service=svc_id1 +29:39.560 [INFO] TestAgent: Synced service: service=svc_id2 +29:39.560 [INFO] TestAgent: Requesting shutdown +29:39.560 [INFO] TestAgent.server: shutting down server +29:39.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.561 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.561 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.561 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.561 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.561 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.561 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.561 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.561 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.561 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.569 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.579Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.579Z [INFO] agent.router.manager: shutting down +29:39.580 [INFO] TestAgent: consul server down +29:39.580 [INFO] TestAgent: shutdown complete +29:39.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27421 network=tcp +29:39.580 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27421 network=udp +29:39.580 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27422 network=tcp +29:40.087 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.087 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_EnableTagOverride (1.35s) === CONT TestAgent_CheckTokens +2021-10-05T16:29:38.941Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:38.970Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.076 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d318d95f-1790-fb80-19ab-6a2ef193764d Address:127.0.0.1:27391}]" +29:39.092 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27391 [Follower]" leader= +29:39.092 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d318d95f-1790-fb80-19ab-6a2ef193764d.dc1 127.0.0.1 +29:39.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d318d95f-1790-fb80-19ab-6a2ef193764d 127.0.0.1 +2021-10-05T16:29:39.145Z [INFO] agent.router: Initializing LAN area manager +29:39.145 [INFO] TestAgent.server: Adding LAN server: server="Node-d318d95f-1790-fb80-19ab-6a2ef193764d (Addr: tcp/127.0.0.1:27391) (DC: dc1)" +29:39.146 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27386 network=udp +29:39.146 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d318d95f-1790-fb80-19ab-6a2ef193764d.dc1 area=wan +29:39.146 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27386 network=tcp +29:39.147 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27387 network=tcp +29:39.147 [INFO] TestAgent: started state syncer +29:39.147 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27392 network=tcp +29:39.152 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.152 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27391 [Candidate]" term=2 +29:39.232 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.232 [DEBUG] TestAgent.server.raft: vote granted: from=d318d95f-1790-fb80-19ab-6a2ef193764d term=2 tally=1 +29:39.232 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.232 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27391 [Leader]" +29:39.234 [INFO] TestAgent.server: cluster leadership acquired +29:39.234 [INFO] TestAgent.server: New leader elected: payload=Node-d318d95f-1790-fb80-19ab-6a2ef193764d +29:39.259 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27391 +29:39.288 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.336 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.336 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.336 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.336 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.336 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.336 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d318d95f-1790-fb80-19ab-6a2ef193764d +29:39.336 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d318d95f-1790-fb80-19ab-6a2ef193764d +29:39.361 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.534 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.546 [INFO] TestAgent: Synced node info +29:39.546 [DEBUG] TestAgent: Check in sync: check=mysql +29:39.553 [INFO] TestAgent: Synced check: check=redis +29:39.561 [INFO] TestAgent: Synced check: check=web +29:39.561 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.561 [DEBUG] TestAgent: Node info in sync +29:39.574 [INFO] TestAgent: Synced check: check=cache +29:39.578 [INFO] TestAgent: Deregistered check: check=lb +29:39.578 [DEBUG] TestAgent: Check in sync: check=mysql +29:39.578 [DEBUG] TestAgent: Check in sync: check=redis +29:39.578 [DEBUG] TestAgent: Check in sync: check=web +29:39.579 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.583 [INFO] TestAgent: Synced node info +29:39.583 [DEBUG] TestAgent: Check in sync: check=web +29:39.583 [DEBUG] TestAgent: Check in sync: check=cache +29:39.583 [DEBUG] TestAgent: Check in sync: check=mysql +29:39.590 [INFO] TestAgent: Deregistered check: check=redis +29:39.591 [INFO] TestAgent: Requesting shutdown +29:39.591 [INFO] TestAgent.server: shutting down server +29:39.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.591 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.591 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.591 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.591 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.591 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.591 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.595 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.599Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.599Z [INFO] agent.router.manager: shutting down +29:39.599 [INFO] TestAgent: consul server down +29:39.599 [INFO] TestAgent: shutdown complete +29:39.599 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27386 network=tcp +29:39.599 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27386 network=udp +29:39.599 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27387 network=tcp +29:40.100 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.100 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Checks (1.39s) === CONT TestAgent_AddCheckFailure -=== CONT TestAgentAntiEntropy_Services_ConnectProxy -=== CONT TestAgentAntiEntropy_Services_WithChecks -=== CONT TestAgentAntiEntropy_Services_ACLDeny +2021-10-05T16:29:38.918Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.034Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.157 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:26239a8f-6625-b86c-41ff-482204c5a006 Address:127.0.0.1:27384}]" +29:39.159 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-26239a8f-6625-b86c-41ff-482204c5a006.dc1 127.0.0.1 +29:39.195 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27384 [Follower]" leader= +29:39.195 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-26239a8f-6625-b86c-41ff-482204c5a006 127.0.0.1 +2021-10-05T16:29:39.195Z [INFO] agent.router: Initializing LAN area manager +29:39.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27379 network=udp +29:39.196 [INFO] TestAgent.server: Adding LAN server: server="Node-26239a8f-6625-b86c-41ff-482204c5a006 (Addr: tcp/127.0.0.1:27384) (DC: dc1)" +29:39.196 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-26239a8f-6625-b86c-41ff-482204c5a006.dc1 area=wan +29:39.196 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27379 network=tcp +29:39.197 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27380 network=tcp +29:39.197 [INFO] TestAgent: started state syncer +29:39.197 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27385 network=tcp +29:39.244 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.244 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27384 [Candidate]" term=2 +29:39.287 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.287 [DEBUG] TestAgent.server.raft: vote granted: from=26239a8f-6625-b86c-41ff-482204c5a006 term=2 tally=1 +29:39.287 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.287 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27384 [Leader]" +29:39.287 [INFO] TestAgent.server: cluster leadership acquired +29:39.288 [INFO] TestAgent.server: New leader elected: payload=Node-26239a8f-6625-b86c-41ff-482204c5a006 +29:39.297 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27384 +29:39.328 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.410 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.410 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.410 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.410 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.410 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.410 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-26239a8f-6625-b86c-41ff-482204c5a006 +29:39.410 [INFO] TestAgent.server: member joined, marking health alive: member=Node-26239a8f-6625-b86c-41ff-482204c5a006 +29:39.410 [INFO] TestAgent: Synced node info +29:39.432 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.550 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.550 [DEBUG] TestAgent: Node info in sync +29:39.550 [DEBUG] TestAgent: Node info in sync +29:39.597 [INFO] TestAgent: Requesting shutdown +29:39.597 [INFO] TestAgent.server: shutting down server +29:39.597 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.597 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.597 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.597 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.597 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.597 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.597 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.597 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.597 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.597 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.603 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.607Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.607Z [INFO] agent.router.manager: shutting down +29:39.607 [INFO] TestAgent: consul server down +29:39.607 [INFO] TestAgent: shutdown complete +29:39.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27379 network=tcp +29:39.607 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27379 network=udp +29:39.608 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27380 network=tcp +29:40.108 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.108 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_sendCoordinate (1.40s) === CONT TestAgent_AliasCheck -=== CONT TestAgent_UpdateCheck_DiscardOutput ---- PASS: TestState_Notify (0.00s) --- PASS: TestAgent_CheckTokens (0.02s) -=== CONT TestAgentAntiEntropy_RemovingServiceAndCheck ---- PASS: TestAgent_ServiceTokens (0.03s) ---- PASS: TestAgent_AliasCheck_ServiceNotification (0.02s) ---- PASS: TestAgent_AliasCheck (0.02s) ---- PASS: TestAgent_AddCheckFailure (0.03s) -=== CONT TestAgent_sendCoordinate - state_test.go:1985: 128 5 100ms -2022-11-07T22:44:38.501Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.526Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.562 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c7568cf3-fc65-ac7a-f5b1-c225875f4d59 Address:127.0.0.1:15195}]" -44:38.562 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15195 [Follower]" leader= -44:38.563 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59.dc1 127.0.0.1 -44:38.564 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59 127.0.0.1 -2022-11-07T22:44:38.564Z [INFO] agent.router: Initializing LAN area manager -44:38.564 [INFO] TestAgent.server: Adding LAN server: server="Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59 (Addr: tcp/127.0.0.1:15195) (DC: dc1)" -44:38.564 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59.dc1 area=wan -44:38.594 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15190 network=tcp -44:38.594 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15190 network=udp -44:38.595 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15191 network=tcp -44:38.595 [INFO] TestAgent: started state syncer -44:38.595 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15196 network=tcp -44:38.630 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.630 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15195 [Candidate]" term=2 -44:38.689 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.689 [DEBUG] TestAgent.server.raft: vote granted: from=c7568cf3-fc65-ac7a-f5b1-c225875f4d59 term=2 tally=1 -44:38.689 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.689 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15195 [Leader]" -44:38.689 [INFO] TestAgent.server: cluster leadership acquired -44:38.689 [INFO] TestAgent.server: New leader elected: payload=Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59 -44:38.707 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15195 -44:38.737 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.786 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.786 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.786 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.786 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59 -44:38.786 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c7568cf3-fc65-ac7a-f5b1-c225875f4d59 -44:38.807 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.810 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.815 [INFO] TestAgent: Synced node info -44:38.820 [INFO] TestAgent: Synced check: check=web -44:38.820 [INFO] TestAgent: Requesting shutdown -44:38.820 [INFO] TestAgent.server: shutting down server -44:38.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.820 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.820 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.820 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.820 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.820 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.820 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.820 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.837 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.843Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.843Z [INFO] agent.router.manager: shutting down -44:38.846 [INFO] TestAgent: consul server down -44:38.846 [INFO] TestAgent: shutdown complete -44:38.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15190 network=tcp -44:38.846 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15190 network=udp -44:38.846 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15191 network=tcp -44:39.346 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.346 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_UpdateCheck_DiscardOutput (0.92s) -2022-11-07T22:44:38.497Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:38.497Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.530Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:38.530Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.594 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4cb78086-44e7-8528-165c-8d18e545a888 Address:127.0.0.1:15188}]" -44:38.594 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15188 [Follower]" leader= -44:38.595 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4cb78086-44e7-8528-165c-8d18e545a888.dc1 127.0.0.1 -44:38.596 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4cb78086-44e7-8528-165c-8d18e545a888 127.0.0.1 -2022-11-07T22:44:38.597Z [INFO] agent.router: Initializing LAN area manager -44:38.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15183 network=udp -44:38.597 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4cb78086-44e7-8528-165c-8d18e545a888.dc1 area=wan -44:38.597 [INFO] TestAgent.server: Adding LAN server: server="Node-4cb78086-44e7-8528-165c-8d18e545a888 (Addr: tcp/127.0.0.1:15188) (DC: dc1)" -44:38.598 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15183 network=tcp -44:38.599 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15184 network=tcp -44:38.599 [INFO] TestAgent: started state syncer -44:38.599 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15189 network=tcp -44:38.647 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:38.650 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.650 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15188 [Candidate]" term=2 -44:38.685 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.685 [DEBUG] TestAgent.server.raft: vote granted: from=4cb78086-44e7-8528-165c-8d18e545a888 term=2 tally=1 -44:38.685 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.685 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15188 [Leader]" -44:38.685 [INFO] TestAgent.server: cluster leadership acquired -44:38.685 [INFO] TestAgent.server: New leader elected: payload=Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.703 [INFO] TestAgent.server: initializing acls -44:38.707 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:38.707 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:38.718 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:38.737 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:38.737 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:38.737 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:38.738 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.738 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4cb78086-44e7-8528-165c-8d18e545a888.dc1 -44:38.738 [INFO] TestAgent.server: Updating LAN server: server="Node-4cb78086-44e7-8528-165c-8d18e545a888 (Addr: tcp/127.0.0.1:15188) (DC: dc1)" -44:38.738 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4cb78086-44e7-8528-165c-8d18e545a888.dc1 area=wan -44:38.756 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.792 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.792 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.792 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.792 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.792 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.792 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.792 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.812 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.812 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.814 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-4cb78086-44e7-8528-165c-8d18e545a888 -44:38.822 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.822 [WARN] TestAgent: Node info update blocked by ACLs: node=4cb78086-44e7-8528-165c-8d18e545a888 accessorID=00000000-0000-0000-0000-000000000002 -44:38.826 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.826 [WARN] TestAgent: Node info update blocked by ACLs: node=4cb78086-44e7-8528-165c-8d18e545a888 accessorID=00000000-0000-0000-0000-000000000002 -44:38.827 [WARN] TestAgent: Service registration blocked by ACLs: service=mysql accessorID= -44:38.839 [INFO] TestAgent: Synced service: service=api -44:38.840 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.840 [WARN] TestAgent: Node info update blocked by ACLs: node=4cb78086-44e7-8528-165c-8d18e545a888 accessorID=00000000-0000-0000-0000-000000000002 -44:38.840 [WARN] TestAgent: Service registration blocked by ACLs: service=mysql accessorID= -44:38.845 [INFO] TestAgent: Deregistered service: service=api -44:38.845 [INFO] TestAgent: Requesting shutdown -44:38.845 [INFO] TestAgent.server: shutting down server -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.845 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.845 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:38.845 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.845 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.852 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.859Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.859Z [INFO] agent.router.manager: shutting down -44:38.859 [INFO] TestAgent: consul server down -44:38.859 [INFO] TestAgent: shutdown complete -44:38.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15183 network=tcp -44:38.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15183 network=udp -44:38.859 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15184 network=tcp -44:39.359 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.359 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Services_ACLDeny (0.93s) -2022-11-07T22:44:38.493Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.523Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.563 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b688e4b-e14c-363e-95c4-0436de08aff0 Address:127.0.0.1:15160}]" -44:38.563 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15160 [Follower]" leader= -44:38.563 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0b688e4b-e14c-363e-95c4-0436de08aff0.dc1 127.0.0.1 -44:38.595 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0b688e4b-e14c-363e-95c4-0436de08aff0 127.0.0.1 -2022-11-07T22:44:38.595Z [INFO] agent.router: Initializing LAN area manager -44:38.595 [INFO] TestAgent.server: Adding LAN server: server="Node-0b688e4b-e14c-363e-95c4-0436de08aff0 (Addr: tcp/127.0.0.1:15160) (DC: dc1)" -44:38.595 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0b688e4b-e14c-363e-95c4-0436de08aff0.dc1 area=wan -44:38.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15155 network=udp -44:38.595 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15155 network=tcp -44:38.596 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15156 network=tcp -44:38.596 [INFO] TestAgent: started state syncer -44:38.596 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15161 network=tcp -44:38.626 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.626 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15160 [Candidate]" term=2 -44:38.688 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.688 [DEBUG] TestAgent.server.raft: vote granted: from=0b688e4b-e14c-363e-95c4-0436de08aff0 term=2 tally=1 -44:38.688 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.688 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15160 [Leader]" -44:38.688 [INFO] TestAgent.server: cluster leadership acquired -44:38.688 [INFO] TestAgent.server: New leader elected: payload=Node-0b688e4b-e14c-363e-95c4-0436de08aff0 -44:38.704 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15160 -44:38.732 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.786 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.786 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.786 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.786 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0b688e4b-e14c-363e-95c4-0436de08aff0 -44:38.786 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0b688e4b-e14c-363e-95c4-0436de08aff0 -44:38.807 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.830 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.842 [INFO] TestAgent: Synced node info -44:38.842 [DEBUG] TestAgent: Node info in sync -44:38.855 [INFO] TestAgent: Requesting shutdown -44:38.855 [INFO] TestAgent.server: shutting down server -44:38.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.855 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.855 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.855 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.855 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.855 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.855 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.855 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.855 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.862 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.871Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.871Z [INFO] agent.router.manager: shutting down -44:38.871 [INFO] TestAgent: consul server down -44:38.871 [INFO] TestAgent: shutdown complete -44:38.871 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15155 network=tcp -44:38.871 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15155 network=udp -44:38.871 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15156 network=tcp -44:39.372 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.372 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_sendCoordinate (0.94s) -2022-11-07T22:44:38.488Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.521Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.558 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ef4e4a12-1fc8-234e-b62d-209ec1a64106 Address:127.0.0.1:15139}]" -44:38.558 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15139 [Follower]" leader= -44:38.559 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106.dc1 127.0.0.1 -44:38.560 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106 127.0.0.1 -2022-11-07T22:44:38.560Z [INFO] agent.router: Initializing LAN area manager -44:38.560 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=udp -44:38.560 [INFO] TestAgent.server: Adding LAN server: server="Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106 (Addr: tcp/127.0.0.1:15139) (DC: dc1)" -44:38.560 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106.dc1 area=wan -44:38.560 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=tcp -44:38.561 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15135 network=tcp -44:38.561 [INFO] TestAgent: started state syncer -44:38.561 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15140 network=tcp -44:38.600 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.600 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15139 [Candidate]" term=2 -44:38.680 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.680 [DEBUG] TestAgent.server.raft: vote granted: from=ef4e4a12-1fc8-234e-b62d-209ec1a64106 term=2 tally=1 -44:38.680 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.680 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15139 [Leader]" -44:38.680 [INFO] TestAgent.server: cluster leadership acquired -44:38.680 [INFO] TestAgent.server: New leader elected: payload=Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106 -44:38.693 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15139 -44:38.722 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.766 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.766 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.766 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.766 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.766 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.766 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106 -44:38.766 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ef4e4a12-1fc8-234e-b62d-209ec1a64106 -44:38.785 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.819 [INFO] TestAgent: Synced node info -44:38.826 [INFO] TestAgent: Synced service: service=svc_id1 -44:38.842 [INFO] TestAgent: Synced service: service=svc_id2 -44:38.859 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.862 [INFO] TestAgent: Synced node info -44:38.871 [INFO] TestAgent: Synced service: service=svc_id1 -44:38.881 [INFO] TestAgent: Synced service: service=svc_id2 -44:38.881 [INFO] TestAgent: Requesting shutdown -44:38.881 [INFO] TestAgent.server: shutting down server -44:38.881 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.881 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.881 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.881 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.881 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.881 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.881 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.881 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.881 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.881 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.881 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.885 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.897Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.897Z [INFO] agent.router.manager: shutting down -44:38.897 [INFO] TestAgent: consul server down -44:38.897 [INFO] TestAgent: shutdown complete -44:38.897 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=tcp -44:38.897 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=udp -44:38.898 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15135 network=tcp -44:39.398 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.398 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_EnableTagOverride (0.97s) -2022-11-07T22:44:38.503Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.533Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.595 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3be1a2d-3c84-51ff-009b-b959f5f51824 Address:127.0.0.1:15125}]" -44:38.595 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15125 [Follower]" leader= -44:38.597 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3be1a2d-3c84-51ff-009b-b959f5f51824.dc1 127.0.0.1 -44:38.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3be1a2d-3c84-51ff-009b-b959f5f51824 127.0.0.1 -2022-11-07T22:44:38.597Z [INFO] agent.router: Initializing LAN area manager -44:38.598 [INFO] TestAgent.server: Adding LAN server: server="Node-f3be1a2d-3c84-51ff-009b-b959f5f51824 (Addr: tcp/127.0.0.1:15125) (DC: dc1)" -44:38.598 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=udp -44:38.598 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3be1a2d-3c84-51ff-009b-b959f5f51824.dc1 area=wan -44:38.598 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=tcp -44:38.599 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15121 network=tcp -44:38.599 [INFO] TestAgent: started state syncer -44:38.599 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15126 network=tcp -44:38.651 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.651 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15125 [Candidate]" term=2 -44:38.689 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.689 [DEBUG] TestAgent.server.raft: vote granted: from=f3be1a2d-3c84-51ff-009b-b959f5f51824 term=2 tally=1 -44:38.689 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.689 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15125 [Leader]" -44:38.689 [INFO] TestAgent.server: cluster leadership acquired -44:38.689 [INFO] TestAgent.server: New leader elected: payload=Node-f3be1a2d-3c84-51ff-009b-b959f5f51824 -44:38.705 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15125 -44:38.734 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.786 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.786 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.786 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.786 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.786 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3be1a2d-3c84-51ff-009b-b959f5f51824 -44:38.786 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f3be1a2d-3c84-51ff-009b-b959f5f51824 -44:38.807 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.852 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.857 [INFO] TestAgent: Synced node info -44:38.857 [DEBUG] TestAgent: Check in sync: check=mysql -44:38.862 [INFO] TestAgent: Synced check: check=redis -44:38.871 [INFO] TestAgent: Synced check: check=web -44:38.881 [INFO] TestAgent: Synced check: check=cache -44:38.885 [INFO] TestAgent: Deregistered check: check=lb -44:38.886 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.889 [INFO] TestAgent: Synced node info -44:38.889 [DEBUG] TestAgent: Check in sync: check=mysql -44:38.896 [INFO] TestAgent: Deregistered check: check=redis -44:38.896 [DEBUG] TestAgent: Check in sync: check=web -44:38.896 [DEBUG] TestAgent: Check in sync: check=cache -44:38.896 [INFO] TestAgent: Requesting shutdown -44:38.896 [INFO] TestAgent.server: shutting down server -44:38.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.896 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.896 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.896 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.896 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.896 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.896 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.896 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.902 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.914Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.914Z [INFO] agent.router.manager: shutting down -44:38.914 [INFO] TestAgent: consul server down -44:38.914 [INFO] TestAgent: shutdown complete -44:38.914 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=tcp -44:38.914 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=udp -44:38.914 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15121 network=tcp -44:39.414 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.414 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Checks (0.99s) -2022-11-07T22:44:38.523Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.551Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.668 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40e4a748-2192-161a-0510-9bf59fe950b5 Address:127.0.0.1:15118}]" -44:38.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= -44:38.668 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f4cfc005-1baa-50c1-28de-f22375595475.dc1 127.0.0.1 -44:38.669 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f4cfc005-1baa-50c1-28de-f22375595475 127.0.0.1 -2022-11-07T22:44:38.669Z [INFO] agent.router: Initializing LAN area manager -44:38.669 [INFO] TestAgent.server: Adding LAN server: server="Node-f4cfc005-1baa-50c1-28de-f22375595475 (Addr: tcp/127.0.0.1:15118) (DC: dc1)" -44:38.669 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f4cfc005-1baa-50c1-28de-f22375595475.dc1 area=wan -44:38.669 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp -44:38.669 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp -44:38.670 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp -44:38.670 [INFO] TestAgent: started state syncer -44:38.670 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp -44:38.717 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.717 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 -44:38.737 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.737 [DEBUG] TestAgent.server.raft: vote granted: from=40e4a748-2192-161a-0510-9bf59fe950b5 term=2 tally=1 -44:38.737 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.737 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" -44:38.737 [INFO] TestAgent.server: cluster leadership acquired -44:38.737 [INFO] TestAgent.server: New leader elected: payload=Node-f4cfc005-1baa-50c1-28de-f22375595475 -44:38.751 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15118 -44:38.751 [INFO] TestAgent: Synced node info -44:38.751 [DEBUG] TestAgent: Node info in sync -44:38.797 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.819 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.819 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.819 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.819 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.819 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.819 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f4cfc005-1baa-50c1-28de-f22375595475 -44:38.820 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f4cfc005-1baa-50c1-28de-f22375595475 -44:38.853 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.879 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.883 [INFO] TestAgent: Synced node info -44:38.889 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.899 [INFO] TestAgent: Synced node info -44:38.899 [INFO] TestAgent: Requesting shutdown -44:38.899 [INFO] TestAgent.server: shutting down server -44:38.899 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.899 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.899 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.899 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.899 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.899 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.899 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.899 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.899 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.899 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.909 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.918Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.918Z [INFO] agent.router.manager: shutting down -44:38.918 [INFO] TestAgent: consul server down -44:38.918 [INFO] TestAgent: shutdown complete -44:38.918 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp -44:38.918 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp -44:38.918 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp -44:39.418 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.418 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_NodeInfo (1.00s) -2022-11-07T22:44:38.496Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:38.496Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.522Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. -2022-11-07T22:44:38.522Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.553 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:81c5c614-45e7-df43-4a70-774adec8453b Address:127.0.0.1:15146}]" -44:38.553 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15146 [Follower]" leader= -44:38.554 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 127.0.0.1 -44:38.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-81c5c614-45e7-df43-4a70-774adec8453b 127.0.0.1 -2022-11-07T22:44:38.554Z [INFO] agent.router: Initializing LAN area manager -44:38.556 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 area=wan -44:38.556 [INFO] TestAgent.server: Adding LAN server: server="Node-81c5c614-45e7-df43-4a70-774adec8453b (Addr: tcp/127.0.0.1:15146) (DC: dc1)" -44:38.556 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15141 network=udp -44:38.557 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15141 network=tcp -44:38.559 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15142 network=tcp -44:38.559 [INFO] TestAgent: started state syncer -44:38.559 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15147 network=tcp -44:38.606 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:38.620 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.620 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15146 [Candidate]" term=2 -44:38.690 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.690 [DEBUG] TestAgent.server.raft: vote granted: from=81c5c614-45e7-df43-4a70-774adec8453b term=2 tally=1 -44:38.690 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.690 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15146 [Leader]" -44:38.690 [INFO] TestAgent.server: cluster leadership acquired -44:38.690 [INFO] TestAgent.server: New leader elected: payload=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.707 [INFO] TestAgent.server: initializing acls -44:38.707 [INFO] TestAgent.server: initializing acls -44:38.726 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:38.726 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:38.726 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:38.726 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:38.754 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:38.754 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:38.767 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:38.767 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:38.767 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:38.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 -44:38.767 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:38.767 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:38.767 [INFO] TestAgent.server: Updating LAN server: server="Node-81c5c614-45e7-df43-4a70-774adec8453b (Addr: tcp/127.0.0.1:15146) (DC: dc1)" -44:38.767 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 area=wan -44:38.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 -44:38.767 [INFO] TestAgent.server: Updating LAN server: server="Node-81c5c614-45e7-df43-4a70-774adec8453b (Addr: tcp/127.0.0.1:15146) (DC: dc1)" -44:38.767 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-81c5c614-45e7-df43-4a70-774adec8453b.dc1 area=wan -44:38.797 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.825 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.825 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.825 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.825 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.825 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.825 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.825 [INFO] TestAgent.server: member joined, marking health alive: member=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.842 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.842 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.849 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.851 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-81c5c614-45e7-df43-4a70-774adec8453b -44:38.863 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.863 [WARN] TestAgent: Node info update blocked by ACLs: node=81c5c614-45e7-df43-4a70-774adec8453b accessorID=00000000-0000-0000-0000-000000000002 -44:38.870 [INFO] TestAgent: Synced service: service=mysql -44:38.875 [INFO] TestAgent: Synced service: service=api -44:38.875 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.875 [WARN] TestAgent: Node info update blocked by ACLs: node=81c5c614-45e7-df43-4a70-774adec8453b accessorID=00000000-0000-0000-0000-000000000002 -44:38.878 [INFO] TestAgent: Synced service: service=mysql -44:38.881 [INFO] TestAgent: Synced service: service=api -44:38.881 [WARN] TestAgent: Check registration blocked by ACLs: check=mysql-check accessorID= -44:38.900 [INFO] TestAgent: Synced check: check=api-check -44:38.900 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=api-check -44:38.900 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth -44:38.900 [WARN] TestAgent: Node info update blocked by ACLs: node=81c5c614-45e7-df43-4a70-774adec8453b accessorID=00000000-0000-0000-0000-000000000002 -44:38.908 [INFO] TestAgent: Synced service: service=mysql -44:38.916 [INFO] TestAgent: Synced service: service=api -44:38.916 [WARN] TestAgent: Check registration blocked by ACLs: check=mysql-check accessorID= -44:38.919 [INFO] TestAgent: Deregistered check: check=api-check -44:38.919 [INFO] TestAgent: Requesting shutdown -44:38.920 [INFO] TestAgent.server: shutting down server -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.920 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:38.920 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.920 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.920 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.920 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:38.923 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.928Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.928Z [INFO] agent.router.manager: shutting down -44:38.928 [INFO] TestAgent: consul server down -44:38.928 [INFO] TestAgent: shutdown complete -44:38.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15141 network=tcp -44:38.928 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15141 network=udp -44:38.928 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15142 network=tcp -44:39.429 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.429 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Checks_ACLDeny (1.01s) -2022-11-07T22:44:38.524Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.718 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63639e55-ada6-041c-4b7d-46acc78682bd Address:127.0.0.1:15181}]" -44:38.718 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15181 [Follower]" leader= -44:38.720 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63639e55-ada6-041c-4b7d-46acc78682bd.dc1 127.0.0.1 -44:38.723 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63639e55-ada6-041c-4b7d-46acc78682bd 127.0.0.1 -2022-11-07T22:44:38.723Z [INFO] agent.router: Initializing LAN area manager -44:38.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15176 network=udp -44:38.724 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63639e55-ada6-041c-4b7d-46acc78682bd.dc1 area=wan -44:38.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15176 network=tcp -44:38.724 [INFO] TestAgent.server: Adding LAN server: server="Node-63639e55-ada6-041c-4b7d-46acc78682bd (Addr: tcp/127.0.0.1:15181) (DC: dc1)" -44:38.726 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15177 network=tcp -44:38.726 [INFO] TestAgent: started state syncer -44:38.726 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15182 network=tcp -44:38.757 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.757 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15181 [Candidate]" term=2 -44:38.776 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.776 [DEBUG] TestAgent.server.raft: vote granted: from=63639e55-ada6-041c-4b7d-46acc78682bd term=2 tally=1 -44:38.776 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.776 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15181 [Leader]" -44:38.777 [INFO] TestAgent.server: cluster leadership acquired -44:38.777 [INFO] TestAgent.server: New leader elected: payload=Node-63639e55-ada6-041c-4b7d-46acc78682bd -44:38.787 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15181 -44:38.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.859 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.859 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.859 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.859 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.859 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.859 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63639e55-ada6-041c-4b7d-46acc78682bd -44:38.859 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63639e55-ada6-041c-4b7d-46acc78682bd -44:38.881 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.889 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.893 [INFO] TestAgent: Synced node info -44:38.908 [INFO] TestAgent: Synced service: service=mysql -44:38.908 [DEBUG] TestAgent: Check in sync: check=mysql -44:38.908 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.909 [DEBUG] TestAgent: Node info in sync -44:38.915 [INFO] TestAgent: Synced service: service=mysql -44:38.922 [INFO] TestAgent: Synced service: service=redis -44:38.922 [DEBUG] TestAgent: Check in sync: check=mysql -44:38.922 [DEBUG] TestAgent: Check in sync: check=redis:1 -44:38.922 [DEBUG] TestAgent: Check in sync: check=redis:2 -44:38.923 [INFO] TestAgent: Requesting shutdown -44:38.923 [INFO] TestAgent.server: shutting down server -44:38.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:38.923 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:38.923 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:38.923 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:38.923 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:38.923 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:38.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:38.923 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:38.923 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:38.923 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:38.923 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:38.926 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:38.930Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:38.930Z [INFO] agent.router.manager: shutting down -44:38.930 [INFO] TestAgent: consul server down -44:38.930 [INFO] TestAgent: shutdown complete -44:38.930 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15176 network=tcp -44:38.930 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15176 network=udp -44:38.930 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15177 network=tcp -44:39.432 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.432 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Services_WithChecks (1.00s) -2022-11-07T22:44:38.524Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.718 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba Address:127.0.0.1:15174}]" -44:38.718 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15174 [Follower]" leader= -44:38.718 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba.dc1 127.0.0.1 -44:38.723 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba 127.0.0.1 -2022-11-07T22:44:38.724Z [INFO] agent.router: Initializing LAN area manager -44:38.724 [INFO] TestAgent.server: Adding LAN server: server="Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba (Addr: tcp/127.0.0.1:15174) (DC: dc1)" -44:38.724 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba.dc1 area=wan -44:38.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15169 network=udp -44:38.724 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15169 network=tcp -44:38.726 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15170 network=tcp -44:38.726 [INFO] TestAgent: started state syncer -44:38.726 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15175 network=tcp -44:38.757 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.757 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15174 [Candidate]" term=2 -44:38.780 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.780 [DEBUG] TestAgent.server.raft: vote granted: from=3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba term=2 tally=1 -44:38.780 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.780 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15174 [Leader]" -44:38.780 [INFO] TestAgent.server: cluster leadership acquired -44:38.780 [INFO] TestAgent.server: New leader elected: payload=Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba -44:38.796 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15174 -44:38.820 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.855 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.855 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.855 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.855 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.855 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.855 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba -44:38.855 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3cc6ed63-e980-a59b-bc88-1f3bbe1a9cba -44:38.874 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.899 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.906 [INFO] TestAgent: Synced node info -44:38.944 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.944 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.944 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.944 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.948 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.948 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.948 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.948 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.952 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.952 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.952 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.952 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -44:38.956 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.969 [INFO] TestAgent: Synced node info -44:38.969 [DEBUG] TestAgent: Service in sync: service=mysql-proxy -44:38.979 [INFO] TestAgent: Synced service: service=redis-proxy -44:38.984 [INFO] TestAgent: Synced service: service=web-proxy -44:38.999 [INFO] TestAgent: Synced service: service=cache-proxy -44:39.003 [INFO] TestAgent: Deregistered service: service=lb-proxy -44:39.004 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.004 [DEBUG] TestAgent: Node info in sync -44:39.004 [DEBUG] TestAgent: Service in sync: service=mysql-proxy -44:39.004 [DEBUG] TestAgent: Service in sync: service=redis-proxy -44:39.004 [DEBUG] TestAgent: Service in sync: service=web-proxy -44:39.016 [INFO] TestAgent: Deregistered service: service=cache-proxy -44:39.016 [INFO] TestAgent: Requesting shutdown -44:39.016 [INFO] TestAgent.server: shutting down server -44:39.016 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.016 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.016 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.016 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.016 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.016 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.016 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.016 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.016 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.020 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.027Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.027Z [INFO] agent.router.manager: shutting down -44:39.027 [INFO] TestAgent: consul server down -44:39.027 [INFO] TestAgent: shutdown complete -44:39.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15169 network=tcp -44:39.027 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15169 network=udp -44:39.027 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15170 network=tcp -44:39.527 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.527 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Services_ConnectProxy (1.10s) -2022-11-07T22:44:38.502Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.530Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.594 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 Address:127.0.0.1:15111}]" -44:38.595 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= -44:38.595 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2.dc1 127.0.0.1 -44:38.596 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 127.0.0.1 -2022-11-07T22:44:38.596Z [INFO] agent.router: Initializing LAN area manager -44:38.596 [INFO] TestAgent.server: Adding LAN server: server="Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" -44:38.596 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp -44:38.597 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2.dc1 area=wan -44:38.597 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp -44:38.598 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp -44:38.598 [INFO] TestAgent: started state syncer -44:38.599 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp -44:38.648 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.648 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 -44:38.689 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.689 [DEBUG] TestAgent.server.raft: vote granted: from=91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 term=2 tally=1 -44:38.689 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.689 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" -44:38.689 [INFO] TestAgent.server: cluster leadership acquired -44:38.689 [INFO] TestAgent.server: New leader elected: payload=Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 -44:38.706 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 -44:38.733 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.777 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.777 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.777 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.777 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.777 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.777 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 -44:38.777 [INFO] TestAgent.server: member joined, marking health alive: member=Node-91bf4bfc-038d-e4f2-3f24-9ea9732db4a2 -44:38.793 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.926 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.930 [INFO] TestAgent: Synced node info -44:38.952 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.952 [DEBUG] TestAgent: Node info in sync -44:38.952 [DEBUG] TestAgent: Node info in sync -44:38.962 [DEBUG] TestAgent: Node info in sync -44:38.977 [INFO] TestAgent: Synced service: service=mysql -44:38.977 [DEBUG] TestAgent: Node info in sync -44:38.991 [INFO] TestAgent: Synced service: service=redis -44:38.991 [DEBUG] TestAgent: Service in sync: service=mysql -44:38.991 [DEBUG] TestAgent: Node info in sync -44:38.991 [DEBUG] TestAgent: Service in sync: service=mysql -44:38.991 [DEBUG] TestAgent: Service in sync: service=redis -44:39.000 [INFO] TestAgent: Synced service: service=web -44:39.000 [DEBUG] TestAgent: Node info in sync -44:39.000 [DEBUG] TestAgent: Service in sync: service=mysql -44:39.000 [DEBUG] TestAgent: Service in sync: service=redis -44:39.000 [DEBUG] TestAgent: Service in sync: service=web -44:39.012 [INFO] TestAgent: Synced service: service=api -44:39.012 [DEBUG] TestAgent: Node info in sync -44:39.012 [DEBUG] TestAgent: Service in sync: service=mysql -44:39.012 [DEBUG] TestAgent: Service in sync: service=redis -44:39.012 [DEBUG] TestAgent: Service in sync: service=web -44:39.012 [DEBUG] TestAgent: Service in sync: service=api -44:39.012 [DEBUG] TestAgent: Service in sync: service=cache -44:39.012 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.019 [INFO] TestAgent: Synced node info -44:39.019 [DEBUG] TestAgent: Service in sync: service=web -44:39.019 [DEBUG] TestAgent: Service in sync: service=api -44:39.030 [INFO] TestAgent: Synced service: service=cache -44:39.033 [INFO] TestAgent: Deregistered service: service=lb -44:39.033 [DEBUG] TestAgent: Service in sync: service=mysql -44:39.033 [DEBUG] TestAgent: Service in sync: service=redis -44:39.034 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.038 [INFO] TestAgent: Synced node info -44:39.038 [DEBUG] TestAgent: Service in sync: service=mysql -44:39.038 [DEBUG] TestAgent: Service in sync: service=redis -44:39.038 [DEBUG] TestAgent: Service in sync: service=web -44:39.041 [INFO] TestAgent: Deregistered service: service=api -44:39.041 [DEBUG] TestAgent: Service in sync: service=cache -44:39.041 [INFO] TestAgent: Requesting shutdown -44:39.041 [DEBUG] TestAgent: Node info in sync -44:39.041 [DEBUG] TestAgent: Service in sync: service=mysql -44:39.041 [DEBUG] TestAgent: Service in sync: service=redis -44:39.041 [DEBUG] TestAgent: Service in sync: service=web -44:39.041 [DEBUG] TestAgent: Service in sync: service=cache -44:39.041 [INFO] TestAgent.server: shutting down server -44:39.041 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.041 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.041 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.041 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.041 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.041 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.041 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.041 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.041 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.045 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.049Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.049Z [INFO] agent.router.manager: shutting down -44:39.049 [INFO] TestAgent: consul server down -44:39.049 [INFO] TestAgent: shutdown complete -44:39.049 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp -44:39.049 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp -44:39.049 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp -44:39.549 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.549 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Services (1.13s) -2022-11-07T22:44:38.523Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.543Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.718 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:882b7243-94d0-ac87-f1d1-416d489652f7 Address:127.0.0.1:15202}]" -44:38.718 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15202 [Follower]" leader= -44:38.719 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-882b7243-94d0-ac87-f1d1-416d489652f7.dc1 127.0.0.1 -44:38.725 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-882b7243-94d0-ac87-f1d1-416d489652f7 127.0.0.1 -2022-11-07T22:44:38.725Z [INFO] agent.router: Initializing LAN area manager -44:38.726 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-882b7243-94d0-ac87-f1d1-416d489652f7.dc1 area=wan -44:38.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15197 network=udp -44:38.726 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15197 network=tcp -44:38.726 [INFO] TestAgent.server: Adding LAN server: server="Node-882b7243-94d0-ac87-f1d1-416d489652f7 (Addr: tcp/127.0.0.1:15202) (DC: dc1)" -44:38.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15198 network=tcp -44:38.727 [INFO] TestAgent: started state syncer -44:38.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15203 network=tcp -44:38.786 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.786 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15202 [Candidate]" term=2 -44:38.805 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.805 [DEBUG] TestAgent.server.raft: vote granted: from=882b7243-94d0-ac87-f1d1-416d489652f7 term=2 tally=1 -44:38.805 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.805 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15202 [Leader]" -44:38.805 [INFO] TestAgent.server: cluster leadership acquired -44:38.805 [INFO] TestAgent.server: New leader elected: payload=Node-882b7243-94d0-ac87-f1d1-416d489652f7 -44:38.816 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15202 -44:38.842 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.879 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.879 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.879 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.879 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.879 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.879 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-882b7243-94d0-ac87-f1d1-416d489652f7 -44:38.879 [INFO] TestAgent.server: member joined, marking health alive: member=Node-882b7243-94d0-ac87-f1d1-416d489652f7 -44:38.901 [INFO] TestAgent: Synced node info -44:38.910 [INFO] TestAgent.server: federation state anti-entropy synced -44:39.055 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.057 [INFO] TestAgent: Synced node info -44:39.060 [INFO] TestAgent: Deregistered service: service=deleted-check-service -44:39.063 [INFO] TestAgent: Deregistered check: check=lb -44:39.063 [INFO] TestAgent: Requesting shutdown -44:39.063 [INFO] TestAgent.server: shutting down server -44:39.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.063 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.063 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.063 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.063 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.063 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.063 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.066 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.069Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.069Z [INFO] agent.router.manager: shutting down -44:39.069 [INFO] TestAgent: consul server down -44:39.069 [INFO] TestAgent: shutdown complete -44:39.069 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15197 network=tcp -44:39.069 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15197 network=udp -44:39.069 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15198 network=tcp -44:39.569 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.569 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_RemovingServiceAndCheck (1.12s) -2022-11-07T22:44:38.526Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.548Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.718 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e582c2e3-f1db-7d23-1ef0-c8b813b6dddd Address:127.0.0.1:15167}]" -44:38.718 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15167 [Follower]" leader= -44:38.724 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd.dc1 127.0.0.1 -44:38.726 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd 127.0.0.1 -2022-11-07T22:44:38.726Z [INFO] agent.router: Initializing LAN area manager -44:38.727 [INFO] TestAgent.server: Adding LAN server: server="Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd (Addr: tcp/127.0.0.1:15167) (DC: dc1)" -44:38.727 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd.dc1 area=wan -44:38.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15162 network=udp -44:38.727 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15162 network=tcp -44:38.727 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15163 network=tcp -44:38.727 [INFO] TestAgent: started state syncer -44:38.727 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15168 network=tcp -44:38.786 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.786 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15167 [Candidate]" term=2 -44:38.801 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.801 [DEBUG] TestAgent.server.raft: vote granted: from=e582c2e3-f1db-7d23-1ef0-c8b813b6dddd term=2 tally=1 -44:38.801 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.801 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15167 [Leader]" -44:38.802 [INFO] TestAgent.server: cluster leadership acquired -44:38.802 [INFO] TestAgent.server: New leader elected: payload=Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd -44:38.808 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15167 -44:38.826 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.875 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.875 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.875 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.875 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.875 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.876 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd -44:38.876 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e582c2e3-f1db-7d23-1ef0-c8b813b6dddd -44:38.901 [INFO] TestAgent.server: federation state anti-entropy synced -44:39.059 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:39.062 [INFO] TestAgent: Synced node info -44:39.062 [DEBUG] TestAgent: Node info in sync -44:39.170 [INFO] TestAgent: Requesting shutdown -44:39.170 [INFO] TestAgent.server: shutting down server -44:39.170 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.170 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.170 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.170 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.170 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.170 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.170 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.170 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.170 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.174 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.182Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.182Z [INFO] agent.router.manager: shutting down -44:39.182 [INFO] TestAgent: consul server down -44:39.182 [INFO] TestAgent: shutdown complete -44:39.182 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15162 network=tcp -44:39.182 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15162 network=udp -44:39.182 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15163 network=tcp -44:39.683 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.683 [INFO] TestAgent: Endpoints down ---- PASS: TestAgent_ServiceWatchCh (1.25s) -2022-11-07T22:44:38.514Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.542Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.680 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b19beaf6-1471-233f-e368-2b4a3932c646 Address:127.0.0.1:15132}]" -44:38.680 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15132 [Follower]" leader= -44:38.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b19beaf6-1471-233f-e368-2b4a3932c646.dc1 127.0.0.1 -44:38.681 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b19beaf6-1471-233f-e368-2b4a3932c646 127.0.0.1 -2022-11-07T22:44:38.681Z [INFO] agent.router: Initializing LAN area manager -44:38.681 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b19beaf6-1471-233f-e368-2b4a3932c646.dc1 area=wan -44:38.681 [INFO] TestAgent.server: Adding LAN server: server="Node-b19beaf6-1471-233f-e368-2b4a3932c646 (Addr: tcp/127.0.0.1:15132) (DC: dc1)" -44:38.681 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=udp -44:38.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=tcp -44:38.719 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15128 network=tcp -44:38.719 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15133 network=tcp -44:38.722 [INFO] TestAgent: started state syncer -44:38.734 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.734 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15132 [Candidate]" term=2 -44:38.749 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.749 [DEBUG] TestAgent.server.raft: vote granted: from=b19beaf6-1471-233f-e368-2b4a3932c646 term=2 tally=1 -44:38.749 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.749 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15132 [Leader]" -44:38.749 [INFO] TestAgent.server: cluster leadership acquired -44:38.750 [INFO] TestAgent.server: New leader elected: payload=Node-b19beaf6-1471-233f-e368-2b4a3932c646 -44:38.759 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15132 -44:38.786 [INFO] TestAgent: Synced node info -44:38.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.829 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.829 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.829 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.829 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.829 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.830 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b19beaf6-1471-233f-e368-2b4a3932c646 -44:38.830 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b19beaf6-1471-233f-e368-2b4a3932c646 -44:38.842 [INFO] TestAgent.server: federation state anti-entropy synced -44:39.171 [INFO] TestAgent: Requesting shutdown -44:39.171 [INFO] TestAgent.server: shutting down server -44:39.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:39.171 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:39.171 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:39.171 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:39.171 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:39.171 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:39.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:39.171 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:39.171 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:39.171 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:39.176 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:39.185Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:39.185Z [INFO] agent.router.manager: shutting down -44:39.186 [INFO] TestAgent: consul server down -44:39.186 [INFO] TestAgent: shutdown complete -44:39.186 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=tcp -44:39.186 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=udp -44:39.186 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15128 network=tcp -44:39.686 [INFO] TestAgent: Waiting for endpoints to shut down -44:39.686 [INFO] TestAgent: Endpoints down ---- PASS: TestAliasNotifications_local (1.26s) -2022-11-07T22:44:38.509Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:38.542Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:38.680 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7f4d88c2-4c3c-b833-8d11-bf7fd495e747 Address:127.0.0.1:15153}]" -44:38.680 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15153 [Follower]" leader= -44:38.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747.dc1 127.0.0.1 -44:38.681 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747 127.0.0.1 -2022-11-07T22:44:38.681Z [INFO] agent.router: Initializing LAN area manager -44:38.681 [INFO] TestAgent.server: Adding LAN server: server="Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747 (Addr: tcp/127.0.0.1:15153) (DC: dc1)" -44:38.681 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747.dc1 area=wan -44:38.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15148 network=tcp -44:38.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15148 network=udp -44:38.718 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15149 network=tcp -44:38.718 [INFO] TestAgent: started state syncer -44:38.718 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15154 network=tcp -44:38.747 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:38.747 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15153 [Candidate]" term=2 -44:38.762 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:38.762 [DEBUG] TestAgent.server.raft: vote granted: from=7f4d88c2-4c3c-b833-8d11-bf7fd495e747 term=2 tally=1 -44:38.762 [INFO] TestAgent.server.raft: election won: tally=1 -44:38.762 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15153 [Leader]" -44:38.762 [INFO] TestAgent.server: cluster leadership acquired -44:38.763 [INFO] TestAgent.server: New leader elected: payload=Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747 -44:38.780 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15153 -44:38.803 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:38.853 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:38.853 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:38.853 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:38.853 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:38.853 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:38.853 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747 -44:38.853 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7f4d88c2-4c3c-b833-8d11-bf7fd495e747 -44:38.875 [INFO] TestAgent.server: federation state anti-entropy synced -44:38.885 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.894 [INFO] TestAgent: Synced node info -44:38.894 [DEBUG] TestAgent: Node info in sync -44:38.898 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:38.898 [DEBUG] TestAgent: Node info in sync -44:38.903 [INFO] TestAgent: Synced check: check=web -44:40.985 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:40.985 [DEBUG] TestAgent: Node info in sync -44:40.991 [INFO] TestAgent: Synced check: check=web -44:40.991 [DEBUG] TestAgent: Node info in sync -44:40.991 [DEBUG] TestAgent: Check in sync: check=web -44:41.064 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:41.067 [INFO] TestAgent: Synced node info -44:41.071 [INFO] TestAgent: Synced check: check=web -44:41.076 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:41.078 [INFO] TestAgent: Synced node info -44:41.078 [DEBUG] TestAgent: Check in sync: check=web -44:41.593 [DEBUG] TestAgent: Node info in sync -44:41.599 [INFO] TestAgent: Synced check: check=web -44:41.609 [INFO] TestAgent: Requesting shutdown -44:41.609 [INFO] TestAgent.server: shutting down server -44:41.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:41.609 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:41.609 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:41.609 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:41.609 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:41.609 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:41.610 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:41.610 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:41.610 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:41.610 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:41.613 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:41.617Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:41.617Z [INFO] agent.router.manager: shutting down -44:41.617 [INFO] TestAgent: consul server down -44:41.617 [INFO] TestAgent: shutdown complete -44:41.617 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15148 network=tcp -44:41.617 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15148 network=udp -44:41.617 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15149 network=tcp -44:42.117 [INFO] TestAgent: Waiting for endpoints to shut down -44:42.117 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentAntiEntropy_Check_DeferSync (3.70s) +=== CONT TestAgent_ServiceTokens +--- PASS: TestAgent_AddCheckFailure (0.01s) +2021-10-05T16:29:38.989Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.156Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.274 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f6629171-7f17-0c89-7f20-65ac847d377e Address:127.0.0.1:27377}]" +29:39.274 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27377 [Follower]" leader= +29:39.275 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f6629171-7f17-0c89-7f20-65ac847d377e.dc1 127.0.0.1 +29:39.276 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f6629171-7f17-0c89-7f20-65ac847d377e 127.0.0.1 +2021-10-05T16:29:39.276Z [INFO] agent.router: Initializing LAN area manager +29:39.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27372 network=udp +29:39.277 [INFO] TestAgent.server: Adding LAN server: server="Node-f6629171-7f17-0c89-7f20-65ac847d377e (Addr: tcp/127.0.0.1:27377) (DC: dc1)" +29:39.277 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f6629171-7f17-0c89-7f20-65ac847d377e.dc1 area=wan +29:39.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27372 network=tcp +29:39.278 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27373 network=tcp +29:39.278 [INFO] TestAgent: started state syncer +29:39.278 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27378 network=tcp +29:39.316 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.316 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27377 [Candidate]" term=2 +29:39.352 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.352 [DEBUG] TestAgent.server.raft: vote granted: from=f6629171-7f17-0c89-7f20-65ac847d377e term=2 tally=1 +29:39.352 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.352 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27377 [Leader]" +29:39.357 [INFO] TestAgent.server: cluster leadership acquired +29:39.357 [INFO] TestAgent.server: New leader elected: payload=Node-f6629171-7f17-0c89-7f20-65ac847d377e +29:39.390 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27377 +29:39.418 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.498 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.498 [INFO] TestAgent: Synced node info +29:39.498 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.498 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.498 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.498 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.498 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f6629171-7f17-0c89-7f20-65ac847d377e +29:39.498 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f6629171-7f17-0c89-7f20-65ac847d377e +29:39.617 [INFO] TestAgent: Requesting shutdown +29:39.617 [INFO] TestAgent.server: shutting down server +29:39.617 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.617 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.617 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.617 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.617 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.617 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +29:39.617 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.617 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.617 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.617 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.624 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.627Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.627Z [INFO] agent.router.manager: shutting down +29:39.627 [INFO] TestAgent: consul server down +29:39.627 [INFO] TestAgent: shutdown complete +29:39.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27372 network=tcp +29:39.627 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27372 network=udp +29:39.627 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27373 network=tcp +29:40.136 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.136 [INFO] TestAgent: Endpoints down +--- PASS: TestAliasNotifications_local (1.43s) +--- PASS: TestAgent_AliasCheck (0.05s) +--- PASS: TestAgent_ServiceTokens (0.05s) +2021-10-05T16:29:38.937Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:38.937Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.092Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:39.092Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.248 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:85e2152d-50ce-5986-6fbe-98df93f8d680 Address:127.0.0.1:27412}]" +29:39.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27412 [Follower]" leader= +29:39.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 127.0.0.1 +29:39.250 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-85e2152d-50ce-5986-6fbe-98df93f8d680 127.0.0.1 +2021-10-05T16:29:39.250Z [INFO] agent.router: Initializing LAN area manager +29:39.251 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27407 network=udp +29:39.251 [INFO] TestAgent.server: Adding LAN server: server="Node-85e2152d-50ce-5986-6fbe-98df93f8d680 (Addr: tcp/127.0.0.1:27412) (DC: dc1)" +29:39.251 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 area=wan +29:39.251 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27407 network=tcp +29:39.260 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27408 network=tcp +29:39.260 [INFO] TestAgent: started state syncer +29:39.272 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27413 network=tcp +29:39.299 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.299 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27412 [Candidate]" term=2 +29:39.302 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:39.350 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.350 [DEBUG] TestAgent.server.raft: vote granted: from=85e2152d-50ce-5986-6fbe-98df93f8d680 term=2 tally=1 +29:39.350 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.350 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27412 [Leader]" +29:39.357 [INFO] TestAgent.server: cluster leadership acquired +29:39.357 [INFO] TestAgent.server: New leader elected: payload=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.380 [INFO] TestAgent.server: initializing acls +29:39.398 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:39.398 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:39.403 [INFO] TestAgent.server: initializing acls +29:39.403 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:39.417 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:39.426 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +29:39.440 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:39.441 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:39.441 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:39.441 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:39.441 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.441 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 +29:39.441 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 area=wan +29:39.441 [INFO] TestAgent.server: Updating LAN server: server="Node-85e2152d-50ce-5986-6fbe-98df93f8d680 (Addr: tcp/127.0.0.1:27412) (DC: dc1)" +29:39.476 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:39.476 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:39.476 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.476 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 +29:39.476 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-85e2152d-50ce-5986-6fbe-98df93f8d680.dc1 area=wan +29:39.476 [INFO] TestAgent.server: Updating LAN server: server="Node-85e2152d-50ce-5986-6fbe-98df93f8d680 (Addr: tcp/127.0.0.1:27412) (DC: dc1)" +29:39.510 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.562 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.562 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.562 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.562 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.562 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.562 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.562 [INFO] TestAgent.server: member joined, marking health alive: member=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.572 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.572 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.584 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.596 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-85e2152d-50ce-5986-6fbe-98df93f8d680 +29:39.609 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +29:39.609 [WARN] TestAgent: Node info update blocked by ACLs: node=85e2152d-50ce-5986-6fbe-98df93f8d680 accessorID=00000000-0000-0000-0000-000000000002 +29:39.616 [INFO] TestAgent: Synced service: service=mysql +29:39.622 [INFO] TestAgent: Synced service: service=api +29:39.623 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +29:39.623 [WARN] TestAgent: Node info update blocked by ACLs: node=85e2152d-50ce-5986-6fbe-98df93f8d680 accessorID=00000000-0000-0000-0000-000000000002 +29:39.639 [INFO] TestAgent: Synced service: service=mysql +29:39.650 [INFO] TestAgent: Synced service: service=api +29:39.668 [INFO] TestAgent: Synced check: check=api-check +29:39.669 [WARN] TestAgent: Check registration blocked by ACLs: check=mysql-check accessorID= +29:39.669 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=api-check +29:39.669 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +29:39.669 [WARN] TestAgent: Node info update blocked by ACLs: node=85e2152d-50ce-5986-6fbe-98df93f8d680 accessorID=00000000-0000-0000-0000-000000000002 +29:39.684 [INFO] TestAgent: Synced service: service=mysql +29:39.708 [INFO] TestAgent: Synced service: service=api +29:39.708 [WARN] TestAgent: Check registration blocked by ACLs: check=mysql-check accessorID= +29:39.722 [INFO] TestAgent: Deregistered check: check=api-check +29:39.722 [INFO] TestAgent: Requesting shutdown +29:39.722 [INFO] TestAgent.server: shutting down server +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:39.722 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:39.722 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:39.722 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:39.722 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:39.729 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:39.735Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:39.735Z [INFO] agent.router.manager: shutting down +29:39.735 [INFO] TestAgent: consul server down +29:39.735 [INFO] TestAgent: shutdown complete +29:39.735 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27407 network=tcp +29:39.735 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27407 network=udp +29:39.736 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27408 network=tcp +29:40.236 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.236 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Checks_ACLDeny (1.52s) +2021-10-05T16:29:40.092Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.115Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.174 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:706b756e-f165-1681-5c72-bfe10f8308bc Address:127.0.0.1:27447}]" +29:40.174 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27447 [Follower]" leader= +29:40.175 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-706b756e-f165-1681-5c72-bfe10f8308bc.dc1 127.0.0.1 +29:40.175 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-706b756e-f165-1681-5c72-bfe10f8308bc 127.0.0.1 +2021-10-05T16:29:40.175Z [INFO] agent.router: Initializing LAN area manager +29:40.175 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27442 network=udp +29:40.176 [INFO] TestAgent.server: Adding LAN server: server="Node-706b756e-f165-1681-5c72-bfe10f8308bc (Addr: tcp/127.0.0.1:27447) (DC: dc1)" +29:40.176 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-706b756e-f165-1681-5c72-bfe10f8308bc.dc1 area=wan +29:40.176 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27442 network=tcp +29:40.176 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27443 network=tcp +29:40.176 [INFO] TestAgent: started state syncer +29:40.176 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27448 network=tcp +29:40.237 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:40.237 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27447 [Candidate]" term=2 +29:40.259 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:40.259 [DEBUG] TestAgent.server.raft: vote granted: from=706b756e-f165-1681-5c72-bfe10f8308bc term=2 tally=1 +29:40.259 [INFO] TestAgent.server.raft: election won: tally=1 +29:40.259 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27447 [Leader]" +29:40.260 [INFO] TestAgent.server: cluster leadership acquired +29:40.260 [INFO] TestAgent.server: New leader elected: payload=Node-706b756e-f165-1681-5c72-bfe10f8308bc +29:40.270 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27447 +29:40.308 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.380 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.380 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.380 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.380 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.380 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.380 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-706b756e-f165-1681-5c72-bfe10f8308bc +29:40.380 [INFO] TestAgent.server: member joined, marking health alive: member=Node-706b756e-f165-1681-5c72-bfe10f8308bc +29:40.406 [INFO] TestAgent: Requesting shutdown +29:40.406 [INFO] TestAgent.server: shutting down server +29:40.406 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.406 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.406 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.406 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.406 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.406 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:40.406 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.406 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.406 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.416 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +29:40.420 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.420 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:29:40.430Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.430Z [INFO] agent.router.manager: shutting down +29:40.430 [INFO] TestAgent: consul server down +29:40.430 [INFO] TestAgent: shutdown complete +29:40.430 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27442 network=tcp +29:40.430 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27442 network=udp +29:40.430 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27443 network=tcp +29:40.930 [INFO] TestAgent: Waiting for endpoints to shut down +29:40.930 [INFO] TestAgent: Endpoints down +--- PASS: TestAgent_ServiceWatchCh (0.87s) +2021-10-05T16:29:40.067Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.094Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.195 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2df35846-e044-bb03-f228-ec1822c3ab51 Address:127.0.0.1:27440}]" +29:40.205 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2df35846-e044-bb03-f228-ec1822c3ab51.dc1 127.0.0.1 +29:40.207 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2df35846-e044-bb03-f228-ec1822c3ab51 127.0.0.1 +2021-10-05T16:29:40.207Z [INFO] agent.router: Initializing LAN area manager +29:40.214 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27440 [Follower]" leader= +29:40.215 [INFO] TestAgent.server: Adding LAN server: server="Node-2df35846-e044-bb03-f228-ec1822c3ab51 (Addr: tcp/127.0.0.1:27440) (DC: dc1)" +29:40.215 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2df35846-e044-bb03-f228-ec1822c3ab51.dc1 area=wan +29:40.215 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27435 network=tcp +29:40.217 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27435 network=udp +29:40.218 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27436 network=tcp +29:40.218 [INFO] TestAgent: started state syncer +29:40.218 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27441 network=tcp +29:40.264 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:40.264 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27440 [Candidate]" term=2 +29:40.316 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:40.316 [DEBUG] TestAgent.server.raft: vote granted: from=2df35846-e044-bb03-f228-ec1822c3ab51 term=2 tally=1 +29:40.316 [INFO] TestAgent.server.raft: election won: tally=1 +29:40.316 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27440 [Leader]" +29:40.320 [INFO] TestAgent.server: cluster leadership acquired +29:40.320 [INFO] TestAgent.server: New leader elected: payload=Node-2df35846-e044-bb03-f228-ec1822c3ab51 +29:40.347 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27440 +29:40.378 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.444 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.444 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.444 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.444 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.444 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.444 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2df35846-e044-bb03-f228-ec1822c3ab51 +29:40.444 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2df35846-e044-bb03-f228-ec1822c3ab51 +29:40.476 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.484 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.499 [INFO] TestAgent: Synced node info +29:40.513 [INFO] TestAgent: Synced service: service=mysql +29:40.513 [DEBUG] TestAgent: Check in sync: check=mysql +29:40.514 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.514 [DEBUG] TestAgent: Node info in sync +29:40.538 [INFO] TestAgent: Synced service: service=mysql +29:40.549 [INFO] TestAgent: Synced service: service=redis +29:40.549 [DEBUG] TestAgent: Check in sync: check=mysql +29:40.549 [DEBUG] TestAgent: Check in sync: check=redis:1 +29:40.549 [DEBUG] TestAgent: Check in sync: check=redis:2 +29:40.549 [INFO] TestAgent: Requesting shutdown +29:40.549 [INFO] TestAgent.server: shutting down server +29:40.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.549 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.549 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.549 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.549 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.549 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.558 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.569Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.569Z [INFO] agent.router.manager: shutting down +29:40.569 [INFO] TestAgent: consul server down +29:40.569 [INFO] TestAgent: shutdown complete +29:40.569 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27435 network=tcp +29:40.569 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27435 network=udp +29:40.569 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27436 network=tcp +29:41.069 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.069 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Services_WithChecks (1.05s) +2021-10-05T16:29:40.161Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.182Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.261 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9853ddfc-c2d0-0830-e75b-5fa1e39947fe Address:127.0.0.1:27454}]" +29:40.261 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27454 [Follower]" leader= +29:40.262 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe.dc1 127.0.0.1 +29:40.263 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe 127.0.0.1 +2021-10-05T16:29:40.263Z [INFO] agent.router: Initializing LAN area manager +29:40.263 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27449 network=udp +29:40.263 [INFO] TestAgent.server: Adding LAN server: server="Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe (Addr: tcp/127.0.0.1:27454) (DC: dc1)" +29:40.264 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe.dc1 area=wan +29:40.264 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27449 network=tcp +29:40.265 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27450 network=tcp +29:40.265 [INFO] TestAgent: started state syncer +29:40.265 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27455 network=tcp +29:40.332 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:40.332 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27454 [Candidate]" term=2 +29:40.360 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:40.360 [DEBUG] TestAgent.server.raft: vote granted: from=9853ddfc-c2d0-0830-e75b-5fa1e39947fe term=2 tally=1 +29:40.360 [INFO] TestAgent.server.raft: election won: tally=1 +29:40.360 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27454 [Leader]" +29:40.361 [INFO] TestAgent.server: cluster leadership acquired +29:40.361 [INFO] TestAgent.server: New leader elected: payload=Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe +29:40.379 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27454 +29:40.428 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.472 [INFO] TestAgent: Synced node info +29:40.488 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.488 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.488 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.488 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.488 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.488 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe +29:40.488 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9853ddfc-c2d0-0830-e75b-5fa1e39947fe +29:40.509 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.509 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.509 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.509 [ERROR] TestAgent.proxycfg: watch error: service_id=mysql-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.532 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.542 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.542 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.542 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.542 [ERROR] TestAgent.proxycfg: watch error: service_id=redis-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.564 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.564 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.564 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.564 [ERROR] TestAgent.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +29:40.589 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.610 [INFO] TestAgent: Synced node info +29:40.614 [INFO] TestAgent: Deregistered service: service=lb-proxy +29:40.614 [DEBUG] TestAgent: Service in sync: service=mysql-proxy +29:40.621 [INFO] TestAgent: Synced service: service=redis-proxy +29:40.626 [INFO] TestAgent: Synced service: service=web-proxy +29:40.632 [INFO] TestAgent: Synced service: service=cache-proxy +29:40.632 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.632 [DEBUG] TestAgent: Node info in sync +29:40.632 [DEBUG] TestAgent: Service in sync: service=mysql-proxy +29:40.632 [DEBUG] TestAgent: Service in sync: service=redis-proxy +29:40.632 [DEBUG] TestAgent: Service in sync: service=web-proxy +29:40.648 [INFO] TestAgent: Deregistered service: service=cache-proxy +29:40.648 [INFO] TestAgent: Requesting shutdown +29:40.649 [INFO] TestAgent.server: shutting down server +29:40.649 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.649 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.649 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.649 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.649 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.649 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.649 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.649 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.649 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.649 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.653 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.658Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.658Z [INFO] agent.router.manager: shutting down +29:40.658 [INFO] TestAgent: consul server down +29:40.658 [INFO] TestAgent: shutdown complete +29:40.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27449 network=tcp +29:40.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27449 network=udp +29:40.658 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27450 network=tcp +29:41.160 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.160 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Services_ConnectProxy (1.08s) +2021-10-05T16:29:38.919Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:39.155Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:39.262 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a95eb5d9-e700-f655-1b10-115b4dae5836 Address:127.0.0.1:27398}]" +29:39.263 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a95eb5d9-e700-f655-1b10-115b4dae5836.dc1 127.0.0.1 +29:39.273 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a95eb5d9-e700-f655-1b10-115b4dae5836 127.0.0.1 +2021-10-05T16:29:39.273Z [INFO] agent.router: Initializing LAN area manager +29:39.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27393 network=udp +29:39.273 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27398 [Follower]" leader= +29:39.276 [INFO] TestAgent.server: Adding LAN server: server="Node-a95eb5d9-e700-f655-1b10-115b4dae5836 (Addr: tcp/127.0.0.1:27398) (DC: dc1)" +29:39.276 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a95eb5d9-e700-f655-1b10-115b4dae5836.dc1 area=wan +29:39.276 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27393 network=tcp +29:39.277 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27394 network=tcp +29:39.277 [INFO] TestAgent: started state syncer +29:39.277 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27399 network=tcp +29:39.320 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:39.320 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27398 [Candidate]" term=2 +29:39.351 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:39.351 [DEBUG] TestAgent.server.raft: vote granted: from=a95eb5d9-e700-f655-1b10-115b4dae5836 term=2 tally=1 +29:39.351 [INFO] TestAgent.server.raft: election won: tally=1 +29:39.351 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27398 [Leader]" +29:39.354 [INFO] TestAgent.server: cluster leadership acquired +29:39.356 [INFO] TestAgent.server: New leader elected: payload=Node-a95eb5d9-e700-f655-1b10-115b4dae5836 +29:39.366 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27398 +29:39.397 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:39.415 [INFO] TestAgent: Synced node info +29:39.415 [DEBUG] TestAgent: Node info in sync +29:39.445 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:39.445 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:39.445 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:39.445 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:39.445 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:39.445 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a95eb5d9-e700-f655-1b10-115b4dae5836 +29:39.445 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a95eb5d9-e700-f655-1b10-115b4dae5836 +29:39.500 [INFO] TestAgent.server: federation state anti-entropy synced +29:39.672 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:39.672 [DEBUG] TestAgent: Node info in sync +29:39.690 [INFO] TestAgent: Synced check: check=web +29:40.141 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.141 [DEBUG] TestAgent: Node info in sync +29:40.154 [INFO] TestAgent: Synced check: check=web +29:40.154 [DEBUG] TestAgent: Node info in sync +29:40.154 [DEBUG] TestAgent: Check in sync: check=web +29:40.161 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.169 [INFO] TestAgent: Synced node info +29:40.176 [INFO] TestAgent: Synced check: check=web +29:40.187 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:40.215 [INFO] TestAgent: Synced node info +29:40.215 [DEBUG] TestAgent: Check in sync: check=web +29:40.689 [DEBUG] TestAgent: Node info in sync +29:40.694 [INFO] TestAgent: Synced check: check=web +29:40.705 [INFO] TestAgent: Requesting shutdown +29:40.705 [INFO] TestAgent.server: shutting down server +29:40.705 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.705 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.705 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.705 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.705 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.707 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.707 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.707 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.707 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.711 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.711Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.711Z [INFO] agent.router.manager: shutting down +29:40.712 [INFO] TestAgent: consul server down +29:40.712 [INFO] TestAgent: shutdown complete +29:40.712 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27393 network=tcp +29:40.712 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27393 network=udp +29:40.712 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27394 network=tcp +29:41.212 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.212 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Check_DeferSync (2.50s) +2021-10-05T16:29:40.101Z [WARN] agent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.101Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:40.159Z [WARN] agent.auto_config: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead. +2021-10-05T16:29:40.159Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:40.248 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2dbe56ed-cafd-c221-d893-bfb4b85859c0 Address:127.0.0.1:27433}]" +29:40.248 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27433 [Follower]" leader= +29:40.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 127.0.0.1 +29:40.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 127.0.0.1 +2021-10-05T16:29:40.249Z [INFO] agent.router: Initializing LAN area manager +29:40.249 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27428 network=udp +29:40.249 [INFO] TestAgent.server: Adding LAN server: server="Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 (Addr: tcp/127.0.0.1:27433) (DC: dc1)" +29:40.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27428 network=tcp +29:40.250 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 area=wan +29:40.250 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27429 network=tcp +29:40.250 [INFO] TestAgent: started state syncer +29:40.250 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27434 network=tcp +29:40.300 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:40.318 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:40.318 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27433 [Candidate]" term=2 +29:40.355 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:40.355 [DEBUG] TestAgent.server.raft: vote granted: from=2dbe56ed-cafd-c221-d893-bfb4b85859c0 term=2 tally=1 +29:40.355 [INFO] TestAgent.server.raft: election won: tally=1 +29:40.355 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27433 [Leader]" +29:40.356 [INFO] TestAgent.server: cluster leadership acquired +29:40.356 [INFO] TestAgent.server: New leader elected: payload=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.359 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +29:40.376 [INFO] TestAgent.server: initializing acls +29:40.381 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:40.381 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:40.400 [INFO] TestAgent.server: initializing acls +29:40.400 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:40.401 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:40.428 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:40.444 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:40.444 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:40.444 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:40.444 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.444 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 +29:40.444 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 area=wan +29:40.445 [INFO] TestAgent.server: Updating LAN server: server="Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 (Addr: tcp/127.0.0.1:27433) (DC: dc1)" +29:40.473 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:40.473 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:40.473 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.473 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 +29:40.473 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0.dc1 area=wan +29:40.473 [INFO] TestAgent.server: Updating LAN server: server="Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 (Addr: tcp/127.0.0.1:27433) (DC: dc1)" +29:40.512 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:40.568 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:40.568 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:40.568 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:40.568 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:40.568 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:40.568 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.568 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.573 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.573 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.580 [INFO] TestAgent.server: federation state anti-entropy synced +29:40.688 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.688 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-2dbe56ed-cafd-c221-d893-bfb4b85859c0 +29:40.694 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +29:40.694 [WARN] TestAgent: Node info update blocked by ACLs: node=2dbe56ed-cafd-c221-d893-bfb4b85859c0 accessorID=00000000-0000-0000-0000-000000000002 +29:40.694 [WARN] TestAgent: Service registration blocked by ACLs: service=mysql accessorID= +29:40.706 [INFO] TestAgent: Synced service: service=api +29:40.706 [DEBUG] TestAgent.acl: dropping check from result due to ACLs: check=serfHealth +29:40.706 [WARN] TestAgent: Node info update blocked by ACLs: node=2dbe56ed-cafd-c221-d893-bfb4b85859c0 accessorID=00000000-0000-0000-0000-000000000002 +29:40.706 [WARN] TestAgent: Service registration blocked by ACLs: service=mysql accessorID= +29:40.715 [INFO] TestAgent: Deregistered service: service=api +29:40.715 [INFO] TestAgent: Requesting shutdown +29:40.716 [INFO] TestAgent.server: shutting down server +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:40.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:40.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:40.716 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:40.716 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:40.721 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:40.727Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:40.727Z [INFO] agent.router.manager: shutting down +29:40.727 [INFO] TestAgent: consul server down +29:40.727 [INFO] TestAgent: shutdown complete +29:40.727 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27428 network=tcp +29:40.727 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27428 network=udp +29:40.727 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27429 network=tcp +29:41.228 [INFO] TestAgent: Waiting for endpoints to shut down +29:41.228 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentAntiEntropy_Services_ACLDeny (1.22s) PASS -ok github.com/hashicorp/consul/agent/local 3.734s +ok github.com/hashicorp/consul/agent/local 2.680s === RUN TestBuild === RUN TestBuild/no_version === RUN TestBuild/bad_version @@ -58761,7 +58969,7 @@ === RUN TestIsConsulServer_Optional --- PASS: TestIsConsulServer_Optional (0.00s) PASS -ok github.com/hashicorp/consul/agent/metadata 0.008s +ok github.com/hashicorp/consul/agent/metadata 0.034s ? github.com/hashicorp/consul/agent/mock [no test files] === RUN TestPeekForTLS_not_TLS === RUN TestPeekForTLS_not_TLS/tcp_rpc_type_byte_0 @@ -58790,17 +58998,17 @@ === RUN TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_5 === RUN TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_6 === RUN TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_7 ---- PASS: TestPeekForTLS_actual_TLS (0.08s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_0 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_1 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_2 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_3 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_4 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_5 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_6 (0.01s) - --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_7 (0.01s) +--- PASS: TestPeekForTLS_actual_TLS (0.17s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_0 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_1 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_2 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_3 (0.03s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_4 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_5 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_6 (0.02s) + --- PASS: TestPeekForTLS_actual_TLS/tcp_rpc_type_byte_7 (0.02s) PASS -ok github.com/hashicorp/consul/agent/pool 0.093s +ok github.com/hashicorp/consul/agent/pool 0.199s === RUN TestManagerInternal_cycleServer --- PASS: TestManagerInternal_cycleServer (0.00s) === RUN TestManagerInternal_getServerList @@ -58834,11 +59042,11 @@ === RUN TestRouter_Routing --- PASS: TestRouter_Routing (0.00s) === RUN TestRouter_Routing_Offline -2022-11-07T22:44:36.648Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node4.dc1 (Addr: tcp/127.0.0.5:8300) (DC: dc1)" error= -2022-11-07T22:44:36.648Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node2.dc1 (Addr: tcp/127.0.0.3:8300) (DC: dc1)" error= -2022-11-07T22:44:36.648Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node1.dc1 (Addr: tcp/127.0.0.2:8300) (DC: dc1)" error= -2022-11-07T22:44:36.648Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node3.dc1 (Addr: tcp/127.0.0.4:8300) (DC: dc1)" error= -2022-11-07T22:44:36.648Z [DEBUG] TestRouter_Routing_Offline.router.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.107Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node3.dc1 (Addr: tcp/127.0.0.4:8300) (DC: dc1)" error= +2021-10-05T16:29:38.107Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node2.dc1 (Addr: tcp/127.0.0.3:8300) (DC: dc1)" error= +2021-10-05T16:29:38.107Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node1.dc1 (Addr: tcp/127.0.0.2:8300) (DC: dc1)" error= +2021-10-05T16:29:38.107Z [DEBUG] TestRouter_Routing_Offline.router.manager: pinging server failed: server="node4.dc1 (Addr: tcp/127.0.0.5:8300) (DC: dc1)" error= +2021-10-05T16:29:38.107Z [DEBUG] TestRouter_Routing_Offline.router.manager: No healthy servers during rebalance, aborting --- PASS: TestRouter_Routing_Offline (0.00s) === RUN TestRouter_GetDatacenters --- PASS: TestRouter_GetDatacenters (0.00s) @@ -58849,10570 +59057,10544 @@ === RUN TestRouter_GetDatacenterMaps --- PASS: TestRouter_GetDatacenterMaps (0.00s) === RUN TestRouter_FindLANServer -2022-11-07T22:44:36.649Z [INFO] TestRouter_FindLANServer.router: Initializing LAN area manager +2021-10-05T16:29:38.110Z [INFO] TestRouter_FindLANServer.router: Initializing LAN area manager --- PASS: TestRouter_FindLANServer (0.00s) === RUN TestServers_AddServer --- PASS: TestServers_AddServer (0.00s) === RUN TestServers_IsOffline -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.649Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: pinging server failed: server="s1 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.111Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.110Z [DEBUG] TestServers_IsOffline.manager: No healthy servers during rebalance, aborting --- PASS: TestServers_IsOffline (0.00s) === RUN TestServers_FindServer -2022-11-07T22:44:36.650Z [WARN] TestServers_FindServer.manager: No servers available -2022-11-07T22:44:36.650Z [DEBUG] TestServers_FindServer.manager: cycled away from server: server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_FindServer.manager: cycled away from server: server="s2 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [WARN] TestServers_FindServer.manager: No servers available +2021-10-05T16:29:38.111Z [DEBUG] TestServers_FindServer.manager: cycled away from server: server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_FindServer.manager: cycled away from server: server="s2 (Addr: /) (DC: )" --- PASS: TestServers_FindServer (0.00s) === RUN TestServers_New --- PASS: TestServers_New (0.00s) === RUN TestServers_NotifyFailedServer -2022-11-07T22:44:36.650Z [DEBUG] TestServers_NotifyFailedServer.manager: cycled away from server: server="s1 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_NotifyFailedServer.manager: cycled away from server: server="s2 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_NotifyFailedServer.manager: cycled away from server: server="s1 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_NotifyFailedServer.manager: cycled away from server: server="s2 (Addr: /) (DC: )" --- PASS: TestServers_NotifyFailedServer (0.00s) === RUN TestServers_NumServers --- PASS: TestServers_NumServers (0.00s) === RUN TestServers_RebalanceServers -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s65 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.650Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s93 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.651Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s45 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s29 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.652Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s03 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s97 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s75 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s76 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s43 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s23 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.653Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s87 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s52 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s85 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.654Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s48 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s01 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s98 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s55 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s62 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s29 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s66 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.655Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s43 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.656Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.657Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s61 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s57 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s38 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s69 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s76 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.658Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s01 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s30 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s54 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s38 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s28 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s81 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s94 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.659Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.660Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s24 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s91 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s40 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s10 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.661Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s91 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s27 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s15 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.662Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s27 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s93 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s16 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s98 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.663Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.664Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s94 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s00 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s08 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s75 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s22 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s80 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.665Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.666Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s42 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s10 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s62 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s02 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.667Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s31 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.668Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.669Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s26 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s78 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s83 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s68 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.670Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s22 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s74 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s95 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s52 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s57 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s93 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s79 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.671Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s62 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s56 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s25 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s68 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s12 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s85 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s24 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s03 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.672Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s40 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s65 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s25 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s67 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s34 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.673Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s46 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s03 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.674Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s23 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s26 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s09 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s38 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s98 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.675Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s93 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s87 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s16 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s75 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s51 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s13 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.676Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s80 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s92 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s56 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s77 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s94 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s47 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s38 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.677Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s36 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s74 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s01 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s85 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.678Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.679Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s28 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s32 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s92 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s10 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.680Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s12 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s61 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s54 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s10 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s34 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s80 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.681Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s55 (Addr: /) (DC: )" error= -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" -2022-11-07T22:44:36.682Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" ---- PASS: TestServers_RebalanceServers (0.04s) +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.111Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.112Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s67 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s29 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s71 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.113Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s17 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s15 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s83 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.114Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.115Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s11 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s79 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.117Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s73 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s76 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s56 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.118Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s14 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s92 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.119Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s09 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s13 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s71 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s48 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s07 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s89 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s72 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.120Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s79 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.121Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s71 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s21 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s13 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.122Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s20 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.123Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.124Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.125Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.126Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.127Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s97 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s68 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.128Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.131Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.132Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.133Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.160Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s51 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s03 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s00 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.161Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.162Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s17 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.163Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.164Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s94 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s16 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s40 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s99 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.168Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.169Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.170Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.177Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s77 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s26 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s84 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s00 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s08 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s69 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s83 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s01 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s87 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s22 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.178Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s54 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s91 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s84 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s42 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s92 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s25 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.179Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.180Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s43 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s07 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s75 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.181Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s62 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.182Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.183Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s45 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.184Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.192Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.193Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.194Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s49 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.195Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.196Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s72 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s48 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s25 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.197Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.198Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s95 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s72 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s60 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.199Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s35 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.200Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.202Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.203Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.204Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s33 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.205Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.206Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.208Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.208Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s39 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.212Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.213Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.214Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.215Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.216Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.217Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.218Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s70 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s12 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s22 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s29 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s91 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s34 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.219Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s06 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.220Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.228Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s79 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.229Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.230Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s03 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.231Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.232Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.236Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s28 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.237Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s10 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s84 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s28 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s47 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.238Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.239Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.240Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.248Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s27 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s73 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s54 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.249Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.250Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s23 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.251Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s62 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.252Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.253Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.254Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.255Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.264Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.265Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s59 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.266Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s72 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.267Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.268Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s72 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s17 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s89 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s39 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.272Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.273Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s19 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s71 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.274Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.275Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.276Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.284Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s99 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.285Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.286Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.287Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.288Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.298Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.299Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s17 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.305Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.316Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.317Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.318Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s26 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s11 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.319Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.320Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.324Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.341Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s95 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: pinging server failed: server="s05 (Addr: /) (DC: )" error= +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.342Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: Rebalanced servers, new active server: number_of_servers=100 active_server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s53 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s25 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s27 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s48 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s55 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s23 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s40 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s69 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s99 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s34 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s68 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s51 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s50 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s36 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s71 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s29 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s96 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s31 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s00 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s38 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s81 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s79 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s41 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s59 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s16 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s33 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s43 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s39 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s78 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s30 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s95 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s28 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s45 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s83 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s76 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s56 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s75 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s82 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s88 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s62 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s98 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s93 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s84 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s44 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s67 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s08 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s46 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s91 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s89 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s24 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s60 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s47 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s77 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s90 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s37 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s21 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s49 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s87 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s57 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s72 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s58 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s94 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s86 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s61 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s63 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s66 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s73 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s85 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s92 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s54 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s65 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s80 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s26 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s20 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s22 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s74 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s64 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s97 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s70 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s42 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s52 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s35 (Addr: /) (DC: )" +2021-10-05T16:29:38.343Z [DEBUG] TestServers_RebalanceServers.manager: cycled away from server: server="s32 (Addr: /) (DC: )" +--- PASS: TestServers_RebalanceServers (0.28s) === RUN TestServers_RebalanceServers_AvoidFailed -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" -2022-11-07T22:44:36.692Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.390Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.389Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s1 (Addr: faux/s1) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: pinging server failed: server="s2 (Addr: faux/s2) (DC: )" error= +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" +2021-10-05T16:29:38.388Z [DEBUG] TestServers_RebalanceServers_AvoidFailed.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="s3 (Addr: faux/s3) (DC: )" --- PASS: TestServers_RebalanceServers_AvoidFailed (0.00s) === RUN TestManager_RemoveServer -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: Rebalanced servers, new active server: number_of_servers=19 active_server="s14 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s08 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s06 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s10 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s05 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s01 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s07 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s17 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s02 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s11 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s03 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s16 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s04 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s12 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s18 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s15 (Addr: /) (DC: )" -2022-11-07T22:44:36.693Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: Rebalanced servers, new active server: number_of_servers=19 active_server="s12 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s17 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s18 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s01 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s03 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s14 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s04 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s10 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s07 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s19 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s15 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s05 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s13 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s11 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s09 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s02 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s06 (Addr: /) (DC: )" +2021-10-05T16:29:38.391Z [DEBUG] TestManager_RemoveServer.manager: cycled away from server: server="s02 (Addr: /) (DC: )" --- PASS: TestManager_RemoveServer (0.00s) PASS -ok github.com/hashicorp/consul/agent/router 0.055s +ok github.com/hashicorp/consul/agent/router 0.431s === RUN TestStructs_ACLCaches === RUN TestStructs_ACLCaches/New === RUN TestStructs_ACLCaches/New/Valid_Sizes @@ -69918,13 +70100,13 @@ --- PASS: TestIngressConfigEntry_Normalize/empty_protocol (0.00s) --- PASS: TestIngressConfigEntry_Normalize/lowercase_protocols (0.00s) === RUN TestIngressConfigEntry_ListRelatedServices -=== RUN TestIngressConfigEntry_ListRelatedServices/one_exact === RUN TestIngressConfigEntry_ListRelatedServices/one_wild === RUN TestIngressConfigEntry_ListRelatedServices/kitchen_sink +=== RUN TestIngressConfigEntry_ListRelatedServices/one_exact --- PASS: TestIngressConfigEntry_ListRelatedServices (0.00s) - --- PASS: TestIngressConfigEntry_ListRelatedServices/one_exact (0.00s) --- PASS: TestIngressConfigEntry_ListRelatedServices/one_wild (0.00s) --- PASS: TestIngressConfigEntry_ListRelatedServices/kitchen_sink (0.00s) + --- PASS: TestIngressConfigEntry_ListRelatedServices/one_exact (0.00s) === RUN TestIngressConfigEntry_Validate === RUN TestIngressConfigEntry_Validate/port_conflict === RUN TestIngressConfigEntry_Validate/http_features:_wildcard @@ -70138,7 +70320,6 @@ --- PASS: TestValidateMeshGatewayMode/remote_(constant) (0.00s) --- PASS: TestValidateMeshGatewayMode/remote_(explicit) (0.00s) === RUN TestIntention_ACLs -=== RUN TestIntention_ACLs/allow-write-with-dest-write === RUN TestIntention_ACLs/deny-write-with-source-write === RUN TestIntention_ACLs/deny-wildcard-write-allow-read === RUN TestIntention_ACLs/allow-wildcard-write @@ -70146,8 +70327,8 @@ === RUN TestIntention_ACLs/all-denied === RUN TestIntention_ACLs/deny-write-read-dest === RUN TestIntention_ACLs/deny-write-read-source +=== RUN TestIntention_ACLs/allow-write-with-dest-write --- PASS: TestIntention_ACLs (0.00s) - --- PASS: TestIntention_ACLs/allow-write-with-dest-write (0.00s) --- PASS: TestIntention_ACLs/deny-write-with-source-write (0.00s) --- PASS: TestIntention_ACLs/deny-wildcard-write-allow-read (0.00s) --- PASS: TestIntention_ACLs/allow-wildcard-write (0.00s) @@ -70155,6 +70336,7 @@ --- PASS: TestIntention_ACLs/all-denied (0.00s) --- PASS: TestIntention_ACLs/deny-write-read-dest (0.00s) --- PASS: TestIntention_ACLs/deny-write-read-source (0.00s) + --- PASS: TestIntention_ACLs/allow-write-with-dest-write (0.00s) === RUN TestIntentionValidate === RUN TestIntentionValidate/long_description === RUN TestIntentionValidate/no_action_set @@ -70200,23 +70382,23 @@ --- PASS: TestServiceDefinitionValidate (0.00s) --- PASS: TestServiceDefinitionValidate/valid (0.00s) === RUN TestStructs_FilterFieldConfigurations +=== RUN TestStructs_FilterFieldConfigurations/ServiceNode +=== RUN TestStructs_FilterFieldConfigurations/HealthCheck +=== RUN TestStructs_FilterFieldConfigurations/CheckServiceNode === RUN TestStructs_FilterFieldConfigurations/NodeInfo === RUN TestStructs_FilterFieldConfigurations/api.AgentService === RUN TestStructs_FilterFieldConfigurations/Intention === RUN TestStructs_FilterFieldConfigurations/Node === RUN TestStructs_FilterFieldConfigurations/NodeService -=== RUN TestStructs_FilterFieldConfigurations/ServiceNode -=== RUN TestStructs_FilterFieldConfigurations/HealthCheck -=== RUN TestStructs_FilterFieldConfigurations/CheckServiceNode ---- PASS: TestStructs_FilterFieldConfigurations (0.00s) +--- PASS: TestStructs_FilterFieldConfigurations (0.01s) + --- PASS: TestStructs_FilterFieldConfigurations/ServiceNode (0.00s) + --- PASS: TestStructs_FilterFieldConfigurations/HealthCheck (0.00s) + --- PASS: TestStructs_FilterFieldConfigurations/CheckServiceNode (0.00s) --- PASS: TestStructs_FilterFieldConfigurations/NodeInfo (0.00s) - --- PASS: TestStructs_FilterFieldConfigurations/api.AgentService (0.00s) + --- PASS: TestStructs_FilterFieldConfigurations/api.AgentService (0.01s) --- PASS: TestStructs_FilterFieldConfigurations/Intention (0.00s) --- PASS: TestStructs_FilterFieldConfigurations/Node (0.00s) --- PASS: TestStructs_FilterFieldConfigurations/NodeService (0.00s) - --- PASS: TestStructs_FilterFieldConfigurations/ServiceNode (0.00s) - --- PASS: TestStructs_FilterFieldConfigurations/HealthCheck (0.00s) - --- PASS: TestStructs_FilterFieldConfigurations/CheckServiceNode (0.00s) === RUN TestEncodeDecode --- PASS: TestEncodeDecode (0.00s) === RUN TestStructs_Implements @@ -70234,71 +70416,71 @@ === RUN TestStructs_ServiceNode_Conversions --- PASS: TestStructs_ServiceNode_Conversions (0.00s) === RUN TestStructs_NodeService_ValidateMeshGateway -=== RUN TestStructs_NodeService_ValidateMeshGateway/zero-port -=== RUN TestStructs_NodeService_ValidateMeshGateway/sidecar-service -=== RUN TestStructs_NodeService_ValidateMeshGateway/proxy-destination-name -=== RUN TestStructs_NodeService_ValidateMeshGateway/proxy-destination-id === RUN TestStructs_NodeService_ValidateMeshGateway/proxy-local-address === RUN TestStructs_NodeService_ValidateMeshGateway/proxy-local-port === RUN TestStructs_NodeService_ValidateMeshGateway/proxy-upstreams === RUN TestStructs_NodeService_ValidateMeshGateway/valid +=== RUN TestStructs_NodeService_ValidateMeshGateway/zero-port +=== RUN TestStructs_NodeService_ValidateMeshGateway/sidecar-service +=== RUN TestStructs_NodeService_ValidateMeshGateway/proxy-destination-name +=== RUN TestStructs_NodeService_ValidateMeshGateway/proxy-destination-id --- PASS: TestStructs_NodeService_ValidateMeshGateway (0.00s) - --- PASS: TestStructs_NodeService_ValidateMeshGateway/zero-port (0.00s) - --- PASS: TestStructs_NodeService_ValidateMeshGateway/sidecar-service (0.00s) - --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-destination-name (0.00s) - --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-destination-id (0.00s) --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-local-address (0.00s) --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-local-port (0.00s) --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-upstreams (0.00s) --- PASS: TestStructs_NodeService_ValidateMeshGateway/valid (0.00s) + --- PASS: TestStructs_NodeService_ValidateMeshGateway/zero-port (0.00s) + --- PASS: TestStructs_NodeService_ValidateMeshGateway/sidecar-service (0.00s) + --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-destination-name (0.00s) + --- PASS: TestStructs_NodeService_ValidateMeshGateway/proxy-destination-id (0.00s) === RUN TestStructs_NodeService_ValidateTerminatingGateway -=== RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-upstreams === RUN TestStructs_NodeService_ValidateTerminatingGateway/valid === RUN TestStructs_NodeService_ValidateTerminatingGateway/sidecar-service === RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-destination-name === RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-destination-id === RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-local-address === RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-local-port +=== RUN TestStructs_NodeService_ValidateTerminatingGateway/proxy-upstreams --- PASS: TestStructs_NodeService_ValidateTerminatingGateway (0.00s) - --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-upstreams (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/valid (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/sidecar-service (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-destination-name (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-destination-id (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-local-address (0.00s) --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-local-port (0.00s) + --- PASS: TestStructs_NodeService_ValidateTerminatingGateway/proxy-upstreams (0.00s) === RUN TestStructs_NodeService_ValidateIngressGateway -=== RUN TestStructs_NodeService_ValidateIngressGateway/proxy-upstreams === RUN TestStructs_NodeService_ValidateIngressGateway/valid === RUN TestStructs_NodeService_ValidateIngressGateway/sidecar-service === RUN TestStructs_NodeService_ValidateIngressGateway/proxy-destination-name === RUN TestStructs_NodeService_ValidateIngressGateway/proxy-destination-id === RUN TestStructs_NodeService_ValidateIngressGateway/proxy-local-address === RUN TestStructs_NodeService_ValidateIngressGateway/proxy-local-port +=== RUN TestStructs_NodeService_ValidateIngressGateway/proxy-upstreams --- PASS: TestStructs_NodeService_ValidateIngressGateway (0.00s) - --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-upstreams (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/valid (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/sidecar-service (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-destination-name (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-destination-id (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-local-address (0.00s) --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-local-port (0.00s) + --- PASS: TestStructs_NodeService_ValidateIngressGateway/proxy-upstreams (0.00s) === RUN TestStructs_NodeService_ValidateExposeConfig +=== RUN TestStructs_NodeService_ValidateExposeConfig/valid +=== RUN TestStructs_NodeService_ValidateExposeConfig/empty_path === RUN TestStructs_NodeService_ValidateExposeConfig/invalid_port_negative === RUN TestStructs_NodeService_ValidateExposeConfig/invalid_port_too_large === RUN TestStructs_NodeService_ValidateExposeConfig/duplicate_paths === RUN TestStructs_NodeService_ValidateExposeConfig/duplicate_ports === RUN TestStructs_NodeService_ValidateExposeConfig/protocol_not_supported -=== RUN TestStructs_NodeService_ValidateExposeConfig/valid -=== RUN TestStructs_NodeService_ValidateExposeConfig/empty_path --- PASS: TestStructs_NodeService_ValidateExposeConfig (0.00s) + --- PASS: TestStructs_NodeService_ValidateExposeConfig/valid (0.00s) + --- PASS: TestStructs_NodeService_ValidateExposeConfig/empty_path (0.00s) --- PASS: TestStructs_NodeService_ValidateExposeConfig/invalid_port_negative (0.00s) --- PASS: TestStructs_NodeService_ValidateExposeConfig/invalid_port_too_large (0.00s) --- PASS: TestStructs_NodeService_ValidateExposeConfig/duplicate_paths (0.00s) --- PASS: TestStructs_NodeService_ValidateExposeConfig/duplicate_ports (0.00s) --- PASS: TestStructs_NodeService_ValidateExposeConfig/protocol_not_supported (0.00s) - --- PASS: TestStructs_NodeService_ValidateExposeConfig/valid (0.00s) - --- PASS: TestStructs_NodeService_ValidateExposeConfig/empty_path (0.00s) === RUN TestStructs_NodeService_ValidateConnectProxy === RUN TestStructs_NodeService_ValidateConnectProxy/valid === RUN TestStructs_NodeService_ValidateConnectProxy/connect-proxy:_no_Proxy.DestinationServiceName @@ -70358,6 +70540,14 @@ === RUN TestStructs_DirEntry_Clone --- PASS: TestStructs_DirEntry_Clone (0.00s) === RUN TestStructs_ValidateServiceAndNodeMetadata +=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed +=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateNodeMetadata +=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_typical +=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_mesh-gateway +=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key +=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateNodeMetadata +=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_typical +=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_mesh-gateway === RUN TestStructs_ValidateServiceAndNodeMetadata/too_many_keys === RUN TestStructs_ValidateServiceAndNodeMetadata/too_many_keys/ValidateNodeMetadata === RUN TestStructs_ValidateServiceAndNodeMetadata/too_many_keys/ValidateServiceMetadata_-_typical @@ -70374,15 +70564,15 @@ === RUN TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateNodeMetadata === RUN TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateServiceMetadata_-_typical === RUN TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateServiceMetadata_-_mesh-gateway -=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed -=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateNodeMetadata -=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_typical -=== RUN TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_mesh-gateway -=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key -=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateNodeMetadata -=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_typical -=== RUN TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_mesh-gateway --- PASS: TestStructs_ValidateServiceAndNodeMetadata (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateNodeMetadata (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_typical (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_mesh-gateway (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateNodeMetadata (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_typical (0.00s) + --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_mesh-gateway (0.00s) --- PASS: TestStructs_ValidateServiceAndNodeMetadata/too_many_keys (0.00s) --- PASS: TestStructs_ValidateServiceAndNodeMetadata/too_many_keys/ValidateNodeMetadata (0.00s) --- PASS: TestStructs_ValidateServiceAndNodeMetadata/too_many_keys/ValidateServiceMetadata_-_typical (0.00s) @@ -70399,14 +70589,6 @@ --- PASS: TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateNodeMetadata (0.00s) --- PASS: TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateServiceMetadata_-_typical (0.00s) --- PASS: TestStructs_ValidateServiceAndNodeMetadata/reserved_key_prefix_allowed_via_an_allowlist_just_for_gateway_-_consul-wan-federation/ValidateServiceMetadata_-_mesh-gateway (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateNodeMetadata (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_typical (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/should_succeed/ValidateServiceMetadata_-_mesh-gateway (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateNodeMetadata (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_typical (0.00s) - --- PASS: TestStructs_ValidateServiceAndNodeMetadata/invalid_key/ValidateServiceMetadata_-_mesh-gateway (0.00s) === RUN TestStructs_validateMetaPair --- PASS: TestStructs_validateMetaPair (0.00s) === RUN TestSpecificServiceRequest_CacheInfo @@ -70448,31 +70630,31 @@ --- PASS: TestNode_BestAddress/address (0.00s) --- PASS: TestNode_BestAddress/wan-address (0.00s) === RUN TestNodeService_BestAddress +=== RUN TestNodeService_BestAddress/no-address === RUN TestNodeService_BestAddress/service-address === RUN TestNodeService_BestAddress/service-wan-address === RUN TestNodeService_BestAddress/service-wan-address-default-port === RUN TestNodeService_BestAddress/service-wan-address-node-lan -=== RUN TestNodeService_BestAddress/no-address --- PASS: TestNodeService_BestAddress (0.00s) + --- PASS: TestNodeService_BestAddress/no-address (0.00s) --- PASS: TestNodeService_BestAddress/service-address (0.00s) --- PASS: TestNodeService_BestAddress/service-wan-address (0.00s) --- PASS: TestNodeService_BestAddress/service-wan-address-default-port (0.00s) --- PASS: TestNodeService_BestAddress/service-wan-address-node-lan (0.00s) - --- PASS: TestNodeService_BestAddress/no-address (0.00s) === RUN TestCheckServiceNode_BestAddress +=== RUN TestCheckServiceNode_BestAddress/service-wan-address-default-port +=== RUN TestCheckServiceNode_BestAddress/service-wan-address-node-lan === RUN TestCheckServiceNode_BestAddress/node-address === RUN TestCheckServiceNode_BestAddress/node-wan-address === RUN TestCheckServiceNode_BestAddress/service-address === RUN TestCheckServiceNode_BestAddress/service-wan-address -=== RUN TestCheckServiceNode_BestAddress/service-wan-address-default-port -=== RUN TestCheckServiceNode_BestAddress/service-wan-address-node-lan --- PASS: TestCheckServiceNode_BestAddress (0.00s) + --- PASS: TestCheckServiceNode_BestAddress/service-wan-address-default-port (0.00s) + --- PASS: TestCheckServiceNode_BestAddress/service-wan-address-node-lan (0.00s) --- PASS: TestCheckServiceNode_BestAddress/node-address (0.00s) --- PASS: TestCheckServiceNode_BestAddress/node-wan-address (0.00s) --- PASS: TestCheckServiceNode_BestAddress/service-address (0.00s) --- PASS: TestCheckServiceNode_BestAddress/service-wan-address (0.00s) - --- PASS: TestCheckServiceNode_BestAddress/service-wan-address-default-port (0.00s) - --- PASS: TestCheckServiceNode_BestAddress/service-wan-address-node-lan (0.00s) === RUN TestNodeService_JSON_Marshal --- PASS: TestNodeService_JSON_Marshal (0.00s) === RUN TestServiceNode_JSON_Marshal @@ -70524,7 +70706,7 @@ === RUN TestGatewayService_IsSame --- PASS: TestGatewayService_IsSame (0.00s) PASS -ok github.com/hashicorp/consul/agent/structs 0.038s +ok github.com/hashicorp/consul/agent/structs 0.094s ? github.com/hashicorp/consul/agent/systemd [no test files] === RUN TestStore_RegularTokens === PAUSE TestStore_RegularTokens @@ -70541,10 +70723,6 @@ === PAUSE TestStore_RegularTokens/set_agent_-_config === RUN TestStore_RegularTokens/set_agent_-_api === PAUSE TestStore_RegularTokens/set_agent_-_api -=== CONT TestStore_Notify -=== CONT TestStore_AgentMasterToken ---- PASS: TestStore_Notify (0.00s) ---- PASS: TestStore_AgentMasterToken (0.00s) === RUN TestStore_RegularTokens/set_user_and_agent === PAUSE TestStore_RegularTokens/set_user_and_agent === RUN TestStore_RegularTokens/set_repl_-_config @@ -70558,595 +70736,599 @@ === RUN TestStore_RegularTokens/set_all === PAUSE TestStore_RegularTokens/set_all === CONT TestStore_RegularTokens/set_user_-_config -=== CONT TestStore_RegularTokens/set_all +=== CONT TestStore_Notify +--- PASS: TestStore_Notify (0.00s) === CONT TestStore_RegularTokens/set_repl_-_config +=== CONT TestStore_RegularTokens/set_repl_-_api === CONT TestStore_RegularTokens/set_user_and_agent -=== CONT TestStore_RegularTokens/set_master_-_config === CONT TestStore_RegularTokens/set_agent_-_api === CONT TestStore_RegularTokens/set_agent_-_config +=== CONT TestStore_AgentMasterToken === CONT TestStore_RegularTokens/set_user_-_api -=== CONT TestStore_RegularTokens/set_repl_-_api +--- PASS: TestStore_AgentMasterToken (0.00s) +=== CONT TestStore_RegularTokens/set_all === CONT TestStore_RegularTokens/set_master_-_api +=== CONT TestStore_RegularTokens/set_master_-_config --- PASS: TestStore_RegularTokens (0.00s) --- PASS: TestStore_RegularTokens/set_user_-_config (0.00s) - --- PASS: TestStore_RegularTokens/set_all (0.00s) - --- PASS: TestStore_RegularTokens/set_user_and_agent (0.00s) --- PASS: TestStore_RegularTokens/set_repl_-_config (0.00s) + --- PASS: TestStore_RegularTokens/set_user_and_agent (0.00s) --- PASS: TestStore_RegularTokens/set_agent_-_api (0.00s) - --- PASS: TestStore_RegularTokens/set_master_-_config (0.00s) + --- PASS: TestStore_RegularTokens/set_repl_-_api (0.00s) --- PASS: TestStore_RegularTokens/set_agent_-_config (0.00s) --- PASS: TestStore_RegularTokens/set_user_-_api (0.00s) - --- PASS: TestStore_RegularTokens/set_repl_-_api (0.00s) + --- PASS: TestStore_RegularTokens/set_all (0.00s) --- PASS: TestStore_RegularTokens/set_master_-_api (0.00s) + --- PASS: TestStore_RegularTokens/set_master_-_config (0.00s) PASS -ok github.com/hashicorp/consul/agent/token 0.006s +ok github.com/hashicorp/consul/agent/token 0.038s === RUN TestClustersFromSnapshot === RUN TestClustersFromSnapshot/envoy-1.14.5 === RUN TestClustersFromSnapshot/envoy-1.14.5/defaults -2022-11-07T22:44:41.982Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.733Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-local-app -2022-11-07T22:44:41.990Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.745Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-local-app-typed -2022-11-07T22:44:41.994Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.753Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-upstream -2022-11-07T22:44:41.999Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.761Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-default-chain -2022-11-07T22:44:42.008Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.769Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-typed -2022-11-07T22:44:42.016Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.777Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-ignores-tls -2022-11-07T22:44:42.023Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.785Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-timeouts -2022-11-07T22:44:42.029Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.810Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-limits-max-connections-only -2022-11-07T22:44:42.037Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.825Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-limits-set-to-zero -2022-11-07T22:44:42.044Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.835Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/custom-limits -2022-11-07T22:44:42.052Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.846Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain -2022-11-07T22:44:42.061Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.855Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni -2022-11-07T22:44:42.068Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.864Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:42.074Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.872Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover -2022-11-07T22:44:42.079Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.880Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.084Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.906Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.089Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.916Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.094Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.924Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.098Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.932Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.104Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.943Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.113Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.951Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.121Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.959Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.127Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.967Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect -2022-11-07T22:44:42.132Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.132Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.975Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:46.975Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/expose-paths-local-app-paths === RUN TestClustersFromSnapshot/envoy-1.14.5/expose-paths-new-cluster-http2 === RUN TestClustersFromSnapshot/envoy-1.14.5/expose-paths-grpc-new-cluster-http1 === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway -2022-11-07T22:44:42.139Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:46.998Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states -2022-11-07T22:44:42.143Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.006Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets -2022-11-07T22:44:42.151Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.031Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.157Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.053Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-timeouts -2022-11-07T22:44:42.162Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.068Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway -2022-11-07T22:44:42.167Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.079Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain -2022-11-07T22:44:42.174Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.103Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni -2022-11-07T22:44:42.178Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.116Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides -2022-11-07T22:44:42.183Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.123Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover -2022-11-07T22:44:42.187Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.137Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.192Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.144Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.197Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.150Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.201Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.156Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.205Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.173Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.211Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.185Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.215Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.198Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.219Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.205Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.223Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.211Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:42.229Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.229Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.217Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.217Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway -2022-11-07T22:44:42.236Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.236Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.233Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.233Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets -2022-11-07T22:44:42.244Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.244Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.245Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:47.245Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.245Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.245Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache === RUN TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets -2022-11-07T22:44:42.250Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.250Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.250Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:47.256Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.256Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.257Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache === RUN TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.257Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.257Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.265Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.266Z [WARN] TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:42.263Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.263Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.274Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.274Z [DEBUG] TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6 === RUN TestClustersFromSnapshot/envoy-1.13.6/defaults -2022-11-07T22:44:42.269Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.306Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-local-app -2022-11-07T22:44:42.274Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.314Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-local-app-typed -2022-11-07T22:44:42.278Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.330Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-upstream -2022-11-07T22:44:42.284Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.339Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-default-chain -2022-11-07T22:44:42.289Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.352Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-typed -2022-11-07T22:44:42.295Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.358Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-ignores-tls -2022-11-07T22:44:42.306Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.363Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-timeouts -2022-11-07T22:44:42.313Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.369Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-limits-max-connections-only -2022-11-07T22:44:42.321Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.375Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-limits-set-to-zero -2022-11-07T22:44:42.328Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.395Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/custom-limits -2022-11-07T22:44:42.337Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.403Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain -2022-11-07T22:44:42.344Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.409Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni -2022-11-07T22:44:42.349Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.420Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:42.356Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.455Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover -2022-11-07T22:44:42.361Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.469Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.366Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.483Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.374Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.493Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.381Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.502Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.389Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.511Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.396Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.519Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.401Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.532Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.415Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.547Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.420Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.554Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect -2022-11-07T22:44:42.425Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.425Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.561Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.561Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/expose-paths-local-app-paths === RUN TestClustersFromSnapshot/envoy-1.13.6/expose-paths-new-cluster-http2 === RUN TestClustersFromSnapshot/envoy-1.13.6/expose-paths-grpc-new-cluster-http1 === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway -2022-11-07T22:44:42.432Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.572Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states -2022-11-07T22:44:42.437Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.578Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets -2022-11-07T22:44:42.451Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.589Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.459Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.595Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-timeouts -2022-11-07T22:44:42.467Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:47.621Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway -2022-11-07T22:44:42.474Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.630Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain -2022-11-07T22:44:42.481Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.654Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni -2022-11-07T22:44:42.486Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.675Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides -2022-11-07T22:44:42.490Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.683Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover -2022-11-07T22:44:42.494Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.693Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.499Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.702Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.507Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.710Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.512Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.723Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.516Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.732Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.520Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.743Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.526Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.758Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.530Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.768Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.534Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.776Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:42.538Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.538Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.786Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.786Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway -2022-11-07T22:44:42.543Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.543Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.801Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.801Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets -2022-11-07T22:44:42.551Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.551Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.551Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:47.818Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.818Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.818Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache === RUN TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets -2022-11-07T22:44:42.557Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.557Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.557Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:47.828Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.828Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.829Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache === RUN TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.563Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.563Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:47.849Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:47.849Z [WARN] TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:42.568Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.568Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.860Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.860Z [DEBUG] TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7 === RUN TestClustersFromSnapshot/envoy-1.12.7/defaults -2022-11-07T22:44:42.573Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.869Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-local-app -2022-11-07T22:44:42.578Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.877Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-local-app-typed -2022-11-07T22:44:42.583Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.886Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-upstream -2022-11-07T22:44:42.587Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.900Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-default-chain -2022-11-07T22:44:42.593Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.909Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-typed -2022-11-07T22:44:42.597Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.921Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-ignores-tls -2022-11-07T22:44:42.602Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.929Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-timeouts -2022-11-07T22:44:42.608Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.942Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-limits-max-connections-only -2022-11-07T22:44:42.617Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.955Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-limits-set-to-zero -2022-11-07T22:44:42.628Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.964Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/custom-limits -2022-11-07T22:44:42.636Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.973Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain -2022-11-07T22:44:42.645Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.983Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni -2022-11-07T22:44:42.655Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:47.991Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:42.663Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.000Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover -2022-11-07T22:44:42.671Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.022Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.679Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.034Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.689Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.041Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.697Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.060Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.704Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.068Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.712Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.092Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.720Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.111Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.728Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.120Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.735Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.130Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect -2022-11-07T22:44:42.740Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.740Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.144Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.144Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/expose-paths-local-app-paths === RUN TestClustersFromSnapshot/envoy-1.12.7/expose-paths-new-cluster-http2 === RUN TestClustersFromSnapshot/envoy-1.12.7/expose-paths-grpc-new-cluster-http1 === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway -2022-11-07T22:44:42.748Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.160Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states -2022-11-07T22:44:42.752Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.171Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets -2022-11-07T22:44:42.761Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.190Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.767Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.200Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-timeouts -2022-11-07T22:44:42.773Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.209Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway -2022-11-07T22:44:42.778Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.218Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain -2022-11-07T22:44:42.789Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.235Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni -2022-11-07T22:44:42.793Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.244Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides -2022-11-07T22:44:42.798Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.251Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover -2022-11-07T22:44:42.802Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.259Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.806Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.266Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.811Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.274Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.815Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.281Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.819Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.289Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.823Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.299Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.827Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.308Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.831Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.316Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.835Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.326Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:42.839Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.839Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.336Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.336Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway -2022-11-07T22:44:42.843Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.843Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:48.356Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:48.356Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets -2022-11-07T22:44:42.853Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.853Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.853Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:48.395Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:48.404Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:48.404Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets -2022-11-07T22:44:42.861Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.861Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:42.861Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:48.427Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:48.428Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:48.431Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache === RUN TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers -2022-11-07T22:44:42.867Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:42.867Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:48.441Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:48.441Z [WARN] TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:42.872Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.872Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.451Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.452Z [DEBUG] TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2 === RUN TestClustersFromSnapshot/envoy-1.11.2/defaults -2022-11-07T22:44:42.878Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.476Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/defaults: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-local-app -2022-11-07T22:44:42.883Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.503Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-local-app: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-local-app-typed -2022-11-07T22:44:42.889Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.535Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-local-app-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-upstream -2022-11-07T22:44:42.893Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.547Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-default-chain -2022-11-07T22:44:42.898Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.566Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-default-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-typed -2022-11-07T22:44:42.903Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.581Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-typed: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-ignores-tls -2022-11-07T22:44:42.907Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.591Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-ignores-tls: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-timeouts -2022-11-07T22:44:42.911Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.597Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-timeouts: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-limits-max-connections-only -2022-11-07T22:44:42.915Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.603Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits-max-connections-only: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-limits-set-to-zero -2022-11-07T22:44:42.920Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.618Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits-set-to-zero: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/custom-limits -2022-11-07T22:44:42.926Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.625Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/custom-limits: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain -2022-11-07T22:44:42.931Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.644Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni -2022-11-07T22:44:42.937Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.659Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:42.943Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.667Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover -2022-11-07T22:44:42.947Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.696Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:42.952Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.710Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.956Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.719Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:42.961Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.728Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:42.967Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.736Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:42.972Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.753Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:42.976Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.766Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:42.981Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.784Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:42.985Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.798Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect -2022-11-07T22:44:42.990Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:42.990Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.807Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.807Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/expose-paths-local-app-paths === RUN TestClustersFromSnapshot/envoy-1.11.2/expose-paths-new-cluster-http2 === RUN TestClustersFromSnapshot/envoy-1.11.2/expose-paths-grpc-new-cluster-http1 === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway -2022-11-07T22:44:42.997Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.821Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states -2022-11-07T22:44:43.002Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.835Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets -2022-11-07T22:44:43.010Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.863Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-ignore-extra-resolvers -2022-11-07T22:44:43.015Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.872Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-ignore-extra-resolvers.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-timeouts -2022-11-07T22:44:43.020Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway +2021-10-05T16:29:48.881Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-timeouts.mesh_gateway: upstream service does not contain any healthy instances: dc=dc6 service=mesh-gateway === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway -2022-11-07T22:44:43.024Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.890Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain -2022-11-07T22:44:43.035Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.927Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni -2022-11-07T22:44:43.039Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.935Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides -2022-11-07T22:44:43.043Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.943Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides: generating cluster for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover -2022-11-07T22:44:43.047Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.951Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.051Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.960Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.057Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.967Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.061Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.976Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.066Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.984Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.070Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:48.991Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.074Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.000Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.078Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.007Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.083Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.044Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:43.087Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.087Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.063Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.063Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating cluster for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway -2022-11-07T22:44:43.091Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:43.091Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:49.080Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:49.080Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-no-services === RUN TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets -2022-11-07T22:44:43.100Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:43.100Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:43.100Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:49.101Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:49.102Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:49.102Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api === RUN TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets -2022-11-07T22:44:43.109Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:43.109Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db -2022-11-07T22:44:43.109Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:49.112Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=cache +2021-10-05T16:29:49.112Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:49.112Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers -2022-11-07T22:44:43.118Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api -2022-11-07T22:44:43.118Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db +2021-10-05T16:29:49.125Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers.terminating_gateway: service contains instances with more than one unique hostname; only "api.altdomain" be resolved by Envoy: dc=dc1 service=api +2021-10-05T16:29:49.126Z [WARN] TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers.terminating_gateway: upstream service does not contain any healthy instances: dc=dc1 service=db === RUN TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:43.123Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.123Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul ---- PASS: TestClustersFromSnapshot (1.15s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5 (0.29s) +2021-10-05T16:29:49.148Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.148Z [DEBUG] TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating cluster for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +--- PASS: TestClustersFromSnapshot (2.43s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5 (0.55s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/defaults (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-local-app (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-local-app-typed (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-local-app-typed (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-upstream (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-default-chain (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-typed (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-upstream-ignores-tls (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-timeouts (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-timeouts (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-limits-max-connections-only (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-limits-set-to-zero (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/custom-limits (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/expose-paths-local-app-paths (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/expose-paths-local-app-paths (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/expose-paths-new-cluster-http2 (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/expose-paths-grpc-new-cluster-http1 (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-timeouts (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway-no-services (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-ignore-extra-resolvers (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/mesh-gateway-service-timeouts (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-gateway-no-services (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-hostname-service-subsets (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.14.5/terminating-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6 (0.31s) + --- PASS: TestClustersFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6 (0.56s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/defaults (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-local-app (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-local-app-typed (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-local-app (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-local-app-typed (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-upstream (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-default-chain (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-typed (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-upstream-ignores-tls (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-timeouts (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-limits-max-connections-only (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-limits-max-connections-only (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-limits-set-to-zero (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/custom-limits (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.04s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/expose-paths-local-app-paths (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/expose-paths-new-cluster-http2 (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/expose-paths-grpc-new-cluster-http1 (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-timeouts (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-ignore-extra-resolvers (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/mesh-gateway-service-timeouts (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-no-services (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-hostname-service-subsets (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7 (0.31s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/defaults (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-local-app (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-local-app-typed (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/terminating-gateway-ignore-extra-resolvers (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7 (0.60s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/defaults (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-local-app (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-local-app-typed (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-default-chain (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-typed (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-typed (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-upstream-ignores-tls (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-timeouts (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/custom-limits-max-connections-only (0.01s) @@ -71155,101 +71337,101 @@ --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/expose-paths-local-app-paths (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/expose-paths-new-cluster-http2 (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/expose-paths-grpc-new-cluster-http1 (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-timeouts (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/mesh-gateway-service-timeouts (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-gateway-no-services (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-hostname-service-subsets (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.12.7/terminating-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2 (0.25s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/defaults (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-local-app (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-local-app-typed (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-default-chain (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-typed (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-ignores-tls (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-timeouts (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-limits-max-connections-only (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2 (0.69s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/defaults (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-local-app (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-local-app-typed (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-default-chain (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-typed (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-upstream-ignores-tls (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-timeouts (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-limits-max-connections-only (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-limits-set-to-zero (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/custom-limits (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.02s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/expose-paths-local-app-paths (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/expose-paths-new-cluster-http2 (0.00s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/expose-paths-grpc-new-cluster-http1 (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-ignore-extra-resolvers (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-timeouts (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-ignore-extra-resolvers (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/mesh-gateway-service-timeouts (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway (0.00s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.02s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway (0.01s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets (0.01s) --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-hostname-service-subsets (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers (0.01s) - --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service (0.00s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/terminating-gateway-ignore-extra-resolvers (0.03s) + --- PASS: TestClustersFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service (0.02s) === RUN TestParseProxyConfig === RUN TestParseProxyConfig/defaults_-_nil === RUN TestParseProxyConfig/defaults_-_empty @@ -71323,320 +71505,320 @@ === RUN Test_endpointsFromSnapshot === RUN Test_endpointsFromSnapshot/envoy-1.14.5 === RUN Test_endpointsFromSnapshot/envoy-1.14.5/defaults -2022-11-07T22:44:43.130Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.167Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway === RUN Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states === RUN Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain -2022-11-07T22:44:43.146Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.196Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni -2022-11-07T22:44:43.152Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.202Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:43.155Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.210Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover -2022-11-07T22:44:43.159Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.225Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.164Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.232Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.168Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.239Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.172Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.246Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.176Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.253Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.180Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.264Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.184Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.274Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.188Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.286Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.192Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.302Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-default-chain-and-custom-cluster -2022-11-07T22:44:43.196Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice +2021-10-05T16:29:49.312Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice === RUN Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect -2022-11-07T22:44:43.200Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.200Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.329Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.329Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-gateway -2022-11-07T22:44:43.223Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.359Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain -2022-11-07T22:44:43.236Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.373Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni -2022-11-07T22:44:43.243Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.380Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides -2022-11-07T22:44:43.250Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.388Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover -2022-11-07T22:44:43.257Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.395Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.264Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.402Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.271Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.409Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.275Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.421Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.279Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.427Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.284Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.433Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.288Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.439Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.292Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.444Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.304Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.449Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:43.311Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.311Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.454Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.454Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway === RUN Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:43.347Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.347Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.481Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.481Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6 === RUN Test_endpointsFromSnapshot/envoy-1.13.6/defaults -2022-11-07T22:44:43.352Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.487Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway === RUN Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states === RUN Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain -2022-11-07T22:44:43.368Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.508Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni -2022-11-07T22:44:43.372Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.525Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:43.376Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.533Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover -2022-11-07T22:44:43.380Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.539Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.389Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.544Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.396Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.549Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.403Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.555Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.410Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.560Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.418Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.565Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.425Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.571Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.432Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.576Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.438Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.581Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-default-chain-and-custom-cluster -2022-11-07T22:44:43.442Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice +2021-10-05T16:29:49.587Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice === RUN Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect -2022-11-07T22:44:43.447Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.447Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.592Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.592Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway -2022-11-07T22:44:43.462Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.643Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain -2022-11-07T22:44:43.470Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.660Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni -2022-11-07T22:44:43.475Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.667Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides -2022-11-07T22:44:43.482Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.678Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover -2022-11-07T22:44:43.489Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.687Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.495Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.695Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.499Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.702Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.503Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.708Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.510Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.715Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.517Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.722Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.524Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.729Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.530Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.738Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.537Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.745Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:43.544Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.544Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.752Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.752Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway === RUN Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:43.575Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.575Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.790Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.790Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7 === RUN Test_endpointsFromSnapshot/envoy-1.12.7/defaults -2022-11-07T22:44:43.579Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.797Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway === RUN Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states === RUN Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain -2022-11-07T22:44:43.596Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.826Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni -2022-11-07T22:44:43.601Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.832Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:43.608Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.842Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover -2022-11-07T22:44:43.616Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.849Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.624Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.856Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.631Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.863Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.636Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.870Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.640Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.877Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.645Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.883Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.649Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.890Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.654Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.897Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.658Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.904Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-default-chain-and-custom-cluster -2022-11-07T22:44:43.663Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice +2021-10-05T16:29:49.911Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice === RUN Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect -2022-11-07T22:44:43.667Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.667Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.917Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.917Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway -2022-11-07T22:44:43.682Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.945Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain -2022-11-07T22:44:43.692Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.958Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni -2022-11-07T22:44:43.696Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.965Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides -2022-11-07T22:44:43.704Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.972Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover -2022-11-07T22:44:43.711Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.979Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.719Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.986Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.726Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:49.993Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.733Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.000Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.740Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.007Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.747Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.019Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.755Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.033Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.761Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.040Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.768Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.058Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:43.775Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.775Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.064Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.064Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway === RUN Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:43.804Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.804Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.113Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.113Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2 === RUN Test_endpointsFromSnapshot/envoy-1.11.2/defaults -2022-11-07T22:44:43.809Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.119Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/defaults: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway === RUN Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states === RUN Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain -2022-11-07T22:44:43.825Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.168Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni -2022-11-07T22:44:43.830Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.183Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides -2022-11-07T22:44:43.834Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.203Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover -2022-11-07T22:44:43.838Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.223Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.843Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.238Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.847Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.244Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.852Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.250Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.856Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.265Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.860Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.277Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.864Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.282Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.869Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.287Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.874Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.293Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-default-chain-and-custom-cluster -2022-11-07T22:44:43.878Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice +2021-10-05T16:29:50.298Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-default-chain-and-custom-cluster: generating endpoints for: cluster=myservice === RUN Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect -2022-11-07T22:44:43.882Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.882Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.304Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.304Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway -2022-11-07T22:44:43.897Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.336Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain -2022-11-07T22:44:43.905Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.347Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni -2022-11-07T22:44:43.909Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.352Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides -2022-11-07T22:44:43.914Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.357Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides: generating endpoints for: cluster=a236e964~db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover -2022-11-07T22:44:43.918Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.363Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway -2022-11-07T22:44:43.923Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.368Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.927Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.373Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway -2022-11-07T22:44:43.931Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.379Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered -2022-11-07T22:44:43.936Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.384Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway -2022-11-07T22:44:43.940Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.389Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered -2022-11-07T22:44:43.944Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.397Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway -2022-11-07T22:44:43.948Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.410Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered -2022-11-07T22:44:43.952Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.424Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect -2022-11-07T22:44:43.956Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.956Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.442Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v1.db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.442Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect: generating endpoints for: cluster=v2.db.default.dc2.internal.11111111-2222-3333-4444-555555555555.consul === RUN Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway === RUN Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-no-services === RUN Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets === RUN Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-default-service-subset === RUN Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service -2022-11-07T22:44:43.977Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:43.977Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul ---- PASS: Test_endpointsFromSnapshot (0.85s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5 (0.22s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/defaults (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-default-chain-and-custom-cluster (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.00s) +2021-10-05T16:29:50.506Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=foo.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:50.506Z [DEBUG] Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service: generating endpoints for: cluster=bar.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +--- PASS: Test_endpointsFromSnapshot (1.35s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5 (0.32s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/defaults (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-failover (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/connect-proxy-with-default-chain-and-custom-cluster (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-service-subsets (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/mesh-gateway-default-service-subset (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-gateway (0.01s) @@ -71647,27 +71829,27 @@ --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-chain-and-failover (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.00s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/terminating-gateway-default-service-subset (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.14.5/ingress-multiple-listeners-duplicate-service (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6 (0.23s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/defaults (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6 (0.31s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/defaults (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-failover (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) @@ -71675,20 +71857,20 @@ --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-default-chain-and-custom-cluster (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/connect-proxy-with-default-chain-and-custom-cluster (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-service-subsets (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-default-service-subset (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/mesh-gateway-default-service-subset (0.03s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-chain-and-failover (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) @@ -71699,99 +71881,99 @@ --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-no-services (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/terminating-gateway-default-service-subset (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7 (0.23s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/defaults (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.13.6/ingress-multiple-listeners-duplicate-service (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7 (0.32s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/defaults (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-failover (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-default-chain-and-custom-cluster (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/connect-proxy-with-default-chain-and-custom-cluster (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-service-subsets (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/mesh-gateway-default-service-subset (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway-no-services (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-gateway-no-services (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-chain-and-failover (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.02s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.02s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-default-service-subset (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/terminating-gateway-default-service-subset (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.12.7/ingress-multiple-listeners-duplicate-service (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2 (0.17s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/defaults (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2 (0.39s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/defaults (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-failover (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-default-chain-and-custom-cluster (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-double-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/connect-proxy-with-default-chain-and-custom-cluster (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-service-subsets (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/mesh-gateway-default-service-subset (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.01s) --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.01s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-default-service-subset (0.00s) - --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service (0.00s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-chain-and-failover (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-remote-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway-triggered (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-double-failover-through-local-gateway-triggered (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.02s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/terminating-gateway-default-service-subset (0.01s) + --- PASS: Test_endpointsFromSnapshot/envoy-1.11.2/ingress-multiple-listeners-duplicate-service (0.01s) === RUN TestDetermineEnvoyVersionFromNode +=== RUN TestDetermineEnvoyVersionFromNode/user_agent_build_version_with_user_agent === RUN TestDetermineEnvoyVersionFromNode/empty === RUN TestDetermineEnvoyVersionFromNode/only_build_version === RUN TestDetermineEnvoyVersionFromNode/user_agent_build_version_but_no_user_agent -=== RUN TestDetermineEnvoyVersionFromNode/user_agent_build_version_with_user_agent --- PASS: TestDetermineEnvoyVersionFromNode (0.00s) + --- PASS: TestDetermineEnvoyVersionFromNode/user_agent_build_version_with_user_agent (0.00s) --- PASS: TestDetermineEnvoyVersionFromNode/empty (0.00s) --- PASS: TestDetermineEnvoyVersionFromNode/only_build_version (0.00s) --- PASS: TestDetermineEnvoyVersionFromNode/user_agent_build_version_but_no_user_agent (0.00s) - --- PASS: TestDetermineEnvoyVersionFromNode/user_agent_build_version_with_user_agent (0.00s) === RUN TestFirstHealthyTarget === RUN TestFirstHealthyTarget/all-ok.default.dc1_and_[] === RUN TestFirstHealthyTarget/all-warn.default.dc1_and_[] @@ -71830,7 +72012,7 @@ === RUN TestListenersFromSnapshot/envoy-1.14.5/custom-upstream === RUN TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed === RUN TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed-ignored-with-disco-chain -2022-11-07T22:44:44.044Z [WARN] TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json +2021-10-05T16:29:50.651Z [WARN] TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json === RUN TestListenersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect === RUN TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain === RUN TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-http-chain @@ -71861,7 +72043,7 @@ === RUN TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets === RUN TestListenersFromSnapshot/envoy-1.14.5/ingress-http-multiple-services === RUN TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-api-cert -2022-11-07T22:44:44.202Z [ERROR] TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= +2021-10-05T16:29:50.978Z [ERROR] TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= === RUN TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tls-listener === RUN TestListenersFromSnapshot/envoy-1.13.6 === RUN TestListenersFromSnapshot/envoy-1.13.6/defaults @@ -71876,7 +72058,7 @@ === RUN TestListenersFromSnapshot/envoy-1.13.6/custom-upstream === RUN TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed === RUN TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed-ignored-with-disco-chain -2022-11-07T22:44:44.296Z [WARN] TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json +2021-10-05T16:29:51.130Z [WARN] TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json === RUN TestListenersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect === RUN TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain === RUN TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-http-chain @@ -71907,7 +72089,7 @@ === RUN TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets === RUN TestListenersFromSnapshot/envoy-1.13.6/ingress-http-multiple-services === RUN TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-api-cert -2022-11-07T22:44:44.454Z [ERROR] TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= +2021-10-05T16:29:51.454Z [ERROR] TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= === RUN TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tls-listener === RUN TestListenersFromSnapshot/envoy-1.12.7 === RUN TestListenersFromSnapshot/envoy-1.12.7/defaults @@ -71922,7 +72104,7 @@ === RUN TestListenersFromSnapshot/envoy-1.12.7/custom-upstream === RUN TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed === RUN TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed-ignored-with-disco-chain -2022-11-07T22:44:44.530Z [WARN] TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json +2021-10-05T16:29:51.582Z [WARN] TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json === RUN TestListenersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect === RUN TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain === RUN TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-http-chain @@ -71953,7 +72135,7 @@ === RUN TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets === RUN TestListenersFromSnapshot/envoy-1.12.7/ingress-http-multiple-services === RUN TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-api-cert -2022-11-07T22:44:44.680Z [ERROR] TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= +2021-10-05T16:29:51.952Z [ERROR] TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= === RUN TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tls-listener === RUN TestListenersFromSnapshot/envoy-1.11.2 === RUN TestListenersFromSnapshot/envoy-1.11.2/defaults @@ -71968,7 +72150,7 @@ === RUN TestListenersFromSnapshot/envoy-1.11.2/custom-upstream === RUN TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed === RUN TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed-ignored-with-disco-chain -2022-11-07T22:44:44.747Z [WARN] TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json +2021-10-05T16:29:52.079Z [WARN] TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed-ignored-with-disco-chain: ignoring escape hatch setting because already configured for: discovery chain=db upstream=db config=envoy_listener_json === RUN TestListenersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect === RUN TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain === RUN TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-http-chain @@ -71999,185 +72181,185 @@ === RUN TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets === RUN TestListenersFromSnapshot/envoy-1.11.2/ingress-http-multiple-services === RUN TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-api-cert -2022-11-07T22:44:44.894Z [ERROR] TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= +2021-10-05T16:29:52.445Z [ERROR] TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-api-cert.terminating_gateway: no client certificate available for linked service, skipping filter chain creation: service=api error= === RUN TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tls-listener ---- PASS: TestListenersFromSnapshot (0.92s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5 (0.23s) +--- PASS: TestListenersFromSnapshot (1.95s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5 (0.49s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/defaults (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-address (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-port (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-address-port (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/http-public-listener (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-address (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-port (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/listener-bind-address-port (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/http-public-listener (0.03s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/http-upstream (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-public-listener (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-public-listener (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-public-listener-typed (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-public-listener-ignores-tls (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-public-listener-ignores-tls (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-upstream (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed-ignored-with-disco-chain (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/custom-upstream-typed-ignored-with-disco-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-http-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-http2-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-grpc-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/expose-paths-local-app-paths (0.00s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/expose-paths-new-cluster-http2 (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-using-federation-states (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-no-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-tagged-addresses (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/mesh-gateway-custom-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-gateway-bind-addrs (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-gateway-bind-addrs (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-custom-and-tagged-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-http-multiple-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-api-cert (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-services (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-custom-and-tagged-addresses (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-service-subsets (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-http-multiple-services (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.14.5/terminating-gateway-no-api-cert (0.02s) --- PASS: TestListenersFromSnapshot/envoy-1.14.5/ingress-with-tls-listener (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6 (0.25s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/defaults (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6 (0.47s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/defaults (0.02s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/listener-bind-address (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/listener-bind-port (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/listener-bind-address-port (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/http-public-listener (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/http-upstream (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-public-listener (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-public-listener-typed (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-public-listener (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-public-listener-typed (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-public-listener-ignores-tls (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-upstream (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/custom-upstream-typed-ignored-with-disco-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-http-chain (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-http-chain (0.02s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-http2-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-grpc-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/expose-paths-local-app-paths (0.00s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/expose-paths-new-cluster-http2 (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-no-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-using-federation-states (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-no-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-tagged-addresses (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/mesh-gateway-custom-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway-bind-addrs (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway-bind-addrs (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-gateway-no-services (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-custom-and-tagged-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-http-multiple-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-services (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-custom-and-tagged-addresses (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-service-subsets (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-http-multiple-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.13.6/terminating-gateway-no-api-cert (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tls-listener (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7 (0.23s) + --- PASS: TestListenersFromSnapshot/envoy-1.13.6/ingress-with-tls-listener (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7 (0.51s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/defaults (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/listener-bind-address (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/listener-bind-port (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/listener-bind-address (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/listener-bind-port (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/listener-bind-address-port (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/http-public-listener (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/http-upstream (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-public-listener (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-public-listener-typed (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-public-listener-ignores-tls (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-upstream (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-upstream (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/custom-upstream-typed-ignored-with-disco-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-http-chain (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-http2-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-grpc-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/expose-paths-local-app-paths (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-http-chain (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-http2-chain (0.03s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-grpc-chain (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/expose-paths-local-app-paths (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/expose-paths-new-cluster-http2 (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-tagged-addresses (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-using-federation-states (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-no-services (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-tagged-addresses (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/mesh-gateway-custom-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway-bind-addrs (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway-bind-addrs (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-gateway-no-services (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.02s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-custom-and-tagged-addresses (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-services (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-custom-and-tagged-addresses (0.02s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-service-subsets (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-http-multiple-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-http-multiple-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.12.7/terminating-gateway-no-api-cert (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tls-listener (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2 (0.21s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/defaults (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/listener-bind-address (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.12.7/ingress-with-tls-listener (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2 (0.49s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/defaults (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/listener-bind-address (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/listener-bind-port (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/listener-bind-address-port (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/listener-bind-address-port (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/http-public-listener (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/http-upstream (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-public-listener (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-public-listener-typed (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-public-listener-ignores-tls (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-public-listener-typed (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-public-listener-ignores-tls (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-upstream (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed-ignored-with-disco-chain (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/custom-upstream-typed-ignored-with-disco-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-http-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-http2-chain (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-grpc-chain (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/connect-proxy-with-tcp-chain-failover-through-local-gateway (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/expose-paths-local-app-paths (0.00s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/expose-paths-new-cluster-http2 (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-tagged-addresses (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-custom-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway-bind-addrs (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.00s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-using-federation-states (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-no-services (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-tagged-addresses (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/mesh-gateway-custom-addresses (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway-bind-addrs (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-gateway-no-services (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-remote-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tcp-chain-failover-through-local-gateway (0.01s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-custom-and-tagged-addresses (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-http-multiple-services (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-service-subsets (0.02s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-http-multiple-services (0.01s) --- PASS: TestListenersFromSnapshot/envoy-1.11.2/terminating-gateway-no-api-cert (0.01s) - --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tls-listener (0.00s) + --- PASS: TestListenersFromSnapshot/envoy-1.11.2/ingress-with-tls-listener (0.01s) === RUN TestRoutesFromSnapshot === RUN TestRoutesFromSnapshot/envoy-1.14.5 === RUN TestRoutesFromSnapshot/envoy-1.14.5/defaults-no-chain @@ -72251,113 +72433,113 @@ === RUN TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-grpc-router === RUN TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-and-router === RUN TestRoutesFromSnapshot/envoy-1.11.2/ingress-http-multiple-services ---- PASS: TestRoutesFromSnapshot (0.40s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5 (0.08s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/defaults-no-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain (0.00s) +--- PASS: TestRoutesFromSnapshot (0.50s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5 (0.13s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/defaults-no-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-overrides (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/splitter-with-resolver-redirect (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-splitter (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/connect-proxy-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-defaults-no-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-defaults-no-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-external-sni (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-splitter-with-resolver-redirect (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-and-splitter (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-http-multiple-services (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6 (0.08s) + --- PASS: TestRoutesFromSnapshot/envoy-1.14.5/ingress-http-multiple-services (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6 (0.10s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/defaults-no-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-external-sni (0.00s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-splitter (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/splitter-with-resolver-redirect (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-splitter (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/connect-proxy-with-chain-and-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-defaults-no-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-external-sni (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-and-overrides (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-splitter-with-resolver-redirect (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-and-splitter (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-and-splitter (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-http-multiple-services (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7 (0.17s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/defaults-no-chain (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.13.6/ingress-http-multiple-services (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7 (0.14s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/defaults-no-chain (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.09s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-external-sni (0.00s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-overrides (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/splitter-with-resolver-redirect (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-splitter (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-splitter (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-grpc-router (0.02s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/connect-proxy-with-chain-and-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-defaults-no-chain (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-and-splitter (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-external-sni (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-and-overrides (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-and-splitter (0.02s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-http-multiple-services (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2 (0.07s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/defaults-no-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-splitter (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.12.7/ingress-http-multiple-services (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2 (0.13s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/defaults-no-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-external-sni (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-overrides (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/splitter-with-resolver-redirect (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-splitter (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/connect-proxy-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-defaults-no-chain (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-defaults-no-chain (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-external-sni (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-and-overrides (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-and-splitter (0.00s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-grpc-router (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-splitter-with-resolver-redirect (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-and-splitter (0.01s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-grpc-router (0.01s) --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-with-chain-and-router (0.01s) - --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-http-multiple-services (0.00s) + --- PASS: TestRoutesFromSnapshot/envoy-1.11.2/ingress-http-multiple-services (0.01s) === RUN TestServer_StreamAggregatedResources_BasicProtocol -2022-11-07T22:44:45.318Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.319Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.344Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.344Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.361Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.361Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul ---- PASS: TestServer_StreamAggregatedResources_BasicProtocol (0.06s) +2021-10-05T16:29:52.973Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:52.975Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.006Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.006Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.026Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.026Z [DEBUG] TestServer_StreamAggregatedResources_BasicProtocol.envoy: generating endpoints for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +--- PASS: TestServer_StreamAggregatedResources_BasicProtocol (0.08s) === RUN TestServer_StreamAggregatedResources_ACLEnforcement server_test.go:396: DM-skipped --- SKIP: TestServer_StreamAggregatedResources_ACLEnforcement (0.00s) === RUN TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest -2022-11-07T22:44:45.378Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.379Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest.envoy: Error handling ADS stream: error="rpc error: code = Unauthenticated desc = unauthenticated: ACL not found" +2021-10-05T16:29:53.047Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.049Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest.envoy: Error handling ADS stream: error="rpc error: code = Unauthenticated desc = unauthenticated: ACL not found" --- PASS: TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedDuringDiscoveryRequest (0.02s) === RUN TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground -2022-11-07T22:44:45.393Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul -2022-11-07T22:44:45.494Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground.envoy: Error handling ADS stream: error="rpc error: code = Unauthenticated desc = unauthenticated: ACL not found" ---- PASS: TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground (0.11s) +2021-10-05T16:29:53.066Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground.envoy: generating cluster for: cluster=db.default.dc1.internal.11111111-2222-3333-4444-555555555555.consul +2021-10-05T16:29:53.175Z [DEBUG] TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground.envoy: Error handling ADS stream: error="rpc error: code = Unauthenticated desc = unauthenticated: ACL not found" +--- PASS: TestServer_StreamAggregatedResources_ACLTokenDeleted_StreamTerminatedInBackground (0.13s) === RUN TestServer_Check === RUN TestServer_Check/auth_allowed -2022-11-07T22:44:45.494Z [DEBUG] TestServer_Check/auth_allowed.envoy: Connect AuthZ ALLOWED: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db reason="default allow" +2021-10-05T16:29:53.175Z [DEBUG] TestServer_Check/auth_allowed.envoy: Connect AuthZ ALLOWED: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db reason="default allow" === RUN TestServer_Check/auth_denied -2022-11-07T22:44:45.494Z [DEBUG] TestServer_Check/auth_denied.envoy: Connect AuthZ DENIED: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db reason="default deny" +2021-10-05T16:29:53.175Z [DEBUG] TestServer_Check/auth_denied.envoy: Connect AuthZ DENIED: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db reason="default deny" === RUN TestServer_Check/no_source === RUN TestServer_Check/no_dest === RUN TestServer_Check/dest_invalid_format -2022-11-07T22:44:45.494Z [DEBUG] TestServer_Check/dest_invalid_format.envoy: Connect AuthZ DENIED: bad destination URI: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=not-a-spiffe-id +2021-10-05T16:29:53.176Z [DEBUG] TestServer_Check/dest_invalid_format.envoy: Connect AuthZ DENIED: bad destination URI: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=not-a-spiffe-id === RUN TestServer_Check/dest_not_a_service_URI -2022-11-07T22:44:45.495Z [DEBUG] TestServer_Check/dest_not_a_service_URI.envoy: Connect AuthZ DENIED: bad destination service ID: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://trust-domain.consul +2021-10-05T16:29:53.176Z [DEBUG] TestServer_Check/dest_not_a_service_URI.envoy: Connect AuthZ DENIED: bad destination service ID: source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://trust-domain.consul === RUN TestServer_Check/ACL_not_got_permission_for_authz_call -2022-11-07T22:44:45.495Z [DEBUG] TestServer_Check/ACL_not_got_permission_for_authz_call.envoy: Connect AuthZ failed ACL check: error="Permission denied" source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web dest=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db +2021-10-05T16:29:53.176Z [DEBUG] TestServer_Check/ACL_not_got_permission_for_authz_call.envoy: Connect AuthZ failed ACL check: error="Permission denied" source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web dest=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db === RUN TestServer_Check/Random_error_running_authz -2022-11-07T22:44:45.495Z [DEBUG] TestServer_Check/Random_error_running_authz.envoy: Connect AuthZ failed: error="gremlin attack" source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db +2021-10-05T16:29:53.176Z [DEBUG] TestServer_Check/Random_error_running_authz.envoy: Connect AuthZ failed: error="gremlin attack" source=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/web destination=spiffe://11111111-2222-3333-4444-555555555555.consul/ns/default/dc/dc1/svc/db --- PASS: TestServer_Check (0.00s) --- PASS: TestServer_Check/auth_allowed (0.00s) --- PASS: TestServer_Check/auth_denied (0.00s) @@ -72368,9 +72550,9 @@ --- PASS: TestServer_Check/ACL_not_got_permission_for_authz_call (0.00s) --- PASS: TestServer_Check/Random_error_running_authz (0.00s) === RUN TestServer_StreamAggregatedResources_IngressEmptyResponse ---- PASS: TestServer_StreamAggregatedResources_IngressEmptyResponse (0.01s) +--- PASS: TestServer_StreamAggregatedResources_IngressEmptyResponse (0.02s) PASS -ok github.com/hashicorp/consul/agent/xds 3.559s +ok github.com/hashicorp/consul/agent/xds 6.616s ? github.com/hashicorp/consul/agent/xds/proxysupport [no test files] ? github.com/hashicorp/consul/command [no test files] ? github.com/hashicorp/consul/command/acl [no test files] @@ -72384,90 +72566,91 @@ --- PASS: TestAgentTokensCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:44:43.677Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:43.692Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:43.728 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f655c03c-d9bd-43a1-f755-ac4e9a8d8210 Address:127.0.0.1:18174}]" -44:43.728 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -44:43.729 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210.dc1 127.0.0.1 -44:43.729 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 127.0.0.1 -2022-11-07T22:44:43.729Z [INFO] agent.router: Initializing LAN area manager -44:43.730 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210.dc1 area=wan -44:43.730 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -44:43.730 [INFO] TestAgent.server: Adding LAN server: server="Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:43.730 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -44:43.732 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -44:43.732 [INFO] TestAgent: started state syncer -44:43.732 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -44:43.775 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:43.775 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -44:43.780 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:43.782 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:43.782 [DEBUG] TestAgent.server.raft: vote granted: from=f655c03c-d9bd-43a1-f755-ac4e9a8d8210 term=2 tally=1 -44:43.782 [INFO] TestAgent.server.raft: election won: tally=1 -44:43.782 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -44:43.782 [INFO] TestAgent.server: cluster leadership acquired -44:43.782 [INFO] TestAgent.server: New leader elected: payload=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 -44:43.787 [INFO] TestAgent.server: initializing acls -44:43.790 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:43.790 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:43.795 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:43.802 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:43.802 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:43.802 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:43.803 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 -44:43.803 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210.dc1 -44:43.803 [INFO] TestAgent.server: Updating LAN server: server="Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:43.803 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210.dc1 area=wan -44:43.810 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:43.827 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:43.827 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:43.827 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:43.827 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:43.827 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:43.827 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 -44:43.827 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 -44:43.831 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f655c03c-d9bd-43a1-f755-ac4e9a8d8210 -44:43.836 [INFO] TestAgent.server: federation state anti-entropy synced -44:43.844 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:43.848 [INFO] TestAgent: Synced node info -44:43.912 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46526 latency=4.173577ms -44:43.913 [INFO] TestAgent: Updated agent's ACL token: token=default -44:43.913 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/default from=127.0.0.1:46528 latency=72.939µs -44:43.913 [INFO] TestAgent: Updated agent's ACL token: token=agent -44:43.914 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/agent from=127.0.0.1:46530 latency=63.955µs -44:43.914 [INFO] TestAgent: Updated agent's ACL token: token=agent_master -44:43.914 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/agent_master from=127.0.0.1:46532 latency=58.056µs -44:43.914 [INFO] TestAgent: Updated agent's ACL token: token=replication -44:43.914 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/replication from=127.0.0.1:46534 latency=31.007µs -44:43.915 [INFO] TestAgent: Requesting shutdown -44:43.915 [INFO] TestAgent.server: shutting down server -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:43.915 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:43.915 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:43.915 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:43.915 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:43.918 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:43.920Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:43.920Z [INFO] agent.router.manager: shutting down -44:43.921 [INFO] TestAgent: consul server down -44:43.921 [INFO] TestAgent: shutdown complete -44:43.921 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -44:43.921 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -44:43.921 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -44:44.421 [INFO] TestAgent: Waiting for endpoints to shut down -44:44.421 [INFO] TestAgent: Endpoints down ---- PASS: TestAgentTokensCommand (0.78s) +2021-10-05T16:29:49.664Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:49.681Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:49.710 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab476ec3-311f-2720-b510-ec39faa83891 Address:127.0.0.1:25321}]" +29:49.711 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab476ec3-311f-2720-b510-ec39faa83891.dc1 127.0.0.1 +29:49.712 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab476ec3-311f-2720-b510-ec39faa83891 127.0.0.1 +2021-10-05T16:29:49.712Z [INFO] agent.router: Initializing LAN area manager +29:49.712 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp +29:49.712 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= +29:49.713 [INFO] TestAgent.server: Adding LAN server: server="Node-ab476ec3-311f-2720-b510-ec39faa83891 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +29:49.713 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab476ec3-311f-2720-b510-ec39faa83891.dc1 area=wan +29:49.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp +29:49.713 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp +29:49.714 [INFO] TestAgent: started state syncer +29:49.714 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp +29:49.763 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:49.782 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:49.782 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 +29:49.790 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:49.790 [DEBUG] TestAgent.server.raft: vote granted: from=ab476ec3-311f-2720-b510-ec39faa83891 term=2 tally=1 +29:49.790 [INFO] TestAgent.server.raft: election won: tally=1 +29:49.790 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" +29:49.791 [INFO] TestAgent.server: cluster leadership acquired +29:49.791 [INFO] TestAgent.server: New leader elected: payload=Node-ab476ec3-311f-2720-b510-ec39faa83891 +29:49.797 [INFO] TestAgent.server: initializing acls +29:49.813 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:49.813 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:49.820 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:49.828 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:49.828 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:49.828 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:49.828 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ab476ec3-311f-2720-b510-ec39faa83891 +29:49.828 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ab476ec3-311f-2720-b510-ec39faa83891.dc1 +29:49.828 [INFO] TestAgent.server: Updating LAN server: server="Node-ab476ec3-311f-2720-b510-ec39faa83891 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +29:49.829 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ab476ec3-311f-2720-b510-ec39faa83891.dc1 area=wan +29:49.844 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:49.863 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:49.863 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:49.863 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:49.863 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:49.863 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:49.863 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab476ec3-311f-2720-b510-ec39faa83891 +29:49.863 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab476ec3-311f-2720-b510-ec39faa83891 +29:49.872 [INFO] TestAgent.server: federation state anti-entropy synced +29:49.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab476ec3-311f-2720-b510-ec39faa83891 +29:50.046 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:50.056 [INFO] TestAgent: Synced node info +29:50.056 [DEBUG] TestAgent: Node info in sync +29:50.078 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:52692 latency=11.056089ms +29:50.084 [INFO] TestAgent: Updated agent's ACL token: token=default +29:50.084 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/default from=127.0.0.1:52694 latency=170.43µs +29:50.085 [INFO] TestAgent: Updated agent's ACL token: token=agent +29:50.085 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/agent from=127.0.0.1:52696 latency=80.6µs +29:50.085 [INFO] TestAgent: Updated agent's ACL token: token=agent_master +29:50.085 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/agent_master from=127.0.0.1:52698 latency=77.195µs +29:50.087 [INFO] TestAgent: Updated agent's ACL token: token=replication +29:50.087 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/token/replication from=127.0.0.1:52700 latency=64.167µs +29:50.092 [INFO] TestAgent: Requesting shutdown +29:50.092 [INFO] TestAgent.server: shutting down server +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:50.092 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:50.092 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:50.092 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:50.092 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:50.099 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:50.105Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:50.105Z [INFO] agent.router.manager: shutting down +29:50.105 [INFO] TestAgent: consul server down +29:50.105 [INFO] TestAgent: shutdown complete +29:50.105 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp +29:50.106 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp +29:50.106 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp +29:50.612 [INFO] TestAgent: Waiting for endpoints to shut down +29:50.612 [INFO] TestAgent: Endpoints down +--- PASS: TestAgentTokensCommand (1.00s) PASS -ok github.com/hashicorp/consul/command/acl/agenttokens 0.801s +ok github.com/hashicorp/consul/command/acl/agenttokens 1.089s ? github.com/hashicorp/consul/command/acl/authmethod [no test files] === RUN TestAuthMethodCreateCommand_noTabs === PAUSE TestAuthMethodCreateCommand_noTabs @@ -72480,482 +72663,488 @@ === RUN TestAuthMethodCreateCommand_config === PAUSE TestAuthMethodCreateCommand_config === CONT TestAuthMethodCreateCommand_noTabs -=== CONT TestAuthMethodCreateCommand_k8s -=== CONT TestAuthMethodCreateCommand_JSON === CONT TestAuthMethodCreateCommand_config -=== CONT TestAuthMethodCreateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestAuthMethodCreateCommand_JSON --- PASS: TestAuthMethodCreateCommand_noTabs (0.00s) +=== CONT TestAuthMethodCreateCommand +=== CONT TestAuthMethodCreateCommand_k8s [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== RUN TestAuthMethodCreateCommand_config/config_file -=== RUN TestAuthMethodCreateCommand_config/config_std-in -=== RUN TestAuthMethodCreateCommand_config/config_string === RUN TestAuthMethodCreateCommand_k8s/k8s_host_required === RUN TestAuthMethodCreateCommand_k8s/k8s_ca_cert_required === RUN TestAuthMethodCreateCommand_k8s/k8s_jwt_required === RUN TestAuthMethodCreateCommand_k8s/create_k8s === RUN TestAuthMethodCreateCommand_k8s/create_k8s_with_cert_file -=== RUN TestAuthMethodCreateCommand_JSON/type_required -=== RUN TestAuthMethodCreateCommand_JSON/create_testing -=== RUN TestAuthMethodCreateCommand_JSON/create_testing_with_max_token_ttl -=== RUN TestAuthMethodCreateCommand_JSON/create_testing_with_token_type_global +=== RUN TestAuthMethodCreateCommand_config/config_file +=== RUN TestAuthMethodCreateCommand_config/config_std-in +=== RUN TestAuthMethodCreateCommand_config/config_string === RUN TestAuthMethodCreateCommand/type_required === RUN TestAuthMethodCreateCommand/name_required === RUN TestAuthMethodCreateCommand/invalid_type === RUN TestAuthMethodCreateCommand/create_testing === RUN TestAuthMethodCreateCommand/create_testing_with_max_token_ttl === RUN TestAuthMethodCreateCommand/create_testing_with_token_type_global -2022-11-07T22:44:46.467Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.480Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.507 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b938f69d-1503-981f-2f23-66afead71278 Address:127.0.0.1:26349}]" -44:46.507 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= -44:46.508 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b938f69d-1503-981f-2f23-66afead71278.dc1 127.0.0.1 -44:46.508 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b938f69d-1503-981f-2f23-66afead71278 127.0.0.1 -2022-11-07T22:44:46.508Z [INFO] agent.router: Initializing LAN area manager -44:46.509 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b938f69d-1503-981f-2f23-66afead71278.dc1 area=wan -44:46.509 [INFO] TestAgent.server: Adding LAN server: server="Node-b938f69d-1503-981f-2f23-66afead71278 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -44:46.509 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp -44:46.509 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp -44:46.509 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp -44:46.509 [INFO] TestAgent: started state syncer -44:46.509 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp -44:46.544 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.544 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 -44:46.553 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.553 [DEBUG] TestAgent.server.raft: vote granted: from=b938f69d-1503-981f-2f23-66afead71278 term=2 tally=1 -44:46.553 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.553 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" -44:46.553 [INFO] TestAgent.server: cluster leadership acquired -44:46.553 [INFO] TestAgent.server: New leader elected: payload=Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.558 [INFO] TestAgent.server: initializing acls -44:46.559 [INFO] TestAgent.server: initializing acls -44:46.562 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.562 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.568 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.568 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.580 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.584 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.596 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.596 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.596 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.596 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.596 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:46.596 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.596 [INFO] TestAgent.server: Updating LAN server: server="Node-b938f69d-1503-981f-2f23-66afead71278 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -44:46.596 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b938f69d-1503-981f-2f23-66afead71278.dc1 -44:46.597 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b938f69d-1503-981f-2f23-66afead71278.dc1 area=wan -44:46.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.597 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b938f69d-1503-981f-2f23-66afead71278.dc1 -44:46.597 [INFO] TestAgent.server: Updating LAN server: server="Node-b938f69d-1503-981f-2f23-66afead71278 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -44:46.597 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b938f69d-1503-981f-2f23-66afead71278.dc1 area=wan -44:46.611 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.629 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.629 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.629 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.629 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.629 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.630 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.630 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.644 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.644 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.644 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b938f69d-1503-981f-2f23-66afead71278 -44:46.664 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:54338 latency=3.15351ms -44:46.666 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-4a196c19-864e-3362-308d-8f7327679be1 from=127.0.0.1:54340 latency=482.034µs -44:46.671 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:54342 latency=3.808377ms -44:46.671 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-e31ddf03-7eea-6331-2909-f3e642f06469 from=127.0.0.1:54340 latency=113.865µs -44:46.675 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:54344 latency=2.78434ms -44:46.677 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-571abb3f-2c1a-c46a-62ff-69a02f928ff9 from=127.0.0.1:54340 latency=1.078772ms -44:46.682 [INFO] TestAgent: Requesting shutdown -44:46.682 [INFO] TestAgent.server: shutting down server -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.682 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.682 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.682 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.682 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.682 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.685 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.689Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.689Z [INFO] agent.router.manager: shutting down -44:46.689 [INFO] TestAgent: consul server down -44:46.689 [INFO] TestAgent: shutdown complete -44:46.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp -44:46.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp -44:46.689 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp -44:47.189 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.189 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodCreateCommand_config (0.77s) - --- PASS: TestAuthMethodCreateCommand_config/config_file (0.01s) - --- PASS: TestAuthMethodCreateCommand_config/config_std-in (0.01s) - --- PASS: TestAuthMethodCreateCommand_config/config_string (0.01s) -2022-11-07T22:44:46.468Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.487Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.514 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 Address:127.0.0.1:26342}]" -44:46.514 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= -44:46.514 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831.dc1 127.0.0.1 -44:46.515 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 127.0.0.1 -2022-11-07T22:44:46.515Z [INFO] agent.router: Initializing LAN area manager -44:46.515 [INFO] TestAgent.server: Adding LAN server: server="Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -44:46.515 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831.dc1 area=wan -44:46.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp -44:46.515 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp -44:46.515 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp -44:46.515 [INFO] TestAgent: started state syncer -44:46.515 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp -44:46.566 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:46.581 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.581 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 -44:46.594 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.594 [DEBUG] TestAgent.server.raft: vote granted: from=a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 term=2 tally=1 -44:46.594 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.594 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" -44:46.594 [INFO] TestAgent.server: cluster leadership acquired -44:46.594 [INFO] TestAgent.server: New leader elected: payload=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 -44:46.601 [INFO] TestAgent.server: initializing acls -44:46.606 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.606 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.615 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.628 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.628 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.628 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.628 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 -44:46.628 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831.dc1 -44:46.629 [INFO] TestAgent.server: Updating LAN server: server="Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -44:46.629 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831.dc1 area=wan -44:46.640 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.657 [INFO] TestAgent: Synced node info -44:46.661 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.661 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.661 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.661 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.661 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.661 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 -44:46.661 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 -44:46.669 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.669 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a8ae3fb2-9ff6-a23a-93bb-3ffe1aef6831 -44:46.853 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56302 latency=6.750397ms -44:46.855 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-175aad81-09a9-1519-5bdb-6157103d6970 from=127.0.0.1:56304 latency=601.367µs -44:46.865 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56306 latency=8.789511ms -44:46.866 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-22bfebd3-cc61-333e-d37d-93403c731a57 from=127.0.0.1:56304 latency=185.931µs -44:46.866 [INFO] TestAgent: Requesting shutdown -44:46.866 [INFO] TestAgent.server: shutting down server -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.866 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.866 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.866 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.866 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.870 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.873Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.873Z [INFO] agent.router.manager: shutting down -44:46.873 [INFO] TestAgent: consul server down -44:46.873 [INFO] TestAgent: shutdown complete -44:46.873 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp -44:46.874 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp -44:46.874 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp -44:47.374 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.374 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodCreateCommand_k8s (0.96s) +=== RUN TestAuthMethodCreateCommand_JSON/type_required +=== RUN TestAuthMethodCreateCommand_JSON/create_testing +=== RUN TestAuthMethodCreateCommand_JSON/create_testing_with_max_token_ttl +=== RUN TestAuthMethodCreateCommand_JSON/create_testing_with_token_type_global +2021-10-05T16:29:50.978Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:51.016Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.120 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ab846f55-0116-5db1-3c8a-7435c41a2c7a Address:127.0.0.1:14111}]" +29:51.121 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a.dc1 127.0.0.1 +29:51.121 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a 127.0.0.1 +2021-10-05T16:29:51.121Z [INFO] agent.router: Initializing LAN area manager +29:51.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14106 network=udp +29:51.122 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14111 [Follower]" leader= +29:51.122 [INFO] TestAgent.server: Adding LAN server: server="Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a (Addr: tcp/127.0.0.1:14111) (DC: dc1)" +29:51.122 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a.dc1 area=wan +29:51.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14106 network=tcp +29:51.123 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14107 network=tcp +29:51.123 [INFO] TestAgent: started state syncer +29:51.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14112 network=tcp +29:51.160 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.160 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14111 [Candidate]" term=2 +29:51.173 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:51.188 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.188 [DEBUG] TestAgent.server.raft: vote granted: from=ab846f55-0116-5db1-3c8a-7435c41a2c7a term=2 tally=1 +29:51.189 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.189 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14111 [Leader]" +29:51.189 [INFO] TestAgent.server: cluster leadership acquired +29:51.189 [INFO] TestAgent.server: New leader elected: payload=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a +29:51.202 [INFO] TestAgent.server: initializing acls +29:51.210 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.210 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.223 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.249 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.249 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.249 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.249 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a +29:51.249 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a.dc1 +29:51.250 [INFO] TestAgent.server: Updating LAN server: server="Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a (Addr: tcp/127.0.0.1:14111) (DC: dc1)" +29:51.250 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a.dc1 area=wan +29:51.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.361 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.362 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.362 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.362 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.362 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.362 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a +29:51.362 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a +29:51.372 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ab846f55-0116-5db1-3c8a-7435c41a2c7a +29:51.387 [INFO] TestAgent.server: federation state anti-entropy synced +29:51.395 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46670 latency=13.068466ms +29:51.405 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-b7bb1b1e-9a3a-8885-71ca-08503e931e62 from=127.0.0.1:46672 latency=7.632881ms +29:51.441 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46674 latency=15.075913ms +29:51.442 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-443245ef-d05e-f18a-d19a-207ea68732e0 from=127.0.0.1:46672 latency=282.133µs +29:51.446 [INFO] TestAgent: Requesting shutdown +29:51.447 [INFO] TestAgent.server: shutting down server +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.447 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.447 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.447 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.447 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.447 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.452 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.461Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.461Z [INFO] agent.router.manager: shutting down +29:51.462 [INFO] TestAgent: consul server down +29:51.462 [INFO] TestAgent: shutdown complete +29:51.462 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14106 network=tcp +29:51.462 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14106 network=udp +29:51.462 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14107 network=tcp +29:51.962 [INFO] TestAgent: Waiting for endpoints to shut down +29:51.962 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodCreateCommand_k8s (1.13s) --- PASS: TestAuthMethodCreateCommand_k8s/k8s_host_required (0.00s) --- PASS: TestAuthMethodCreateCommand_k8s/k8s_ca_cert_required (0.00s) --- PASS: TestAuthMethodCreateCommand_k8s/k8s_jwt_required (0.00s) - --- PASS: TestAuthMethodCreateCommand_k8s/create_k8s (0.01s) - --- PASS: TestAuthMethodCreateCommand_k8s/create_k8s_with_cert_file (0.01s) -2022-11-07T22:44:46.467Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.482Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.509 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c59df83b-cab2-ff89-d47e-c6b70b18f003 Address:127.0.0.1:26363}]" -44:46.509 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26363 [Follower]" leader= -44:46.509 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c59df83b-cab2-ff89-d47e-c6b70b18f003.dc1 127.0.0.1 -44:46.510 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 127.0.0.1 -2022-11-07T22:44:46.510Z [INFO] agent.router: Initializing LAN area manager -44:46.510 [INFO] TestAgent.server: Adding LAN server: server="Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 (Addr: tcp/127.0.0.1:26363) (DC: dc1)" -44:46.510 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=udp -44:46.510 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003.dc1 area=wan -44:46.510 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=tcp -44:46.512 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26359 network=tcp -44:46.512 [INFO] TestAgent: started state syncer -44:46.512 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26364 network=tcp -44:46.560 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:46.567 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.567 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26363 [Candidate]" term=2 -44:46.576 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.576 [DEBUG] TestAgent.server.raft: vote granted: from=c59df83b-cab2-ff89-d47e-c6b70b18f003 term=2 tally=1 -44:46.576 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.576 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26363 [Leader]" -44:46.576 [INFO] TestAgent.server: cluster leadership acquired -44:46.576 [INFO] TestAgent.server: New leader elected: payload=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 -44:46.582 [INFO] TestAgent.server: initializing acls -44:46.589 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.589 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.597 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.607 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.607 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.607 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.607 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 -44:46.607 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c59df83b-cab2-ff89-d47e-c6b70b18f003.dc1 -44:46.607 [INFO] TestAgent.server: Updating LAN server: server="Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 (Addr: tcp/127.0.0.1:26363) (DC: dc1)" -44:46.608 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003.dc1 area=wan -44:46.620 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.642 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.642 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.642 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.642 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.642 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.642 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 -44:46.642 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 -44:46.648 [INFO] TestAgent: Synced node info -44:46.653 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c59df83b-cab2-ff89-d47e-c6b70b18f003 -44:46.657 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.923 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48574 latency=3.789323ms -44:46.924 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-1618fde3-300e-4758-4ce5-3d368c5eb114 from=127.0.0.1:48576 latency=200.667µs -44:46.929 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48578 latency=3.94145ms -44:46.930 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-2dc67e7f-663c-ae09-ec88-20e846072836 from=127.0.0.1:48576 latency=739.415µs -44:46.934 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48580 latency=2.960667ms -44:46.935 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-77827b24-ebd9-029a-c867-9e42db1ecd3a from=127.0.0.1:48576 latency=163.457µs -44:46.935 [INFO] TestAgent: Requesting shutdown -44:46.935 [INFO] TestAgent.server: shutting down server -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.935 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.935 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.935 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.935 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.941 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.944Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.944Z [INFO] agent.router.manager: shutting down -44:46.944 [INFO] TestAgent: consul server down -44:46.944 [INFO] TestAgent: shutdown complete -44:46.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=tcp -44:46.944 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=udp -44:46.944 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26359 network=tcp -44:47.444 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.444 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodCreateCommand_JSON (1.03s) - --- PASS: TestAuthMethodCreateCommand_JSON/type_required (0.00s) - --- PASS: TestAuthMethodCreateCommand_JSON/create_testing (0.01s) - --- PASS: TestAuthMethodCreateCommand_JSON/create_testing_with_max_token_ttl (0.01s) - --- PASS: TestAuthMethodCreateCommand_JSON/create_testing_with_token_type_global (0.00s) -2022-11-07T22:44:46.465Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:46.485Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:46.513 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:81d8f675-bf64-98e3-c80c-f5b21d8c93b4 Address:127.0.0.1:26356}]" -44:46.513 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26356 [Follower]" leader= -44:46.513 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 127.0.0.1 -44:46.514 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 127.0.0.1 -2022-11-07T22:44:46.514Z [INFO] agent.router: Initializing LAN area manager -44:46.514 [INFO] TestAgent.server: Adding LAN server: server="Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 (Addr: tcp/127.0.0.1:26356) (DC: dc1)" -44:46.514 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 area=wan -44:46.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=udp -44:46.514 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=tcp -44:46.514 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26352 network=tcp -44:46.514 [INFO] TestAgent: started state syncer -44:46.514 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26357 network=tcp -44:46.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:46.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26356 [Candidate]" term=2 -44:46.564 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:46.564 [DEBUG] TestAgent.server.raft: vote granted: from=81d8f675-bf64-98e3-c80c-f5b21d8c93b4 term=2 tally=1 -44:46.564 [INFO] TestAgent.server.raft: election won: tally=1 -44:46.564 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26356 [Leader]" -44:46.564 [INFO] TestAgent.server: cluster leadership acquired -44:46.564 [INFO] TestAgent.server: initializing acls -44:46.564 [INFO] TestAgent.server: New leader elected: payload=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.571 [INFO] TestAgent.server: initializing acls -44:46.571 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.571 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:46.571 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:46.588 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.588 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:46.592 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.592 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:46.592 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:46.592 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:46.592 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.592 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 -44:46.592 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 area=wan -44:46.592 [INFO] TestAgent.server: Updating LAN server: server="Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 (Addr: tcp/127.0.0.1:26356) (DC: dc1)" -44:46.597 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:46.597 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.597 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 -44:46.597 [INFO] TestAgent.server: Updating LAN server: server="Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 (Addr: tcp/127.0.0.1:26356) (DC: dc1)" -44:46.597 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4.dc1 area=wan -44:46.610 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:46.630 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:46.630 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:46.630 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:46.630 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:46.630 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:46.630 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.630 [INFO] TestAgent.server: member joined, marking health alive: member=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.636 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.637 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-81d8f675-bf64-98e3-c80c-f5b21d8c93b4 -44:46.645 [INFO] TestAgent.server: federation state anti-entropy synced -44:46.691 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:46.694 [INFO] TestAgent: Synced node info -44:46.814 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:46.814 [DEBUG] TestAgent: Node info in sync -44:46.814 [DEBUG] TestAgent: Node info in sync -44:46.954 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51338 error="Invalid Auth Method: Type should be one of: [jwt kubernetes testing]" -44:46.954 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51338 latency=529.699µs -44:46.959 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51340 latency=3.017221ms -44:46.960 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-bb28dc16-dbce-c15e-b31e-efe6d6b3d310 from=127.0.0.1:51342 latency=166.041µs -44:46.964 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51344 latency=2.956261ms -44:46.965 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-e2b4012a-a454-308b-5c77-f901aa9e4c80 from=127.0.0.1:51342 latency=393.983µs -44:46.969 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51346 latency=3.088663ms -44:46.969 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-10ca4fdd-aff3-1a3f-f132-53d6b2bcb3b9 from=127.0.0.1:51342 latency=119.43µs -44:46.969 [INFO] TestAgent: Requesting shutdown -44:46.969 [INFO] TestAgent.server: shutting down server -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:46.969 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:46.969 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:46.969 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:46.969 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:46.969 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:46.970 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:46.970 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:46.970 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:46.970 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:46.974 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:46.977Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:46.977Z [INFO] agent.router.manager: shutting down -44:46.978 [INFO] TestAgent: consul server down -44:46.978 [INFO] TestAgent: shutdown complete -44:46.978 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=tcp -44:46.978 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=udp -44:46.978 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26352 network=tcp -44:47.478 [INFO] TestAgent: Waiting for endpoints to shut down -44:47.478 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodCreateCommand (1.06s) + --- PASS: TestAuthMethodCreateCommand_k8s/create_k8s (0.03s) + --- PASS: TestAuthMethodCreateCommand_k8s/create_k8s_with_cert_file (0.04s) +2021-10-05T16:29:50.982Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:51.018Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.085 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7227c63f-9b5c-588d-4847-95af215d9f27 Address:127.0.0.1:14090}]" +29:51.092 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 127.0.0.1 +29:51.093 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +29:51.120 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7227c63f-9b5c-588d-4847-95af215d9f27 127.0.0.1 +2021-10-05T16:29:51.121Z [INFO] agent.router: Initializing LAN area manager +29:51.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +29:51.121 [INFO] TestAgent.server: Adding LAN server: server="Node-7227c63f-9b5c-588d-4847-95af215d9f27 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +29:51.121 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 area=wan +29:51.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +29:51.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +29:51.122 [INFO] TestAgent: started state syncer +29:51.122 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +29:51.143 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.143 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +29:51.166 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.166 [DEBUG] TestAgent.server.raft: vote granted: from=7227c63f-9b5c-588d-4847-95af215d9f27 term=2 tally=1 +29:51.166 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.166 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +29:51.167 [INFO] TestAgent.server: cluster leadership acquired +29:51.167 [INFO] TestAgent.server: New leader elected: payload=Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.173 [INFO] TestAgent.server: initializing acls +29:51.186 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.186 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.187 [INFO] TestAgent.server: initializing acls +29:51.187 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.221 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.221 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.244 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.244 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.244 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.244 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:51.245 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.245 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 +29:51.245 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 area=wan +29:51.245 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.245 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.245 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 +29:51.246 [INFO] TestAgent.server: Updating LAN server: server="Node-7227c63f-9b5c-588d-4847-95af215d9f27 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +29:51.246 [INFO] TestAgent.server: Updating LAN server: server="Node-7227c63f-9b5c-588d-4847-95af215d9f27 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +29:51.246 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7227c63f-9b5c-588d-4847-95af215d9f27.dc1 area=wan +29:51.266 [INFO] TestAgent: Synced node info +29:51.266 [DEBUG] TestAgent: Node info in sync +29:51.275 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.311 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.311 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.311 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.311 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.311 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.311 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.311 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.322 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.322 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7227c63f-9b5c-588d-4847-95af215d9f27 +29:51.335 [INFO] TestAgent.server: federation state anti-entropy synced +29:51.446 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43054 latency=6.979704ms +29:51.447 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-8558741c-7ef9-ed6d-989b-91edca0b4b96 from=127.0.0.1:43056 latency=285.698µs +29:51.453 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43058 latency=5.147477ms +29:51.454 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-293fcd9c-3aa6-bbdb-025c-10d2b2476c9f from=127.0.0.1:43056 latency=202.898µs +29:51.466 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43060 latency=6.159696ms +29:51.473 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-8a4d2705-1f39-d7cf-6317-61d4b8c5d8a1 from=127.0.0.1:43056 latency=265.71µs +29:51.473 [INFO] TestAgent: Requesting shutdown +29:51.473 [INFO] TestAgent.server: shutting down server +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.473 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.473 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.473 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.473 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.480 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.485Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.485Z [INFO] agent.router.manager: shutting down +29:51.485 [INFO] TestAgent: consul server down +29:51.485 [INFO] TestAgent: shutdown complete +29:51.485 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +29:51.485 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +29:51.485 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +29:51.985 [INFO] TestAgent: Waiting for endpoints to shut down +29:51.985 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodCreateCommand_config (1.15s) + --- PASS: TestAuthMethodCreateCommand_config/config_file (0.01s) + --- PASS: TestAuthMethodCreateCommand_config/config_std-in (0.01s) + --- PASS: TestAuthMethodCreateCommand_config/config_string (0.02s) +2021-10-05T16:29:50.915Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:50.980Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.072 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f7f871c-7bb7-3035-cc9e-0d741046e684 Address:127.0.0.1:14097}]" +29:51.074 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= +29:51.075 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 127.0.0.1 +29:51.120 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 127.0.0.1 +2021-10-05T16:29:51.120Z [INFO] agent.router: Initializing LAN area manager +29:51.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp +29:51.121 [INFO] TestAgent.server: Adding LAN server: server="Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +29:51.121 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 area=wan +29:51.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp +29:51.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp +29:51.122 [INFO] TestAgent: started state syncer +29:51.122 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp +29:51.134 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.134 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 +29:51.155 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.155 [DEBUG] TestAgent.server.raft: vote granted: from=8f7f871c-7bb7-3035-cc9e-0d741046e684 term=2 tally=1 +29:51.155 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.155 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" +29:51.160 [INFO] TestAgent.server: cluster leadership acquired +29:51.160 [INFO] TestAgent.server: New leader elected: payload=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.167 [INFO] TestAgent.server: initializing acls +29:51.173 [INFO] TestAgent.server: initializing acls +29:51.190 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.190 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.190 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.190 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.210 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.224 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.224 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.224 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.224 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.224 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +29:51.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.224 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 +29:51.224 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 area=wan +29:51.224 [INFO] TestAgent.server: Updating LAN server: server="Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +29:51.224 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.225 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 +29:51.231 [INFO] TestAgent.server: Updating LAN server: server="Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +29:51.231 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684.dc1 area=wan +29:51.270 [INFO] TestAgent: Synced node info +29:51.270 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.270 [DEBUG] TestAgent: Node info in sync +29:51.313 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.313 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.313 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.313 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.313 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.314 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.314 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.323 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.323 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f7f871c-7bb7-3035-cc9e-0d741046e684 +29:51.462 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +29:51.462 [DEBUG] TestAgent: Node info in sync +29:51.525 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58382 error="Invalid Auth Method: Type should be one of: [jwt kubernetes testing]" +29:51.526 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58382 latency=1.481317ms +29:51.540 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58384 latency=11.076104ms +29:51.541 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-43604b2a-728f-9313-85a6-39f8bb6f9160 from=127.0.0.1:58386 latency=348.248µs +29:51.554 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58388 latency=11.59789ms +29:51.555 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-f1465fd4-9a07-b888-c452-20a89cce2154 from=127.0.0.1:58386 latency=193.404µs +29:51.572 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58390 latency=15.206573ms +29:51.575 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-b4484306-0ffd-3461-e769-4b714dd580e9 from=127.0.0.1:58386 latency=479.683µs +29:51.576 [INFO] TestAgent: Requesting shutdown +29:51.576 [INFO] TestAgent.server: shutting down server +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.576 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.576 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.576 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.576 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.589 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.595Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.595Z [INFO] agent.router.manager: shutting down +29:51.596 [INFO] TestAgent: consul server down +29:51.596 [INFO] TestAgent: shutdown complete +29:51.596 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp +29:51.596 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp +29:51.596 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp +29:52.099 [INFO] TestAgent: Waiting for endpoints to shut down +29:52.099 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodCreateCommand (1.26s) --- PASS: TestAuthMethodCreateCommand/type_required (0.00s) --- PASS: TestAuthMethodCreateCommand/name_required (0.00s) --- PASS: TestAuthMethodCreateCommand/invalid_type (0.00s) - --- PASS: TestAuthMethodCreateCommand/create_testing (0.00s) - --- PASS: TestAuthMethodCreateCommand/create_testing_with_max_token_ttl (0.00s) - --- PASS: TestAuthMethodCreateCommand/create_testing_with_token_type_global (0.00s) + --- PASS: TestAuthMethodCreateCommand/create_testing (0.01s) + --- PASS: TestAuthMethodCreateCommand/create_testing_with_max_token_ttl (0.01s) + --- PASS: TestAuthMethodCreateCommand/create_testing_with_token_type_global (0.02s) +2021-10-05T16:29:50.982Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:51.013Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:51.064 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b277b9c-bcba-375b-8c78-c1e550e35769 Address:127.0.0.1:14104}]" +29:51.065 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5b277b9c-bcba-375b-8c78-c1e550e35769.dc1 127.0.0.1 +29:51.067 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5b277b9c-bcba-375b-8c78-c1e550e35769 127.0.0.1 +2021-10-05T16:29:51.067Z [INFO] agent.router: Initializing LAN area manager +29:51.072 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14099 network=udp +29:51.072 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14104 [Follower]" leader= +29:51.072 [INFO] TestAgent.server: Adding LAN server: server="Node-5b277b9c-bcba-375b-8c78-c1e550e35769 (Addr: tcp/127.0.0.1:14104) (DC: dc1)" +29:51.073 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5b277b9c-bcba-375b-8c78-c1e550e35769.dc1 area=wan +29:51.072 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14099 network=tcp +29:51.093 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14100 network=tcp +29:51.122 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:51.122 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14104 [Candidate]" term=2 +29:51.123 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:51.123 [INFO] TestAgent: started state syncer +29:51.123 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14105 network=tcp +29:51.141 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:51.141 [DEBUG] TestAgent.server.raft: vote granted: from=5b277b9c-bcba-375b-8c78-c1e550e35769 term=2 tally=1 +29:51.141 [INFO] TestAgent.server.raft: election won: tally=1 +29:51.141 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14104 [Leader]" +29:51.141 [INFO] TestAgent.server: cluster leadership acquired +29:51.141 [INFO] TestAgent.server: New leader elected: payload=Node-5b277b9c-bcba-375b-8c78-c1e550e35769 +29:51.157 [INFO] TestAgent.server: initializing acls +29:51.163 [INFO] TestAgent.server: Created ACL 'global-management' policy +29:51.163 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +29:51.173 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +29:51.181 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +29:51.201 [INFO] TestAgent.server: Created ACL anonymous token from configuration +29:51.201 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +29:51.201 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +29:51.201 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5b277b9c-bcba-375b-8c78-c1e550e35769 +29:51.202 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5b277b9c-bcba-375b-8c78-c1e550e35769.dc1 +29:51.202 [INFO] TestAgent.server: Updating LAN server: server="Node-5b277b9c-bcba-375b-8c78-c1e550e35769 (Addr: tcp/127.0.0.1:14104) (DC: dc1)" +29:51.202 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5b277b9c-bcba-375b-8c78-c1e550e35769.dc1 area=wan +29:51.224 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +29:51.290 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +29:51.290 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +29:51.290 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +29:51.290 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +29:51.290 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +29:51.290 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b277b9c-bcba-375b-8c78-c1e550e35769 +29:51.291 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5b277b9c-bcba-375b-8c78-c1e550e35769 +29:51.314 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b277b9c-bcba-375b-8c78-c1e550e35769 +29:51.336 [INFO] TestAgent.server: federation state anti-entropy synced +29:51.577 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:38130 latency=15.758621ms +29:51.579 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-d7fc6165-168a-368b-bc49-3b369981081d from=127.0.0.1:38132 latency=327.74µs +29:51.590 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:38134 latency=6.943242ms +29:51.592 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-7bc38728-7b88-8f51-8030-f3a3048d7240 from=127.0.0.1:38132 latency=429.634µs +29:51.598 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:38136 latency=4.762684ms +29:51.599 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-5cbcf4da-597d-6126-78c0-b15196308128 from=127.0.0.1:38132 latency=201.465µs +29:51.599 [INFO] TestAgent: Requesting shutdown +29:51.599 [INFO] TestAgent.server: shutting down server +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +29:51.599 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +29:51.599 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +29:51.600 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +29:51.600 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +29:51.604 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:29:51.612Z [INFO] agent.router.manager: shutting down +2021-10-05T16:29:51.612Z [INFO] agent.router.manager: shutting down +29:51.612 [INFO] TestAgent: consul server down +29:51.612 [INFO] TestAgent: shutdown complete +29:51.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14099 network=tcp +29:51.612 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14099 network=udp +29:51.612 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14100 network=tcp +29:52.116 [INFO] TestAgent: Waiting for endpoints to shut down +29:52.116 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodCreateCommand_JSON (1.28s) + --- PASS: TestAuthMethodCreateCommand_JSON/type_required (0.00s) + --- PASS: TestAuthMethodCreateCommand_JSON/create_testing (0.02s) + --- PASS: TestAuthMethodCreateCommand_JSON/create_testing_with_max_token_ttl (0.01s) + --- PASS: TestAuthMethodCreateCommand_JSON/create_testing_with_token_type_global (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/authmethod/create 1.087s +ok github.com/hashicorp/consul/command/acl/authmethod/create 1.384s === RUN TestAuthMethodDeleteCommand_noTabs === PAUSE TestAuthMethodDeleteCommand_noTabs === RUN TestAuthMethodDeleteCommand === PAUSE TestAuthMethodDeleteCommand === CONT TestAuthMethodDeleteCommand_noTabs -=== CONT TestAuthMethodDeleteCommand --- PASS: TestAuthMethodDeleteCommand_noTabs (0.00s) +=== CONT TestAuthMethodDeleteCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestAuthMethodDeleteCommand/name_required === RUN TestAuthMethodDeleteCommand/delete_notfound === RUN TestAuthMethodDeleteCommand/delete_works -2022-11-07T22:44:48.648Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:48.663Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:48.684 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:79c98399-5aab-fec9-8360-a8c0cf76aeab Address:127.0.0.1:20216}]" -44:48.684 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= -44:48.684 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-79c98399-5aab-fec9-8360-a8c0cf76aeab.dc1 127.0.0.1 -44:48.685 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-79c98399-5aab-fec9-8360-a8c0cf76aeab 127.0.0.1 -2022-11-07T22:44:48.685Z [INFO] agent.router: Initializing LAN area manager -44:48.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp -44:48.685 [INFO] TestAgent.server: Adding LAN server: server="Node-79c98399-5aab-fec9-8360-a8c0cf76aeab (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -44:48.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp -44:48.685 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab.dc1 area=wan -44:48.686 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp -44:48.686 [INFO] TestAgent: started state syncer -44:48.686 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp -44:48.735 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:48.744 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:48.744 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 -44:48.755 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:48.755 [DEBUG] TestAgent.server.raft: vote granted: from=79c98399-5aab-fec9-8360-a8c0cf76aeab term=2 tally=1 -44:48.755 [INFO] TestAgent.server.raft: election won: tally=1 -44:48.755 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" -44:48.755 [INFO] TestAgent.server: cluster leadership acquired -44:48.755 [INFO] TestAgent.server: New leader elected: payload=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab -44:48.762 [INFO] TestAgent.server: initializing acls -44:48.766 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:48.766 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:48.771 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:48.782 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:48.782 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:48.782 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:48.782 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-79c98399-5aab-fec9-8360-a8c0cf76aeab -44:48.782 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-79c98399-5aab-fec9-8360-a8c0cf76aeab.dc1 -44:48.782 [INFO] TestAgent.server: Updating LAN server: server="Node-79c98399-5aab-fec9-8360-a8c0cf76aeab (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -44:48.782 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab.dc1 area=wan -44:48.793 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:48.808 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:48.808 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:48.808 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:48.808 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:48.808 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:48.808 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab -44:48.808 [INFO] TestAgent.server: member joined, marking health alive: member=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab -44:48.818 [INFO] TestAgent.server: federation state anti-entropy synced -44:48.818 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-79c98399-5aab-fec9-8360-a8c0cf76aeab -44:48.899 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:48.905 [INFO] TestAgent: Synced node info -44:49.076 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/notfound from=127.0.0.1:42900 latency=832.797µs -44:49.081 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:42902 latency=4.158005ms -44:49.085 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/test-6376c966-6b1d-25d5-0b4f-5ac4a9b2b410 from=127.0.0.1:42904 latency=3.033253ms -44:49.086 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-6376c966-6b1d-25d5-0b4f-5ac4a9b2b410 from=127.0.0.1:42902 latency=92.762µs -44:49.086 [INFO] TestAgent: Requesting shutdown -44:49.086 [INFO] TestAgent.server: shutting down server -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:49.086 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:49.086 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:49.086 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:49.086 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:49.086 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:49.086 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:49.086 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:49.087 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:49.087 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:49.092 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:49.095Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:49.095Z [INFO] agent.router.manager: shutting down -44:49.095 [INFO] TestAgent: consul server down -44:49.095 [INFO] TestAgent: shutdown complete -44:49.095 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp -44:49.095 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp -44:49.095 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp -44:49.596 [INFO] TestAgent: Waiting for endpoints to shut down -44:49.596 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodDeleteCommand (0.99s) +2021-10-05T16:29:59.820Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:29:59.832Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +29:59.882 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b0e15c3-d049-9051-2bcb-5c90be815f1e Address:127.0.0.1:18174}]" +29:59.886 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 127.0.0.1 +29:59.887 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e 127.0.0.1 +2021-10-05T16:29:59.887Z [INFO] agent.router: Initializing LAN area manager +29:59.888 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= +29:59.890 [INFO] TestAgent.server: Adding LAN server: server="Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +29:59.890 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp +29:59.891 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 area=wan +29:59.893 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp +29:59.893 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp +29:59.893 [INFO] TestAgent: started state syncer +29:59.894 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp +29:59.942 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +29:59.956 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +29:59.956 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 +29:59.997 [DEBUG] TestAgent.server.raft: votes: needed=1 +29:59.997 [DEBUG] TestAgent.server.raft: vote granted: from=0b0e15c3-d049-9051-2bcb-5c90be815f1e term=2 tally=1 +29:59.997 [INFO] TestAgent.server.raft: election won: tally=1 +29:59.997 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" +30:00.000 [INFO] TestAgent.server: cluster leadership acquired +30:00.000 [INFO] TestAgent.server: New leader elected: payload=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.018 [INFO] TestAgent.server: initializing acls +30:00.022 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:00.022 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:00.034 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:00.043 [INFO] TestAgent.server: initializing acls +30:00.043 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:00.046 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:00.046 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:00.046 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:00.046 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:00.047 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.047 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 +30:00.047 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 area=wan +30:00.047 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:00.047 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.047 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 +30:00.047 [INFO] TestAgent.server: Updating LAN server: server="Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +30:00.047 [INFO] TestAgent.server: Updating LAN server: server="Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +30:00.047 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e.dc1 area=wan +30:00.056 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:00.076 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:00.076 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:00.076 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:00.076 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:00.076 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:00.076 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.076 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.079 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.079 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0b0e15c3-d049-9051-2bcb-5c90be815f1e +30:00.085 [INFO] TestAgent.server: federation state anti-entropy synced +30:00.098 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/notfound from=127.0.0.1:51538 latency=867.775µs +30:00.108 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:51540 latency=9.379965ms +30:00.144 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/auth-method/test-df28dc21-d617-b9da-7f59-ca8833f018ef from=127.0.0.1:51542 latency=26.779753ms +30:00.148 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-df28dc21-d617-b9da-7f59-ca8833f018ef from=127.0.0.1:51540 latency=113.526µs +30:00.148 [INFO] TestAgent: Requesting shutdown +30:00.148 [INFO] TestAgent.server: shutting down server +30:00.148 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:00.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:00.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:00.148 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:00.148 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:00.149 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:00.149 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:00.149 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:00.149 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:00.149 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:00.160 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:00.164Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:00.164Z [INFO] agent.router.manager: shutting down +30:00.164 [INFO] TestAgent: consul server down +30:00.164 [INFO] TestAgent: shutdown complete +30:00.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp +30:00.164 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp +30:00.164 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp +30:00.668 [INFO] TestAgent: Waiting for endpoints to shut down +30:00.668 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodDeleteCommand (0.97s) --- PASS: TestAuthMethodDeleteCommand/name_required (0.00s) --- PASS: TestAuthMethodDeleteCommand/delete_notfound (0.00s) - --- PASS: TestAuthMethodDeleteCommand/delete_works (0.01s) + --- PASS: TestAuthMethodDeleteCommand/delete_works (0.05s) PASS -ok github.com/hashicorp/consul/command/acl/authmethod/delete 1.091s +ok github.com/hashicorp/consul/command/acl/authmethod/delete 1.120s === RUN TestAuthMethodListCommand_noTabs === PAUSE TestAuthMethodListCommand_noTabs === RUN TestAuthMethodListCommand @@ -72963,188 +73152,198 @@ === RUN TestAuthMethodListCommand_JSON === PAUSE TestAuthMethodListCommand_JSON === CONT TestAuthMethodListCommand_noTabs +--- PASS: TestAuthMethodListCommand_noTabs (0.00s) === CONT TestAuthMethodListCommand_JSON -=== CONT TestAuthMethodListCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestAuthMethodListCommand_noTabs (0.00s) +=== CONT TestAuthMethodListCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== RUN TestAuthMethodListCommand_JSON/found_some === RUN TestAuthMethodListCommand/found_none +=== RUN TestAuthMethodListCommand_JSON/found_some === RUN TestAuthMethodListCommand/found_some -2022-11-07T22:44:49.608Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:49.618Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:49.645 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0028d2c-29bc-a654-d0b2-032c7891c472 Address:127.0.0.1:14090}]" -44:49.645 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= -44:49.647 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a0028d2c-29bc-a654-d0b2-032c7891c472.dc1 127.0.0.1 -44:49.649 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a0028d2c-29bc-a654-d0b2-032c7891c472 127.0.0.1 -2022-11-07T22:44:49.649Z [INFO] agent.router: Initializing LAN area manager -44:49.649 [INFO] TestAgent.server: Adding LAN server: server="Node-a0028d2c-29bc-a654-d0b2-032c7891c472 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -44:49.649 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a0028d2c-29bc-a654-d0b2-032c7891c472.dc1 area=wan -44:49.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp -44:49.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp -44:49.649 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp -44:49.649 [INFO] TestAgent: started state syncer -44:49.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp -44:49.699 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:49.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:49.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 -44:49.717 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:49.717 [DEBUG] TestAgent.server.raft: vote granted: from=a0028d2c-29bc-a654-d0b2-032c7891c472 term=2 tally=1 -44:49.717 [INFO] TestAgent.server.raft: election won: tally=1 -44:49.717 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" -44:49.718 [INFO] TestAgent.server: cluster leadership acquired -44:49.718 [INFO] TestAgent.server: New leader elected: payload=Node-a0028d2c-29bc-a654-d0b2-032c7891c472 -44:49.724 [INFO] TestAgent.server: initializing acls -44:49.726 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:49.726 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:49.731 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:49.743 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:49.743 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:49.743 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:49.743 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a0028d2c-29bc-a654-d0b2-032c7891c472 -44:49.743 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a0028d2c-29bc-a654-d0b2-032c7891c472.dc1 -44:49.743 [INFO] TestAgent.server: Updating LAN server: server="Node-a0028d2c-29bc-a654-d0b2-032c7891c472 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -44:49.743 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a0028d2c-29bc-a654-d0b2-032c7891c472.dc1 area=wan -44:49.753 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:49.768 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:49.768 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:49.768 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:49.768 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:49.768 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:49.768 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0028d2c-29bc-a654-d0b2-032c7891c472 -44:49.768 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a0028d2c-29bc-a654-d0b2-032c7891c472 -44:49.778 [INFO] TestAgent.server: federation state anti-entropy synced -44:49.778 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a0028d2c-29bc-a654-d0b2-032c7891c472 -44:49.798 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55928 latency=6.793092ms -44:49.804 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55928 latency=3.407936ms -44:49.808 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55928 latency=3.543491ms -44:49.813 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55928 latency=3.673146ms -44:49.818 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55928 latency=4.178065ms -44:49.819 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:55930 latency=198.235µs -44:49.819 [INFO] TestAgent: Requesting shutdown -44:49.820 [INFO] TestAgent.server: shutting down server -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:49.820 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:49.820 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:49.820 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:49.820 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:49.820 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:49.824 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:49.827Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:49.827Z [INFO] agent.router.manager: shutting down -44:49.827 [INFO] TestAgent: consul server down -44:49.827 [INFO] TestAgent: shutdown complete -44:49.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp -44:49.827 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp -44:49.827 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp -44:50.327 [INFO] TestAgent: Waiting for endpoints to shut down -44:50.327 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodListCommand_JSON (0.76s) +2021-10-05T16:30:00.824Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:00.863Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:00.914 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5e6bce4a-e560-b6ee-7776-ee43e43636b9 Address:127.0.0.1:11027}]" +30:00.915 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 127.0.0.1 +30:00.915 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 127.0.0.1 +2021-10-05T16:30:00.915Z [INFO] agent.router: Initializing LAN area manager +30:00.916 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=udp +30:00.916 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11027 [Follower]" leader= +30:00.916 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=tcp +30:00.916 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 area=wan +30:00.916 [INFO] TestAgent.server: Adding LAN server: server="Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +30:00.917 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11023 network=tcp +30:00.917 [INFO] TestAgent: started state syncer +30:00.917 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11028 network=tcp +30:00.968 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:00.980 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:00.980 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11027 [Candidate]" term=2 +30:00.999 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:00.999 [DEBUG] TestAgent.server.raft: vote granted: from=5e6bce4a-e560-b6ee-7776-ee43e43636b9 term=2 tally=1 +30:00.999 [INFO] TestAgent.server.raft: election won: tally=1 +30:00.999 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11027 [Leader]" +30:00.999 [INFO] TestAgent.server: cluster leadership acquired +30:00.999 [INFO] TestAgent.server: New leader elected: payload=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.029 [INFO] TestAgent.server: initializing acls +30:01.040 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:01.040 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:01.054 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:01.069 [INFO] TestAgent.server: initializing acls +30:01.069 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:01.081 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:01.081 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:01.081 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:01.081 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:01.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.081 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 +30:01.081 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 area=wan +30:01.081 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:01.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.082 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 +30:01.084 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9.dc1 area=wan +30:01.084 [INFO] TestAgent.server: Updating LAN server: server="Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +30:01.084 [INFO] TestAgent.server: Updating LAN server: server="Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +30:01.109 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:01.118 [INFO] TestAgent: Synced node info +30:01.118 [DEBUG] TestAgent: Node info in sync +30:01.131 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:01.131 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:01.131 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:01.131 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:01.131 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:01.132 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.132 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.169 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.169 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e6bce4a-e560-b6ee-7776-ee43e43636b9 +30:01.181 [INFO] TestAgent.server: federation state anti-entropy synced +30:01.223 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59632 latency=18.877551ms +30:01.240 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59632 latency=12.500389ms +30:01.264 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59632 latency=14.175026ms +30:01.285 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59632 latency=16.277298ms +30:01.308 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59632 latency=22.755059ms +30:01.315 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:59638 latency=319.212µs +30:01.315 [INFO] TestAgent: Requesting shutdown +30:01.316 [INFO] TestAgent.server: shutting down server +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:01.316 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:01.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:01.320 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:01.320 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:01.324 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:01.336Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:01.336Z [INFO] agent.router.manager: shutting down +30:01.340 [INFO] TestAgent: consul server down +30:01.340 [INFO] TestAgent: shutdown complete +30:01.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=tcp +30:01.340 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=udp +30:01.340 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11023 network=tcp +30:01.844 [INFO] TestAgent: Waiting for endpoints to shut down +30:01.844 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodListCommand_JSON (1.15s) --- PASS: TestAuthMethodListCommand_JSON/found_some (0.00s) -2022-11-07T22:44:49.608Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:49.618Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:49.647 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e Address:127.0.0.1:14097}]" -44:49.647 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= -44:49.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 127.0.0.1 -44:49.649 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e 127.0.0.1 -2022-11-07T22:44:49.649Z [INFO] agent.router: Initializing LAN area manager -44:49.649 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 area=wan -44:49.649 [INFO] TestAgent.server: Adding LAN server: server="Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e (Addr: tcp/127.0.0.1:14097) (DC: dc1)" -44:49.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp -44:49.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp -44:49.650 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp -44:49.650 [INFO] TestAgent: started state syncer -44:49.650 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp -44:49.685 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:49.685 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 -44:49.694 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:49.694 [DEBUG] TestAgent.server.raft: vote granted: from=d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e term=2 tally=1 -44:49.694 [INFO] TestAgent.server.raft: election won: tally=1 -44:49.694 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" -44:49.694 [INFO] TestAgent.server: cluster leadership acquired -44:49.694 [INFO] TestAgent.server: New leader elected: payload=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.699 [INFO] TestAgent.server: initializing acls -44:49.700 [INFO] TestAgent.server: initializing acls -44:49.706 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:49.706 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:49.706 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:49.706 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:49.716 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:49.716 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:49.723 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:49.723 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:49.723 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:49.723 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:49.723 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:49.723 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.723 [INFO] TestAgent.server: Updating LAN server: server="Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e (Addr: tcp/127.0.0.1:14097) (DC: dc1)" -44:49.724 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 -44:49.724 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.724 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 area=wan -44:49.724 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 -44:49.724 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e.dc1 area=wan -44:49.724 [INFO] TestAgent.server: Updating LAN server: server="Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e (Addr: tcp/127.0.0.1:14097) (DC: dc1)" -44:49.735 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:49.751 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:49.751 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:49.751 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:49.751 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:49.751 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:49.751 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.751 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.759 [INFO] TestAgent.server: federation state anti-entropy synced -44:49.759 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.759 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d5ccc8e5-0da1-96a9-e634-0bf724a2fb9e -44:49.890 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:49.893 [INFO] TestAgent: Synced node info -44:50.008 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:37876 latency=1.014272ms -44:50.012 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37878 latency=3.536858ms -44:50.016 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37878 latency=4.10466ms -44:50.021 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37878 latency=4.842185ms -44:50.027 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37878 latency=5.168196ms -44:50.031 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37878 latency=3.550289ms -44:50.032 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:37880 latency=224.97µs -44:50.034 [INFO] TestAgent: Requesting shutdown -44:50.034 [INFO] TestAgent.server: shutting down server -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:50.034 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:50.034 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:50.034 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:50.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:50.034 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:50.034 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:50.034 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:50.035 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:50.035 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:50.038 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:50.041Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:50.041Z [INFO] agent.router.manager: shutting down -44:50.041 [INFO] TestAgent: consul server down -44:50.041 [INFO] TestAgent: shutdown complete -44:50.041 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp -44:50.041 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp -44:50.041 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp -44:50.542 [INFO] TestAgent: Waiting for endpoints to shut down -44:50.542 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodListCommand (0.97s) - --- PASS: TestAuthMethodListCommand/found_none (0.00s) - --- PASS: TestAuthMethodListCommand/found_some (0.00s) +2021-10-05T16:30:00.814Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:00.870Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:00.944 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 Address:127.0.0.1:11034}]" +30:00.945 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 127.0.0.1 +30:00.945 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 127.0.0.1 +2021-10-05T16:30:00.945Z [INFO] agent.router: Initializing LAN area manager +30:00.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=udp +30:00.946 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11034 [Follower]" leader= +30:00.946 [INFO] TestAgent.server: Adding LAN server: server="Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:00.946 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 area=wan +30:00.946 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=tcp +30:00.947 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11030 network=tcp +30:00.947 [INFO] TestAgent: started state syncer +30:00.947 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11035 network=tcp +30:00.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:01.015 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:01.015 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11034 [Candidate]" term=2 +30:01.052 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:01.052 [DEBUG] TestAgent.server.raft: vote granted: from=b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 term=2 tally=1 +30:01.052 [INFO] TestAgent.server.raft: election won: tally=1 +30:01.052 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11034 [Leader]" +30:01.052 [INFO] TestAgent.server: cluster leadership acquired +30:01.052 [INFO] TestAgent.server: New leader elected: payload=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.072 [INFO] TestAgent.server: initializing acls +30:01.100 [INFO] TestAgent.server: initializing acls +30:01.101 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:01.101 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:01.122 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:01.123 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:01.123 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:01.144 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:01.144 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:01.144 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:01.144 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:01.144 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.144 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 +30:01.145 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 area=wan +30:01.145 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:01.145 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.145 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 +30:01.148 [INFO] TestAgent.server: Updating LAN server: server="Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:01.148 [INFO] TestAgent.server: Updating LAN server: server="Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:01.148 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3.dc1 area=wan +30:01.200 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:01.248 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:01.248 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:01.248 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:01.248 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:01.248 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:01.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.269 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.269 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8296bc8-b245-fbcd-d7fc-db5ba32ad9b3 +30:01.292 [INFO] TestAgent.server: federation state anti-entropy synced +30:01.293 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:46514 latency=7.326343ms +30:01.297 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:01.312 [INFO] TestAgent: Synced node info +30:01.321 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46516 latency=24.222138ms +30:01.328 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46516 latency=5.939929ms +30:01.343 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46516 latency=14.467444ms +30:01.353 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46516 latency=9.733248ms +30:01.366 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46516 latency=9.536534ms +30:01.373 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-methods from=127.0.0.1:46520 latency=342.213µs +30:01.373 [INFO] TestAgent: Requesting shutdown +30:01.373 [INFO] TestAgent.server: shutting down server +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:01.373 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:01.373 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:01.373 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:01.374 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:01.383 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:01.389Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:01.389Z [INFO] agent.router.manager: shutting down +30:01.389 [INFO] TestAgent: consul server down +30:01.389 [INFO] TestAgent: shutdown complete +30:01.389 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=tcp +30:01.389 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=udp +30:01.389 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11030 network=tcp +30:01.892 [INFO] TestAgent: Waiting for endpoints to shut down +30:01.892 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodListCommand (1.19s) + --- PASS: TestAuthMethodListCommand/found_none (0.01s) + --- PASS: TestAuthMethodListCommand/found_some (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/authmethod/list 1.000s +ok github.com/hashicorp/consul/command/acl/authmethod/list 1.318s === RUN TestAuthMethodReadCommand_noTabs === PAUSE TestAuthMethodReadCommand_noTabs === RUN TestAuthMethodReadCommand @@ -73152,172 +73351,171 @@ === RUN TestAuthMethodReadCommand_JSON === PAUSE TestAuthMethodReadCommand_JSON === CONT TestAuthMethodReadCommand_noTabs +--- PASS: TestAuthMethodReadCommand_noTabs (0.00s) === CONT TestAuthMethodReadCommand_JSON === CONT TestAuthMethodReadCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestAuthMethodReadCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== RUN TestAuthMethodReadCommand_JSON/read_by_name === RUN TestAuthMethodReadCommand/name_required === RUN TestAuthMethodReadCommand/not_found === RUN TestAuthMethodReadCommand/read_by_name -=== RUN TestAuthMethodReadCommand_JSON/read_by_name -2022-11-07T22:44:51.560Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:51.573Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:51.600 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b58acb5e-d007-ba03-4d25-631beacad54f Address:127.0.0.1:18174}]" -44:51.600 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -44:51.601 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b58acb5e-d007-ba03-4d25-631beacad54f.dc1 127.0.0.1 -44:51.602 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b58acb5e-d007-ba03-4d25-631beacad54f 127.0.0.1 -2022-11-07T22:44:51.602Z [INFO] agent.router: Initializing LAN area manager -44:51.602 [INFO] TestAgent.server: Adding LAN server: server="Node-b58acb5e-d007-ba03-4d25-631beacad54f (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:51.602 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b58acb5e-d007-ba03-4d25-631beacad54f.dc1 area=wan -44:51.603 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -44:51.603 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -44:51.603 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -44:51.603 [INFO] TestAgent: started state syncer -44:51.603 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -44:51.653 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:51.655 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:51.655 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -44:51.663 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:51.663 [DEBUG] TestAgent.server.raft: vote granted: from=b58acb5e-d007-ba03-4d25-631beacad54f term=2 tally=1 -44:51.663 [INFO] TestAgent.server.raft: election won: tally=1 -44:51.663 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -44:51.663 [INFO] TestAgent.server: cluster leadership acquired -44:51.663 [INFO] TestAgent.server: New leader elected: payload=Node-b58acb5e-d007-ba03-4d25-631beacad54f -44:51.668 [INFO] TestAgent.server: initializing acls -44:51.671 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:51.671 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:51.677 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:51.684 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:51.684 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:51.684 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:51.684 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b58acb5e-d007-ba03-4d25-631beacad54f -44:51.685 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b58acb5e-d007-ba03-4d25-631beacad54f.dc1 -44:51.685 [INFO] TestAgent.server: Updating LAN server: server="Node-b58acb5e-d007-ba03-4d25-631beacad54f (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:51.685 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b58acb5e-d007-ba03-4d25-631beacad54f.dc1 area=wan -44:51.694 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:51.711 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:51.711 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:51.711 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:51.711 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:51.711 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:51.711 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b58acb5e-d007-ba03-4d25-631beacad54f -44:51.711 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b58acb5e-d007-ba03-4d25-631beacad54f -44:51.717 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b58acb5e-d007-ba03-4d25-631beacad54f -44:51.721 [INFO] TestAgent.server: federation state anti-entropy synced -44:51.753 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/notfound from=127.0.0.1:46792 latency=320.794µs -44:51.758 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46794 latency=5.185626ms -44:51.760 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-8d7cb2e6-e2b8-1308-554d-b04223acf7bb from=127.0.0.1:46796 latency=500.986µs -44:51.760 [INFO] TestAgent: Requesting shutdown -44:51.760 [INFO] TestAgent.server: shutting down server -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:51.760 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:51.760 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:51.761 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:51.761 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:51.761 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:51.764 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:51.767Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:51.767Z [INFO] agent.router.manager: shutting down -44:51.767 [INFO] TestAgent: consul server down -44:51.767 [INFO] TestAgent: shutdown complete -44:51.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -44:51.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -44:51.767 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -44:52.267 [INFO] TestAgent: Waiting for endpoints to shut down -44:52.267 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodReadCommand (0.74s) +2021-10-05T16:30:02.691Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:02.727Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:02.779 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:237a9d16-7963-ce7d-1e66-00bf3c86f639 Address:127.0.0.1:14090}]" +30:02.779 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +30:02.780 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-237a9d16-7963-ce7d-1e66-00bf3c86f639.dc1 127.0.0.1 +30:02.781 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 127.0.0.1 +2021-10-05T16:30:02.781Z [INFO] agent.router: Initializing LAN area manager +30:02.781 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +30:02.781 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639.dc1 area=wan +30:02.781 [INFO] TestAgent.server: Adding LAN server: server="Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +30:02.782 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +30:02.782 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +30:02.782 [INFO] TestAgent: started state syncer +30:02.782 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +30:02.825 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:02.825 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +30:02.832 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:02.836 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:02.836 [DEBUG] TestAgent.server.raft: vote granted: from=237a9d16-7963-ce7d-1e66-00bf3c86f639 term=2 tally=1 +30:02.836 [INFO] TestAgent.server.raft: election won: tally=1 +30:02.836 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +30:02.836 [INFO] TestAgent.server: cluster leadership acquired +30:02.836 [INFO] TestAgent.server: New leader elected: payload=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 +30:02.856 [INFO] TestAgent.server: initializing acls +30:02.872 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:02.872 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:02.892 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:02.920 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:02.920 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:02.921 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:02.922 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 +30:02.922 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-237a9d16-7963-ce7d-1e66-00bf3c86f639.dc1 +30:02.924 [INFO] TestAgent.server: Updating LAN server: server="Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +30:02.924 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639.dc1 area=wan +30:02.952 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:02.952 [INFO] TestAgent: Synced node info +30:03.014 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:03.014 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:03.014 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:03.014 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:03.014 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:03.014 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 +30:03.014 [INFO] TestAgent.server: member joined, marking health alive: member=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 +30:03.023 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-237a9d16-7963-ce7d-1e66-00bf3c86f639 +30:03.032 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.039 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43158 latency=14.506221ms +30:03.042 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-3730235f-6b62-8f3e-9a7e-a1057c29d1ac from=127.0.0.1:43164 latency=330.656µs +30:03.048 [INFO] TestAgent: Requesting shutdown +30:03.048 [INFO] TestAgent.server: shutting down server +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.048 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:03.048 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.048 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.048 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:03.060 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.066Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.066Z [INFO] agent.router.manager: shutting down +30:03.066 [INFO] TestAgent: consul server down +30:03.066 [INFO] TestAgent: shutdown complete +30:03.066 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +30:03.066 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +30:03.066 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +30:03.567 [INFO] TestAgent: Waiting for endpoints to shut down +30:03.567 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodReadCommand_JSON (1.00s) + --- PASS: TestAuthMethodReadCommand_JSON/read_by_name (0.06s) +2021-10-05T16:30:02.692Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:02.721Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:02.788 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:44beda08-5e41-42c5-b67a-9276aa6e5512 Address:127.0.0.1:14097}]" +30:02.788 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-44beda08-5e41-42c5-b67a-9276aa6e5512.dc1 127.0.0.1 +30:02.789 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-44beda08-5e41-42c5-b67a-9276aa6e5512 127.0.0.1 +2021-10-05T16:30:02.789Z [INFO] agent.router: Initializing LAN area manager +30:02.789 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp +30:02.789 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= +30:02.789 [INFO] TestAgent.server: Adding LAN server: server="Node-44beda08-5e41-42c5-b67a-9276aa6e5512 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +30:02.790 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-44beda08-5e41-42c5-b67a-9276aa6e5512.dc1 area=wan +30:02.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp +30:02.790 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp +30:02.790 [INFO] TestAgent: started state syncer +30:02.790 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp +30:02.840 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:02.844 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:02.844 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 +30:02.883 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:02.883 [DEBUG] TestAgent.server.raft: vote granted: from=44beda08-5e41-42c5-b67a-9276aa6e5512 term=2 tally=1 +30:02.883 [INFO] TestAgent.server.raft: election won: tally=1 +30:02.883 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" +30:02.885 [INFO] TestAgent.server: cluster leadership acquired +30:02.886 [INFO] TestAgent.server: New leader elected: payload=Node-44beda08-5e41-42c5-b67a-9276aa6e5512 +30:02.891 [INFO] TestAgent.server: initializing acls +30:02.896 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:02.896 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:02.903 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:02.912 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:02.912 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:02.912 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:02.913 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-44beda08-5e41-42c5-b67a-9276aa6e5512 +30:02.913 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-44beda08-5e41-42c5-b67a-9276aa6e5512.dc1 +30:02.919 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-44beda08-5e41-42c5-b67a-9276aa6e5512.dc1 area=wan +30:02.919 [INFO] TestAgent.server: Updating LAN server: server="Node-44beda08-5e41-42c5-b67a-9276aa6e5512 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +30:02.935 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:02.986 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:02.986 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:02.986 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:02.986 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:02.986 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:02.986 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44beda08-5e41-42c5-b67a-9276aa6e5512 +30:02.986 [INFO] TestAgent.server: member joined, marking health alive: member=Node-44beda08-5e41-42c5-b67a-9276aa6e5512 +30:02.997 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-44beda08-5e41-42c5-b67a-9276aa6e5512 +30:03.014 [INFO] TestAgent.server: federation state anti-entropy synced +30:03.033 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/notfound from=127.0.0.1:58480 latency=482.621µs +30:03.055 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:58482 latency=21.052088ms +30:03.065 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-d9d5bc6c-7ca2-aea7-fba3-2a837c00f1e8 from=127.0.0.1:58486 latency=1.990881ms +30:03.069 [INFO] TestAgent: Requesting shutdown +30:03.069 [INFO] TestAgent.server: shutting down server +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:03.069 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:03.069 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:03.069 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:03.069 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:03.069 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:03.084 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:03.092Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:03.092Z [INFO] agent.router.manager: shutting down +30:03.092 [INFO] TestAgent: consul server down +30:03.092 [INFO] TestAgent: shutdown complete +30:03.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp +30:03.092 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp +30:03.092 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp +30:03.596 [INFO] TestAgent: Waiting for endpoints to shut down +30:03.596 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodReadCommand (1.03s) --- PASS: TestAuthMethodReadCommand/name_required (0.00s) - --- PASS: TestAuthMethodReadCommand/not_found (0.00s) - --- PASS: TestAuthMethodReadCommand/read_by_name (0.01s) -2022-11-07T22:44:51.560Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:51.573Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:51.602 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e2f4de85-7342-8830-329b-11bf4f6aeb80 Address:127.0.0.1:18181}]" -44:51.602 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18181 [Follower]" leader= -44:51.603 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e2f4de85-7342-8830-329b-11bf4f6aeb80.dc1 127.0.0.1 -44:51.603 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 127.0.0.1 -2022-11-07T22:44:51.603Z [INFO] agent.router: Initializing LAN area manager -44:51.604 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80.dc1 area=wan -44:51.604 [INFO] TestAgent.server: Adding LAN server: server="Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -44:51.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=udp -44:51.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=tcp -44:51.604 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18177 network=tcp -44:51.604 [INFO] TestAgent: started state syncer -44:51.604 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18182 network=tcp -44:51.654 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:51.667 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:51.667 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18181 [Candidate]" term=2 -44:51.676 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:51.676 [DEBUG] TestAgent.server.raft: vote granted: from=e2f4de85-7342-8830-329b-11bf4f6aeb80 term=2 tally=1 -44:51.676 [INFO] TestAgent.server.raft: election won: tally=1 -44:51.676 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18181 [Leader]" -44:51.676 [INFO] TestAgent.server: cluster leadership acquired -44:51.677 [INFO] TestAgent.server: New leader elected: payload=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 -44:51.682 [INFO] TestAgent.server: initializing acls -44:51.684 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:51.684 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:51.691 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:51.703 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:51.703 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:51.703 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:51.703 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 -44:51.703 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e2f4de85-7342-8830-329b-11bf4f6aeb80.dc1 -44:51.703 [INFO] TestAgent.server: Updating LAN server: server="Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -44:51.703 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80.dc1 area=wan -44:51.712 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:51.729 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:51.729 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:51.729 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:51.729 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:51.729 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:51.729 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 -44:51.729 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 -44:51.731 [INFO] TestAgent.server: federation state anti-entropy synced -44:51.738 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2f4de85-7342-8830-329b-11bf4f6aeb80 -44:51.852 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:51.858 [INFO] TestAgent: Synced node info -44:51.976 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:53804 latency=3.55602ms -44:51.979 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-18a830e7-0fc2-017f-331a-affb7fe5dc4d from=127.0.0.1:53806 latency=1.749643ms -44:51.981 [INFO] TestAgent: Requesting shutdown -44:51.981 [INFO] TestAgent.server: shutting down server -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:51.981 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:51.981 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:51.981 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:51.981 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:51.985 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:51.988Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:51.988Z [INFO] agent.router.manager: shutting down -44:51.988 [INFO] TestAgent: consul server down -44:51.988 [INFO] TestAgent: shutdown complete -44:51.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=tcp -44:51.988 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=udp -44:51.988 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18177 network=tcp -44:52.488 [INFO] TestAgent: Waiting for endpoints to shut down -44:52.488 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodReadCommand_JSON (0.96s) - --- PASS: TestAuthMethodReadCommand_JSON/read_by_name (0.01s) + --- PASS: TestAuthMethodReadCommand/not_found (0.02s) + --- PASS: TestAuthMethodReadCommand/read_by_name (0.04s) PASS -ok github.com/hashicorp/consul/command/acl/authmethod/read 0.995s +ok github.com/hashicorp/consul/command/acl/authmethod/read 1.153s === RUN TestAuthMethodUpdateCommand_noTabs === PAUSE TestAuthMethodUpdateCommand_noTabs === RUN TestAuthMethodUpdateCommand @@ -73333,587 +73531,598 @@ === RUN TestAuthMethodUpdateCommand_config === PAUSE TestAuthMethodUpdateCommand_config === CONT TestAuthMethodUpdateCommand_noTabs +=== CONT TestAuthMethodUpdateCommand_k8s_noMerge +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestAuthMethodUpdateCommand_noMerge === CONT TestAuthMethodUpdateCommand_k8s -=== CONT TestAuthMethodUpdateCommand +--- PASS: TestAuthMethodUpdateCommand_noTabs (0.00s) === CONT TestAuthMethodUpdateCommand_JSON -=== CONT TestAuthMethodUpdateCommand_noMerge === CONT TestAuthMethodUpdateCommand_config -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestAuthMethodUpdateCommand_k8s_noMerge +=== CONT TestAuthMethodUpdateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestAuthMethodUpdateCommand_noTabs (0.00s) -=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_host === RUN TestAuthMethodUpdateCommand_noMerge/update_without_name === RUN TestAuthMethodUpdateCommand_noMerge/update_nonexistent_method === RUN TestAuthMethodUpdateCommand_noMerge/update_all_fields +=== RUN TestAuthMethodUpdateCommand_k8s/update_all_fields +=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_host === RUN TestAuthMethodUpdateCommand_config/config_file === RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_ca_cert -=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_jwt -=== RUN TestAuthMethodUpdateCommand_config/config_stdin -=== RUN TestAuthMethodUpdateCommand_config/config_string -=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields === RUN TestAuthMethodUpdateCommand/update_without_name === RUN TestAuthMethodUpdateCommand/update_nonexistent_method -=== RUN TestAuthMethodUpdateCommand_config/config_with_no_merge -=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields_with_cert_file +=== RUN TestAuthMethodUpdateCommand_k8s/update_all_fields_with_cert_file === RUN TestAuthMethodUpdateCommand/update_all_fields +=== RUN TestAuthMethodUpdateCommand_JSON/update_without_name +=== RUN TestAuthMethodUpdateCommand_JSON/update_all_fields +=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_jwt === RUN TestAuthMethodUpdateCommand/update_config_field_and_prove_no_merging_happens -=== RUN TestAuthMethodUpdateCommand_k8s/update_all_fields -=== RUN TestAuthMethodUpdateCommand_k8s/update_all_fields_with_cert_file +=== RUN TestAuthMethodUpdateCommand_config/config_stdin +=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields === RUN TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_host +=== RUN TestAuthMethodUpdateCommand_config/config_string +=== RUN TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields_with_cert_file === RUN TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_ca_cert +=== RUN TestAuthMethodUpdateCommand_config/config_with_no_merge === RUN TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_jwt -=== RUN TestAuthMethodUpdateCommand_JSON/update_without_name -=== RUN TestAuthMethodUpdateCommand_JSON/update_all_fields -2022-11-07T22:44:53.790Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.846 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c0b546ef-d4f2-b15f-771c-5ff1abe17b33 Address:127.0.0.1:15132}]" -44:53.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15132 [Follower]" leader= -44:53.847 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33.dc1 127.0.0.1 -44:53.848 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 127.0.0.1 -2022-11-07T22:44:53.848Z [INFO] agent.router: Initializing LAN area manager -44:53.849 [INFO] TestAgent.server: Adding LAN server: server="Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 (Addr: tcp/127.0.0.1:15132) (DC: dc1)" -44:53.849 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33.dc1 area=wan -44:53.850 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=tcp -44:53.850 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=udp -44:53.850 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15128 network=tcp -44:53.882 [INFO] TestAgent: started state syncer -44:53.882 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15133 network=tcp -44:53.900 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:53.914 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.914 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15132 [Candidate]" term=2 -44:53.935 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.935 [DEBUG] TestAgent.server.raft: vote granted: from=c0b546ef-d4f2-b15f-771c-5ff1abe17b33 term=2 tally=1 -44:53.935 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.935 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15132 [Leader]" -44:53.935 [INFO] TestAgent.server: cluster leadership acquired -44:53.935 [INFO] TestAgent.server: New leader elected: payload=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 -44:53.944 [INFO] TestAgent.server: initializing acls -44:53.947 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.947 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.956 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.976 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.976 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.976 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.977 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 -44:53.977 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33.dc1 -44:53.977 [INFO] TestAgent.server: Updating LAN server: server="Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 (Addr: tcp/127.0.0.1:15132) (DC: dc1)" -44:53.977 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33.dc1 area=wan -44:53.991 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.014 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.014 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.014 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.014 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.014 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.014 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 -44:54.014 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 -44:54.018 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.028 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c0b546ef-d4f2-b15f-771c-5ff1abe17b33 -44:54.033 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.034 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-50705b75-ce6b-1a9f-b150-94b30cb113bb from=127.0.0.1:46074 latency=381.862µs -44:54.042 [INFO] TestAgent: Synced node info -44:54.042 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46076 latency=7.912314ms -44:54.043 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-321c4f85-82ca-1b58-b4a6-fc15ba7dea31 from=127.0.0.1:46082 latency=135.926µs -44:54.048 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-321c4f85-82ca-1b58-b4a6-fc15ba7dea31 from=127.0.0.1:46082 latency=4.408147ms -44:54.049 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-321c4f85-82ca-1b58-b4a6-fc15ba7dea31 from=127.0.0.1:46076 latency=249.243µs -44:54.049 [INFO] TestAgent: Requesting shutdown -44:54.049 [INFO] TestAgent.server: shutting down server -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.049 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.049 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.049 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.049 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.053 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.059Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.059Z [INFO] agent.router.manager: shutting down -44:54.059 [INFO] TestAgent: consul server down -44:54.059 [INFO] TestAgent: shutdown complete -44:54.059 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=tcp -44:54.059 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=udp -44:54.059 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15128 network=tcp -44:54.560 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.560 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand_noMerge (0.81s) +2021-10-05T16:30:10.181Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.262Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.380 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:55971041-5f11-4c3b-63b7-4b27dd51cfc1 Address:127.0.0.1:11034}]" +30:10.381 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 127.0.0.1 +30:10.381 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 127.0.0.1 +2021-10-05T16:30:10.381Z [INFO] agent.router: Initializing LAN area manager +30:10.382 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=udp +30:10.382 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11034 [Follower]" leader= +30:10.382 [INFO] TestAgent.server: Adding LAN server: server="Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:10.382 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 area=wan +30:10.382 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11029 network=tcp +30:10.383 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11030 network=tcp +30:10.383 [INFO] TestAgent: started state syncer +30:10.383 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11035 network=tcp +30:10.424 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.424 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11034 [Candidate]" term=2 +30:10.432 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:10.468 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.468 [DEBUG] TestAgent.server.raft: vote granted: from=55971041-5f11-4c3b-63b7-4b27dd51cfc1 term=2 tally=1 +30:10.468 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.468 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11034 [Leader]" +30:10.472 [INFO] TestAgent.server: cluster leadership acquired +30:10.472 [INFO] TestAgent.server: New leader elected: payload=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 +30:10.508 [INFO] TestAgent.server: initializing acls +30:10.528 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.528 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.536 [INFO] TestAgent.server: initializing acls +30:10.536 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.553 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.584 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.584 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.584 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.584 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 +30:10.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 +30:10.585 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.585 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:10.585 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 +30:10.585 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 +30:10.585 [INFO] TestAgent.server: Updating LAN server: server="Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:10.585 [INFO] TestAgent.server: Updating LAN server: server="Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 (Addr: tcp/127.0.0.1:11034) (DC: dc1)" +30:10.586 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 area=wan +30:10.586 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1.dc1 area=wan +30:10.632 [INFO] TestAgent: Synced node info +30:10.632 [DEBUG] TestAgent: Node info in sync +30:10.641 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.642 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-34697363-e44f-ca56-4998-865bb9ce785c from=127.0.0.1:46578 latency=2.531148ms +30:10.703 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46580 latency=54.650783ms +30:10.706 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-dd01bd48-b496-a00b-cd55-56c732ef62ab from=127.0.0.1:46582 latency=607.212µs +30:10.755 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.759 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-dd01bd48-b496-a00b-cd55-56c732ef62ab from=127.0.0.1:46582 latency=51.936444ms +30:10.761 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-dd01bd48-b496-a00b-cd55-56c732ef62ab from=127.0.0.1:46580 latency=471.646µs +30:10.761 [INFO] TestAgent: Requesting shutdown +30:10.761 [INFO] TestAgent.server: shutting down server +30:10.761 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:10.761 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:10.761 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:10.761 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:10.761 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:10.761 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.761 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.761 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.761 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.761 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 +30:10.762 [INFO] TestAgent.server: member joined, marking health alive: member=Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 +30:10.768 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:10.773Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:10.773Z [INFO] agent.router.manager: shutting down +30:10.780 [INFO] TestAgent: consul server down +30:10.780 [INFO] TestAgent: shutdown complete +30:10.780 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=tcp +30:10.780 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11029 network=udp +30:10.780 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11030 network=tcp +30:10.780 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-55971041-5f11-4c3b-63b7-4b27dd51cfc1 127.0.0.1 11032 map[acls:1 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:55971041-5f11-4c3b-63b7-4b27dd51cfc1 port:11034 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:11033] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +30:10.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:10.780 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:10.780 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:10.780 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:10.782 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:10.782 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:10.782 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.284 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.284 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand_noMerge (1.36s) --- PASS: TestAuthMethodUpdateCommand_noMerge/update_without_name (0.00s) - --- PASS: TestAuthMethodUpdateCommand_noMerge/update_nonexistent_method (0.00s) - --- PASS: TestAuthMethodUpdateCommand_noMerge/update_all_fields (0.02s) -2022-11-07T22:44:53.788Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.846 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:46b7bcbd-88ae-5c6a-913a-45ad2f00a31a Address:127.0.0.1:15146}]" -44:53.846 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15146 [Follower]" leader= -44:53.847 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a.dc1 127.0.0.1 -44:53.848 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a 127.0.0.1 -2022-11-07T22:44:53.848Z [INFO] agent.router: Initializing LAN area manager -44:53.848 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a.dc1 area=wan -44:53.848 [INFO] TestAgent.server: Adding LAN server: server="Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a (Addr: tcp/127.0.0.1:15146) (DC: dc1)" -44:53.848 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15141 network=udp -44:53.849 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15141 network=tcp -44:53.850 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15142 network=tcp -44:53.850 [INFO] TestAgent: started state syncer -44:53.850 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15147 network=tcp -44:53.889 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.889 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15146 [Candidate]" term=2 -44:53.898 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:53.908 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.908 [DEBUG] TestAgent.server.raft: vote granted: from=46b7bcbd-88ae-5c6a-913a-45ad2f00a31a term=2 tally=1 -44:53.908 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.908 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15146 [Leader]" -44:53.908 [INFO] TestAgent.server: cluster leadership acquired -44:53.908 [INFO] TestAgent.server: New leader elected: payload=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a -44:53.921 [INFO] TestAgent.server: initializing acls -44:53.928 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.928 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.941 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.956 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.956 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.956 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.956 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a -44:53.957 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a.dc1 -44:53.957 [INFO] TestAgent.server: Updating LAN server: server="Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a (Addr: tcp/127.0.0.1:15146) (DC: dc1)" -44:53.957 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a.dc1 area=wan -44:53.976 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.009 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.009 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.009 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.009 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.009 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.009 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a -44:54.009 [INFO] TestAgent.server: member joined, marking health alive: member=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a -44:54.018 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-46b7bcbd-88ae-5c6a-913a-45ad2f00a31a -44:54.026 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.048 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39752 latency=7.155064ms -44:54.049 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test19033109-8768-4114-fe77-fc59445331d6 from=127.0.0.1:39758 latency=308.077µs -44:54.053 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test19033109-8768-4114-fe77-fc59445331d6 from=127.0.0.1:39758 latency=3.56932ms -44:54.053 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test19033109-8768-4114-fe77-fc59445331d6 from=127.0.0.1:39752 latency=121.145µs -44:54.059 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39752 latency=4.689588ms -44:54.060 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test29007d69-072d-fa61-7928-2385a9a1dd04 from=127.0.0.1:39760 latency=192.343µs -44:54.064 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test29007d69-072d-fa61-7928-2385a9a1dd04 from=127.0.0.1:39760 latency=3.524199ms -44:54.065 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test29007d69-072d-fa61-7928-2385a9a1dd04 from=127.0.0.1:39752 latency=167.322µs -44:54.069 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39752 latency=3.538066ms -44:54.070 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/testea22f644-c4a1-86f9-bece-25468c1772f8 from=127.0.0.1:39764 latency=164.127µs -44:54.074 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/testea22f644-c4a1-86f9-bece-25468c1772f8 from=127.0.0.1:39764 latency=3.961712ms -44:54.075 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/testea22f644-c4a1-86f9-bece-25468c1772f8 from=127.0.0.1:39752 latency=221.975µs -44:54.080 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39752 latency=3.98614ms -44:54.081 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/testef738730-d555-425e-7aed-da6011fce7ab from=127.0.0.1:39772 latency=151.28µs -44:54.086 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/testef738730-d555-425e-7aed-da6011fce7ab from=127.0.0.1:39772 latency=3.94216ms -44:54.086 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/testef738730-d555-425e-7aed-da6011fce7ab from=127.0.0.1:39752 latency=152.898µs -44:54.087 [INFO] TestAgent: Requesting shutdown -44:54.087 [INFO] TestAgent.server: shutting down server -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.087 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.087 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.087 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.087 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.087 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:54.090 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.096Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.096Z [INFO] agent.router.manager: shutting down -44:54.096 [INFO] TestAgent: consul server down -44:54.096 [INFO] TestAgent: shutdown complete -44:54.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15141 network=tcp -44:54.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15141 network=udp -44:54.096 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15142 network=tcp -44:54.597 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.597 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand_config (0.85s) - --- PASS: TestAuthMethodUpdateCommand_config/config_file (0.01s) - --- PASS: TestAuthMethodUpdateCommand_config/config_stdin (0.01s) - --- PASS: TestAuthMethodUpdateCommand_config/config_string (0.01s) - --- PASS: TestAuthMethodUpdateCommand_config/config_with_no_merge (0.01s) -2022-11-07T22:44:53.792Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.848 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ec40dfe5-355b-5a96-bf8a-9ba7291bf251 Address:127.0.0.1:15125}]" -44:53.850 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15125 [Follower]" leader= -44:53.850 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 127.0.0.1 -44:53.882 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 127.0.0.1 -2022-11-07T22:44:53.882Z [INFO] agent.router: Initializing LAN area manager -44:53.883 [INFO] TestAgent.server: Adding LAN server: server="Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 (Addr: tcp/127.0.0.1:15125) (DC: dc1)" -44:53.883 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 area=wan -44:53.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=udp -44:53.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=tcp -44:53.883 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15121 network=tcp -44:53.883 [INFO] TestAgent: started state syncer -44:53.883 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15126 network=tcp -44:53.895 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.895 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15125 [Candidate]" term=2 -44:53.912 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.912 [DEBUG] TestAgent.server.raft: vote granted: from=ec40dfe5-355b-5a96-bf8a-9ba7291bf251 term=2 tally=1 -44:53.912 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.912 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15125 [Leader]" -44:53.912 [INFO] TestAgent.server: cluster leadership acquired -44:53.912 [INFO] TestAgent.server: New leader elected: payload=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:53.928 [INFO] TestAgent.server: initializing acls -44:53.933 [INFO] TestAgent.server: initializing acls -44:53.933 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.933 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.943 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.943 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.961 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.961 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.968 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.968 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.968 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.968 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:53.968 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:53.969 [INFO] TestAgent.server: Updating LAN server: server="Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 (Addr: tcp/127.0.0.1:15125) (DC: dc1)" -44:53.969 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 -44:53.969 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 area=wan -44:53.976 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.977 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:53.977 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 -44:53.977 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251.dc1 area=wan -44:53.977 [INFO] TestAgent.server: Updating LAN server: server="Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 (Addr: tcp/127.0.0.1:15125) (DC: dc1)" -44:54.005 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.030 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.030 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.030 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.030 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.030 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.030 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:54.030 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:54.040 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.040 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:54.040 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ec40dfe5-355b-5a96-bf8a-9ba7291bf251 -44:54.078 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-29999204-16a8-82f2-3146-82a03c3428ac from=127.0.0.1:41778 latency=552.76µs -44:54.083 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:41780 latency=3.470921ms -44:54.084 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41784 latency=275.968µs -44:54.088 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41784 latency=3.405783ms -44:54.088 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41780 latency=117.877µs -44:54.090 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41788 latency=149.536µs -44:54.093 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41788 latency=2.982968ms -44:54.094 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-97432988-1726-09c8-e694-70783362bb8a from=127.0.0.1:41780 latency=141.399µs -44:54.094 [INFO] TestAgent: Requesting shutdown -44:54.094 [INFO] TestAgent.server: shutting down server -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.094 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.094 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.094 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.094 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.094 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:54.099 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.104Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.104Z [INFO] agent.router.manager: shutting down -44:54.104 [INFO] TestAgent: consul server down -44:54.104 [INFO] TestAgent: shutdown complete -44:54.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=tcp -44:54.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=udp -44:54.104 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15121 network=tcp -44:54.604 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.604 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand (0.86s) - --- PASS: TestAuthMethodUpdateCommand/update_without_name (0.00s) - --- PASS: TestAuthMethodUpdateCommand/update_nonexistent_method (0.00s) - --- PASS: TestAuthMethodUpdateCommand/update_all_fields (0.01s) - --- PASS: TestAuthMethodUpdateCommand/update_config_field_and_prove_no_merging_happens (0.01s) -2022-11-07T22:44:53.786Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.802Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.837 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:336b6c9d-c975-4212-95cd-df7efd2987c7 Address:127.0.0.1:15139}]" -44:53.837 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15139 [Follower]" leader= -44:53.838 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-336b6c9d-c975-4212-95cd-df7efd2987c7.dc1 127.0.0.1 -44:53.839 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-336b6c9d-c975-4212-95cd-df7efd2987c7 127.0.0.1 -2022-11-07T22:44:53.839Z [INFO] agent.router: Initializing LAN area manager -44:53.839 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-336b6c9d-c975-4212-95cd-df7efd2987c7.dc1 area=wan -44:53.839 [INFO] TestAgent.server: Adding LAN server: server="Node-336b6c9d-c975-4212-95cd-df7efd2987c7 (Addr: tcp/127.0.0.1:15139) (DC: dc1)" -44:53.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=tcp -44:53.839 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=udp -44:53.839 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15135 network=tcp -44:53.839 [INFO] TestAgent: started state syncer -44:53.839 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15140 network=tcp -44:53.883 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.883 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15139 [Candidate]" term=2 -44:53.889 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:53.892 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.892 [DEBUG] TestAgent.server.raft: vote granted: from=336b6c9d-c975-4212-95cd-df7efd2987c7 term=2 tally=1 -44:53.892 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.892 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15139 [Leader]" -44:53.892 [INFO] TestAgent.server: cluster leadership acquired -44:53.892 [INFO] TestAgent.server: New leader elected: payload=Node-336b6c9d-c975-4212-95cd-df7efd2987c7 -44:53.903 [INFO] TestAgent.server: initializing acls -44:53.906 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -44:53.909 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.909 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.918 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.933 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.933 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.933 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.933 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-336b6c9d-c975-4212-95cd-df7efd2987c7 -44:53.933 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-336b6c9d-c975-4212-95cd-df7efd2987c7.dc1 -44:53.933 [INFO] TestAgent.server: Updating LAN server: server="Node-336b6c9d-c975-4212-95cd-df7efd2987c7 (Addr: tcp/127.0.0.1:15139) (DC: dc1)" -44:53.933 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-336b6c9d-c975-4212-95cd-df7efd2987c7.dc1 area=wan -44:53.952 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:53.987 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:53.987 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:53.987 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:53.987 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:53.987 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:53.987 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-336b6c9d-c975-4212-95cd-df7efd2987c7 -44:53.987 [INFO] TestAgent.server: member joined, marking health alive: member=Node-336b6c9d-c975-4212-95cd-df7efd2987c7 -44:54.002 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.002 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-336b6c9d-c975-4212-95cd-df7efd2987c7 -44:54.036 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56184 latency=5.950461ms -44:54.039 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-79403619-832f-0639-608c-394d0686666e from=127.0.0.1:56190 latency=1.477022ms -44:54.047 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56184 latency=6.628123ms -44:54.048 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-52e2af4b-aee6-a986-773b-ee483fff06f0 from=127.0.0.1:56196 latency=162.003µs -44:54.063 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56184 latency=14.166811ms -44:54.065 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-1bfe114c-9d87-d864-afb3-6dcf7de92190 from=127.0.0.1:56202 latency=237.484µs -44:54.070 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56184 latency=4.242336ms -44:54.071 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-591b0250-3d1b-33fc-9f6a-cddbb4022940 from=127.0.0.1:56206 latency=192.271µs -44:54.080 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-591b0250-3d1b-33fc-9f6a-cddbb4022940 from=127.0.0.1:56206 latency=8.193754ms -44:54.081 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-591b0250-3d1b-33fc-9f6a-cddbb4022940 from=127.0.0.1:56184 latency=308.373µs -44:54.086 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56184 latency=4.214053ms -44:54.088 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-e1e5c70d-fd4c-ae12-51fb-9f098847b865 from=127.0.0.1:56216 latency=319.881µs -44:54.097 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-e1e5c70d-fd4c-ae12-51fb-9f098847b865 from=127.0.0.1:56216 latency=7.946993ms -44:54.097 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-e1e5c70d-fd4c-ae12-51fb-9f098847b865 from=127.0.0.1:56184 latency=212.526µs -44:54.098 [INFO] TestAgent: Requesting shutdown -44:54.098 [INFO] TestAgent.server: shutting down server -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.098 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.098 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.098 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.098 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.101 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.106Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.106Z [INFO] agent.router.manager: shutting down -44:54.107 [INFO] TestAgent: consul server down -44:54.107 [INFO] TestAgent: shutdown complete -44:54.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=tcp -44:54.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=udp -44:54.107 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15135 network=tcp -44:54.607 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.607 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand_k8s_noMerge (0.86s) - --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_host (0.01s) - --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_ca_cert (0.01s) - --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_jwt (0.02s) - --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields (0.02s) - --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields_with_cert_file (0.02s) -2022-11-07T22:44:53.788Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.845 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d245638a-8176-270a-f411-181b48bb6e10 Address:127.0.0.1:15111}]" -44:53.845 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= -44:53.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d245638a-8176-270a-f411-181b48bb6e10.dc1 127.0.0.1 -44:53.846 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d245638a-8176-270a-f411-181b48bb6e10 127.0.0.1 -2022-11-07T22:44:53.846Z [INFO] agent.router: Initializing LAN area manager -44:53.846 [INFO] TestAgent.server: Adding LAN server: server="Node-d245638a-8176-270a-f411-181b48bb6e10 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" -44:53.846 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d245638a-8176-270a-f411-181b48bb6e10.dc1 area=wan -44:53.846 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp -44:53.847 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp -44:53.847 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp -44:53.848 [INFO] TestAgent: started state syncer -44:53.848 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp -44:53.896 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:53.904 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.904 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 -44:53.922 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.922 [DEBUG] TestAgent.server.raft: vote granted: from=d245638a-8176-270a-f411-181b48bb6e10 term=2 tally=1 -44:53.922 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.922 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" -44:53.922 [INFO] TestAgent.server: cluster leadership acquired -44:53.923 [INFO] TestAgent.server: New leader elected: payload=Node-d245638a-8176-270a-f411-181b48bb6e10 -44:53.936 [INFO] TestAgent.server: initializing acls -44:53.939 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.939 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.946 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.962 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.962 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.962 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.962 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d245638a-8176-270a-f411-181b48bb6e10 -44:53.962 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d245638a-8176-270a-f411-181b48bb6e10.dc1 -44:53.962 [INFO] TestAgent.server: Updating LAN server: server="Node-d245638a-8176-270a-f411-181b48bb6e10 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" -44:53.962 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d245638a-8176-270a-f411-181b48bb6e10.dc1 area=wan -44:53.981 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.009 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.009 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.009 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.009 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.009 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.010 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d245638a-8176-270a-f411-181b48bb6e10 -44:54.010 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d245638a-8176-270a-f411-181b48bb6e10 -44:54.019 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d245638a-8176-270a-f411-181b48bb6e10 -44:54.030 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.147 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.152 [INFO] TestAgent: Synced node info -44:54.152 [DEBUG] TestAgent: Node info in sync -44:54.233 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46504 latency=4.468784ms -44:54.234 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f8097057-4729-20e8-020c-0b13a7fe1e5b from=127.0.0.1:46506 latency=230.668µs -44:54.241 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-f8097057-4729-20e8-020c-0b13a7fe1e5b from=127.0.0.1:46506 latency=6.101925ms -44:54.242 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f8097057-4729-20e8-020c-0b13a7fe1e5b from=127.0.0.1:46504 latency=155.491µs -44:54.251 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46504 latency=8.26145ms -44:54.252 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-df107666-a261-115e-dd92-b0d0ffef7789 from=127.0.0.1:46508 latency=219.555µs -44:54.258 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-df107666-a261-115e-dd92-b0d0ffef7789 from=127.0.0.1:46508 latency=5.684301ms -44:54.259 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-df107666-a261-115e-dd92-b0d0ffef7789 from=127.0.0.1:46504 latency=152.488µs -44:54.264 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46504 latency=3.668494ms -44:54.265 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-a23af6bf-4093-482e-fc9d-65b51d71923f from=127.0.0.1:46510 latency=180.833µs -44:54.271 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-a23af6bf-4093-482e-fc9d-65b51d71923f from=127.0.0.1:46510 latency=5.610166ms -44:54.272 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-a23af6bf-4093-482e-fc9d-65b51d71923f from=127.0.0.1:46504 latency=162.578µs -44:54.276 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46504 latency=3.51809ms -44:54.279 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-e71afd73-c3aa-29c2-d3f1-3321d2f0ce8b from=127.0.0.1:46512 latency=190.386µs -44:54.286 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-e71afd73-c3aa-29c2-d3f1-3321d2f0ce8b from=127.0.0.1:46512 latency=6.334977ms -44:54.286 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-e71afd73-c3aa-29c2-d3f1-3321d2f0ce8b from=127.0.0.1:46504 latency=161.376µs -44:54.291 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46504 latency=3.548152ms -44:54.292 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-8b410c39-3a1d-c048-e741-9f4af82f0f95 from=127.0.0.1:46514 latency=225.79µs -44:54.298 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-8b410c39-3a1d-c048-e741-9f4af82f0f95 from=127.0.0.1:46514 latency=5.378564ms -44:54.299 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-8b410c39-3a1d-c048-e741-9f4af82f0f95 from=127.0.0.1:46504 latency=131.504µs -44:54.299 [INFO] TestAgent: Requesting shutdown -44:54.300 [INFO] TestAgent.server: shutting down server -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.300 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.300 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.300 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.300 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.303 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.306Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.306Z [INFO] agent.router.manager: shutting down -44:54.306 [INFO] TestAgent: consul server down -44:54.306 [INFO] TestAgent: shutdown complete -44:54.306 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp -44:54.306 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp -44:54.306 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp -44:54.807 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.807 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand_k8s (1.06s) - --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields (0.01s) - --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_with_cert_file (0.02s) - --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_host (0.01s) - --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_ca_cert (0.01s) - --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_jwt (0.01s) -2022-11-07T22:44:53.790Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:53.804Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:53.849 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f2f45c92-10e1-affd-7a7f-49eed344899f Address:127.0.0.1:15118}]" -44:53.849 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= -44:53.850 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 127.0.0.1 -44:53.882 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f2f45c92-10e1-affd-7a7f-49eed344899f 127.0.0.1 -2022-11-07T22:44:53.882Z [INFO] agent.router: Initializing LAN area manager -44:53.882 [INFO] TestAgent.server: Adding LAN server: server="Node-f2f45c92-10e1-affd-7a7f-49eed344899f (Addr: tcp/127.0.0.1:15118) (DC: dc1)" -44:53.883 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 area=wan -44:53.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp -44:53.883 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp -44:53.883 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp -44:53.883 [INFO] TestAgent: started state syncer -44:53.883 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp -44:53.895 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:53.895 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 -44:53.922 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:53.922 [DEBUG] TestAgent.server.raft: vote granted: from=f2f45c92-10e1-affd-7a7f-49eed344899f term=2 tally=1 -44:53.922 [INFO] TestAgent.server.raft: election won: tally=1 -44:53.922 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" -44:53.922 [INFO] TestAgent.server: cluster leadership acquired -44:53.922 [INFO] TestAgent.server: New leader elected: payload=Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:53.933 [INFO] TestAgent.server: initializing acls -44:53.936 [INFO] TestAgent.server: initializing acls -44:53.949 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.949 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.949 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:53.949 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:53.967 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.981 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:53.981 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:53.981 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:53.981 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:53.981 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:53.981 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:53.981 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 -44:53.981 [INFO] TestAgent.server: Updating LAN server: server="Node-f2f45c92-10e1-affd-7a7f-49eed344899f (Addr: tcp/127.0.0.1:15118) (DC: dc1)" -44:53.981 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 area=wan -44:53.981 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:53.981 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 -44:53.981 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f2f45c92-10e1-affd-7a7f-49eed344899f.dc1 area=wan -44:53.981 [INFO] TestAgent.server: Updating LAN server: server="Node-f2f45c92-10e1-affd-7a7f-49eed344899f (Addr: tcp/127.0.0.1:15118) (DC: dc1)" -44:54.000 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.021 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.021 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.021 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.021 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.021 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.021 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:54.021 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:54.032 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.032 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:54.032 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f2f45c92-10e1-affd-7a7f-49eed344899f -44:54.311 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.314 [INFO] TestAgent: Synced node info -44:54.314 [DEBUG] TestAgent: Node info in sync -44:54.318 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:52910 latency=3.464672ms -44:54.319 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-a1ef0bd6-0bf7-c5bd-0b5f-b528a98afa3c from=127.0.0.1:52912 latency=127.739µs -44:54.323 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-a1ef0bd6-0bf7-c5bd-0b5f-b528a98afa3c from=127.0.0.1:52912 latency=3.016138ms -44:54.323 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-a1ef0bd6-0bf7-c5bd-0b5f-b528a98afa3c from=127.0.0.1:52910 latency=137.978µs -44:54.324 [INFO] TestAgent: Requesting shutdown -44:54.324 [INFO] TestAgent.server: shutting down server -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.324 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.324 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.324 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.324 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.327 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.330Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.330Z [INFO] agent.router.manager: shutting down -44:54.330 [INFO] TestAgent: consul server down -44:54.330 [INFO] TestAgent: shutdown complete -44:54.330 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp -44:54.330 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp -44:54.331 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp -44:54.831 [INFO] TestAgent: Waiting for endpoints to shut down -44:54.831 [INFO] TestAgent: Endpoints down ---- PASS: TestAuthMethodUpdateCommand_JSON (1.08s) + --- PASS: TestAuthMethodUpdateCommand_noMerge/update_nonexistent_method (0.01s) + --- PASS: TestAuthMethodUpdateCommand_noMerge/update_all_fields (0.12s) +2021-10-05T16:30:10.181Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.317Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.410 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f97b425f-67ff-55b6-821c-3453c2fad690 Address:127.0.0.1:11048}]" +30:10.411 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 127.0.0.1 +30:10.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f97b425f-67ff-55b6-821c-3453c2fad690 127.0.0.1 +2021-10-05T16:30:10.411Z [INFO] agent.router: Initializing LAN area manager +30:10.412 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11048 [Follower]" leader= +30:10.412 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=udp +30:10.412 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11043 network=tcp +30:10.412 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 area=wan +30:10.412 [INFO] TestAgent.server: Adding LAN server: server="Node-f97b425f-67ff-55b6-821c-3453c2fad690 (Addr: tcp/127.0.0.1:11048) (DC: dc1)" +30:10.413 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11044 network=tcp +30:10.413 [INFO] TestAgent: started state syncer +30:10.413 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11049 network=tcp +30:10.464 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:10.480 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.480 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11048 [Candidate]" term=2 +30:10.528 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.528 [DEBUG] TestAgent.server.raft: vote granted: from=f97b425f-67ff-55b6-821c-3453c2fad690 term=2 tally=1 +30:10.528 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.528 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11048 [Leader]" +30:10.528 [INFO] TestAgent.server: cluster leadership acquired +30:10.528 [INFO] TestAgent.server: New leader elected: payload=Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.564 [INFO] TestAgent.server: initializing acls +30:10.584 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.584 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.584 [INFO] TestAgent.server: initializing acls +30:10.584 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.644 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.648 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.660 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.660 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.660 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.660 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.660 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 +30:10.661 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 area=wan +30:10.661 [INFO] TestAgent.server: Updating LAN server: server="Node-f97b425f-67ff-55b6-821c-3453c2fad690 (Addr: tcp/127.0.0.1:11048) (DC: dc1)" +30:10.680 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.680 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:10.680 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.680 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 +30:10.680 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f97b425f-67ff-55b6-821c-3453c2fad690.dc1 area=wan +30:10.680 [INFO] TestAgent.server: Updating LAN server: server="Node-f97b425f-67ff-55b6-821c-3453c2fad690 (Addr: tcp/127.0.0.1:11048) (DC: dc1)" +30:10.727 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.796 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.796 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.796 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.796 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.796 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.796 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.796 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.809 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.809 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f97b425f-67ff-55b6-821c-3453c2fad690 +30:10.824 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.849 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39348 latency=23.233129ms +30:10.855 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-a1161b7f-62b5-9949-8a92-e108cfbf818d from=127.0.0.1:39354 latency=522.364µs +30:10.876 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-a1161b7f-62b5-9949-8a92-e108cfbf818d from=127.0.0.1:39354 latency=20.893339ms +30:10.882 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-a1161b7f-62b5-9949-8a92-e108cfbf818d from=127.0.0.1:39348 latency=146.994µs +30:10.882 [INFO] TestAgent: Requesting shutdown +30:10.882 [INFO] TestAgent.server: shutting down server +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:10.882 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:10.882 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:10.882 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:10.882 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:10.882 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:10.902 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:10.903Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:10.903Z [INFO] agent.router.manager: shutting down +30:10.903 [INFO] TestAgent: consul server down +30:10.903 [INFO] TestAgent: shutdown complete +30:10.903 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=tcp +30:10.903 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11043 network=udp +30:10.903 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11044 network=tcp +30:11.403 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.403 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand_JSON (1.48s) --- PASS: TestAuthMethodUpdateCommand_JSON/update_without_name (0.00s) - --- PASS: TestAuthMethodUpdateCommand_JSON/update_all_fields (0.01s) + --- PASS: TestAuthMethodUpdateCommand_JSON/update_all_fields (0.06s) +2021-10-05T16:30:10.123Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.278Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.383 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5d3e2f1d-9218-d9e5-bdc4-f5832637820a Address:127.0.0.1:11062}]" +30:10.384 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11062 [Follower]" leader= +30:10.389 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 127.0.0.1 +30:10.390 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a 127.0.0.1 +2021-10-05T16:30:10.390Z [INFO] agent.router: Initializing LAN area manager +30:10.396 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 area=wan +30:10.404 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=tcp +30:10.405 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11057 network=udp +30:10.405 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11058 network=tcp +30:10.405 [INFO] TestAgent: started state syncer +30:10.406 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11063 network=tcp +30:10.406 [INFO] TestAgent.server: Adding LAN server: server="Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a (Addr: tcp/127.0.0.1:11062) (DC: dc1)" +30:10.435 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.435 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11062 [Candidate]" term=2 +30:10.447 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:10.452 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.452 [DEBUG] TestAgent.server.raft: vote granted: from=5d3e2f1d-9218-d9e5-bdc4-f5832637820a term=2 tally=1 +30:10.452 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.452 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11062 [Leader]" +30:10.454 [INFO] TestAgent.server: cluster leadership acquired +30:10.454 [INFO] TestAgent.server: New leader elected: payload=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.480 [INFO] TestAgent.server: initializing acls +30:10.503 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.503 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.528 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.548 [INFO] TestAgent.server: initializing acls +30:10.548 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.560 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.560 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.560 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.561 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.561 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 +30:10.561 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 area=wan +30:10.561 [INFO] TestAgent.server: Updating LAN server: server="Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a (Addr: tcp/127.0.0.1:11062) (DC: dc1)" +30:10.594 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.594 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:10.595 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.595 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 +30:10.595 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a.dc1 area=wan +30:10.595 [INFO] TestAgent.server: Updating LAN server: server="Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a (Addr: tcp/127.0.0.1:11062) (DC: dc1)" +30:10.629 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.666 [INFO] TestAgent: Synced node info +30:10.666 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.666 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.666 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.666 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.666 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.666 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.666 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5d3e2f1d-9218-d9e5-bdc4-f5832637820a +30:10.708 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.794 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-8b3f503f-5078-c224-e7cd-6e7418312f7c from=127.0.0.1:43916 latency=683.247µs +30:10.807 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43918 latency=12.033185ms +30:10.825 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43922 latency=2.274179ms +30:10.844 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43922 latency=18.580678ms +30:10.854 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43918 latency=481.201µs +30:10.856 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43934 latency=248.491µs +30:10.900 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43934 latency=35.834369ms +30:10.902 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test-974961ca-79ba-3a73-834e-2656249038c9 from=127.0.0.1:43918 latency=341.309µs +30:10.902 [INFO] TestAgent: Requesting shutdown +30:10.903 [INFO] TestAgent.server: shutting down server +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:10.903 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:10.903 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:10.903 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:10.903 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:10.911 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:10.913Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:10.913Z [INFO] agent.router.manager: shutting down +30:10.913 [INFO] TestAgent: consul server down +30:10.913 [INFO] TestAgent: shutdown complete +30:10.913 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=tcp +30:10.913 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11057 network=udp +30:10.913 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11058 network=tcp +30:11.416 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.416 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand (1.49s) + --- PASS: TestAuthMethodUpdateCommand/update_without_name (0.00s) + --- PASS: TestAuthMethodUpdateCommand/update_nonexistent_method (0.01s) + --- PASS: TestAuthMethodUpdateCommand/update_all_fields (0.05s) + --- PASS: TestAuthMethodUpdateCommand/update_config_field_and_prove_no_merging_happens (0.05s) +2021-10-05T16:30:10.132Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.222Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.344 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40ec56d2-673a-8339-7e4f-a4293df61bd1 Address:127.0.0.1:11027}]" +30:10.344 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11027 [Follower]" leader= +30:10.345 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-40ec56d2-673a-8339-7e4f-a4293df61bd1.dc1 127.0.0.1 +30:10.345 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 127.0.0.1 +2021-10-05T16:30:10.346Z [INFO] agent.router: Initializing LAN area manager +30:10.346 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=udp +30:10.346 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1.dc1 area=wan +30:10.346 [INFO] TestAgent.server: Adding LAN server: server="Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +30:10.346 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=tcp +30:10.347 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11023 network=tcp +30:10.347 [INFO] TestAgent: started state syncer +30:10.347 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11028 network=tcp +30:10.400 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:10.413 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.413 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11027 [Candidate]" term=2 +30:10.431 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.431 [DEBUG] TestAgent.server.raft: vote granted: from=40ec56d2-673a-8339-7e4f-a4293df61bd1 term=2 tally=1 +30:10.431 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.431 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11027 [Leader]" +30:10.432 [INFO] TestAgent.server: cluster leadership acquired +30:10.433 [INFO] TestAgent.server: New leader elected: payload=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 +30:10.444 [INFO] TestAgent.server: initializing acls +30:10.453 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.453 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.468 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.492 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.492 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.492 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.492 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 +30:10.492 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-40ec56d2-673a-8339-7e4f-a4293df61bd1.dc1 +30:10.493 [INFO] TestAgent.server: Updating LAN server: server="Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +30:10.493 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1.dc1 area=wan +30:10.517 [INFO] TestAgent: Synced node info +30:10.528 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.593 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.593 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.593 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.593 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.593 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.593 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 +30:10.593 [INFO] TestAgent.server: member joined, marking health alive: member=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 +30:10.616 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-40ec56d2-673a-8339-7e4f-a4293df61bd1 +30:10.628 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.766 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59706 latency=37.700732ms +30:10.774 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-ffacfc36-97d6-126a-556a-14cf5e9fadc5 from=127.0.0.1:59714 latency=464.517µs +30:10.809 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59706 latency=33.659993ms +30:10.826 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-ee1a1245-86fb-c417-97aa-ca87307d440b from=127.0.0.1:59724 latency=387.824µs +30:10.856 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59706 latency=29.460414ms +30:10.864 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-7aa7df28-c128-1f7c-34b7-23067f0ad4c7 from=127.0.0.1:59736 latency=446.31µs +30:10.917 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59706 latency=48.839288ms +30:10.921 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-3c8d807f-1cc6-2eb6-9622-6b966cb65957 from=127.0.0.1:59740 latency=416.852µs +30:10.941 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-3c8d807f-1cc6-2eb6-9622-6b966cb65957 from=127.0.0.1:59740 latency=19.080367ms +30:10.949 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-3c8d807f-1cc6-2eb6-9622-6b966cb65957 from=127.0.0.1:59706 latency=678.714µs +30:10.985 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59706 latency=34.767048ms +30:10.987 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-83d909ac-c0b8-5898-ef4d-27828e25126f from=127.0.0.1:59746 latency=329.207µs +30:11.004 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-83d909ac-c0b8-5898-ef4d-27828e25126f from=127.0.0.1:59746 latency=14.828658ms +30:11.013 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-83d909ac-c0b8-5898-ef4d-27828e25126f from=127.0.0.1:59706 latency=1.076966ms +30:11.014 [INFO] TestAgent: Requesting shutdown +30:11.014 [INFO] TestAgent.server: shutting down server +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:11.014 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:11.014 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:11.014 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:11.014 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.037 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:11.040Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:11.040Z [INFO] agent.router.manager: shutting down +30:11.040 [INFO] TestAgent: consul server down +30:11.040 [INFO] TestAgent: shutdown complete +30:11.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=tcp +30:11.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=udp +30:11.040 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11023 network=tcp +30:11.543 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.543 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand_k8s_noMerge (1.62s) + --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_host (0.05s) + --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_ca_cert (0.05s) + --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_missing_k8s_jwt (0.04s) + --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields (0.08s) + --- PASS: TestAuthMethodUpdateCommand_k8s_noMerge/update_all_fields_with_cert_file (0.06s) +2021-10-05T16:30:10.138Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.299Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.440 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:24b5d716-0393-5765-d731-2ee9bfe72fe2 Address:127.0.0.1:11055}]" +30:10.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11055 [Follower]" leader= +30:10.441 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-24b5d716-0393-5765-d731-2ee9bfe72fe2.dc1 127.0.0.1 +30:10.442 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 127.0.0.1 +2021-10-05T16:30:10.442Z [INFO] agent.router: Initializing LAN area manager +30:10.442 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=udp +30:10.442 [INFO] TestAgent.server: Adding LAN server: server="Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 (Addr: tcp/127.0.0.1:11055) (DC: dc1)" +30:10.443 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2.dc1 area=wan +30:10.443 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11050 network=tcp +30:10.443 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11051 network=tcp +30:10.443 [INFO] TestAgent: started state syncer +30:10.443 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11056 network=tcp +30:10.482 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.482 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11055 [Candidate]" term=2 +30:10.493 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:10.519 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.519 [DEBUG] TestAgent.server.raft: vote granted: from=24b5d716-0393-5765-d731-2ee9bfe72fe2 term=2 tally=1 +30:10.519 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.519 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11055 [Leader]" +30:10.519 [INFO] TestAgent.server: cluster leadership acquired +30:10.519 [INFO] TestAgent.server: New leader elected: payload=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 +30:10.530 [INFO] TestAgent.server: initializing acls +30:10.536 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.536 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.564 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.592 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.592 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.592 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.593 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 +30:10.593 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-24b5d716-0393-5765-d731-2ee9bfe72fe2.dc1 +30:10.595 [INFO] TestAgent.server: Updating LAN server: server="Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 (Addr: tcp/127.0.0.1:11055) (DC: dc1)" +30:10.606 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2.dc1 area=wan +30:10.644 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.720 [INFO] TestAgent: Synced node info +30:10.759 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.759 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.759 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.759 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.759 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.759 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 +30:10.759 [INFO] TestAgent.server: member joined, marking health alive: member=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 +30:10.793 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24b5d716-0393-5765-d731-2ee9bfe72fe2 +30:10.821 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48124 latency=57.230311ms +30:10.835 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test0f2822f2-d38c-818e-e7fa-358c74368315 from=127.0.0.1:48144 latency=389.282µs +30:10.848 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test0f2822f2-d38c-818e-e7fa-358c74368315 from=127.0.0.1:48144 latency=13.151316ms +30:10.855 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test0f2822f2-d38c-818e-e7fa-358c74368315 from=127.0.0.1:48124 latency=342.35µs +30:10.896 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48124 latency=40.830247ms +30:10.917 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test8325447e-93cf-48e6-95d4-32eac2a021a0 from=127.0.0.1:48152 latency=689.273µs +30:10.923 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test8325447e-93cf-48e6-95d4-32eac2a021a0 from=127.0.0.1:48152 latency=4.284155ms +30:10.927 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test8325447e-93cf-48e6-95d4-32eac2a021a0 from=127.0.0.1:48124 latency=926.934µs +30:10.954 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48124 latency=25.820256ms +30:10.972 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test91942e81-4a78-1fa7-c6d4-7207124bdf62 from=127.0.0.1:48158 latency=14.242276ms +30:10.992 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test91942e81-4a78-1fa7-c6d4-7207124bdf62 from=127.0.0.1:48158 latency=19.372237ms +30:10.996 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test91942e81-4a78-1fa7-c6d4-7207124bdf62 from=127.0.0.1:48124 latency=260.437µs +30:11.012 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48124 latency=15.0946ms +30:11.016 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test94ef1880-d083-7459-ec6f-86b953acbb68 from=127.0.0.1:48164 latency=281.527µs +30:11.040 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/test94ef1880-d083-7459-ec6f-86b953acbb68 from=127.0.0.1:48164 latency=23.546335ms +30:11.041 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/test94ef1880-d083-7459-ec6f-86b953acbb68 from=127.0.0.1:48124 latency=213.732µs +30:11.042 [INFO] TestAgent: Requesting shutdown +30:11.042 [INFO] TestAgent.server: shutting down server +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:11.042 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:11.042 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:11.042 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:11.042 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.047 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:11.047Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:11.047Z [INFO] agent.router.manager: shutting down +30:11.048 [INFO] TestAgent: consul server down +30:11.048 [INFO] TestAgent: shutdown complete +30:11.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=tcp +30:11.048 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11050 network=udp +30:11.048 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11051 network=tcp +30:11.551 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.551 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand_config (1.62s) + --- PASS: TestAuthMethodUpdateCommand_config/config_file (0.11s) + --- PASS: TestAuthMethodUpdateCommand_config/config_stdin (0.07s) + --- PASS: TestAuthMethodUpdateCommand_config/config_string (0.07s) + --- PASS: TestAuthMethodUpdateCommand_config/config_with_no_merge (0.05s) +2021-10-05T16:30:10.099Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:10.231Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:10.347 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:71d0a91c-0ee8-5579-0b7f-e222ae97a63b Address:127.0.0.1:11041}]" +30:10.347 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11041 [Follower]" leader= +30:10.347 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 127.0.0.1 +30:10.398 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b 127.0.0.1 +2021-10-05T16:30:10.398Z [INFO] agent.router: Initializing LAN area manager +30:10.399 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=udp +30:10.399 [INFO] TestAgent.server: Adding LAN server: server="Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b (Addr: tcp/127.0.0.1:11041) (DC: dc1)" +30:10.399 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 area=wan +30:10.406 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11036 network=tcp +30:10.406 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:10.406 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11041 [Candidate]" term=2 +30:10.406 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11037 network=tcp +30:10.406 [INFO] TestAgent: started state syncer +30:10.406 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11042 network=tcp +30:10.448 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:10.448 [DEBUG] TestAgent.server.raft: vote granted: from=71d0a91c-0ee8-5579-0b7f-e222ae97a63b term=2 tally=1 +30:10.448 [INFO] TestAgent.server.raft: election won: tally=1 +30:10.448 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11041 [Leader]" +30:10.448 [INFO] TestAgent.server: cluster leadership acquired +30:10.448 [INFO] TestAgent.server: New leader elected: payload=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.452 [INFO] TestAgent.server: initializing acls +30:10.480 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:10.480 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.480 [INFO] TestAgent.server: initializing acls +30:10.480 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:10.514 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.553 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:10.553 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:10.553 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:10.553 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:10.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.554 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 +30:10.554 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 area=wan +30:10.554 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:10.554 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.554 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 +30:10.555 [INFO] TestAgent.server: Updating LAN server: server="Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b (Addr: tcp/127.0.0.1:11041) (DC: dc1)" +30:10.556 [INFO] TestAgent.server: Updating LAN server: server="Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b (Addr: tcp/127.0.0.1:11041) (DC: dc1)" +30:10.556 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b.dc1 area=wan +30:10.618 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:10.647 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:10.647 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:10.647 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:10.647 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:10.647 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:10.647 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.647 [INFO] TestAgent.server: member joined, marking health alive: member=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.659 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.659 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-71d0a91c-0ee8-5579-0b7f-e222ae97a63b +30:10.680 [INFO] TestAgent.server: federation state anti-entropy synced +30:10.688 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:10.700 [INFO] TestAgent: Synced node info +30:10.700 [DEBUG] TestAgent: Node info in sync +30:10.742 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48230 latency=25.774064ms +30:10.761 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f35f482a-f940-1969-90cc-2555b845ce15 from=127.0.0.1:48238 latency=485.546µs +30:10.781 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-f35f482a-f940-1969-90cc-2555b845ce15 from=127.0.0.1:48238 latency=19.332909ms +30:10.797 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f35f482a-f940-1969-90cc-2555b845ce15 from=127.0.0.1:48230 latency=2.859851ms +30:10.827 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48230 latency=25.883935ms +30:10.831 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-74dec523-90bd-df95-759d-af5197fc02fd from=127.0.0.1:48254 latency=429.828µs +30:10.873 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-74dec523-90bd-df95-759d-af5197fc02fd from=127.0.0.1:48254 latency=32.29205ms +30:10.881 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-74dec523-90bd-df95-759d-af5197fc02fd from=127.0.0.1:48230 latency=557.772µs +30:10.920 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48230 latency=32.481495ms +30:10.923 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-1ab28ef8-6946-f8b2-bf08-955e6f128537 from=127.0.0.1:48268 latency=314.466µs +30:10.949 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-1ab28ef8-6946-f8b2-bf08-955e6f128537 from=127.0.0.1:48268 latency=25.93694ms +30:10.951 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-1ab28ef8-6946-f8b2-bf08-955e6f128537 from=127.0.0.1:48230 latency=338.793µs +30:10.992 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48230 latency=35.642761ms +30:10.997 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-ac1e85d3-433c-6de3-8250-fbbd28cdcb4d from=127.0.0.1:48274 latency=253.753µs +30:11.020 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-ac1e85d3-433c-6de3-8250-fbbd28cdcb4d from=127.0.0.1:48274 latency=22.230859ms +30:11.025 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-ac1e85d3-433c-6de3-8250-fbbd28cdcb4d from=127.0.0.1:48230 latency=296.383µs +30:11.048 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48230 latency=22.683391ms +30:11.050 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f8f99bfc-38fc-f866-f126-7c996f750395 from=127.0.0.1:48278 latency=403.475µs +30:11.059 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method/k8s-f8f99bfc-38fc-f866-f126-7c996f750395 from=127.0.0.1:48278 latency=7.69423ms +30:11.060 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/auth-method/k8s-f8f99bfc-38fc-f866-f126-7c996f750395 from=127.0.0.1:48230 latency=277.805µs +30:11.061 [INFO] TestAgent: Requesting shutdown +30:11.061 [INFO] TestAgent.server: shutting down server +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:11.061 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:11.061 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:11.061 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:11.061 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:11.078 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:11.079Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:11.079Z [INFO] agent.router.manager: shutting down +30:11.079 [INFO] TestAgent: consul server down +30:11.079 [INFO] TestAgent: shutdown complete +30:11.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=tcp +30:11.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11036 network=udp +30:11.080 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11037 network=tcp +30:11.580 [INFO] TestAgent: Waiting for endpoints to shut down +30:11.580 [INFO] TestAgent: Endpoints down +--- PASS: TestAuthMethodUpdateCommand_k8s (1.65s) + --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields (0.09s) + --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_with_cert_file (0.08s) + --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_host (0.07s) + --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_ca_cert (0.07s) + --- PASS: TestAuthMethodUpdateCommand_k8s/update_all_fields_but_k8s_jwt (0.04s) PASS -ok github.com/hashicorp/consul/command/acl/authmethod/update 1.116s +ok github.com/hashicorp/consul/command/acl/authmethod/update 1.786s ? github.com/hashicorp/consul/command/acl/bindingrule [no test files] === RUN TestBindingRuleCreateCommand_noTabs === PAUSE TestBindingRuleCreateCommand_noTabs @@ -73922,10 +74131,10 @@ === RUN TestBindingRuleCreateCommand_JSON === PAUSE TestBindingRuleCreateCommand_JSON === CONT TestBindingRuleCreateCommand_noTabs +--- PASS: TestBindingRuleCreateCommand_noTabs (0.00s) === CONT TestBindingRuleCreateCommand_JSON -=== CONT TestBindingRuleCreateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestBindingRuleCreateCommand_noTabs (0.00s) +=== CONT TestBindingRuleCreateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestBindingRuleCreateCommand/method_is_required @@ -73934,418 +74143,419 @@ === RUN TestBindingRuleCreateCommand/must_use_roughly_valid_selector === RUN TestBindingRuleCreateCommand/create_it_with_no_selector === RUN TestBindingRuleCreateCommand/create_it_with_a_match_selector -=== RUN TestBindingRuleCreateCommand/create_it_with_type_role === RUN TestBindingRuleCreateCommand_JSON/create_it_with_no_selector -2022-11-07T22:44:54.607Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:54.618Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:54.649 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:083ad127-0cef-db28-1a8d-bef96849619a Address:127.0.0.1:18181}]" -44:54.649 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18181 [Follower]" leader= -44:54.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-083ad127-0cef-db28-1a8d-bef96849619a.dc1 127.0.0.1 -44:54.649 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-083ad127-0cef-db28-1a8d-bef96849619a 127.0.0.1 -2022-11-07T22:44:54.649Z [INFO] agent.router: Initializing LAN area manager -44:54.650 [INFO] TestAgent.server: Adding LAN server: server="Node-083ad127-0cef-db28-1a8d-bef96849619a (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -44:54.650 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-083ad127-0cef-db28-1a8d-bef96849619a.dc1 area=wan -44:54.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=udp -44:54.650 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=tcp -44:54.650 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18177 network=tcp -44:54.650 [INFO] TestAgent: started state syncer -44:54.650 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18182 network=tcp -44:54.700 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:54.712 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:54.712 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18181 [Candidate]" term=2 -44:54.723 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:54.723 [DEBUG] TestAgent.server.raft: vote granted: from=083ad127-0cef-db28-1a8d-bef96849619a term=2 tally=1 -44:54.723 [INFO] TestAgent.server.raft: election won: tally=1 -44:54.723 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18181 [Leader]" -44:54.723 [INFO] TestAgent.server: cluster leadership acquired -44:54.723 [INFO] TestAgent.server: New leader elected: payload=Node-083ad127-0cef-db28-1a8d-bef96849619a -44:54.732 [INFO] TestAgent.server: initializing acls -44:54.737 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:54.737 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:54.746 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:54.761 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:54.761 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:54.761 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:54.761 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-083ad127-0cef-db28-1a8d-bef96849619a -44:54.761 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-083ad127-0cef-db28-1a8d-bef96849619a.dc1 -44:54.761 [INFO] TestAgent.server: Updating LAN server: server="Node-083ad127-0cef-db28-1a8d-bef96849619a (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -44:54.761 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-083ad127-0cef-db28-1a8d-bef96849619a.dc1 area=wan -44:54.774 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.793 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.793 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.793 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.793 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.793 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.793 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-083ad127-0cef-db28-1a8d-bef96849619a -44:54.793 [INFO] TestAgent.server: member joined, marking health alive: member=Node-083ad127-0cef-db28-1a8d-bef96849619a -44:54.796 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.804 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-083ad127-0cef-db28-1a8d-bef96849619a -44:54.818 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:53860 latency=6.022631ms -44:54.821 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:53862 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" -44:54.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:53862 latency=592.383µs -44:54.825 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:53864 latency=3.151348ms -44:54.830 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:53866 latency=3.293626ms -44:54.835 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:53868 latency=4.119329ms -44:54.836 [INFO] TestAgent: Requesting shutdown -44:54.836 [INFO] TestAgent.server: shutting down server -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:54.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:54.836 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:54.836 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:54.836 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:54.836 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:54.840 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:54.845Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:54.845Z [INFO] agent.router.manager: shutting down -44:54.845 [INFO] TestAgent: consul server down -44:54.845 [INFO] TestAgent: shutdown complete -44:54.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=tcp -44:54.845 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=udp -44:54.845 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18177 network=tcp -44:55.345 [INFO] TestAgent: Waiting for endpoints to shut down -44:55.345 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleCreateCommand (0.78s) +=== RUN TestBindingRuleCreateCommand/create_it_with_type_role +2021-10-05T16:30:13.235Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:13.273Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:13.315 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:209c7822-3b9c-9e8b-f958-e16e61a393c6 Address:127.0.0.1:10006}]" +30:13.315 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-209c7822-3b9c-9e8b-f958-e16e61a393c6.dc1 127.0.0.1 +30:13.316 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= +30:13.316 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 127.0.0.1 +2021-10-05T16:30:13.316Z [INFO] agent.router: Initializing LAN area manager +30:13.317 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp +30:13.317 [INFO] TestAgent.server: Adding LAN server: server="Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" +30:13.317 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6.dc1 area=wan +30:13.317 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp +30:13.318 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp +30:13.318 [INFO] TestAgent: started state syncer +30:13.318 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp +30:13.367 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:13.370 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:13.370 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 +30:13.383 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:13.383 [DEBUG] TestAgent.server.raft: vote granted: from=209c7822-3b9c-9e8b-f958-e16e61a393c6 term=2 tally=1 +30:13.383 [INFO] TestAgent.server.raft: election won: tally=1 +30:13.383 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" +30:13.383 [INFO] TestAgent.server: cluster leadership acquired +30:13.383 [INFO] TestAgent.server: New leader elected: payload=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 +30:13.402 [INFO] TestAgent.server: initializing acls +30:13.418 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:13.418 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:13.431 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:13.446 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:13.446 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:13.446 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:13.446 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 +30:13.446 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-209c7822-3b9c-9e8b-f958-e16e61a393c6.dc1 +30:13.447 [INFO] TestAgent.server: Updating LAN server: server="Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" +30:13.447 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6.dc1 area=wan +30:13.475 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:13.506 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:13.506 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:13.506 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:13.506 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:13.506 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:13.506 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 +30:13.506 [INFO] TestAgent.server: member joined, marking health alive: member=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 +30:13.514 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-209c7822-3b9c-9e8b-f958-e16e61a393c6 +30:13.519 [INFO] TestAgent.server: federation state anti-entropy synced +30:13.526 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:13.543 [INFO] TestAgent: Synced node info +30:13.738 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:47554 latency=17.910072ms +30:13.766 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47556 latency=13.643614ms +30:13.768 [INFO] TestAgent: Requesting shutdown +30:13.768 [INFO] TestAgent.server: shutting down server +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:13.768 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:13.768 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:13.768 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:13.768 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:13.781 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:13.788Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:13.788Z [INFO] agent.router.manager: shutting down +30:13.788 [INFO] TestAgent: consul server down +30:13.788 [INFO] TestAgent: shutdown complete +30:13.788 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp +30:13.788 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp +30:13.788 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp +30:14.290 [INFO] TestAgent: Waiting for endpoints to shut down +30:14.290 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleCreateCommand_JSON (1.16s) + --- PASS: TestBindingRuleCreateCommand_JSON/create_it_with_no_selector (0.03s) +2021-10-05T16:30:13.225Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:13.258Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:13.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08c9f4e9-8b43-143e-a72d-5bfa3b682687 Address:127.0.0.1:10013}]" +30:13.321 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 127.0.0.1 +30:13.322 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 127.0.0.1 +2021-10-05T16:30:13.322Z [INFO] agent.router: Initializing LAN area manager +30:13.322 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp +30:13.322 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= +30:13.322 [INFO] TestAgent.server: Adding LAN server: server="Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +30:13.322 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 area=wan +30:13.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp +30:13.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp +30:13.323 [INFO] TestAgent: started state syncer +30:13.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp +30:13.358 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:13.358 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 +30:13.367 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:13.367 [DEBUG] TestAgent.server.raft: vote granted: from=08c9f4e9-8b43-143e-a72d-5bfa3b682687 term=2 tally=1 +30:13.367 [INFO] TestAgent.server.raft: election won: tally=1 +30:13.367 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" +30:13.367 [INFO] TestAgent.server: cluster leadership acquired +30:13.368 [INFO] TestAgent.server: New leader elected: payload=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.373 [INFO] TestAgent.server: initializing acls +30:13.376 [INFO] TestAgent.server: initializing acls +30:13.389 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:13.389 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:13.389 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:13.389 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:13.416 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:13.430 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:13.430 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:13.430 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:13.431 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.431 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 +30:13.432 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:13.432 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:13.432 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.432 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 +30:13.432 [INFO] TestAgent.server: Updating LAN server: server="Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +30:13.432 [INFO] TestAgent.server: Updating LAN server: server="Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +30:13.432 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 area=wan +30:13.432 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687.dc1 area=wan +30:13.452 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:13.493 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:13.493 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:13.493 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:13.493 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:13.493 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:13.493 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.493 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.510 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.510 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08c9f4e9-8b43-143e-a72d-5bfa3b682687 +30:13.510 [INFO] TestAgent.server: federation state anti-entropy synced +30:13.656 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:13.666 [INFO] TestAgent: Synced node info +30:13.666 [DEBUG] TestAgent: Node info in sync +30:13.684 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:47854 latency=8.150391ms +30:13.700 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47856 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" +30:13.700 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47856 latency=1.098417ms +30:13.708 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47858 latency=6.118298ms +30:13.737 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47860 latency=24.029723ms +30:13.776 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47866 latency=34.483055ms +30:13.781 [INFO] TestAgent: Requesting shutdown +30:13.781 [INFO] TestAgent.server: shutting down server +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:13.781 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:13.781 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:13.782 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:13.782 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:13.789 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:13.794Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:13.794Z [INFO] agent.router.manager: shutting down +30:13.794 [INFO] TestAgent: consul server down +30:13.794 [INFO] TestAgent: shutdown complete +30:13.794 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp +30:13.794 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp +30:13.794 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp +30:14.298 [INFO] TestAgent: Waiting for endpoints to shut down +30:14.298 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleCreateCommand (1.17s) --- PASS: TestBindingRuleCreateCommand/method_is_required (0.00s) --- PASS: TestBindingRuleCreateCommand/bind_type_required (0.00s) --- PASS: TestBindingRuleCreateCommand/bind_name_required (0.00s) - --- PASS: TestBindingRuleCreateCommand/must_use_roughly_valid_selector (0.00s) - --- PASS: TestBindingRuleCreateCommand/create_it_with_no_selector (0.00s) - --- PASS: TestBindingRuleCreateCommand/create_it_with_a_match_selector (0.00s) - --- PASS: TestBindingRuleCreateCommand/create_it_with_type_role (0.01s) -2022-11-07T22:44:54.606Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:54.617Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:54.645 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 Address:127.0.0.1:18174}]" -44:54.645 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -44:54.647 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6.dc1 127.0.0.1 -44:54.649 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 127.0.0.1 -2022-11-07T22:44:54.649Z [INFO] agent.router: Initializing LAN area manager -44:54.649 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6.dc1 area=wan -44:54.649 [INFO] TestAgent.server: Adding LAN server: server="Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:54.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -44:54.649 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -44:54.650 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -44:54.650 [INFO] TestAgent: started state syncer -44:54.650 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -44:54.699 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:54.711 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:54.711 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -44:54.724 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:54.724 [DEBUG] TestAgent.server.raft: vote granted: from=02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 term=2 tally=1 -44:54.724 [INFO] TestAgent.server.raft: election won: tally=1 -44:54.724 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -44:54.724 [INFO] TestAgent.server: cluster leadership acquired -44:54.724 [INFO] TestAgent.server: New leader elected: payload=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 -44:54.734 [INFO] TestAgent.server: initializing acls -44:54.742 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:54.742 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:54.750 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:54.765 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:54.765 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:54.765 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:54.765 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 -44:54.765 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6.dc1 -44:54.765 [INFO] TestAgent.server: Updating LAN server: server="Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -44:54.766 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6.dc1 area=wan -44:54.775 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:54.795 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:54.795 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:54.795 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:54.795 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:54.795 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:54.795 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 -44:54.795 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 -44:54.801 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02ac0ce2-6cc3-babc-c915-5e0d7b5dc2b6 -44:54.806 [INFO] TestAgent.server: federation state anti-entropy synced -44:54.907 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:54.912 [INFO] TestAgent: Synced node info -44:55.125 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46864 latency=4.021863ms -44:55.129 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46866 latency=3.156133ms -44:55.130 [INFO] TestAgent: Requesting shutdown -44:55.130 [INFO] TestAgent.server: shutting down server -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:55.130 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:55.130 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:55.130 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:55.130 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:55.137 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:55.141Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:55.141Z [INFO] agent.router.manager: shutting down -44:55.141 [INFO] TestAgent: consul server down -44:55.141 [INFO] TestAgent: shutdown complete -44:55.141 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -44:55.141 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -44:55.141 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -44:55.641 [INFO] TestAgent: Waiting for endpoints to shut down -44:55.641 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleCreateCommand_JSON (1.07s) - --- PASS: TestBindingRuleCreateCommand_JSON/create_it_with_no_selector (0.00s) + --- PASS: TestBindingRuleCreateCommand/must_use_roughly_valid_selector (0.01s) + --- PASS: TestBindingRuleCreateCommand/create_it_with_no_selector (0.01s) + --- PASS: TestBindingRuleCreateCommand/create_it_with_a_match_selector (0.03s) + --- PASS: TestBindingRuleCreateCommand/create_it_with_type_role (0.04s) PASS -ok github.com/hashicorp/consul/command/acl/bindingrule/create 1.105s +ok github.com/hashicorp/consul/command/acl/bindingrule/create 1.320s === RUN TestBindingRuleDeleteCommand_noTabs === PAUSE TestBindingRuleDeleteCommand_noTabs === RUN TestBindingRuleDeleteCommand === PAUSE TestBindingRuleDeleteCommand === CONT TestBindingRuleDeleteCommand_noTabs === CONT TestBindingRuleDeleteCommand -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... --- PASS: TestBindingRuleDeleteCommand_noTabs (0.00s) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestBindingRuleDeleteCommand/id_required === RUN TestBindingRuleDeleteCommand/delete_works === RUN TestBindingRuleDeleteCommand/delete_works_via_prefixes === RUN TestBindingRuleDeleteCommand/delete_fails_when_prefix_matches_more_than_one_rule -2022-11-07T22:44:56.592Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:56.607Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:56.632 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ebc53d64-4551-e827-c4b3-23e1cd8faa66 Address:127.0.0.1:21237}]" -44:56.632 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= -44:56.632 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66.dc1 127.0.0.1 -44:56.633 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 127.0.0.1 -2022-11-07T22:44:56.633Z [INFO] agent.router: Initializing LAN area manager -44:56.633 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66.dc1 area=wan -44:56.633 [INFO] TestAgent.server: Adding LAN server: server="Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -44:56.633 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp -44:56.633 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp -44:56.633 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp -44:56.633 [INFO] TestAgent: started state syncer -44:56.633 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp -44:56.676 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:56.676 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 -44:56.683 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:56.686 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:56.686 [DEBUG] TestAgent.server.raft: vote granted: from=ebc53d64-4551-e827-c4b3-23e1cd8faa66 term=2 tally=1 -44:56.686 [INFO] TestAgent.server.raft: election won: tally=1 -44:56.686 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" -44:56.686 [INFO] TestAgent.server: cluster leadership acquired -44:56.686 [INFO] TestAgent.server: New leader elected: payload=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 -44:56.693 [INFO] TestAgent.server: initializing acls -44:56.696 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:56.696 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:56.702 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:56.710 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:56.710 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:56.710 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:56.710 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 -44:56.710 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66.dc1 -44:56.712 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66.dc1 area=wan -44:56.712 [INFO] TestAgent.server: Updating LAN server: server="Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -44:56.721 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:56.739 [INFO] TestAgent: Synced node info -44:56.739 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:56.739 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:56.739 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:56.739 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:56.739 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:56.739 [DEBUG] TestAgent: Node info in sync -44:56.739 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 -44:56.739 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 -44:56.747 [INFO] TestAgent.server: federation state anti-entropy synced -44:56.748 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ebc53d64-4551-e827-c4b3-23e1cd8faa66 -44:57.084 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:39872 latency=42.821247ms -44:57.089 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=4.506847ms -44:57.095 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/45a09069-4fa2-edf5-76fa-ab06e1fe01b6 from=127.0.0.1:39874 latency=4.967742ms -44:57.096 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/45a09069-4fa2-edf5-76fa-ab06e1fe01b6 from=127.0.0.1:39872 latency=66.02µs -44:57.099 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.138895ms -44:57.100 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:39876 latency=181.235µs -44:57.104 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/5ef473e2-d407-99b1-ed75-457fd1706ae8 from=127.0.0.1:39876 latency=2.913382ms -44:57.104 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5ef473e2-d407-99b1-ed75-457fd1706ae8 from=127.0.0.1:39872 latency=45.881µs -44:57.104 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=171.725µs -44:57.109 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=4.626681ms -44:57.110 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=141.387µs -44:57.113 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.245209ms -44:57.114 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=121.967µs -44:57.118 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.230435ms -44:57.118 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=125.304µs -44:57.121 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.043351ms -44:57.122 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=121.844µs -44:57.126 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.193217ms -44:57.126 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=126.737µs -44:57.130 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.064989ms -44:57.135 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=505.737µs -44:57.141 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=5.706272ms -44:57.142 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=174.465µs -44:57.145 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.251601ms -44:57.146 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=399.937µs -44:57.150 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:39872 latency=3.138313ms -44:57.150 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:39872 latency=124.354µs -44:57.151 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:39878 latency=169.272µs -44:57.152 [INFO] TestAgent: Requesting shutdown -44:57.152 [INFO] TestAgent.server: shutting down server -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:57.152 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:57.152 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:57.152 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:57.152 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:57.155 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:57.159Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:57.159Z [INFO] agent.router.manager: shutting down -44:57.159 [INFO] TestAgent: consul server down -44:57.159 [INFO] TestAgent: shutdown complete -44:57.159 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp -44:57.159 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp -44:57.159 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp -44:57.659 [INFO] TestAgent: Waiting for endpoints to shut down -44:57.659 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleDeleteCommand (1.10s) +2021-10-05T16:30:13.659Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:13.676Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:13.715 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cf0e2de1-0d15-181c-2f02-6565348c3593 Address:127.0.0.1:21237}]" +30:13.715 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= +30:13.716 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cf0e2de1-0d15-181c-2f02-6565348c3593.dc1 127.0.0.1 +30:13.717 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cf0e2de1-0d15-181c-2f02-6565348c3593 127.0.0.1 +2021-10-05T16:30:13.717Z [INFO] agent.router: Initializing LAN area manager +30:13.717 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp +30:13.718 [INFO] TestAgent.server: Adding LAN server: server="Node-cf0e2de1-0d15-181c-2f02-6565348c3593 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +30:13.718 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cf0e2de1-0d15-181c-2f02-6565348c3593.dc1 area=wan +30:13.718 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp +30:13.718 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp +30:13.718 [INFO] TestAgent: started state syncer +30:13.718 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp +30:13.768 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:13.781 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:13.781 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 +30:13.809 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:13.809 [DEBUG] TestAgent.server.raft: vote granted: from=cf0e2de1-0d15-181c-2f02-6565348c3593 term=2 tally=1 +30:13.809 [INFO] TestAgent.server.raft: election won: tally=1 +30:13.809 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" +30:13.809 [INFO] TestAgent.server: cluster leadership acquired +30:13.810 [INFO] TestAgent.server: New leader elected: payload=Node-cf0e2de1-0d15-181c-2f02-6565348c3593 +30:13.821 [INFO] TestAgent.server: initializing acls +30:13.826 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:13.826 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:13.833 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:13.841 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:13.841 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:13.841 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:13.841 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cf0e2de1-0d15-181c-2f02-6565348c3593 +30:13.841 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cf0e2de1-0d15-181c-2f02-6565348c3593.dc1 +30:13.842 [INFO] TestAgent.server: Updating LAN server: server="Node-cf0e2de1-0d15-181c-2f02-6565348c3593 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +30:13.842 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cf0e2de1-0d15-181c-2f02-6565348c3593.dc1 area=wan +30:13.850 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:13.872 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:13.872 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:13.872 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:13.872 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:13.872 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:13.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf0e2de1-0d15-181c-2f02-6565348c3593 +30:13.872 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cf0e2de1-0d15-181c-2f02-6565348c3593 +30:13.886 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cf0e2de1-0d15-181c-2f02-6565348c3593 +30:13.904 [INFO] TestAgent.server: federation state anti-entropy synced +30:13.924 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:33480 latency=15.563058ms +30:13.929 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=4.20662ms +30:13.993 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/a6e578ce-3f71-b807-6b78-0151faa74335 from=127.0.0.1:33482 latency=16.873813ms +30:13.994 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/a6e578ce-3f71-b807-6b78-0151faa74335 from=127.0.0.1:33480 latency=118.016µs +30:14.001 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=6.780217ms +30:14.006 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:33484 latency=374.855µs +30:14.020 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/d0916522-c88f-6317-7ac9-193d7358719a from=127.0.0.1:33484 latency=13.287242ms +30:14.024 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/d0916522-c88f-6317-7ac9-193d7358719a from=127.0.0.1:33480 latency=100.757µs +30:14.025 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:33480 latency=208.798µs +30:14.044 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=18.996695ms +30:14.052 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:33480 latency=286.535µs +30:14.076 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=20.041429ms +30:14.078 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:33480 latency=304.187µs +30:14.084 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=5.465596ms +30:14.085 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:33480 latency=259.604µs +30:14.089 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:33480 latency=4.033705ms +30:14.090 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:33480 latency=283.307µs +30:14.091 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:33486 latency=225.22µs +30:14.092 [INFO] TestAgent: Requesting shutdown +30:14.092 [INFO] TestAgent.server: shutting down server +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:14.092 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:14.092 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:14.092 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:14.092 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:14.092 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:14.097 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:14.112Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:14.112Z [INFO] agent.router.manager: shutting down +30:14.112 [INFO] TestAgent: consul server down +30:14.112 [INFO] TestAgent: shutdown complete +30:14.112 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp +30:14.112 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp +30:14.112 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp +30:14.616 [INFO] TestAgent: Waiting for endpoints to shut down +30:14.616 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleDeleteCommand (1.05s) --- PASS: TestBindingRuleDeleteCommand/id_required (0.00s) - --- PASS: TestBindingRuleDeleteCommand/delete_works (0.01s) - --- PASS: TestBindingRuleDeleteCommand/delete_works_via_prefixes (0.01s) - --- PASS: TestBindingRuleDeleteCommand/delete_fails_when_prefix_matches_more_than_one_rule (0.05s) + --- PASS: TestBindingRuleDeleteCommand/delete_works (0.07s) + --- PASS: TestBindingRuleDeleteCommand/delete_works_via_prefixes (0.03s) + --- PASS: TestBindingRuleDeleteCommand/delete_fails_when_prefix_matches_more_than_one_rule (0.07s) PASS -ok github.com/hashicorp/consul/command/acl/bindingrule/delete 1.129s +ok github.com/hashicorp/consul/command/acl/bindingrule/delete 1.185s === RUN TestBindingRuleListCommand_noTabs === PAUSE TestBindingRuleListCommand_noTabs === RUN TestBindingRuleListCommand === PAUSE TestBindingRuleListCommand === CONT TestBindingRuleListCommand_noTabs +--- PASS: TestBindingRuleListCommand_noTabs (0.00s) === CONT TestBindingRuleListCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestBindingRuleListCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestBindingRuleListCommand/normal === RUN TestBindingRuleListCommand/filter_by_method_1 === RUN TestBindingRuleListCommand/filter_by_method_2 === RUN TestBindingRuleListCommand/normal_json_formatted -2022-11-07T22:44:58.872Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:58.882Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:58.903 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f3ec19f9-a099-70cc-6929-ec67097a0c82 Address:127.0.0.1:22258}]" -44:58.903 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= -44:58.903 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 127.0.0.1 -44:58.904 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 127.0.0.1 -2022-11-07T22:44:58.904Z [INFO] agent.router: Initializing LAN area manager -44:58.904 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp -44:58.904 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp -44:58.904 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 area=wan -44:58.904 [INFO] TestAgent.server: Adding LAN server: server="Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" -44:58.905 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp -44:58.905 [INFO] TestAgent: started state syncer -44:58.906 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp -44:58.939 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:58.939 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 -44:58.947 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:58.947 [DEBUG] TestAgent.server.raft: vote granted: from=f3ec19f9-a099-70cc-6929-ec67097a0c82 term=2 tally=1 -44:58.947 [INFO] TestAgent.server.raft: election won: tally=1 -44:58.947 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" -44:58.948 [INFO] TestAgent.server: cluster leadership acquired -44:58.948 [INFO] TestAgent.server: New leader elected: payload=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:58.953 [INFO] TestAgent.server: initializing acls -44:58.955 [INFO] TestAgent.server: initializing acls -44:58.961 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:58.961 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:58.961 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:58.961 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:58.969 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:58.974 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:58.974 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:58.974 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:58.974 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:58.974 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -44:58.974 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:58.974 [INFO] TestAgent.server: Updating LAN server: server="Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" -44:58.974 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 -44:58.974 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 area=wan -44:58.974 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:58.974 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 -44:58.975 [INFO] TestAgent.server: Updating LAN server: server="Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" -44:58.975 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82.dc1 area=wan -44:58.983 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:58.998 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:58.998 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:58.998 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:58.998 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:58.998 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:58.998 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:58.998 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:59.002 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:59.002 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f3ec19f9-a099-70cc-6929-ec67097a0c82 -44:59.033 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37752 latency=5.593263ms -44:59.037 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37752 latency=3.442214ms -44:59.043 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.241041ms -44:59.047 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=2.937573ms -44:59.050 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -44:59.051 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.168249ms -44:59.058 [INFO] TestAgent: Synced node info -44:59.058 [DEBUG] TestAgent: Node info in sync -44:59.059 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=7.778793ms -44:59.064 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=4.705777ms -44:59.069 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.241332ms -44:59.073 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.127083ms -44:59.080 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=6.610673ms -44:59.084 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.182333ms -44:59.088 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37752 latency=3.912085ms -44:59.089 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:37754 latency=209.776µs -44:59.090 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test-1 from=127.0.0.1:37756 latency=178.869µs -44:59.091 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test-2 from=127.0.0.1:37758 latency=224.562µs -44:59.093 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:37760 latency=213.235µs -44:59.093 [INFO] TestAgent: Requesting shutdown -44:59.093 [INFO] TestAgent.server: shutting down server -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:59.093 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:59.093 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:59.093 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:59.094 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:59.099 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:59.104Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:59.104Z [INFO] agent.router.manager: shutting down -44:59.104 [INFO] TestAgent: consul server down -44:59.104 [INFO] TestAgent: shutdown complete -44:59.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp -44:59.104 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp -44:59.104 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp -44:59.605 [INFO] TestAgent: Waiting for endpoints to shut down -44:59.605 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleListCommand (0.77s) - --- PASS: TestBindingRuleListCommand/normal (0.00s) - --- PASS: TestBindingRuleListCommand/filter_by_method_1 (0.00s) +2021-10-05T16:30:21.395Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:21.419Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:21.477 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 Address:127.0.0.1:15111}]" +30:21.478 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 127.0.0.1 +30:21.479 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 127.0.0.1 +2021-10-05T16:30:21.479Z [INFO] agent.router: Initializing LAN area manager +30:21.479 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +30:21.479 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +30:21.479 [INFO] TestAgent.server: Adding LAN server: server="Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +30:21.479 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 area=wan +30:21.479 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +30:21.480 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +30:21.480 [INFO] TestAgent: started state syncer +30:21.480 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +30:21.532 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:21.532 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +30:21.532 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:21.580 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:21.580 [DEBUG] TestAgent.server.raft: vote granted: from=d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 term=2 tally=1 +30:21.580 [INFO] TestAgent.server.raft: election won: tally=1 +30:21.580 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +30:21.581 [INFO] TestAgent.server: cluster leadership acquired +30:21.581 [INFO] TestAgent.server: New leader elected: payload=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.624 [INFO] TestAgent.server: initializing acls +30:21.633 [INFO] TestAgent.server: initializing acls +30:21.700 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:21.708 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:21.708 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:21.712 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:21.712 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:21.756 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:21.760 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:21.786 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:21.786 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:21.786 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:21.786 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:21.786 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.786 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 +30:21.786 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 area=wan +30:21.786 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:21.786 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.786 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 +30:21.788 [INFO] TestAgent.server: Updating LAN server: server="Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +30:21.788 [INFO] TestAgent.server: Updating LAN server: server="Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +30:21.788 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5.dc1 area=wan +30:21.815 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:21.854 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:21.854 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:21.854 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:21.854 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:21.854 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:21.854 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.854 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.861 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.861 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6cff4f9-f94b-87dc-cd78-486bf0ea3bd5 +30:21.866 [INFO] TestAgent.server: federation state anti-entropy synced +30:21.884 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:54478 latency=7.390046ms +30:21.889 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:54478 latency=4.015192ms +30:21.893 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=3.421085ms +30:21.897 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=3.461996ms +30:21.903 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=3.241398ms +30:21.912 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=3.425165ms +30:21.936 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=20.035926ms +30:21.957 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=16.593041ms +30:21.980 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=12.627269ms +30:22.002 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=14.153184ms +30:22.010 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=7.030642ms +30:22.028 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:54478 latency=15.993427ms +30:22.033 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:54480 latency=433.99µs +30:22.044 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test-1 from=127.0.0.1:54482 latency=380.402µs +30:22.047 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test-2 from=127.0.0.1:54484 latency=281.58µs +30:22.052 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:54486 latency=342.575µs +30:22.056 [INFO] TestAgent: Requesting shutdown +30:22.057 [INFO] TestAgent.server: shutting down server +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:22.057 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:22.057 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:22.057 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:22.057 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:22.068 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:22.069Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:22.069Z [INFO] agent.router.manager: shutting down +30:22.070 [INFO] TestAgent: consul server down +30:22.070 [INFO] TestAgent: shutdown complete +30:22.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +30:22.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +30:22.070 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +30:22.572 [INFO] TestAgent: Waiting for endpoints to shut down +30:22.572 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleListCommand (1.29s) + --- PASS: TestBindingRuleListCommand/normal (0.01s) + --- PASS: TestBindingRuleListCommand/filter_by_method_1 (0.01s) --- PASS: TestBindingRuleListCommand/filter_by_method_2 (0.00s) - --- PASS: TestBindingRuleListCommand/normal_json_formatted (0.00s) + --- PASS: TestBindingRuleListCommand/normal_json_formatted (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/bindingrule/list 0.796s +ok github.com/hashicorp/consul/command/acl/bindingrule/list 1.496s === RUN TestBindingRuleReadCommand_noTabs === PAUSE TestBindingRuleReadCommand_noTabs === RUN TestBindingRuleReadCommand === PAUSE TestBindingRuleReadCommand === CONT TestBindingRuleReadCommand_noTabs +--- PASS: TestBindingRuleReadCommand_noTabs (0.00s) === CONT TestBindingRuleReadCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestBindingRuleReadCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestBindingRuleReadCommand/id_required @@ -74353,94 +74563,94 @@ === RUN TestBindingRuleReadCommand/read_by_id === RUN TestBindingRuleReadCommand/read_by_id_prefix === RUN TestBindingRuleReadCommand/read_by_id_json_formatted -2022-11-07T22:44:59.638Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:44:59.649Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -44:59.670 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eec91163-19d4-f0e3-8601-155e68c45113 Address:127.0.0.1:20216}]" -44:59.671 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= -44:59.671 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eec91163-19d4-f0e3-8601-155e68c45113.dc1 127.0.0.1 -44:59.671 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eec91163-19d4-f0e3-8601-155e68c45113 127.0.0.1 -2022-11-07T22:44:59.671Z [INFO] agent.router: Initializing LAN area manager -44:59.672 [INFO] TestAgent.server: Adding LAN server: server="Node-eec91163-19d4-f0e3-8601-155e68c45113 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -44:59.672 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eec91163-19d4-f0e3-8601-155e68c45113.dc1 area=wan -44:59.672 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp -44:59.672 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp -44:59.672 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp -44:59.673 [INFO] TestAgent: started state syncer -44:59.673 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp -44:59.722 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -44:59.738 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -44:59.738 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 -44:59.746 [DEBUG] TestAgent.server.raft: votes: needed=1 -44:59.746 [DEBUG] TestAgent.server.raft: vote granted: from=eec91163-19d4-f0e3-8601-155e68c45113 term=2 tally=1 -44:59.746 [INFO] TestAgent.server.raft: election won: tally=1 -44:59.746 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" -44:59.746 [INFO] TestAgent.server: cluster leadership acquired -44:59.748 [INFO] TestAgent.server: New leader elected: payload=Node-eec91163-19d4-f0e3-8601-155e68c45113 -44:59.753 [INFO] TestAgent.server: initializing acls -44:59.756 [INFO] TestAgent.server: Created ACL 'global-management' policy -44:59.756 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -44:59.762 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -44:59.769 [INFO] TestAgent.server: Created ACL anonymous token from configuration -44:59.769 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -44:59.769 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -44:59.770 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-eec91163-19d4-f0e3-8601-155e68c45113 -44:59.770 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-eec91163-19d4-f0e3-8601-155e68c45113.dc1 -44:59.770 [INFO] TestAgent.server: Updating LAN server: server="Node-eec91163-19d4-f0e3-8601-155e68c45113 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -44:59.770 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-eec91163-19d4-f0e3-8601-155e68c45113.dc1 area=wan -44:59.778 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -44:59.795 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -44:59.795 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -44:59.795 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -44:59.795 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -44:59.795 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -44:59.795 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eec91163-19d4-f0e3-8601-155e68c45113 -44:59.795 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eec91163-19d4-f0e3-8601-155e68c45113 -44:59.802 [INFO] TestAgent.server: federation state anti-entropy synced -44:59.803 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eec91163-19d4-f0e3-8601-155e68c45113 -44:59.938 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43010 latency=6.428375ms -44:59.940 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/c4cfc688-53da-5be2-6722-ad93f6bd6f1f from=127.0.0.1:43012 latency=138.207µs -44:59.944 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:43010 latency=3.304079ms -44:59.948 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/1f916abb-085e-2caa-0443-c2a160fc5263 from=127.0.0.1:43014 latency=817.083µs -44:59.952 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:43010 latency=3.30934ms -44:59.953 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:43016 latency=251.948µs -44:59.954 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/d456685e-31d8-d4b9-7813-915093659b51 from=127.0.0.1:43016 latency=857.522µs -44:59.961 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:43010 latency=4.182582ms -44:59.963 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e449aea9-8c29-2c42-2fc3-8a04b9c7702f from=127.0.0.1:43018 latency=346.81µs -44:59.963 [INFO] TestAgent: Requesting shutdown -44:59.963 [INFO] TestAgent.server: shutting down server -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -44:59.963 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -44:59.963 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -44:59.963 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -44:59.963 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -44:59.963 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -44:59.967 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:44:59.970Z [INFO] agent.router.manager: shutting down -2022-11-07T22:44:59.970Z [INFO] agent.router.manager: shutting down -44:59.970 [INFO] TestAgent: consul server down -44:59.970 [INFO] TestAgent: shutdown complete -44:59.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp -44:59.970 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp -44:59.970 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp -45:00.470 [INFO] TestAgent: Waiting for endpoints to shut down -45:00.470 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleReadCommand (0.87s) +2021-10-05T16:30:23.886Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:23.907Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:23.938 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d Address:127.0.0.1:12048}]" +30:23.939 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d.dc1 127.0.0.1 +30:23.940 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= +30:23.940 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d 127.0.0.1 +2021-10-05T16:30:23.940Z [INFO] agent.router: Initializing LAN area manager +30:23.948 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d.dc1 area=wan +30:23.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp +30:23.954 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp +30:23.955 [INFO] TestAgent.server: Adding LAN server: server="Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +30:23.968 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp +30:23.968 [INFO] TestAgent: started state syncer +30:23.968 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp +30:24.003 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:24.003 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 +30:24.004 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:24.033 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:24.033 [DEBUG] TestAgent.server.raft: vote granted: from=4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d term=2 tally=1 +30:24.033 [INFO] TestAgent.server.raft: election won: tally=1 +30:24.033 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" +30:24.033 [INFO] TestAgent.server: cluster leadership acquired +30:24.034 [INFO] TestAgent.server: New leader elected: payload=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d +30:24.042 [INFO] TestAgent.server: initializing acls +30:24.046 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:24.046 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:24.056 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:24.061 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:24.088 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:24.088 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:24.088 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:24.088 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d +30:24.088 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d.dc1 +30:24.089 [INFO] TestAgent.server: Updating LAN server: server="Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +30:24.089 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d.dc1 area=wan +30:24.118 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:24.176 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:24.176 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:24.176 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:24.176 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:24.176 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:24.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d +30:24.176 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d +30:24.191 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cb6c8a9-0313-5bf2-13a6-dc177b20ab8d +30:24.200 [INFO] TestAgent.server: federation state anti-entropy synced +30:24.297 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:56730 latency=7.976275ms +30:24.302 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/f1e38026-9913-633b-69fe-64939c8490b5 from=127.0.0.1:56732 latency=197.726µs +30:24.324 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:56730 latency=16.76499ms +30:24.337 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/56aba3bb-f296-59ce-6c0e-1e731ced4123 from=127.0.0.1:56734 latency=351.085µs +30:24.364 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:56730 latency=20.149253ms +30:24.370 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:56736 latency=1.216696ms +30:24.371 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/26d3310f-d6b6-17b4-4064-2fa64954231e from=127.0.0.1:56736 latency=365.225µs +30:24.394 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:56730 latency=13.839467ms +30:24.396 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/8038af23-46b9-f8b9-e20c-2a683cef8894 from=127.0.0.1:56738 latency=331.768µs +30:24.400 [INFO] TestAgent: Requesting shutdown +30:24.400 [INFO] TestAgent.server: shutting down server +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:24.400 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:24.400 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:24.400 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:24.400 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:24.404 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:24.412Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:24.412Z [INFO] agent.router.manager: shutting down +30:24.412 [INFO] TestAgent: consul server down +30:24.412 [INFO] TestAgent: shutdown complete +30:24.412 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp +30:24.412 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp +30:24.412 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp +30:24.913 [INFO] TestAgent: Waiting for endpoints to shut down +30:24.913 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleReadCommand (1.16s) --- PASS: TestBindingRuleReadCommand/id_required (0.00s) --- PASS: TestBindingRuleReadCommand/read_by_id_not_found (0.00s) - --- PASS: TestBindingRuleReadCommand/read_by_id (0.01s) - --- PASS: TestBindingRuleReadCommand/read_by_id_prefix (0.01s) - --- PASS: TestBindingRuleReadCommand/read_by_id_json_formatted (0.01s) + --- PASS: TestBindingRuleReadCommand/read_by_id (0.04s) + --- PASS: TestBindingRuleReadCommand/read_by_id_prefix (0.04s) + --- PASS: TestBindingRuleReadCommand/read_by_id_json_formatted (0.02s) PASS -ok github.com/hashicorp/consul/command/acl/bindingrule/read 0.896s +ok github.com/hashicorp/consul/command/acl/bindingrule/read 1.402s === RUN TestBindingRuleUpdateCommand_noTabs === PAUSE TestBindingRuleUpdateCommand_noTabs === RUN TestBindingRuleUpdateCommand @@ -74448,313 +74658,317 @@ === RUN TestBindingRuleUpdateCommand_noMerge === PAUSE TestBindingRuleUpdateCommand_noMerge === CONT TestBindingRuleUpdateCommand_noTabs +--- PASS: TestBindingRuleUpdateCommand_noTabs (0.00s) === CONT TestBindingRuleUpdateCommand_noMerge -=== CONT TestBindingRuleUpdateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestBindingRuleUpdateCommand_noTabs (0.00s) +=== CONT TestBindingRuleUpdateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestBindingRuleUpdateCommand_noMerge/rule_id_required === RUN TestBindingRuleUpdateCommand_noMerge/rule_id_partial_matches_nothing === RUN TestBindingRuleUpdateCommand_noMerge/rule_id_exact_match_doesn't_exist === RUN TestBindingRuleUpdateCommand_noMerge/rule_id_partial_matches_multiple -=== RUN TestBindingRuleUpdateCommand_noMerge/must_use_roughly_valid_selector -=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields -=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_-_partial === RUN TestBindingRuleUpdateCommand/rule_id_required === RUN TestBindingRuleUpdateCommand/rule_id_partial_matches_nothing === RUN TestBindingRuleUpdateCommand/rule_id_exact_match_doesn't_exist === RUN TestBindingRuleUpdateCommand/rule_id_partial_matches_multiple -=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_description === RUN TestBindingRuleUpdateCommand/must_use_roughly_valid_selector -=== RUN TestBindingRuleUpdateCommand_noMerge/missing_bind_name -=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_selector +=== RUN TestBindingRuleUpdateCommand_noMerge/must_use_roughly_valid_selector === RUN TestBindingRuleUpdateCommand/update_all_fields +=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields +=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_-_partial === RUN TestBindingRuleUpdateCommand/update_all_fields_-_partial === RUN TestBindingRuleUpdateCommand/update_all_fields_but_description === RUN TestBindingRuleUpdateCommand/update_all_fields_but_bind_name +=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_description === RUN TestBindingRuleUpdateCommand/update_all_fields_but_must_exist === RUN TestBindingRuleUpdateCommand/update_all_fields_but_selector +=== RUN TestBindingRuleUpdateCommand_noMerge/missing_bind_name +=== RUN TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_selector === RUN TestBindingRuleUpdateCommand/update_all_fields_clear_selector === RUN TestBindingRuleUpdateCommand/update_all_fields_json_formatted -2022-11-07T22:45:01.734Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.746Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.776 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b8b6028-62ac-a983-4cde-29cac27f3345 Address:127.0.0.1:24300}]" -45:01.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24300 [Follower]" leader= -45:01.777 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b8b6028-62ac-a983-4cde-29cac27f3345.dc1 127.0.0.1 -45:01.778 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b8b6028-62ac-a983-4cde-29cac27f3345 127.0.0.1 -2022-11-07T22:45:01.778Z [INFO] agent.router: Initializing LAN area manager -45:01.778 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b8b6028-62ac-a983-4cde-29cac27f3345.dc1 area=wan -45:01.778 [INFO] TestAgent.server: Adding LAN server: server="Node-4b8b6028-62ac-a983-4cde-29cac27f3345 (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:01.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=tcp -45:01.778 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=udp -45:01.779 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24296 network=tcp -45:01.779 [INFO] TestAgent: started state syncer -45:01.779 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24301 network=tcp -45:01.819 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.819 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24300 [Candidate]" term=2 -45:01.828 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:01.839 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.839 [DEBUG] TestAgent.server.raft: vote granted: from=4b8b6028-62ac-a983-4cde-29cac27f3345 term=2 tally=1 -45:01.839 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.839 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24300 [Leader]" -45:01.839 [INFO] TestAgent.server: cluster leadership acquired -45:01.839 [INFO] TestAgent.server: New leader elected: payload=Node-4b8b6028-62ac-a983-4cde-29cac27f3345 -45:01.847 [INFO] TestAgent.server: initializing acls -45:01.850 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:01.850 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:01.856 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:01.867 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:01.867 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:01.867 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:01.867 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4b8b6028-62ac-a983-4cde-29cac27f3345 -45:01.867 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4b8b6028-62ac-a983-4cde-29cac27f3345.dc1 -45:01.867 [INFO] TestAgent.server: Updating LAN server: server="Node-4b8b6028-62ac-a983-4cde-29cac27f3345 (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:01.867 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4b8b6028-62ac-a983-4cde-29cac27f3345.dc1 area=wan -45:01.877 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.893 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.893 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.893 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.893 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.893 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.893 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b8b6028-62ac-a983-4cde-29cac27f3345 -45:01.893 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b8b6028-62ac-a983-4cde-29cac27f3345 -45:01.898 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b8b6028-62ac-a983-4cde-29cac27f3345 -45:01.965 [INFO] TestAgent.server: federation state anti-entropy synced -45:02.100 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46338 latency=6.873532ms -45:02.103 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:46340 latency=326.194µs -45:02.104 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/6a6ec2d6-3df2-31d3-4e76-4c0a5d1dec0d from=127.0.0.1:46342 latency=110.926µs -45:02.105 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=760.06µs -45:02.109 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.417271ms -45:02.109 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=126.001µs -45:02.113 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.269728ms -45:02.114 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=139.6µs -45:02.117 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.125332ms -45:02.118 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=193.685µs -45:02.121 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.287359ms -45:02.122 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=177.878µs -45:02.126 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.493991ms -45:02.126 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=201.993µs -45:02.127 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:46344 latency=176.089µs -45:02.131 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=2.993702ms -45:02.132 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/cb456fe9-53aa-51d1-fc95-44686530c700 from=127.0.0.1:46346 latency=130.644µs -45:02.132 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/cb456fe9-53aa-51d1-fc95-44686530c700 from=127.0.0.1:46346 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" -45:02.134 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/cb456fe9-53aa-51d1-fc95-44686530c700 from=127.0.0.1:46346 latency=1.80139ms -45:02.140 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46338 latency=3.906554ms -45:02.141 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:02.141 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/731eebe8-6168-f37f-0d5e-17c771f4b07b from=127.0.0.1:46348 latency=149.504µs -45:02.147 [INFO] TestAgent: Synced node info -45:02.147 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/731eebe8-6168-f37f-0d5e-17c771f4b07b from=127.0.0.1:46348 latency=5.664247ms -45:02.148 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/731eebe8-6168-f37f-0d5e-17c771f4b07b from=127.0.0.1:46338 latency=243.6µs -45:02.149 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:46338 latency=519.78µs -45:02.152 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/542c7323-3a30-bbb4-20bd-dd5b4dae39e1 from=127.0.0.1:46338 latency=3.024898ms -45:02.156 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/731eebe8-6168-f37f-0d5e-17c771f4b07b from=127.0.0.1:46350 latency=3.048679ms -45:02.159 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/7a891905-889b-de05-e397-8094aab8f8ca from=127.0.0.1:46352 latency=2.988865ms -45:02.163 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/cb456fe9-53aa-51d1-fc95-44686530c700 from=127.0.0.1:46354 latency=3.037101ms -45:02.166 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/d0a9fddc-7b6b-3e34-5ed6-934476cf3ee7 from=127.0.0.1:46356 latency=2.93349ms -45:02.170 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/d470c22b-f536-ce5c-4e80-1ae3d51327b9 from=127.0.0.1:46360 latency=3.552946ms -45:02.173 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/eb8ececf-026e-7f5f-73c5-63d5cb90870f from=127.0.0.1:46362 latency=2.676633ms -45:02.177 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46368 latency=3.166948ms -45:02.178 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:46370 latency=154.601µs -45:02.179 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/b1299e2b-95f2-ca74-f174-7756ee5fdbe8 from=127.0.0.1:46370 latency=110.911µs -45:02.183 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/b1299e2b-95f2-ca74-f174-7756ee5fdbe8 from=127.0.0.1:46370 latency=3.887308ms -45:02.183 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/b1299e2b-95f2-ca74-f174-7756ee5fdbe8 from=127.0.0.1:46368 latency=166.953µs -45:02.189 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46368 latency=5.54633ms -45:02.190 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/48a43785-bc6a-6882-1439-2aae466b55e8 from=127.0.0.1:46372 latency=172.9µs -45:02.194 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/48a43785-bc6a-6882-1439-2aae466b55e8 from=127.0.0.1:46372 latency=3.068036ms -45:02.194 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/48a43785-bc6a-6882-1439-2aae466b55e8 from=127.0.0.1:46368 latency=94.639µs -45:02.198 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46368 latency=3.39647ms -45:02.199 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/a1627009-c1fc-8c29-d2e8-f6b396f417c9 from=127.0.0.1:46376 latency=114.136µs -45:02.207 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46368 latency=7.401696ms -45:02.208 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/288f7530-eb52-810c-7685-6c54a8996556 from=127.0.0.1:46380 latency=177.19µs -45:02.211 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/288f7530-eb52-810c-7685-6c54a8996556 from=127.0.0.1:46380 latency=2.757204ms -45:02.211 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/288f7530-eb52-810c-7685-6c54a8996556 from=127.0.0.1:46368 latency=118.223µs -45:02.211 [INFO] TestAgent: Requesting shutdown -45:02.212 [INFO] TestAgent.server: shutting down server -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:02.212 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:02.212 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:02.212 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:02.212 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:02.215 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:02.219Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:02.219Z [INFO] agent.router.manager: shutting down -45:02.219 [INFO] TestAgent: consul server down -45:02.219 [INFO] TestAgent: shutdown complete -45:02.219 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=tcp -45:02.219 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=udp -45:02.219 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24296 network=tcp -45:02.719 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.719 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleUpdateCommand_noMerge (1.02s) +2021-10-05T16:30:24.604Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:24.639Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.709 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 Address:127.0.0.1:28384}]" +30:24.709 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5.dc1 127.0.0.1 +30:24.710 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 127.0.0.1 +2021-10-05T16:30:24.710Z [INFO] agent.router: Initializing LAN area manager +30:24.710 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=udp +30:24.710 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28384 [Follower]" leader= +30:24.710 [INFO] TestAgent.server: Adding LAN server: server="Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" +30:24.710 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5.dc1 area=wan +30:24.711 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=tcp +30:24.711 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28380 network=tcp +30:24.711 [INFO] TestAgent: started state syncer +30:24.711 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28385 network=tcp +30:24.761 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:24.763 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:24.763 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28384 [Candidate]" term=2 +30:24.792 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:24.792 [DEBUG] TestAgent.server.raft: vote granted: from=bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 term=2 tally=1 +30:24.792 [INFO] TestAgent.server.raft: election won: tally=1 +30:24.792 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28384 [Leader]" +30:24.792 [INFO] TestAgent.server: cluster leadership acquired +30:24.792 [INFO] TestAgent.server: New leader elected: payload=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 +30:24.815 [INFO] TestAgent.server: initializing acls +30:24.822 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:24.822 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:24.832 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:24.850 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:24.850 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:24.850 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:24.850 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 +30:24.850 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5.dc1 +30:24.850 [INFO] TestAgent.server: Updating LAN server: server="Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" +30:24.868 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5.dc1 area=wan +30:24.873 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:24.914 [INFO] TestAgent: Synced node info +30:24.914 [DEBUG] TestAgent: Node info in sync +30:24.928 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:24.928 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:24.928 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:24.928 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:24.928 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:24.928 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 +30:24.928 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 +30:24.949 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bb2d0d30-8d0a-2514-e9ac-06e530c7a8f5 +30:24.954 [INFO] TestAgent.server: federation state anti-entropy synced +30:25.042 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59370 latency=6.015074ms +30:25.047 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:59372 latency=697.23µs +30:25.048 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/b0ada926-c6cf-f8ef-4444-3e8e3bc3d009 from=127.0.0.1:59374 latency=263.122µs +30:25.056 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=5.819956ms +30:25.057 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:25.058 [DEBUG] TestAgent: Node info in sync +30:25.070 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=11.474555ms +30:25.072 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=191.426µs +30:25.078 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=6.138313ms +30:25.079 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=196.927µs +30:25.083 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=4.369477ms +30:25.085 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=1.185057ms +30:25.112 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=26.235022ms +30:25.113 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=175.651µs +30:25.128 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=13.826773ms +30:25.129 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=553.879µs +30:25.136 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=6.884576ms +30:25.141 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=528.34µs +30:25.146 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:59384 latency=625.086µs +30:25.163 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=15.95146ms +30:25.184 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/2ee27ebd-c93e-1e5d-3f5d-d8144b9d290a from=127.0.0.1:59388 latency=789.493µs +30:25.185 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/2ee27ebd-c93e-1e5d-3f5d-d8144b9d290a from=127.0.0.1:59388 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" +30:25.185 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/2ee27ebd-c93e-1e5d-3f5d-d8144b9d290a from=127.0.0.1:59388 latency=606.34µs +30:25.189 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59370 latency=3.349365ms +30:25.190 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e562a0c1-c141-2d5e-33f9-fa910349454e from=127.0.0.1:59390 latency=200.002µs +30:25.194 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/e562a0c1-c141-2d5e-33f9-fa910349454e from=127.0.0.1:59390 latency=3.655474ms +30:25.195 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e562a0c1-c141-2d5e-33f9-fa910349454e from=127.0.0.1:59370 latency=162.287µs +30:25.195 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:59370 latency=175.899µs +30:25.210 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/2ee27ebd-c93e-1e5d-3f5d-d8144b9d290a from=127.0.0.1:59370 latency=9.876838ms +30:25.224 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/7d933d55-73c9-1c68-98f1-8925c064960f from=127.0.0.1:59392 latency=13.624044ms +30:25.235 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/ab78337e-a7b5-ac49-74f0-5dc1651779ff from=127.0.0.1:59396 latency=9.909974ms +30:25.253 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/b3530851-6cf7-3056-a086-d3cd2d5fda73 from=127.0.0.1:59400 latency=17.032219ms +30:25.263 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/b493da55-bf3f-9685-346b-556b45e5d4e8 from=127.0.0.1:59410 latency=6.981449ms +30:25.273 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/e04290f4-7f02-740e-8f67-2da91f512c26 from=127.0.0.1:59414 latency=9.01894ms +30:25.288 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/e562a0c1-c141-2d5e-33f9-fa910349454e from=127.0.0.1:59418 latency=14.84386ms +30:25.295 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/f6869b15-6e57-12a9-f43d-b4e437d7f5ed from=127.0.0.1:59420 latency=5.420392ms +30:25.313 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59422 latency=17.211801ms +30:25.317 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:59426 latency=514.753µs +30:25.321 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/d28b1f25-59ac-f425-0c97-4c2a7d318851 from=127.0.0.1:59426 latency=205.01µs +30:25.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/d28b1f25-59ac-f425-0c97-4c2a7d318851 from=127.0.0.1:59426 latency=6.248126ms +30:25.328 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/d28b1f25-59ac-f425-0c97-4c2a7d318851 from=127.0.0.1:59422 latency=174.415µs +30:25.343 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59422 latency=15.070937ms +30:25.349 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e4d52d11-e451-b865-8255-794fec4882e3 from=127.0.0.1:59432 latency=193.295µs +30:25.362 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/e4d52d11-e451-b865-8255-794fec4882e3 from=127.0.0.1:59432 latency=11.731674ms +30:25.363 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e4d52d11-e451-b865-8255-794fec4882e3 from=127.0.0.1:59422 latency=434.48µs +30:25.370 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59422 latency=6.213212ms +30:25.372 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/08fcb160-4bcc-c539-eed5-8cedacbc165e from=127.0.0.1:59436 latency=526.312µs +30:25.377 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59422 latency=4.614234ms +30:25.378 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/be9045d2-6e3b-2e02-8288-25fb11b226c8 from=127.0.0.1:59438 latency=199.122µs +30:25.401 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/be9045d2-6e3b-2e02-8288-25fb11b226c8 from=127.0.0.1:59438 latency=21.984825ms +30:25.402 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/be9045d2-6e3b-2e02-8288-25fb11b226c8 from=127.0.0.1:59422 latency=201.865µs +30:25.402 [INFO] TestAgent: Requesting shutdown +30:25.402 [INFO] TestAgent.server: shutting down server +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:25.402 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:25.402 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:25.402 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:25.402 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:25.411 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:25.419Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:25.419Z [INFO] agent.router.manager: shutting down +30:25.420 [INFO] TestAgent: consul server down +30:25.420 [INFO] TestAgent: shutdown complete +30:25.420 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=tcp +30:25.420 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=udp +30:25.420 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28380 network=tcp +30:25.924 [INFO] TestAgent: Waiting for endpoints to shut down +30:25.924 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleUpdateCommand_noMerge (1.44s) --- PASS: TestBindingRuleUpdateCommand_noMerge/rule_id_required (0.00s) --- PASS: TestBindingRuleUpdateCommand_noMerge/rule_id_partial_matches_nothing (0.00s) --- PASS: TestBindingRuleUpdateCommand_noMerge/rule_id_exact_match_doesn't_exist (0.00s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/rule_id_partial_matches_multiple (0.02s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/must_use_roughly_valid_selector (0.01s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/rule_id_partial_matches_multiple (0.10s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/must_use_roughly_valid_selector (0.04s) --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields (0.01s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_-_partial (0.04s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_description (0.01s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/missing_bind_name (0.00s) - --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_selector (0.01s) -2022-11-07T22:45:01.735Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:01.747Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:01.773 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67791c36-1391-443b-d6f9-0634edcdf470 Address:127.0.0.1:24307}]" -45:01.774 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24307 [Follower]" leader= -45:01.774 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67791c36-1391-443b-d6f9-0634edcdf470.dc1 127.0.0.1 -45:01.775 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67791c36-1391-443b-d6f9-0634edcdf470 127.0.0.1 -2022-11-07T22:45:01.775Z [INFO] agent.router: Initializing LAN area manager -45:01.776 [INFO] TestAgent.server: Adding LAN server: server="Node-67791c36-1391-443b-d6f9-0634edcdf470 (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:01.776 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=udp -45:01.776 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=tcp -45:01.776 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67791c36-1391-443b-d6f9-0634edcdf470.dc1 area=wan -45:01.778 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24303 network=tcp -45:01.778 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24308 network=tcp -45:01.778 [INFO] TestAgent: started state syncer -45:01.827 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:01.834 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:01.834 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24307 [Candidate]" term=2 -45:01.848 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:01.848 [DEBUG] TestAgent.server.raft: vote granted: from=67791c36-1391-443b-d6f9-0634edcdf470 term=2 tally=1 -45:01.848 [INFO] TestAgent.server.raft: election won: tally=1 -45:01.848 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24307 [Leader]" -45:01.848 [INFO] TestAgent.server: cluster leadership acquired -45:01.848 [INFO] TestAgent.server: New leader elected: payload=Node-67791c36-1391-443b-d6f9-0634edcdf470 -45:01.855 [INFO] TestAgent.server: initializing acls -45:01.857 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:01.857 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:01.864 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:01.871 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:01.871 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:01.871 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:01.871 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-67791c36-1391-443b-d6f9-0634edcdf470 -45:01.871 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-67791c36-1391-443b-d6f9-0634edcdf470.dc1 -45:01.871 [INFO] TestAgent.server: Updating LAN server: server="Node-67791c36-1391-443b-d6f9-0634edcdf470 (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:01.871 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-67791c36-1391-443b-d6f9-0634edcdf470.dc1 area=wan -45:01.882 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:01.898 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:01.898 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:01.898 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:01.898 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:01.898 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:01.898 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67791c36-1391-443b-d6f9-0634edcdf470 -45:01.898 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67791c36-1391-443b-d6f9-0634edcdf470 -45:01.901 [INFO] TestAgent.server: federation state anti-entropy synced -45:01.906 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67791c36-1391-443b-d6f9-0634edcdf470 -45:02.058 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:02.063 [INFO] TestAgent: Synced node info -45:02.063 [DEBUG] TestAgent: Node info in sync -45:02.169 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:47698 latency=3.579345ms -45:02.171 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:47704 latency=218.238µs -45:02.172 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/d461b05b-bace-21c3-8634-d2b18350fe18 from=127.0.0.1:47706 latency=66.676µs -45:02.172 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=111.197µs -45:02.177 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=4.333165ms -45:02.177 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=109.062µs -45:02.180 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=2.760947ms -45:02.181 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=139.5µs -45:02.184 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=3.058257ms -45:02.185 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=161.616µs -45:02.189 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=3.486812ms -45:02.189 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=118.485µs -45:02.192 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=3.022488ms -45:02.193 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=131.282µs -45:02.194 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:47714 latency=142.023µs -45:02.199 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=4.55684ms -45:02.203 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/12ed3781-12a9-41c9-c308-8ba9fce6f24a from=127.0.0.1:47718 latency=144.463µs -45:02.205 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/12ed3781-12a9-41c9-c308-8ba9fce6f24a from=127.0.0.1:47718 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" -45:02.205 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/12ed3781-12a9-41c9-c308-8ba9fce6f24a from=127.0.0.1:47718 latency=911.584µs -45:02.209 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47698 latency=3.377109ms -45:02.210 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/80c1f805-6430-b47e-bc25-e865a88022f8 from=127.0.0.1:47722 latency=253.718µs -45:02.214 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/80c1f805-6430-b47e-bc25-e865a88022f8 from=127.0.0.1:47722 latency=3.202379ms -45:02.214 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/80c1f805-6430-b47e-bc25-e865a88022f8 from=127.0.0.1:47698 latency=116.194µs -45:02.215 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:47698 latency=149.721µs -45:02.219 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/12ed3781-12a9-41c9-c308-8ba9fce6f24a from=127.0.0.1:47698 latency=3.724775ms -45:02.224 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/1513e5f9-395c-9bad-9a32-58d041f46760 from=127.0.0.1:47724 latency=3.857868ms -45:02.227 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/5015c7a5-8204-9d5a-5301-8d6c379c53af from=127.0.0.1:47726 latency=3.203583ms -45:02.231 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/5509c026-2f7e-2de2-0ae4-3214e3c8b1f2 from=127.0.0.1:47728 latency=2.768199ms -45:02.234 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/640ee5e5-1c99-0fde-3f0a-09cf59b27728 from=127.0.0.1:47730 latency=2.709533ms -45:02.237 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/7cc42d2e-e5e1-805e-6b23-283065124889 from=127.0.0.1:47732 latency=2.900701ms -45:02.240 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/80c1f805-6430-b47e-bc25-e865a88022f8 from=127.0.0.1:47734 latency=2.719156ms -45:02.246 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=4.768045ms -45:02.248 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:47738 latency=567.266µs -45:02.249 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/b64608b0-24bb-92e6-f10f-c88a41b71d90 from=127.0.0.1:47738 latency=300.236µs -45:02.253 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/b64608b0-24bb-92e6-f10f-c88a41b71d90 from=127.0.0.1:47738 latency=3.660091ms -45:02.253 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/b64608b0-24bb-92e6-f10f-c88a41b71d90 from=127.0.0.1:47736 latency=83.864µs -45:02.257 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=3.053103ms -45:02.258 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5dce85b2-d1ba-f257-94d3-30638c292ebf from=127.0.0.1:47740 latency=140.803µs -45:02.264 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/5dce85b2-d1ba-f257-94d3-30638c292ebf from=127.0.0.1:47740 latency=5.125877ms -45:02.264 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5dce85b2-d1ba-f257-94d3-30638c292ebf from=127.0.0.1:47736 latency=126.368µs -45:02.268 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=3.051727ms -45:02.269 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/37b57906-72e6-92fe-de7c-460721bc8109 from=127.0.0.1:47742 latency=315.519µs -45:02.272 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/37b57906-72e6-92fe-de7c-460721bc8109 from=127.0.0.1:47742 latency=3.032338ms -45:02.273 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/37b57906-72e6-92fe-de7c-460721bc8109 from=127.0.0.1:47736 latency=110.278µs -45:02.276 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=2.95614ms -45:02.277 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/27b8a8cf-87e3-fa0c-287d-4f8b60ed6a89 from=127.0.0.1:47744 latency=137.288µs -45:02.280 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/27b8a8cf-87e3-fa0c-287d-4f8b60ed6a89 from=127.0.0.1:47744 latency=2.983095ms -45:02.281 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/27b8a8cf-87e3-fa0c-287d-4f8b60ed6a89 from=127.0.0.1:47736 latency=124.273µs -45:02.285 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=3.264192ms -45:02.286 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/2b67d404-0327-60e5-6c39-861179c56c93 from=127.0.0.1:47746 latency=154.21µs -45:02.289 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/2b67d404-0327-60e5-6c39-861179c56c93 from=127.0.0.1:47746 latency=3.111947ms -45:02.290 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/2b67d404-0327-60e5-6c39-861179c56c93 from=127.0.0.1:47736 latency=116.248µs -45:02.298 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=7.985814ms -45:02.299 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/eec9ffcd-c3f3-81f5-b7fb-c6f5268ef60b from=127.0.0.1:47748 latency=144.595µs -45:02.302 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/eec9ffcd-c3f3-81f5-b7fb-c6f5268ef60b from=127.0.0.1:47748 latency=2.787481ms -45:02.303 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/eec9ffcd-c3f3-81f5-b7fb-c6f5268ef60b from=127.0.0.1:47736 latency=107.66µs -45:02.306 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:47736 latency=3.241829ms -45:02.307 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/4a982701-e9a2-ebf4-299a-44295addfe8c from=127.0.0.1:47750 latency=130.058µs -45:02.311 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/4a982701-e9a2-ebf4-299a-44295addfe8c from=127.0.0.1:47750 latency=3.038143ms -45:02.311 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/4a982701-e9a2-ebf4-299a-44295addfe8c from=127.0.0.1:47736 latency=108.365µs -45:02.312 [INFO] TestAgent: Requesting shutdown -45:02.312 [INFO] TestAgent.server: shutting down server -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:02.312 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:02.312 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:02.312 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:02.312 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:02.317 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:02.320Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:02.320Z [INFO] agent.router.manager: shutting down -45:02.320 [INFO] TestAgent: consul server down -45:02.320 [INFO] TestAgent: shutdown complete -45:02.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=tcp -45:02.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=udp -45:02.320 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24303 network=tcp -45:02.820 [INFO] TestAgent: Waiting for endpoints to shut down -45:02.820 [INFO] TestAgent: Endpoints down ---- PASS: TestBindingRuleUpdateCommand (1.12s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_-_partial (0.13s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_description (0.04s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/missing_bind_name (0.01s) + --- PASS: TestBindingRuleUpdateCommand_noMerge/update_all_fields_but_selector (0.03s) +2021-10-05T16:30:24.593Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:24.635Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:24.697 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7c92acb4-2343-67d6-f69a-3496710602db Address:127.0.0.1:28391}]" +30:24.698 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28391 [Follower]" leader= +30:24.698 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7c92acb4-2343-67d6-f69a-3496710602db.dc1 127.0.0.1 +30:24.699 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7c92acb4-2343-67d6-f69a-3496710602db 127.0.0.1 +2021-10-05T16:30:24.699Z [INFO] agent.router: Initializing LAN area manager +30:24.699 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=udp +30:24.699 [INFO] TestAgent.server: Adding LAN server: server="Node-7c92acb4-2343-67d6-f69a-3496710602db (Addr: tcp/127.0.0.1:28391) (DC: dc1)" +30:24.699 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7c92acb4-2343-67d6-f69a-3496710602db.dc1 area=wan +30:24.699 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=tcp +30:24.700 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28387 network=tcp +30:24.700 [INFO] TestAgent: started state syncer +30:24.700 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28392 network=tcp +30:24.750 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:24.750 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:24.750 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28391 [Candidate]" term=2 +30:24.769 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:24.769 [DEBUG] TestAgent.server.raft: vote granted: from=7c92acb4-2343-67d6-f69a-3496710602db term=2 tally=1 +30:24.769 [INFO] TestAgent.server.raft: election won: tally=1 +30:24.769 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28391 [Leader]" +30:24.769 [INFO] TestAgent.server: cluster leadership acquired +30:24.769 [INFO] TestAgent.server: New leader elected: payload=Node-7c92acb4-2343-67d6-f69a-3496710602db +30:24.777 [INFO] TestAgent.server: initializing acls +30:24.780 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:24.780 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:24.796 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:24.820 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:24.820 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:24.820 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:24.820 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7c92acb4-2343-67d6-f69a-3496710602db +30:24.820 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7c92acb4-2343-67d6-f69a-3496710602db.dc1 +30:24.824 [INFO] TestAgent.server: Updating LAN server: server="Node-7c92acb4-2343-67d6-f69a-3496710602db (Addr: tcp/127.0.0.1:28391) (DC: dc1)" +30:24.824 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7c92acb4-2343-67d6-f69a-3496710602db.dc1 area=wan +30:24.840 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:24.897 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:24.897 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:24.897 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:24.897 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:24.897 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:24.898 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7c92acb4-2343-67d6-f69a-3496710602db +30:24.898 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7c92acb4-2343-67d6-f69a-3496710602db +30:24.907 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7c92acb4-2343-67d6-f69a-3496710602db +30:24.918 [INFO] TestAgent.server: federation state anti-entropy synced +30:25.058 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:44718 latency=25.910649ms +30:25.061 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:25.065 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:44726 latency=459.361µs +30:25.067 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5f8510a5-3641-4918-ac7d-017dce47a123 from=127.0.0.1:44728 latency=231.03µs +30:25.071 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=460.986µs +30:25.076 [INFO] TestAgent: Synced node info +30:25.092 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=20.290191ms +30:25.093 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=305.845µs +30:25.100 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=6.206196ms +30:25.100 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=161.012µs +30:25.107 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=6.610433ms +30:25.108 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=312.361µs +30:25.112 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=4.110138ms +30:25.113 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=145.248µs +30:25.127 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=12.841642ms +30:25.129 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=196.805µs +30:25.130 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:44730 latency=407.587µs +30:25.143 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=11.651341ms +30:25.146 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/f2256858-ef26-74e8-5feb-f7f28a2eb192 from=127.0.0.1:44732 latency=174.377µs +30:25.149 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/acl/binding-rule/f2256858-ef26-74e8-5feb-f7f28a2eb192 from=127.0.0.1:44732 error="invalid Binding Rule: Selector is invalid: 1:4 (3): no match found, expected: "!=", ".", "==", "[", [ \t\r\n] or [a-zA-Z0-9_]" +30:25.159 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/f2256858-ef26-74e8-5feb-f7f28a2eb192 from=127.0.0.1:44732 latency=11.207596ms +30:25.172 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44718 latency=6.788453ms +30:25.180 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/9f01a34b-4405-a2eb-a733-be451b7656a0 from=127.0.0.1:44736 latency=484.847µs +30:25.196 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/9f01a34b-4405-a2eb-a733-be451b7656a0 from=127.0.0.1:44736 latency=11.834645ms +30:25.200 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/9f01a34b-4405-a2eb-a733-be451b7656a0 from=127.0.0.1:44718 latency=267.037µs +30:25.201 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules?authmethod=test from=127.0.0.1:44718 latency=251.189µs +30:25.215 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/47db3aa6-51e1-94c4-5815-93a5c999c8a9 from=127.0.0.1:44718 latency=13.406433ms +30:25.226 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/67e94764-f98b-6acd-f5b0-c5848f68450e from=127.0.0.1:44744 latency=10.971586ms +30:25.236 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/77e8e39f-42f3-3a81-ad4a-860ab736cc25 from=127.0.0.1:44748 latency=9.629279ms +30:25.241 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/9f01a34b-4405-a2eb-a733-be451b7656a0 from=127.0.0.1:44752 latency=3.848956ms +30:25.245 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/cce4f155-54d8-0f2d-3caf-6b81f8485f72 from=127.0.0.1:44754 latency=4.10427ms +30:25.250 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/ce451d69-6904-88e8-613f-e471c46887cb from=127.0.0.1:44756 latency=4.188141ms +30:25.257 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/binding-rule/f2256858-ef26-74e8-5feb-f7f28a2eb192 from=127.0.0.1:44758 latency=5.890792ms +30:25.269 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=11.579412ms +30:25.273 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rules from=127.0.0.1:44766 latency=261.908µs +30:25.274 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/c5825ead-ee38-dc87-caa0-91f68ac13283 from=127.0.0.1:44766 latency=144.725µs +30:25.285 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/c5825ead-ee38-dc87-caa0-91f68ac13283 from=127.0.0.1:44766 latency=10.760639ms +30:25.286 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/c5825ead-ee38-dc87-caa0-91f68ac13283 from=127.0.0.1:44762 latency=578.861µs +30:25.295 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=8.527847ms +30:25.298 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/83c2504c-e91f-734d-9f4e-4191eb0e2476 from=127.0.0.1:44774 latency=960.587µs +30:25.311 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/83c2504c-e91f-734d-9f4e-4191eb0e2476 from=127.0.0.1:44774 latency=10.784969ms +30:25.312 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/83c2504c-e91f-734d-9f4e-4191eb0e2476 from=127.0.0.1:44762 latency=243.098µs +30:25.317 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=5.13489ms +30:25.321 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/026fd178-52e2-dbe8-7e09-9186f3babc41 from=127.0.0.1:44778 latency=417.289µs +30:25.329 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/026fd178-52e2-dbe8-7e09-9186f3babc41 from=127.0.0.1:44778 latency=7.307577ms +30:25.330 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/026fd178-52e2-dbe8-7e09-9186f3babc41 from=127.0.0.1:44762 latency=134.222µs +30:25.347 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=16.447165ms +30:25.349 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/077f0d00-f179-75d9-7222-c45b2588a3e6 from=127.0.0.1:44780 latency=267.685µs +30:25.361 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/077f0d00-f179-75d9-7222-c45b2588a3e6 from=127.0.0.1:44780 latency=11.291762ms +30:25.363 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/077f0d00-f179-75d9-7222-c45b2588a3e6 from=127.0.0.1:44762 latency=563.732µs +30:25.368 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=4.402718ms +30:25.369 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e472f3e8-fb06-17d4-b9f7-c69ee5d68947 from=127.0.0.1:44784 latency=292.779µs +30:25.376 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/e472f3e8-fb06-17d4-b9f7-c69ee5d68947 from=127.0.0.1:44784 latency=5.45578ms +30:25.377 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/e472f3e8-fb06-17d4-b9f7-c69ee5d68947 from=127.0.0.1:44762 latency=466.591µs +30:25.398 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=20.560918ms +30:25.400 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/9e15f94c-f944-5ebb-aa39-a50bafa639d7 from=127.0.0.1:44790 latency=397.648µs +30:25.408 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/9e15f94c-f944-5ebb-aa39-a50bafa639d7 from=127.0.0.1:44790 latency=6.101751ms +30:25.412 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/9e15f94c-f944-5ebb-aa39-a50bafa639d7 from=127.0.0.1:44762 latency=228.059µs +30:25.417 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:44762 latency=4.518735ms +30:25.418 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5d7905a0-4a1e-c1ec-aedd-2c95012b07b1 from=127.0.0.1:44792 latency=463.695µs +30:25.424 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule/5d7905a0-4a1e-c1ec-aedd-2c95012b07b1 from=127.0.0.1:44792 latency=5.312862ms +30:25.425 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/binding-rule/5d7905a0-4a1e-c1ec-aedd-2c95012b07b1 from=127.0.0.1:44762 latency=152.36µs +30:25.425 [INFO] TestAgent: Requesting shutdown +30:25.425 [INFO] TestAgent.server: shutting down server +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:25.425 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:25.425 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:25.425 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:25.425 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:25.431 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:25.435Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:25.435Z [INFO] agent.router.manager: shutting down +30:25.435 [INFO] TestAgent: consul server down +30:25.435 [INFO] TestAgent: shutdown complete +30:25.435 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=tcp +30:25.435 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=udp +30:25.435 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28387 network=tcp +30:25.936 [INFO] TestAgent: Waiting for endpoints to shut down +30:25.936 [INFO] TestAgent: Endpoints down +--- PASS: TestBindingRuleUpdateCommand (1.46s) --- PASS: TestBindingRuleUpdateCommand/rule_id_required (0.00s) --- PASS: TestBindingRuleUpdateCommand/rule_id_partial_matches_nothing (0.00s) --- PASS: TestBindingRuleUpdateCommand/rule_id_exact_match_doesn't_exist (0.00s) - --- PASS: TestBindingRuleUpdateCommand/rule_id_partial_matches_multiple (0.02s) - --- PASS: TestBindingRuleUpdateCommand/must_use_roughly_valid_selector (0.01s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields (0.01s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields_-_partial (0.04s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_description (0.01s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_bind_name (0.01s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_must_exist (0.01s) + --- PASS: TestBindingRuleUpdateCommand/rule_id_partial_matches_multiple (0.06s) + --- PASS: TestBindingRuleUpdateCommand/must_use_roughly_valid_selector (0.03s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields (0.04s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields_-_partial (0.09s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_description (0.03s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_bind_name (0.02s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_must_exist (0.03s) --- PASS: TestBindingRuleUpdateCommand/update_all_fields_but_selector (0.01s) - --- PASS: TestBindingRuleUpdateCommand/update_all_fields_clear_selector (0.01s) + --- PASS: TestBindingRuleUpdateCommand/update_all_fields_clear_selector (0.03s) --- PASS: TestBindingRuleUpdateCommand/update_all_fields_json_formatted (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/bindingrule/update 1.150s +ok github.com/hashicorp/consul/command/acl/bindingrule/update 1.569s === RUN TestBootstrapCommand_noTabs === PAUSE TestBootstrapCommand_noTabs === RUN TestBootstrapCommand_Pretty @@ -74762,162 +74976,175 @@ === RUN TestBootstrapCommand_JSON === PAUSE TestBootstrapCommand_JSON === CONT TestBootstrapCommand_noTabs +--- PASS: TestBootstrapCommand_noTabs (0.00s) === CONT TestBootstrapCommand_JSON -=== CONT TestBootstrapCommand_Pretty [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestBootstrapCommand_noTabs (0.00s) +=== CONT TestBootstrapCommand_Pretty [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:03.702Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:03.713Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:03.742 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a89d8be-2eb5-e74a-2a21-012ce74be44e Address:127.0.0.1:26342}]" -45:03.742 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= -45:03.742 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e.dc1 127.0.0.1 -45:03.743 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e 127.0.0.1 -2022-11-07T22:45:03.743Z [INFO] agent.router: Initializing LAN area manager -45:03.743 [INFO] TestAgent.server: Adding LAN server: server="Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:03.743 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e.dc1 area=wan -45:03.743 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp -45:03.743 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp -45:03.744 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp -45:03.745 [INFO] TestAgent: started state syncer -45:03.745 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp -45:03.793 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:03.803 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:03.803 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 -45:03.812 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:03.812 [DEBUG] TestAgent.server.raft: vote granted: from=4a89d8be-2eb5-e74a-2a21-012ce74be44e term=2 tally=1 -45:03.812 [INFO] TestAgent.server.raft: election won: tally=1 -45:03.812 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" -45:03.812 [INFO] TestAgent.server: cluster leadership acquired -45:03.812 [INFO] TestAgent.server: New leader elected: payload=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e -45:03.818 [INFO] TestAgent.server: initializing acls -45:03.821 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:03.826 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:03.826 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:03.826 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:03.826 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e -45:03.826 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e.dc1 -45:03.827 [INFO] TestAgent.server: Updating LAN server: server="Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:03.827 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e.dc1 area=wan -45:03.842 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:03.857 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:03.857 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:03.858 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:03.858 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:03.858 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:03.858 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e -45:03.858 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e -45:03.869 [INFO] TestAgent.server: federation state anti-entropy synced -45:03.869 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a89d8be-2eb5-e74a-2a21-012ce74be44e -45:03.887 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestBootstrapCommand_Pretty-TestAgent-agent873872440/acl-bootstrap-reset: no such file or directory" -45:03.890 [INFO] TestAgent.server.acl: ACL bootstrap completed -45:03.891 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:56520 latency=4.214035ms -45:03.892 [INFO] TestAgent: Requesting shutdown -45:03.892 [INFO] TestAgent.server: shutting down server -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:03.892 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:03.892 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:03.892 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:03.892 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:03.892 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:03.896 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:03.900Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:03.900Z [INFO] agent.router.manager: shutting down -45:03.900 [INFO] TestAgent: consul server down -45:03.900 [INFO] TestAgent: shutdown complete -45:03.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp -45:03.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp -45:03.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp -45:04.400 [INFO] TestAgent: Waiting for endpoints to shut down -45:04.400 [INFO] TestAgent: Endpoints down ---- PASS: TestBootstrapCommand_Pretty (0.74s) -2022-11-07T22:45:03.701Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:03.713Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:03.744 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a3eac8a1-dfaa-f462-276e-ea141e511b2b Address:127.0.0.1:26349}]" -45:03.745 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= -45:03.745 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b.dc1 127.0.0.1 -45:03.747 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b 127.0.0.1 -2022-11-07T22:45:03.747Z [INFO] agent.router: Initializing LAN area manager -45:03.747 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b.dc1 area=wan -45:03.747 [INFO] TestAgent.server: Adding LAN server: server="Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -45:03.747 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp -45:03.747 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp -45:03.747 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp -45:03.747 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp -45:03.747 [INFO] TestAgent: started state syncer -45:03.797 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:03.815 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:03.815 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 -45:03.825 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:03.825 [DEBUG] TestAgent.server.raft: vote granted: from=a3eac8a1-dfaa-f462-276e-ea141e511b2b term=2 tally=1 -45:03.825 [INFO] TestAgent.server.raft: election won: tally=1 -45:03.825 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" -45:03.825 [INFO] TestAgent.server: cluster leadership acquired -45:03.825 [INFO] TestAgent.server: New leader elected: payload=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b -45:03.831 [INFO] TestAgent.server: initializing acls -45:03.835 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:03.839 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:03.839 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:03.839 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:03.840 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b -45:03.840 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b.dc1 -45:03.840 [INFO] TestAgent.server: Updating LAN server: server="Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -45:03.840 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b.dc1 area=wan -45:03.853 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:03.867 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:03.867 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:03.867 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:03.867 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:03.867 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:03.868 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b -45:03.868 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b -45:03.873 [INFO] TestAgent.server: federation state anti-entropy synced -45:03.878 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a3eac8a1-dfaa-f462-276e-ea141e511b2b -45:03.976 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:03.980 [INFO] TestAgent: Synced node info -45:03.980 [DEBUG] TestAgent: Node info in sync -45:04.202 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestBootstrapCommand_JSON-TestAgent-agent042025527/acl-bootstrap-reset: no such file or directory" -45:04.205 [INFO] TestAgent.server.acl: ACL bootstrap completed -45:04.205 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:54566 latency=3.693945ms -45:04.206 [INFO] TestAgent: Requesting shutdown -45:04.206 [INFO] TestAgent.server: shutting down server -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:04.206 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:04.206 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:04.206 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:04.206 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:04.209 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:04.212Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:04.212Z [INFO] agent.router.manager: shutting down -45:04.212 [INFO] TestAgent: consul server down -45:04.212 [INFO] TestAgent: shutdown complete -45:04.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp -45:04.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp -45:04.212 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp -45:04.713 [INFO] TestAgent: Waiting for endpoints to shut down -45:04.713 [INFO] TestAgent: Endpoints down ---- PASS: TestBootstrapCommand_JSON (1.05s) +2021-10-05T16:30:32.379Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:32.419Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:32.499 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:713fbd61-00e4-41f6-f815-2221e2c78bf1 Address:127.0.0.1:25328}]" +30:32.499 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 127.0.0.1 +30:32.500 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25328 [Follower]" leader= +30:32.500 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 127.0.0.1 +2021-10-05T16:30:32.500Z [INFO] agent.router: Initializing LAN area manager +30:32.500 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=udp +30:32.500 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 area=wan +30:32.500 [INFO] TestAgent.server: Adding LAN server: server="Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +30:32.500 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=tcp +30:32.501 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25324 network=tcp +30:32.501 [INFO] TestAgent: started state syncer +30:32.501 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25329 network=tcp +30:32.552 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:32.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:32.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25328 [Candidate]" term=2 +30:32.600 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:32.600 [DEBUG] TestAgent.server.raft: vote granted: from=713fbd61-00e4-41f6-f815-2221e2c78bf1 term=2 tally=1 +30:32.600 [INFO] TestAgent.server.raft: election won: tally=1 +30:32.600 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25328 [Leader]" +30:32.600 [INFO] TestAgent.server: cluster leadership acquired +30:32.600 [INFO] TestAgent.server: New leader elected: payload=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.624 [INFO] TestAgent.server: initializing acls +30:32.640 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:32.652 [INFO] TestAgent.server: initializing acls +30:32.654 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:32.654 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:32.654 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:32.654 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.655 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 +30:32.655 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 area=wan +30:32.655 [INFO] TestAgent.server: Updating LAN server: server="Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +30:32.740 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:32.740 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:32.740 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.740 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 +30:32.740 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1.dc1 area=wan +30:32.741 [INFO] TestAgent.server: Updating LAN server: server="Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +30:32.751 [INFO] TestAgent: Synced node info +30:32.751 [DEBUG] TestAgent: Node info in sync +30:32.779 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:32.840 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestBootstrapCommand_Pretty-TestAgent-agent851786111/acl-bootstrap-reset: no such file or directory" +30:32.864 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:32.864 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:32.864 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:32.864 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:32.864 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:32.864 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.864 [INFO] TestAgent.server: member joined, marking health alive: member=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.868 [INFO] TestAgent.server.acl: ACL bootstrap completed +30:32.869 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:36236 latency=28.644747ms +30:32.872 [INFO] TestAgent: Requesting shutdown +30:32.872 [INFO] TestAgent.server: shutting down server +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:32.872 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:32.872 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:32.873 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:32.873 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:32.873 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:32.873 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:32.873 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:32.876 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:32.877 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:32.877 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-713fbd61-00e4-41f6-f815-2221e2c78bf1 +30:32.884 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:32.884Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:32.884Z [INFO] agent.router.manager: shutting down +30:32.884 [INFO] TestAgent: consul server down +30:32.884 [INFO] TestAgent: shutdown complete +30:32.884 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=tcp +30:32.884 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=udp +30:32.884 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25324 network=tcp +30:33.385 [INFO] TestAgent: Waiting for endpoints to shut down +30:33.385 [INFO] TestAgent: Endpoints down +--- PASS: TestBootstrapCommand_Pretty (1.14s) +2021-10-05T16:30:32.363Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:32.406Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:32.496 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a469b4ff-6f26-0dcf-4469-63624c27ec50 Address:127.0.0.1:25321}]" +30:32.497 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 127.0.0.1 +30:32.497 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 127.0.0.1 +2021-10-05T16:30:32.497Z [INFO] agent.router: Initializing LAN area manager +30:32.498 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp +30:32.498 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= +30:32.498 [INFO] TestAgent.server: Adding LAN server: server="Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +30:32.498 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 area=wan +30:32.498 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp +30:32.499 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp +30:32.499 [INFO] TestAgent: started state syncer +30:32.499 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp +30:32.552 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:32.564 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:32.564 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 +30:32.604 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:32.604 [DEBUG] TestAgent.server.raft: vote granted: from=a469b4ff-6f26-0dcf-4469-63624c27ec50 term=2 tally=1 +30:32.604 [INFO] TestAgent.server.raft: election won: tally=1 +30:32.604 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" +30:32.608 [INFO] TestAgent.server: cluster leadership acquired +30:32.608 [INFO] TestAgent.server: New leader elected: payload=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.632 [INFO] TestAgent.server: initializing acls +30:32.648 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:32.652 [INFO] TestAgent.server: initializing acls +30:32.700 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:32.700 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:32.700 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:32.701 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.701 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 +30:32.701 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:32.701 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:32.702 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.702 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 +30:32.702 [INFO] TestAgent.server: Updating LAN server: server="Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +30:32.702 [INFO] TestAgent.server: Updating LAN server: server="Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +30:32.702 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 area=wan +30:32.702 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50.dc1 area=wan +30:32.752 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:32.854 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:32.854 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:32.854 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:32.854 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:32.854 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:32.854 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.854 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.871 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.871 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a469b4ff-6f26-0dcf-4469-63624c27ec50 +30:32.871 [INFO] TestAgent.server: federation state anti-entropy synced +30:32.895 [WARN] TestAgent.server.acl: failed to remove bootstrap file: error="remove /tmp/consul-test/TestBootstrapCommand_JSON-TestAgent-agent270794501/acl-bootstrap-reset: no such file or directory" +30:32.901 [INFO] TestAgent.server.acl: ACL bootstrap completed +30:32.916 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/bootstrap from=127.0.0.1:53080 latency=21.565666ms +30:32.924 [INFO] TestAgent: Requesting shutdown +30:32.924 [INFO] TestAgent.server: shutting down server +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:32.924 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:32.924 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:32.924 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:32.924 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:32.924 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:32.929 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:32.929Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:32.929Z [INFO] agent.router.manager: shutting down +30:32.929 [INFO] TestAgent: consul server down +30:32.929 [INFO] TestAgent: shutdown complete +30:32.929 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp +30:32.929 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp +30:32.930 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp +30:33.432 [INFO] TestAgent: Waiting for endpoints to shut down +30:33.432 [INFO] TestAgent: Endpoints down +--- PASS: TestBootstrapCommand_JSON (1.19s) PASS -ok github.com/hashicorp/consul/command/acl/bootstrap 1.081s +ok github.com/hashicorp/consul/command/acl/bootstrap 1.293s ? github.com/hashicorp/consul/command/acl/policy [no test files] === RUN TestPolicyCreateCommand_noTabs === PAUSE TestPolicyCreateCommand_noTabs @@ -74926,250 +75153,268 @@ === RUN TestPolicyCreateCommand_JSON === PAUSE TestPolicyCreateCommand_JSON === CONT TestPolicyCreateCommand_noTabs +--- PASS: TestPolicyCreateCommand_noTabs (0.00s) === CONT TestPolicyCreateCommand_JSON -=== CONT TestPolicyCreateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestPolicyCreateCommand_noTabs (0.00s) +=== CONT TestPolicyCreateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:04.566Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.576Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:04.604 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:303f5c19-1594-aaa7-ad7d-f647a82bd35a Address:127.0.0.1:13076}]" -45:04.604 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13076 [Follower]" leader= -45:04.605 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a.dc1 127.0.0.1 -45:04.605 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a 127.0.0.1 -2022-11-07T22:45:04.605Z [INFO] agent.router: Initializing LAN area manager -45:04.605 [INFO] TestAgent.server: Adding LAN server: server="Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a (Addr: tcp/127.0.0.1:13076) (DC: dc1)" -45:04.605 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a.dc1 area=wan -45:04.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=udp -45:04.605 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=tcp -45:04.606 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13072 network=tcp -45:04.606 [INFO] TestAgent: started state syncer -45:04.606 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13077 network=tcp -45:04.656 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:04.665 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:04.665 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13076 [Candidate]" term=2 -45:04.677 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:04.677 [DEBUG] TestAgent.server.raft: vote granted: from=303f5c19-1594-aaa7-ad7d-f647a82bd35a term=2 tally=1 -45:04.677 [INFO] TestAgent.server.raft: election won: tally=1 -45:04.677 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13076 [Leader]" -45:04.677 [INFO] TestAgent.server: cluster leadership acquired -45:04.677 [INFO] TestAgent.server: New leader elected: payload=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a -45:04.682 [INFO] TestAgent.server: initializing acls -45:04.686 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:04.686 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:04.690 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:04.698 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:04.698 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:04.698 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:04.698 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a -45:04.698 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a.dc1 -45:04.698 [INFO] TestAgent.server: Updating LAN server: server="Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a (Addr: tcp/127.0.0.1:13076) (DC: dc1)" -45:04.698 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a.dc1 area=wan -45:04.707 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:04.724 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:04.724 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:04.724 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:04.724 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:04.724 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:04.724 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a -45:04.724 [INFO] TestAgent.server: member joined, marking health alive: member=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a -45:04.729 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-303f5c19-1594-aaa7-ad7d-f647a82bd35a -45:04.738 [INFO] TestAgent.server: federation state anti-entropy synced -45:04.738 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:46476 latency=5.762868ms -45:04.738 [INFO] TestAgent: Requesting shutdown -45:04.738 [INFO] TestAgent.server: shutting down server -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:04.738 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:04.738 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:04.738 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:04.738 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:04.738 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:04.742 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:04.745Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:04.745Z [INFO] agent.router.manager: shutting down -45:04.745 [INFO] TestAgent: consul server down -45:04.745 [INFO] TestAgent: shutdown complete -45:04.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=tcp -45:04.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=udp -45:04.745 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13072 network=tcp -45:05.245 [INFO] TestAgent: Waiting for endpoints to shut down -45:05.245 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyCreateCommand_JSON (0.72s) -2022-11-07T22:45:04.566Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:04.577Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:04.601 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1aa80d6e-313a-06ac-4703-437ce333d48c Address:127.0.0.1:13069}]" -45:04.601 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13069 [Follower]" leader= -45:04.603 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1aa80d6e-313a-06ac-4703-437ce333d48c.dc1 127.0.0.1 -45:04.604 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1aa80d6e-313a-06ac-4703-437ce333d48c 127.0.0.1 -2022-11-07T22:45:04.604Z [INFO] agent.router: Initializing LAN area manager -45:04.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=udp -45:04.604 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1aa80d6e-313a-06ac-4703-437ce333d48c.dc1 area=wan -45:04.604 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=tcp -45:04.604 [INFO] TestAgent.server: Adding LAN server: server="Node-1aa80d6e-313a-06ac-4703-437ce333d48c (Addr: tcp/127.0.0.1:13069) (DC: dc1)" -45:04.605 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13065 network=tcp -45:04.605 [INFO] TestAgent: started state syncer -45:04.605 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13070 network=tcp -45:04.649 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:04.649 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13069 [Candidate]" term=2 -45:04.654 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:04.657 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:04.657 [DEBUG] TestAgent.server.raft: vote granted: from=1aa80d6e-313a-06ac-4703-437ce333d48c term=2 tally=1 -45:04.657 [INFO] TestAgent.server.raft: election won: tally=1 -45:04.657 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13069 [Leader]" -45:04.657 [INFO] TestAgent.server: cluster leadership acquired -45:04.657 [INFO] TestAgent.server: New leader elected: payload=Node-1aa80d6e-313a-06ac-4703-437ce333d48c -45:04.663 [INFO] TestAgent.server: initializing acls -45:04.666 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:04.666 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:04.673 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:04.674 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -45:04.681 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:04.681 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:04.681 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:04.681 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1aa80d6e-313a-06ac-4703-437ce333d48c -45:04.681 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1aa80d6e-313a-06ac-4703-437ce333d48c.dc1 -45:04.681 [INFO] TestAgent.server: Updating LAN server: server="Node-1aa80d6e-313a-06ac-4703-437ce333d48c (Addr: tcp/127.0.0.1:13069) (DC: dc1)" -45:04.681 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1aa80d6e-313a-06ac-4703-437ce333d48c.dc1 area=wan -45:04.690 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:04.707 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:04.707 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:04.707 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:04.707 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:04.707 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:04.707 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1aa80d6e-313a-06ac-4703-437ce333d48c -45:04.707 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1aa80d6e-313a-06ac-4703-437ce333d48c -45:04.717 [INFO] TestAgent.server: federation state anti-entropy synced -45:04.717 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1aa80d6e-313a-06ac-4703-437ce333d48c -45:04.749 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:34406 latency=7.702141ms -45:04.751 [INFO] TestAgent: Requesting shutdown -45:04.752 [INFO] TestAgent.server: shutting down server -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:04.752 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:04.752 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:04.752 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:04.752 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:04.756 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:04.763Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:04.763Z [INFO] agent.router.manager: shutting down -45:04.763 [INFO] TestAgent: consul server down -45:04.763 [INFO] TestAgent: shutdown complete -45:04.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=tcp -45:04.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=udp -45:04.763 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13065 network=tcp -45:05.263 [INFO] TestAgent: Waiting for endpoints to shut down -45:05.263 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyCreateCommand (0.73s) +2021-10-05T16:30:35.502Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:35.534Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:35.577 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:958161b7-5fb2-05a7-e1cd-30d8ec148e9a Address:127.0.0.1:27363}]" +30:35.578 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 127.0.0.1 +30:35.579 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a 127.0.0.1 +2021-10-05T16:30:35.579Z [INFO] agent.router: Initializing LAN area manager +30:35.579 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +30:35.579 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +30:35.579 [INFO] TestAgent.server: Adding LAN server: server="Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +30:35.579 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 area=wan +30:35.579 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +30:35.580 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +30:35.580 [INFO] TestAgent: started state syncer +30:35.580 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +30:35.629 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:35.636 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:35.636 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +30:35.680 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:35.680 [DEBUG] TestAgent.server.raft: vote granted: from=958161b7-5fb2-05a7-e1cd-30d8ec148e9a term=2 tally=1 +30:35.680 [INFO] TestAgent.server.raft: election won: tally=1 +30:35.680 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +30:35.688 [INFO] TestAgent.server: cluster leadership acquired +30:35.688 [INFO] TestAgent.server: New leader elected: payload=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.696 [INFO] TestAgent.server: initializing acls +30:35.698 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:35.698 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:35.707 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:35.733 [INFO] TestAgent.server: initializing acls +30:35.733 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:35.738 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:35.738 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:35.738 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:35.738 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.738 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 +30:35.738 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 area=wan +30:35.738 [INFO] TestAgent.server: Updating LAN server: server="Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +30:35.748 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:35.748 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:35.748 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.748 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 +30:35.748 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a.dc1 area=wan +30:35.748 [INFO] TestAgent.server: Updating LAN server: server="Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +30:35.767 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:35.809 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:35.809 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:35.809 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:35.809 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:35.809 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:35.809 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.809 [INFO] TestAgent.server: member joined, marking health alive: member=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.821 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-958161b7-5fb2-05a7-e1cd-30d8ec148e9a +30:35.840 [INFO] TestAgent.server: federation state anti-entropy synced +30:35.864 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:35.872 [INFO] TestAgent: Synced node info +30:35.872 [DEBUG] TestAgent: Node info in sync +30:35.881 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:57780 latency=4.947956ms +30:35.882 [INFO] TestAgent: Requesting shutdown +30:35.882 [INFO] TestAgent.server: shutting down server +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:35.882 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:35.882 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:35.882 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:35.882 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:35.888 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:35.900Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:35.900Z [INFO] agent.router.manager: shutting down +30:35.900 [INFO] TestAgent: consul server down +30:35.900 [INFO] TestAgent: shutdown complete +30:35.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +30:35.900 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +30:35.900 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +30:36.402 [INFO] TestAgent: Waiting for endpoints to shut down +30:36.402 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyCreateCommand_JSON (0.99s) +2021-10-05T16:30:35.500Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:35.516Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:35.566 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca5126e7-f8c6-5030-daf9-05341adbb47a Address:127.0.0.1:27370}]" +30:35.567 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 127.0.0.1 +30:35.567 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a 127.0.0.1 +2021-10-05T16:30:35.567Z [INFO] agent.router: Initializing LAN area manager +30:35.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=udp +30:35.568 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27370 [Follower]" leader= +30:35.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=tcp +30:35.568 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 area=wan +30:35.568 [INFO] TestAgent.server: Adding LAN server: server="Node-ca5126e7-f8c6-5030-daf9-05341adbb47a (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +30:35.569 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27366 network=tcp +30:35.569 [INFO] TestAgent: started state syncer +30:35.569 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27371 network=tcp +30:35.619 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:35.627 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:35.627 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27370 [Candidate]" term=2 +30:35.668 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:35.668 [DEBUG] TestAgent.server.raft: vote granted: from=ca5126e7-f8c6-5030-daf9-05341adbb47a term=2 tally=1 +30:35.668 [INFO] TestAgent.server.raft: election won: tally=1 +30:35.668 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27370 [Leader]" +30:35.668 [INFO] TestAgent.server: cluster leadership acquired +30:35.668 [INFO] TestAgent.server: New leader elected: payload=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.694 [INFO] TestAgent.server: initializing acls +30:35.705 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:35.705 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:35.719 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:35.719 [INFO] TestAgent.server: initializing acls +30:35.719 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:35.749 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:35.749 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:35.749 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:35.749 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:35.749 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.749 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 +30:35.749 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 area=wan +30:35.749 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:35.750 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.750 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 +30:35.752 [INFO] TestAgent.server: Updating LAN server: server="Node-ca5126e7-f8c6-5030-daf9-05341adbb47a (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +30:35.752 [INFO] TestAgent.server: Updating LAN server: server="Node-ca5126e7-f8c6-5030-daf9-05341adbb47a (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +30:35.752 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a.dc1 area=wan +30:35.769 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:35.822 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:35.822 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:35.822 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:35.822 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:35.822 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:35.822 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.822 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.832 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.832 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca5126e7-f8c6-5030-daf9-05341adbb47a +30:35.844 [INFO] TestAgent.server: federation state anti-entropy synced +30:35.922 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:34070 latency=9.190131ms +30:35.929 [INFO] TestAgent: Requesting shutdown +30:35.929 [INFO] TestAgent.server: shutting down server +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:35.929 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:35.929 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:35.929 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:35.929 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:35.929 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:35.934 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:35.939Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:35.939Z [INFO] agent.router.manager: shutting down +30:35.939 [INFO] TestAgent: consul server down +30:35.939 [INFO] TestAgent: shutdown complete +30:35.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=tcp +30:35.939 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=udp +30:35.939 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27366 network=tcp +30:36.440 [INFO] TestAgent: Waiting for endpoints to shut down +30:36.440 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyCreateCommand (1.02s) PASS -ok github.com/hashicorp/consul/command/acl/policy/create 0.762s +ok github.com/hashicorp/consul/command/acl/policy/create 1.109s === RUN TestPolicyDeleteCommand_noTabs === PAUSE TestPolicyDeleteCommand_noTabs === RUN TestPolicyDeleteCommand === PAUSE TestPolicyDeleteCommand === CONT TestPolicyDeleteCommand_noTabs === CONT TestPolicyDeleteCommand -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... --- PASS: TestPolicyDeleteCommand_noTabs (0.00s) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:06.897Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:06.909Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:06.934 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 Address:127.0.0.1:17153}]" -45:06.934 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= -45:06.935 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685.dc1 127.0.0.1 -45:06.935 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 127.0.0.1 -2022-11-07T22:45:06.935Z [INFO] agent.router: Initializing LAN area manager -45:06.935 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685.dc1 area=wan -45:06.935 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=tcp -45:06.935 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=udp -45:06.935 [INFO] TestAgent.server: Adding LAN server: server="Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 (Addr: tcp/127.0.0.1:17153) (DC: dc1)" -45:06.936 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17149 network=tcp -45:06.936 [INFO] TestAgent: started state syncer -45:06.936 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17154 network=tcp -45:06.985 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:06.989 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:06.989 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 -45:06.998 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:06.998 [DEBUG] TestAgent.server.raft: vote granted: from=3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 term=2 tally=1 -45:06.998 [INFO] TestAgent.server.raft: election won: tally=1 -45:06.998 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" -45:06.998 [INFO] TestAgent.server: cluster leadership acquired -45:06.998 [INFO] TestAgent.server: New leader elected: payload=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 -45:07.004 [INFO] TestAgent.server: initializing acls -45:07.007 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:07.007 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:07.013 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:07.020 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:07.020 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:07.020 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:07.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 -45:07.021 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685.dc1 -45:07.021 [INFO] TestAgent.server: Updating LAN server: server="Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 (Addr: tcp/127.0.0.1:17153) (DC: dc1)" -45:07.021 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685.dc1 area=wan -45:07.029 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:07.042 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:07.042 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:07.042 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:07.042 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:07.042 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:07.042 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 -45:07.042 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 -45:07.047 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ec45fd3-6d9a-f26a-f32f-7e3e9c32e685 -45:07.050 [INFO] TestAgent.server: federation state anti-entropy synced -45:07.070 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:07.075 [INFO] TestAgent: Synced node info -45:07.075 [DEBUG] TestAgent: Node info in sync -45:07.078 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35386 latency=8.335193ms -45:07.083 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/309b7bf3-b11b-11e5-2775-3df121be5874 from=127.0.0.1:35388 latency=3.286205ms -45:07.083 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/309b7bf3-b11b-11e5-2775-3df121be5874 from=127.0.0.1:35386 error="ACL not found" -45:07.084 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/309b7bf3-b11b-11e5-2775-3df121be5874 from=127.0.0.1:35386 latency=928.039µs -45:07.086 [INFO] TestAgent: Requesting shutdown -45:07.086 [INFO] TestAgent.server: shutting down server -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:07.087 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:07.087 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:07.087 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:07.087 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:07.090 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:07.093Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:07.093Z [INFO] agent.router.manager: shutting down -45:07.093 [INFO] TestAgent: consul server down -45:07.093 [INFO] TestAgent: shutdown complete -45:07.093 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp -45:07.093 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp -45:07.093 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp -45:07.593 [INFO] TestAgent: Waiting for endpoints to shut down -45:07.593 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyDeleteCommand (0.73s) +2021-10-05T16:30:36.178Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:36.217Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:36.277 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c3e06800-3f79-def5-5985-bc54a77d26b5 Address:127.0.0.1:26342}]" +30:36.282 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c3e06800-3f79-def5-5985-bc54a77d26b5.dc1 127.0.0.1 +30:36.283 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c3e06800-3f79-def5-5985-bc54a77d26b5 127.0.0.1 +2021-10-05T16:30:36.283Z [INFO] agent.router: Initializing LAN area manager +30:36.287 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp +30:36.287 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= +30:36.288 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp +30:36.289 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp +30:36.289 [INFO] TestAgent: started state syncer +30:36.294 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c3e06800-3f79-def5-5985-bc54a77d26b5.dc1 area=wan +30:36.300 [INFO] TestAgent.server: Adding LAN server: server="Node-c3e06800-3f79-def5-5985-bc54a77d26b5 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +30:36.300 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp +30:36.332 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:36.332 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 +30:36.338 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:36.349 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:36.349 [DEBUG] TestAgent.server.raft: vote granted: from=c3e06800-3f79-def5-5985-bc54a77d26b5 term=2 tally=1 +30:36.349 [INFO] TestAgent.server.raft: election won: tally=1 +30:36.349 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" +30:36.349 [INFO] TestAgent.server: cluster leadership acquired +30:36.349 [INFO] TestAgent.server: New leader elected: payload=Node-c3e06800-3f79-def5-5985-bc54a77d26b5 +30:36.356 [INFO] TestAgent.server: initializing acls +30:36.359 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:36.359 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:36.365 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:36.393 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:36.393 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:36.393 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:36.394 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c3e06800-3f79-def5-5985-bc54a77d26b5 +30:36.394 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c3e06800-3f79-def5-5985-bc54a77d26b5.dc1 +30:36.395 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c3e06800-3f79-def5-5985-bc54a77d26b5.dc1 area=wan +30:36.395 [INFO] TestAgent.server: Updating LAN server: server="Node-c3e06800-3f79-def5-5985-bc54a77d26b5 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +30:36.414 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:36.434 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:36.434 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:36.434 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:36.434 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:36.434 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:36.434 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c3e06800-3f79-def5-5985-bc54a77d26b5 +30:36.434 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c3e06800-3f79-def5-5985-bc54a77d26b5 +30:36.439 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c3e06800-3f79-def5-5985-bc54a77d26b5 +30:36.449 [INFO] TestAgent.server: federation state anti-entropy synced +30:36.624 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:56692 latency=18.454724ms +30:36.653 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/policy/925ec0c1-82bd-1b54-9eb7-26ab889f8f36 from=127.0.0.1:56694 latency=23.416961ms +30:36.659 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/policy/925ec0c1-82bd-1b54-9eb7-26ab889f8f36 from=127.0.0.1:56692 error="ACL not found" +30:36.659 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/925ec0c1-82bd-1b54-9eb7-26ab889f8f36 from=127.0.0.1:56692 latency=532.166µs +30:36.659 [INFO] TestAgent: Requesting shutdown +30:36.659 [INFO] TestAgent.server: shutting down server +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:36.659 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:36.659 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:36.660 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:36.660 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:36.660 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:36.668 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:36.683Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:36.683Z [INFO] agent.router.manager: shutting down +30:36.683 [INFO] TestAgent: consul server down +30:36.683 [INFO] TestAgent: shutdown complete +30:36.683 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp +30:36.683 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp +30:36.683 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp +30:37.184 [INFO] TestAgent: Waiting for endpoints to shut down +30:37.184 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyDeleteCommand (1.09s) PASS -ok github.com/hashicorp/consul/command/acl/policy/delete 0.760s +ok github.com/hashicorp/consul/command/acl/policy/delete 1.176s === RUN TestPolicyListCommand_noTabs === PAUSE TestPolicyListCommand_noTabs === RUN TestPolicyListCommand @@ -75183,167 +75428,164 @@ --- PASS: TestPolicyListCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:08.871Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:08.882Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:08.903 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dcc5f232-91d0-6456-c528-e3af05b62a8c Address:127.0.0.1:10013}]" -45:08.903 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= -45:08.904 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dcc5f232-91d0-6456-c528-e3af05b62a8c.dc1 127.0.0.1 -45:08.905 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dcc5f232-91d0-6456-c528-e3af05b62a8c 127.0.0.1 -2022-11-07T22:45:08.905Z [INFO] agent.router: Initializing LAN area manager -45:08.905 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c.dc1 area=wan -45:08.905 [INFO] TestAgent.server: Adding LAN server: server="Node-dcc5f232-91d0-6456-c528-e3af05b62a8c (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:08.905 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp -45:08.905 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp -45:08.905 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp -45:08.905 [INFO] TestAgent: started state syncer -45:08.905 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp -45:08.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:08.957 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:08.957 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 -45:08.966 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:08.966 [DEBUG] TestAgent.server.raft: vote granted: from=dcc5f232-91d0-6456-c528-e3af05b62a8c term=2 tally=1 -45:08.966 [INFO] TestAgent.server.raft: election won: tally=1 -45:08.966 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" -45:08.966 [INFO] TestAgent.server: cluster leadership acquired -45:08.966 [INFO] TestAgent.server: New leader elected: payload=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c -45:08.972 [INFO] TestAgent.server: initializing acls -45:08.975 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:08.975 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:08.982 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:08.992 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:08.992 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:08.992 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:08.992 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dcc5f232-91d0-6456-c528-e3af05b62a8c -45:08.992 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dcc5f232-91d0-6456-c528-e3af05b62a8c.dc1 -45:08.992 [INFO] TestAgent.server: Updating LAN server: server="Node-dcc5f232-91d0-6456-c528-e3af05b62a8c (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:08.992 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c.dc1 area=wan -45:09.004 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:09.019 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:09.019 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:09.019 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:09.019 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:09.019 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:09.019 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c -45:09.019 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c -45:09.022 [INFO] TestAgent.server: federation state anti-entropy synced -45:09.030 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dcc5f232-91d0-6456-c528-e3af05b62a8c -45:09.196 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44778 latency=6.295473ms -45:09.198 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:09.202 [INFO] TestAgent: Synced node info -45:09.205 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44778 latency=5.624779ms -45:09.209 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44778 latency=3.375864ms -45:09.213 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44778 latency=3.519336ms -45:09.217 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44778 latency=3.440141ms -45:09.218 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:44780 latency=308.482µs -45:09.218 [INFO] TestAgent: Requesting shutdown -45:09.218 [INFO] TestAgent.server: shutting down server -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:09.218 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:09.218 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:09.218 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:09.218 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:09.222 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:09.225Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:09.225Z [INFO] agent.router.manager: shutting down -45:09.225 [INFO] TestAgent: consul server down -45:09.225 [INFO] TestAgent: shutdown complete -45:09.225 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp -45:09.225 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp -45:09.225 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp -45:09.726 [INFO] TestAgent: Waiting for endpoints to shut down -45:09.726 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyListCommand (0.89s) -2022-11-07T22:45:08.870Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:08.881Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:08.903 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a97c5bef-dfde-edb8-e5f1-e133b4ef0843 Address:127.0.0.1:10006}]" -45:08.903 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -45:08.905 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843.dc1 127.0.0.1 -45:08.905 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 127.0.0.1 -2022-11-07T22:45:08.905Z [INFO] agent.router: Initializing LAN area manager -45:08.905 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843.dc1 area=wan -45:08.905 [INFO] TestAgent.server: Adding LAN server: server="Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:08.905 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -45:08.905 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -45:08.906 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -45:08.906 [INFO] TestAgent: started state syncer -45:08.906 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -45:08.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:08.964 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:08.964 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -45:08.973 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:08.973 [DEBUG] TestAgent.server.raft: vote granted: from=a97c5bef-dfde-edb8-e5f1-e133b4ef0843 term=2 tally=1 -45:08.973 [INFO] TestAgent.server.raft: election won: tally=1 -45:08.973 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -45:08.973 [INFO] TestAgent.server: cluster leadership acquired -45:08.973 [INFO] TestAgent.server: New leader elected: payload=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 -45:08.979 [INFO] TestAgent.server: initializing acls -45:08.982 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:08.982 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:08.991 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:08.999 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:08.999 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:08.999 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:09.000 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 -45:09.000 [INFO] TestAgent.server: Updating LAN server: server="Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:09.000 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843.dc1 -45:09.000 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843.dc1 area=wan -45:09.009 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:09.027 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:09.027 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:09.027 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:09.027 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:09.027 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:09.027 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 -45:09.027 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 -45:09.032 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a97c5bef-dfde-edb8-e5f1-e133b4ef0843 -45:09.036 [INFO] TestAgent.server: federation state anti-entropy synced -45:09.292 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:09.298 [INFO] TestAgent: Synced node info -45:09.298 [DEBUG] TestAgent: Node info in sync -45:09.318 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36564 latency=3.431515ms -45:09.323 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36564 latency=3.072583ms -45:09.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36564 latency=3.46042ms -45:09.330 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36564 latency=3.257839ms -45:09.334 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36564 latency=3.020871ms -45:09.335 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:36566 latency=218.007µs -45:09.335 [INFO] TestAgent: Requesting shutdown -45:09.335 [INFO] TestAgent.server: shutting down server -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:09.335 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:09.335 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:09.335 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:09.335 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:09.339 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:09.344Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:09.345Z [INFO] agent.router.manager: shutting down -45:09.345 [INFO] TestAgent: consul server down -45:09.345 [INFO] TestAgent: shutdown complete -45:09.345 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -45:09.345 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -45:09.345 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -45:09.845 [INFO] TestAgent: Waiting for endpoints to shut down -45:09.845 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyListCommand_JSON (1.01s) +2021-10-05T16:30:42.702Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:42.728Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:42.768 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f97886b1-973f-8710-ecd1-86671a7921a7 Address:127.0.0.1:21244}]" +30:42.769 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f97886b1-973f-8710-ecd1-86671a7921a7.dc1 127.0.0.1 +30:42.770 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f97886b1-973f-8710-ecd1-86671a7921a7 127.0.0.1 +2021-10-05T16:30:42.770Z [INFO] agent.router: Initializing LAN area manager +30:42.770 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp +30:42.770 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= +30:42.770 [INFO] TestAgent.server: Adding LAN server: server="Node-f97886b1-973f-8710-ecd1-86671a7921a7 (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +30:42.770 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f97886b1-973f-8710-ecd1-86671a7921a7.dc1 area=wan +30:42.770 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp +30:42.771 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp +30:42.771 [INFO] TestAgent: started state syncer +30:42.771 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp +30:42.820 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:42.828 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 +30:42.832 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:42.871 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:42.871 [DEBUG] TestAgent.server.raft: vote granted: from=f97886b1-973f-8710-ecd1-86671a7921a7 term=2 tally=1 +30:42.871 [INFO] TestAgent.server.raft: election won: tally=1 +30:42.871 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" +30:42.871 [INFO] TestAgent.server: cluster leadership acquired +30:42.871 [INFO] TestAgent.server: New leader elected: payload=Node-f97886b1-973f-8710-ecd1-86671a7921a7 +30:42.878 [INFO] TestAgent.server: initializing acls +30:42.882 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:42.882 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:42.893 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:42.906 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:42.906 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:42.906 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:42.906 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f97886b1-973f-8710-ecd1-86671a7921a7 +30:42.906 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f97886b1-973f-8710-ecd1-86671a7921a7.dc1 +30:42.907 [INFO] TestAgent.server: Updating LAN server: server="Node-f97886b1-973f-8710-ecd1-86671a7921a7 (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +30:42.907 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f97886b1-973f-8710-ecd1-86671a7921a7.dc1 area=wan +30:42.921 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:42.945 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:42.945 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:42.945 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:42.945 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:42.945 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:42.945 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f97886b1-973f-8710-ecd1-86671a7921a7 +30:42.945 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f97886b1-973f-8710-ecd1-86671a7921a7 +30:42.977 [INFO] TestAgent.server: federation state anti-entropy synced +30:42.977 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f97886b1-973f-8710-ecd1-86671a7921a7 +30:43.048 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35066 latency=23.961507ms +30:43.072 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35066 latency=8.652331ms +30:43.077 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35066 latency=3.595951ms +30:43.096 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35066 latency=16.04471ms +30:43.103 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:35066 latency=4.544007ms +30:43.104 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:35070 latency=329.293µs +30:43.105 [INFO] TestAgent: Requesting shutdown +30:43.105 [INFO] TestAgent.server: shutting down server +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:43.105 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:43.105 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:43.105 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:43.105 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:43.105 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:43.111 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:43.116Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:43.116Z [INFO] agent.router.manager: shutting down +30:43.116 [INFO] TestAgent: consul server down +30:43.116 [INFO] TestAgent: shutdown complete +30:43.116 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp +30:43.116 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp +30:43.116 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp +30:43.620 [INFO] TestAgent: Waiting for endpoints to shut down +30:43.620 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyListCommand_JSON (1.03s) +2021-10-05T16:30:42.698Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:42.731Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:42.804 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3bbc2ecf-123a-2e17-5736-c0df9dc9a281 Address:127.0.0.1:21237}]" +30:42.804 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281.dc1 127.0.0.1 +30:42.805 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 127.0.0.1 +2021-10-05T16:30:42.805Z [INFO] agent.router: Initializing LAN area manager +30:42.805 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp +30:42.805 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= +30:42.806 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281.dc1 area=wan +30:42.806 [INFO] TestAgent.server: Adding LAN server: server="Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +30:42.806 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp +30:42.806 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp +30:42.806 [INFO] TestAgent: started state syncer +30:42.806 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp +30:42.844 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:42.844 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 +30:42.856 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:42.879 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:42.879 [DEBUG] TestAgent.server.raft: vote granted: from=3bbc2ecf-123a-2e17-5736-c0df9dc9a281 term=2 tally=1 +30:42.879 [INFO] TestAgent.server.raft: election won: tally=1 +30:42.879 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" +30:42.879 [INFO] TestAgent.server: cluster leadership acquired +30:42.879 [INFO] TestAgent.server: New leader elected: payload=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 +30:42.888 [INFO] TestAgent.server: initializing acls +30:42.893 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:42.894 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:42.903 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:42.916 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:42.916 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:42.916 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:42.917 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 +30:42.917 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281.dc1 +30:42.917 [INFO] TestAgent.server: Updating LAN server: server="Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +30:42.917 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281.dc1 area=wan +30:42.944 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:43.000 [INFO] TestAgent: Synced node info +30:43.020 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:43.020 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:43.020 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:43.020 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:43.020 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:43.020 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 +30:43.020 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 +30:43.060 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:33668 latency=27.782694ms +30:43.060 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3bbc2ecf-123a-2e17-5736-c0df9dc9a281 +30:43.063 [INFO] TestAgent.server: federation state anti-entropy synced +30:43.092 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:33668 latency=16.382351ms +30:43.102 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:33668 latency=5.819018ms +30:43.107 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:33668 latency=5.076616ms +30:43.120 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:33668 latency=10.932441ms +30:43.129 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policies from=127.0.0.1:33672 latency=643.25µs +30:43.129 [INFO] TestAgent: Requesting shutdown +30:43.129 [INFO] TestAgent.server: shutting down server +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:43.129 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:43.129 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:43.130 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:43.130 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:43.140 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:43.145Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:43.145Z [INFO] agent.router.manager: shutting down +30:43.145 [INFO] TestAgent: consul server down +30:43.145 [INFO] TestAgent: shutdown complete +30:43.145 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp +30:43.145 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp +30:43.145 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp +30:43.646 [INFO] TestAgent: Waiting for endpoints to shut down +30:43.646 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyListCommand (1.05s) PASS -ok github.com/hashicorp/consul/command/acl/policy/list 1.039s +ok github.com/hashicorp/consul/command/acl/policy/list 1.153s === RUN TestPolicyReadCommand_noTabs === PAUSE TestPolicyReadCommand_noTabs === RUN TestPolicyReadCommand @@ -75351,163 +75593,174 @@ === RUN TestPolicyReadCommand_JSON === PAUSE TestPolicyReadCommand_JSON === CONT TestPolicyReadCommand_noTabs +--- PASS: TestPolicyReadCommand_noTabs (0.00s) === CONT TestPolicyReadCommand_JSON -=== CONT TestPolicyReadCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestPolicyReadCommand_noTabs (0.00s) +=== CONT TestPolicyReadCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:09.322Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:09.332Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:09.362 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd342852-bee6-fed6-22e2-ba456c6b7a7a Address:127.0.0.1:28384}]" -45:09.362 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28384 [Follower]" leader= -45:09.362 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a.dc1 127.0.0.1 -45:09.363 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a 127.0.0.1 -2022-11-07T22:45:09.363Z [INFO] agent.router: Initializing LAN area manager -45:09.363 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a.dc1 area=wan -45:09.363 [INFO] TestAgent.server: Adding LAN server: server="Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a (Addr: tcp/127.0.0.1:28384) (DC: dc1)" -45:09.363 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=udp -45:09.363 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=tcp -45:09.363 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28380 network=tcp -45:09.363 [INFO] TestAgent: started state syncer -45:09.363 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28385 network=tcp -45:09.413 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:09.421 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:09.421 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28384 [Candidate]" term=2 -45:09.430 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:09.430 [DEBUG] TestAgent.server.raft: vote granted: from=dd342852-bee6-fed6-22e2-ba456c6b7a7a term=2 tally=1 -45:09.430 [INFO] TestAgent.server.raft: election won: tally=1 -45:09.430 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28384 [Leader]" -45:09.430 [INFO] TestAgent.server: cluster leadership acquired -45:09.430 [INFO] TestAgent.server: New leader elected: payload=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a -45:09.435 [INFO] TestAgent.server: initializing acls -45:09.438 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:09.438 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:09.446 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:09.456 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:09.456 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:09.456 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:09.457 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a -45:09.457 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a.dc1 -45:09.457 [INFO] TestAgent.server: Updating LAN server: server="Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a (Addr: tcp/127.0.0.1:28384) (DC: dc1)" -45:09.457 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a.dc1 area=wan -45:09.466 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:09.483 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:09.483 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:09.483 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:09.483 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:09.483 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:09.483 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a -45:09.483 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a -45:09.490 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd342852-bee6-fed6-22e2-ba456c6b7a7a -45:09.494 [INFO] TestAgent.server: federation state anti-entropy synced -45:09.526 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:37152 latency=5.747735ms -45:09.527 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/82633a04-4da1-a327-a937-ce495ab31afd from=127.0.0.1:37154 latency=188.261µs -45:09.527 [INFO] TestAgent: Requesting shutdown -45:09.527 [INFO] TestAgent.server: shutting down server -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:09.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:09.527 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:09.527 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:09.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:09.527 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:09.530 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:09.534Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:09.534Z [INFO] agent.router.manager: shutting down -45:09.534 [INFO] TestAgent: consul server down -45:09.534 [INFO] TestAgent: shutdown complete -45:09.534 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=tcp -45:09.534 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=udp -45:09.534 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28380 network=tcp -45:10.034 [INFO] TestAgent: Waiting for endpoints to shut down -45:10.034 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyReadCommand_JSON (0.76s) -2022-11-07T22:45:09.321Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:09.331Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:09.359 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b6819d5-d20d-8426-402c-d634e4ba8021 Address:127.0.0.1:28391}]" -45:09.360 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28391 [Follower]" leader= -45:09.360 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b6819d5-d20d-8426-402c-d634e4ba8021.dc1 127.0.0.1 -45:09.360 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b6819d5-d20d-8426-402c-d634e4ba8021 127.0.0.1 -2022-11-07T22:45:09.360Z [INFO] agent.router: Initializing LAN area manager -45:09.361 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b6819d5-d20d-8426-402c-d634e4ba8021.dc1 area=wan -45:09.361 [INFO] TestAgent.server: Adding LAN server: server="Node-4b6819d5-d20d-8426-402c-d634e4ba8021 (Addr: tcp/127.0.0.1:28391) (DC: dc1)" -45:09.361 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=udp -45:09.361 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=tcp -45:09.362 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28387 network=tcp -45:09.362 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28392 network=tcp -45:09.362 [INFO] TestAgent: started state syncer -45:09.411 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:09.420 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:09.420 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28391 [Candidate]" term=2 -45:09.428 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:09.428 [DEBUG] TestAgent.server.raft: vote granted: from=4b6819d5-d20d-8426-402c-d634e4ba8021 term=2 tally=1 -45:09.428 [INFO] TestAgent.server.raft: election won: tally=1 -45:09.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28391 [Leader]" -45:09.428 [INFO] TestAgent.server: cluster leadership acquired -45:09.429 [INFO] TestAgent.server: New leader elected: payload=Node-4b6819d5-d20d-8426-402c-d634e4ba8021 -45:09.433 [INFO] TestAgent.server: initializing acls -45:09.437 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:09.437 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:09.443 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:09.454 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:09.454 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:09.454 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:09.455 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4b6819d5-d20d-8426-402c-d634e4ba8021 -45:09.455 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4b6819d5-d20d-8426-402c-d634e4ba8021.dc1 -45:09.455 [INFO] TestAgent.server: Updating LAN server: server="Node-4b6819d5-d20d-8426-402c-d634e4ba8021 (Addr: tcp/127.0.0.1:28391) (DC: dc1)" -45:09.455 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4b6819d5-d20d-8426-402c-d634e4ba8021.dc1 area=wan -45:09.464 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:09.482 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:09.482 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:09.482 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:09.482 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:09.482 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:09.482 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b6819d5-d20d-8426-402c-d634e4ba8021 -45:09.482 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b6819d5-d20d-8426-402c-d634e4ba8021 -45:09.485 [INFO] TestAgent.server: federation state anti-entropy synced -45:09.492 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b6819d5-d20d-8426-402c-d634e4ba8021 -45:09.560 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:09.565 [INFO] TestAgent: Synced node info -45:09.586 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:52774 latency=3.766693ms -45:09.591 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/6197c05c-3a13-682c-33b9-566ae7f5be73 from=127.0.0.1:52776 latency=1.188873ms -45:09.591 [INFO] TestAgent: Requesting shutdown -45:09.591 [INFO] TestAgent.server: shutting down server -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:09.591 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:09.591 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:09.591 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:09.591 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:09.594 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:09.598Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:09.598Z [INFO] agent.router.manager: shutting down -45:09.598 [INFO] TestAgent: consul server down -45:09.598 [INFO] TestAgent: shutdown complete -45:09.598 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=tcp -45:09.598 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=udp -45:09.598 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28387 network=tcp -45:10.098 [INFO] TestAgent: Waiting for endpoints to shut down -45:10.098 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyReadCommand (0.82s) +2021-10-05T16:30:46.759Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:46.864Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:46.922 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:78b9685c-2057-f66f-b123-22d2f8d4ea48 Address:127.0.0.1:15118}]" +30:46.923 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= +30:46.923 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 127.0.0.1 +30:46.924 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 127.0.0.1 +2021-10-05T16:30:46.924Z [INFO] agent.router: Initializing LAN area manager +30:46.924 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp +30:46.924 [INFO] TestAgent.server: Adding LAN server: server="Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +30:46.924 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 area=wan +30:46.925 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp +30:46.925 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp +30:46.925 [INFO] TestAgent: started state syncer +30:46.925 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp +30:46.976 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:46.979 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:46.979 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 +30:47.015 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:47.015 [DEBUG] TestAgent.server.raft: vote granted: from=78b9685c-2057-f66f-b123-22d2f8d4ea48 term=2 tally=1 +30:47.015 [INFO] TestAgent.server.raft: election won: tally=1 +30:47.015 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" +30:47.016 [INFO] TestAgent.server: cluster leadership acquired +30:47.016 [INFO] TestAgent.server: New leader elected: payload=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.028 [INFO] TestAgent.server: initializing acls +30:47.036 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:47.036 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.048 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.078 [INFO] TestAgent.server: initializing acls +30:47.078 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.078 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:47.078 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:47.078 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:47.078 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.078 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 +30:47.079 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 area=wan +30:47.079 [INFO] TestAgent.server: Updating LAN server: server="Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +30:47.079 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.080 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.080 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 +30:47.081 [INFO] TestAgent.server: Updating LAN server: server="Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +30:47.081 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48.dc1 area=wan +30:47.107 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:47.139 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:47.139 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:47.139 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:47.139 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:47.139 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:47.139 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.139 [INFO] TestAgent.server: member joined, marking health alive: member=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.161 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.162 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78b9685c-2057-f66f-b123-22d2f8d4ea48 +30:47.176 [INFO] TestAgent.server: federation state anti-entropy synced +30:47.211 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:47.222 [INFO] TestAgent: Synced node info +30:47.222 [DEBUG] TestAgent: Node info in sync +30:47.223 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36660 latency=18.622516ms +30:47.225 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/500d66e7-03b6-4b0e-0030-4c57358e2cd1 from=127.0.0.1:36662 latency=462.745µs +30:47.225 [INFO] TestAgent: Requesting shutdown +30:47.225 [INFO] TestAgent.server: shutting down server +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:47.225 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:47.225 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:47.225 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:47.225 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:47.237 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:47.245Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:47.245Z [INFO] agent.router.manager: shutting down +30:47.245 [INFO] TestAgent: consul server down +30:47.245 [INFO] TestAgent: shutdown complete +30:47.245 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp +30:47.245 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp +30:47.245 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp +30:47.748 [INFO] TestAgent: Waiting for endpoints to shut down +30:47.748 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyReadCommand (1.12s) +2021-10-05T16:30:46.770Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:46.849Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:46.929 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:606e5d9c-cadf-8e6f-4e99-9644882902c2 Address:127.0.0.1:15111}]" +30:46.930 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-606e5d9c-cadf-8e6f-4e99-9644882902c2.dc1 127.0.0.1 +30:46.931 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 127.0.0.1 +2021-10-05T16:30:46.931Z [INFO] agent.router: Initializing LAN area manager +30:46.931 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +30:46.931 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +30:46.931 [INFO] TestAgent.server: Adding LAN server: server="Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +30:46.931 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2.dc1 area=wan +30:46.932 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +30:46.932 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +30:46.932 [INFO] TestAgent: started state syncer +30:46.932 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +30:46.976 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:46.976 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +30:46.982 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:46.988 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:46.988 [DEBUG] TestAgent.server.raft: vote granted: from=606e5d9c-cadf-8e6f-4e99-9644882902c2 term=2 tally=1 +30:46.988 [INFO] TestAgent.server.raft: election won: tally=1 +30:46.988 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +30:46.988 [INFO] TestAgent.server: cluster leadership acquired +30:46.988 [INFO] TestAgent.server: New leader elected: payload=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 +30:46.995 [INFO] TestAgent.server: initializing acls +30:46.999 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:46.999 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.016 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.040 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.040 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:47.040 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:47.040 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 +30:47.040 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-606e5d9c-cadf-8e6f-4e99-9644882902c2.dc1 +30:47.041 [INFO] TestAgent.server: Updating LAN server: server="Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +30:47.041 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2.dc1 area=wan +30:47.068 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:47.118 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:47.118 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:47.118 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:47.118 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:47.118 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:47.118 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 +30:47.118 [INFO] TestAgent.server: member joined, marking health alive: member=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 +30:47.129 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-606e5d9c-cadf-8e6f-4e99-9644882902c2 +30:47.142 [INFO] TestAgent.server: federation state anti-entropy synced +30:47.339 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:54702 latency=24.121809ms +30:47.340 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:47.345 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/db3a47ad-fe57-a786-a892-61f70aedb0b6 from=127.0.0.1:54726 latency=1.83276ms +30:47.347 [INFO] TestAgent: Requesting shutdown +30:47.347 [INFO] TestAgent.server: shutting down server +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:47.347 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:47.347 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:47.347 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:47.347 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:47.356 [INFO] TestAgent: Synced node info +30:47.356 [DEBUG] TestAgent: Node info in sync +30:47.360 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:47.367Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:47.367Z [INFO] agent.router.manager: shutting down +30:47.367 [INFO] TestAgent: consul server down +30:47.367 [INFO] TestAgent: shutdown complete +30:47.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +30:47.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +30:47.367 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +30:47.868 [INFO] TestAgent: Waiting for endpoints to shut down +30:47.868 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyReadCommand_JSON (1.24s) PASS -ok github.com/hashicorp/consul/command/acl/policy/read 0.848s +ok github.com/hashicorp/consul/command/acl/policy/read 1.558s === RUN TestPolicyUpdateCommand_noTabs === PAUSE TestPolicyUpdateCommand_noTabs === RUN TestPolicyUpdateCommand @@ -75515,207 +75768,220 @@ === RUN TestPolicyUpdateCommand_JSON === PAUSE TestPolicyUpdateCommand_JSON === CONT TestPolicyUpdateCommand_noTabs +--- PASS: TestPolicyUpdateCommand_noTabs (0.00s) === CONT TestPolicyUpdateCommand_JSON -=== CONT TestPolicyUpdateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestPolicyUpdateCommand_noTabs (0.00s) +=== CONT TestPolicyUpdateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:11.657Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:11.668Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:11.710 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a Address:127.0.0.1:12055}]" -45:11.710 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12055 [Follower]" leader= -45:11.710 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a.dc1 127.0.0.1 -45:11.712 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a 127.0.0.1 -2022-11-07T22:45:11.712Z [INFO] agent.router: Initializing LAN area manager -45:11.713 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a.dc1 area=wan -45:11.713 [INFO] TestAgent.server: Adding LAN server: server="Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a (Addr: tcp/127.0.0.1:12055) (DC: dc1)" -45:11.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=tcp -45:11.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=udp -45:11.713 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12051 network=tcp -45:11.713 [INFO] TestAgent: started state syncer -45:11.713 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12056 network=tcp -45:11.752 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:11.752 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12055 [Candidate]" term=2 -45:11.763 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:11.765 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:11.765 [DEBUG] TestAgent.server.raft: vote granted: from=ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a term=2 tally=1 -45:11.765 [INFO] TestAgent.server.raft: election won: tally=1 -45:11.765 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12055 [Leader]" -45:11.765 [INFO] TestAgent.server: cluster leadership acquired -45:11.765 [INFO] TestAgent.server: New leader elected: payload=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a -45:11.775 [INFO] TestAgent.server: initializing acls -45:11.779 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:11.779 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:11.785 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:11.797 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:11.797 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:11.797 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:11.797 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a -45:11.797 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a.dc1 -45:11.797 [INFO] TestAgent.server: Updating LAN server: server="Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a (Addr: tcp/127.0.0.1:12055) (DC: dc1)" -45:11.797 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a.dc1 area=wan -45:11.810 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:11.835 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:11.835 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:11.835 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:11.835 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:11.835 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:11.835 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a -45:11.835 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a -45:11.846 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ed2fd31b-ebb0-0c73-dcdf-23a106a2cf2a -45:11.851 [INFO] TestAgent.server: federation state anti-entropy synced -45:12.001 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:12.006 [INFO] TestAgent: Synced node info -45:12.032 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47956 latency=3.781523ms -45:12.033 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/7504a682-2223-9bc6-6258-3dca6b99aa79 from=127.0.0.1:47958 latency=236.1µs -45:12.038 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/7504a682-2223-9bc6-6258-3dca6b99aa79 from=127.0.0.1:47958 latency=4.098951ms -45:12.038 [INFO] TestAgent: Requesting shutdown -45:12.038 [INFO] TestAgent.server: shutting down server -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:12.038 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:12.038 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:12.038 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:12.038 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:12.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:12.049Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:12.049Z [INFO] agent.router.manager: shutting down -45:12.049 [INFO] TestAgent: consul server down -45:12.049 [INFO] TestAgent: shutdown complete -45:12.049 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=tcp -45:12.049 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=udp -45:12.049 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12051 network=tcp -45:12.549 [INFO] TestAgent: Waiting for endpoints to shut down -45:12.549 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyUpdateCommand (0.93s) -2022-11-07T22:45:11.658Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:11.669Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:11.713 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3a0210f5-bdc2-4d21-2471-054efc70ac18 Address:127.0.0.1:12048}]" -45:11.713 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= -45:11.714 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3a0210f5-bdc2-4d21-2471-054efc70ac18.dc1 127.0.0.1 -45:11.714 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 127.0.0.1 -2022-11-07T22:45:11.714Z [INFO] agent.router: Initializing LAN area manager -45:11.714 [INFO] TestAgent.server: Adding LAN server: server="Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" -45:11.714 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18.dc1 area=wan -45:11.715 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp -45:11.715 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp -45:11.715 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp -45:11.715 [INFO] TestAgent: started state syncer -45:11.715 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp -45:11.765 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:11.772 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:11.772 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 -45:11.787 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:11.787 [DEBUG] TestAgent.server.raft: vote granted: from=3a0210f5-bdc2-4d21-2471-054efc70ac18 term=2 tally=1 -45:11.787 [INFO] TestAgent.server.raft: election won: tally=1 -45:11.787 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" -45:11.787 [INFO] TestAgent.server: cluster leadership acquired -45:11.787 [INFO] TestAgent.server: New leader elected: payload=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 -45:11.797 [INFO] TestAgent.server: initializing acls -45:11.800 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:11.800 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:11.806 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:11.820 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:11.820 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:11.820 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:11.820 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 -45:11.820 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-3a0210f5-bdc2-4d21-2471-054efc70ac18.dc1 -45:11.821 [INFO] TestAgent.server: Updating LAN server: server="Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" -45:11.821 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18.dc1 area=wan -45:11.834 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:11.855 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:11.855 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:11.855 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:11.855 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:11.855 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:11.855 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 -45:11.855 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 -45:11.861 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3a0210f5-bdc2-4d21-2471-054efc70ac18 -45:11.868 [INFO] TestAgent.server: federation state anti-entropy synced -45:11.929 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:11.936 [INFO] TestAgent: Synced node info -45:11.936 [DEBUG] TestAgent: Node info in sync -45:11.963 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:11.963 [DEBUG] TestAgent: Node info in sync -45:12.054 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:37586 latency=6.002184ms -45:12.058 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/f348802c-3bdd-70db-9b04-95b1e12ff78c from=127.0.0.1:37588 latency=3.403413ms -45:12.065 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/f348802c-3bdd-70db-9b04-95b1e12ff78c from=127.0.0.1:37588 latency=4.636043ms -45:12.065 [INFO] TestAgent: Requesting shutdown -45:12.065 [INFO] TestAgent.server: shutting down server -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:12.065 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:12.065 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:12.065 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:12.065 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:12.072 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:12.076Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:12.076Z [INFO] agent.router.manager: shutting down -45:12.076 [INFO] TestAgent: consul server down -45:12.076 [INFO] TestAgent: shutdown complete -45:12.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp -45:12.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp -45:12.076 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp -45:12.577 [INFO] TestAgent: Waiting for endpoints to shut down -45:12.577 [INFO] TestAgent: Endpoints down ---- PASS: TestPolicyUpdateCommand_JSON (0.95s) +2021-10-05T16:30:47.376Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:47.390Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:47.431 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15390cc5-6d80-e55e-4abe-730910848a06 Address:127.0.0.1:31447}]" +30:47.432 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= +30:47.432 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 127.0.0.1 +30:47.433 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15390cc5-6d80-e55e-4abe-730910848a06 127.0.0.1 +2021-10-05T16:30:47.433Z [INFO] agent.router: Initializing LAN area manager +30:47.433 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp +30:47.433 [INFO] TestAgent.server: Adding LAN server: server="Node-15390cc5-6d80-e55e-4abe-730910848a06 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" +30:47.433 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 area=wan +30:47.433 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp +30:47.434 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp +30:47.434 [INFO] TestAgent: started state syncer +30:47.434 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp +30:47.484 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:47.500 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:47.500 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 +30:47.552 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:47.552 [DEBUG] TestAgent.server.raft: vote granted: from=15390cc5-6d80-e55e-4abe-730910848a06 term=2 tally=1 +30:47.552 [INFO] TestAgent.server.raft: election won: tally=1 +30:47.552 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" +30:47.554 [INFO] TestAgent.server: cluster leadership acquired +30:47.554 [INFO] TestAgent.server: New leader elected: payload=Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.568 [INFO] TestAgent.server: initializing acls +30:47.573 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:47.573 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.580 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.584 [INFO] TestAgent.server: initializing acls +30:47.584 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.603 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.603 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:47.603 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:47.603 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:47.604 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.604 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.604 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 +30:47.604 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 area=wan +30:47.604 [INFO] TestAgent.server: Updating LAN server: server="Node-15390cc5-6d80-e55e-4abe-730910848a06 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" +30:47.604 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.604 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 +30:47.604 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-15390cc5-6d80-e55e-4abe-730910848a06.dc1 area=wan +30:47.605 [INFO] TestAgent.server: Updating LAN server: server="Node-15390cc5-6d80-e55e-4abe-730910848a06 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" +30:47.622 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:47.624 [INFO] TestAgent: Synced node info +30:47.666 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:47.666 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:47.666 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:47.666 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:47.666 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:47.666 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.666 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.672 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.672 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15390cc5-6d80-e55e-4abe-730910848a06 +30:47.680 [INFO] TestAgent.server: federation state anti-entropy synced +30:47.707 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44854 latency=22.95165ms +30:47.711 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/75a24662-586b-6d6f-747e-05b540f11bcc from=127.0.0.1:44978 latency=742.106µs +30:47.715 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/75a24662-586b-6d6f-747e-05b540f11bcc from=127.0.0.1:44978 latency=3.90454ms +30:47.716 [INFO] TestAgent: Requesting shutdown +30:47.716 [INFO] TestAgent.server: shutting down server +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:47.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:47.716 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:47.716 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:47.720 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:47.727Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:47.727Z [INFO] agent.router.manager: shutting down +30:47.728 [INFO] TestAgent: consul server down +30:47.728 [INFO] TestAgent: shutdown complete +30:47.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp +30:47.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp +30:47.728 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp +30:48.232 [INFO] TestAgent: Waiting for endpoints to shut down +30:48.232 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyUpdateCommand_JSON (0.94s) +2021-10-05T16:30:47.406Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:47.442Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:47.507 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:529bd658-c5cc-b098-ffa7-0e97d9594c64 Address:127.0.0.1:31454}]" +30:47.508 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31454 [Follower]" leader= +30:47.508 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 127.0.0.1 +30:47.517 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 127.0.0.1 +2021-10-05T16:30:47.517Z [INFO] agent.router: Initializing LAN area manager +30:47.523 [INFO] TestAgent.server: Adding LAN server: server="Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 (Addr: tcp/127.0.0.1:31454) (DC: dc1)" +30:47.523 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 area=wan +30:47.523 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=tcp +30:47.524 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=udp +30:47.525 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31450 network=tcp +30:47.525 [INFO] TestAgent: started state syncer +30:47.527 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31455 network=tcp +30:47.560 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:47.560 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31454 [Candidate]" term=2 +30:47.572 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:47.572 [DEBUG] TestAgent.server.raft: vote granted: from=529bd658-c5cc-b098-ffa7-0e97d9594c64 term=2 tally=1 +30:47.572 [INFO] TestAgent.server.raft: election won: tally=1 +30:47.572 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31454 [Leader]" +30:47.572 [INFO] TestAgent.server: cluster leadership acquired +30:47.572 [INFO] TestAgent.server: New leader elected: payload=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.574 [INFO] TestAgent.server: initializing acls +30:47.580 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:47.580 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.580 [INFO] TestAgent.server: initializing acls +30:47.580 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:47.590 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.605 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:47.610 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.610 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:47.610 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:47.610 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:47.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.611 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 +30:47.611 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 area=wan +30:47.611 [INFO] TestAgent.server: Updating LAN server: server="Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 (Addr: tcp/127.0.0.1:31454) (DC: dc1)" +30:47.615 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:47.615 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.615 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 +30:47.615 [INFO] TestAgent.server: Updating LAN server: server="Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 (Addr: tcp/127.0.0.1:31454) (DC: dc1)" +30:47.615 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64.dc1 area=wan +30:47.630 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:47.652 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:47.652 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:47.652 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:47.652 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:47.652 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:47.652 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.652 [INFO] TestAgent.server: member joined, marking health alive: member=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.671 [INFO] TestAgent.server: federation state anti-entropy synced +30:47.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.679 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-529bd658-c5cc-b098-ffa7-0e97d9594c64 +30:47.697 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:45112 latency=6.123685ms +30:47.703 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/policy/a4e976f7-b181-14fc-d78a-311e4024c148 from=127.0.0.1:45126 latency=1.96208ms +30:47.714 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy/a4e976f7-b181-14fc-d78a-311e4024c148 from=127.0.0.1:45126 latency=4.728397ms +30:47.716 [INFO] TestAgent: Requesting shutdown +30:47.716 [INFO] TestAgent.server: shutting down server +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:47.716 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:47.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:47.717 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:47.717 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:47.717 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:47.728 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:47.735Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:47.735Z [INFO] agent.router.manager: shutting down +30:47.736 [INFO] TestAgent: consul server down +30:47.736 [INFO] TestAgent: shutdown complete +30:47.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=tcp +30:47.736 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=udp +30:47.736 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31450 network=tcp +30:48.240 [INFO] TestAgent: Waiting for endpoints to shut down +30:48.240 [INFO] TestAgent: Endpoints down +--- PASS: TestPolicyUpdateCommand (0.95s) PASS -ok github.com/hashicorp/consul/command/acl/policy/update 0.981s +ok github.com/hashicorp/consul/command/acl/policy/update 1.094s === RUN TestFormatRole === RUN TestFormatRole/basic -=== RUN TestFormatRole/basic/pretty === RUN TestFormatRole/basic/pretty-meta === RUN TestFormatRole/basic/json +=== RUN TestFormatRole/basic/pretty === RUN TestFormatRole/complex +=== RUN TestFormatRole/complex/json === RUN TestFormatRole/complex/pretty === RUN TestFormatRole/complex/pretty-meta -=== RUN TestFormatRole/complex/json --- PASS: TestFormatRole (0.00s) --- PASS: TestFormatRole/basic (0.00s) - --- PASS: TestFormatRole/basic/pretty (0.00s) --- PASS: TestFormatRole/basic/pretty-meta (0.00s) --- PASS: TestFormatRole/basic/json (0.00s) + --- PASS: TestFormatRole/basic/pretty (0.00s) --- PASS: TestFormatRole/complex (0.00s) + --- PASS: TestFormatRole/complex/json (0.00s) --- PASS: TestFormatRole/complex/pretty (0.00s) --- PASS: TestFormatRole/complex/pretty-meta (0.00s) - --- PASS: TestFormatRole/complex/json (0.00s) === RUN TestFormatTokenList === RUN TestFormatTokenList/basic +=== RUN TestFormatTokenList/basic/pretty-meta === RUN TestFormatTokenList/basic/json === RUN TestFormatTokenList/basic/pretty -=== RUN TestFormatTokenList/basic/pretty-meta === RUN TestFormatTokenList/complex === RUN TestFormatTokenList/complex/pretty === RUN TestFormatTokenList/complex/pretty-meta === RUN TestFormatTokenList/complex/json --- PASS: TestFormatTokenList (0.00s) --- PASS: TestFormatTokenList/basic (0.00s) + --- PASS: TestFormatTokenList/basic/pretty-meta (0.00s) --- PASS: TestFormatTokenList/basic/json (0.00s) --- PASS: TestFormatTokenList/basic/pretty (0.00s) - --- PASS: TestFormatTokenList/basic/pretty-meta (0.00s) --- PASS: TestFormatTokenList/complex (0.00s) --- PASS: TestFormatTokenList/complex/pretty (0.00s) --- PASS: TestFormatTokenList/complex/pretty-meta (0.00s) --- PASS: TestFormatTokenList/complex/json (0.00s) PASS -ok github.com/hashicorp/consul/command/acl/role 0.010s +ok github.com/hashicorp/consul/command/acl/role 0.016s === RUN TestRoleCreateCommand_noTabs === PAUSE TestRoleCreateCommand_noTabs === RUN TestRoleCreateCommand_Pretty @@ -75724,9 +75990,9 @@ === PAUSE TestRoleCreateCommand_JSON === CONT TestRoleCreateCommand_noTabs === CONT TestRoleCreateCommand_JSON -=== CONT TestRoleCreateCommand_Pretty -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... --- PASS: TestRoleCreateCommand_noTabs (0.00s) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestRoleCreateCommand_Pretty [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestRoleCreateCommand_Pretty/policy-name @@ -75734,166 +76000,168 @@ === RUN TestRoleCreateCommand_Pretty/service-identity === RUN TestRoleCreateCommand_Pretty/dc-scoped-service-identity === RUN TestRoleCreateCommand_Pretty/node-identity -2022-11-07T22:45:14.223Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:14.233Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:14.264 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88aafcc1-a0b3-bb93-5fb2-ac100fc04837 Address:127.0.0.1:30426}]" -45:14.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= -45:14.266 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837.dc1 127.0.0.1 -45:14.267 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 127.0.0.1 -2022-11-07T22:45:14.267Z [INFO] agent.router: Initializing LAN area manager -45:14.267 [INFO] TestAgent.server: Adding LAN server: server="Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:14.267 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837.dc1 area=wan -45:14.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp -45:14.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp -45:14.267 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp -45:14.268 [INFO] TestAgent: started state syncer -45:14.268 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp -45:14.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:14.333 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:14.333 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 -45:14.346 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:14.346 [DEBUG] TestAgent.server.raft: vote granted: from=88aafcc1-a0b3-bb93-5fb2-ac100fc04837 term=2 tally=1 -45:14.346 [INFO] TestAgent.server.raft: election won: tally=1 -45:14.346 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" -45:14.346 [INFO] TestAgent.server: cluster leadership acquired -45:14.346 [INFO] TestAgent.server: New leader elected: payload=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 -45:14.361 [INFO] TestAgent.server: initializing acls -45:14.366 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:14.366 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:14.373 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:14.387 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:14.387 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:14.387 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:14.387 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 -45:14.387 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837.dc1 -45:14.387 [INFO] TestAgent.server: Updating LAN server: server="Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:14.387 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837.dc1 area=wan -45:14.404 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:14.425 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:14.426 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:14.426 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:14.426 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:14.426 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:14.426 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 -45:14.426 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 -45:14.436 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88aafcc1-a0b3-bb93-5fb2-ac100fc04837 -45:14.442 [INFO] TestAgent.server: federation state anti-entropy synced -45:14.464 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:34996 latency=7.894763ms -45:14.471 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:34998 latency=3.760758ms -45:14.471 [INFO] TestAgent: Requesting shutdown -45:14.471 [INFO] TestAgent.server: shutting down server -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:14.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:14.471 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:14.471 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:14.471 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:14.471 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:14.475 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:14.483Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:14.483Z [INFO] agent.router.manager: shutting down -45:14.483 [INFO] TestAgent: consul server down -45:14.483 [INFO] TestAgent: shutdown complete -45:14.483 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp -45:14.483 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp -45:14.483 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp -45:14.983 [INFO] TestAgent: Waiting for endpoints to shut down -45:14.983 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleCreateCommand_JSON (0.80s) -2022-11-07T22:45:14.223Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:14.233Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:14.265 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:01309a93-cfc0-2d5a-cc29-21a2725ad05f Address:127.0.0.1:30433}]" -45:14.265 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30433 [Follower]" leader= -45:14.266 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f.dc1 127.0.0.1 -45:14.267 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f 127.0.0.1 -2022-11-07T22:45:14.267Z [INFO] agent.router: Initializing LAN area manager -45:14.267 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f.dc1 area=wan -45:14.267 [INFO] TestAgent.server: Adding LAN server: server="Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:14.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=tcp -45:14.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=udp -45:14.267 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30429 network=tcp -45:14.267 [INFO] TestAgent: started state syncer -45:14.268 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30434 network=tcp -45:14.305 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:14.305 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30433 [Candidate]" term=2 -45:14.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:14.318 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:14.318 [DEBUG] TestAgent.server.raft: vote granted: from=01309a93-cfc0-2d5a-cc29-21a2725ad05f term=2 tally=1 -45:14.318 [INFO] TestAgent.server.raft: election won: tally=1 -45:14.318 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30433 [Leader]" -45:14.318 [INFO] TestAgent.server: cluster leadership acquired -45:14.319 [INFO] TestAgent.server: New leader elected: payload=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f -45:14.328 [INFO] TestAgent.server: initializing acls -45:14.333 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:14.333 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:14.339 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:14.358 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:14.358 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:14.358 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:14.358 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f -45:14.359 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f.dc1 -45:14.359 [INFO] TestAgent.server: Updating LAN server: server="Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:14.359 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f.dc1 area=wan -45:14.372 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:14.397 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:14.397 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:14.397 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:14.397 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:14.397 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:14.398 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f -45:14.398 [INFO] TestAgent.server: member joined, marking health alive: member=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f -45:14.402 [INFO] TestAgent.server: federation state anti-entropy synced -45:14.411 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-01309a93-cfc0-2d5a-cc29-21a2725ad05f -45:14.471 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:14.477 [INFO] TestAgent: Synced node info -45:14.597 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36914 latency=4.364632ms -45:14.604 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36916 latency=5.878361ms -45:14.609 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36918 latency=4.343136ms -45:14.617 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36920 latency=5.472026ms -45:14.623 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36922 latency=4.514841ms -45:14.630 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36924 latency=5.646322ms -45:14.630 [INFO] TestAgent: Requesting shutdown -45:14.630 [INFO] TestAgent.server: shutting down server -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:14.630 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:14.630 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:14.630 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:14.630 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:14.637 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:14.643Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:14.643Z [INFO] agent.router.manager: shutting down -45:14.643 [INFO] TestAgent: consul server down -45:14.643 [INFO] TestAgent: shutdown complete -45:14.643 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=tcp -45:14.643 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=udp -45:14.643 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30429 network=tcp -45:15.143 [INFO] TestAgent: Waiting for endpoints to shut down -45:15.143 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleCreateCommand_Pretty (0.96s) +2021-10-05T16:30:55.174Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:55.190Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:55.264 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:daba52f5-46f6-7489-a6ed-c22fb04ae340 Address:127.0.0.1:14090}]" +30:55.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +30:55.265 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-daba52f5-46f6-7489-a6ed-c22fb04ae340.dc1 127.0.0.1 +30:55.266 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 127.0.0.1 +2021-10-05T16:30:55.266Z [INFO] agent.router: Initializing LAN area manager +30:55.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +30:55.267 [INFO] TestAgent.server: Adding LAN server: server="Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +30:55.267 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340.dc1 area=wan +30:55.267 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +30:55.268 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +30:55.268 [INFO] TestAgent: started state syncer +30:55.268 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +30:55.311 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:55.311 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +30:55.318 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:55.326 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:55.326 [DEBUG] TestAgent.server.raft: vote granted: from=daba52f5-46f6-7489-a6ed-c22fb04ae340 term=2 tally=1 +30:55.326 [INFO] TestAgent.server.raft: election won: tally=1 +30:55.326 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +30:55.326 [INFO] TestAgent.server: cluster leadership acquired +30:55.326 [INFO] TestAgent.server: New leader elected: payload=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 +30:55.343 [INFO] TestAgent.server: initializing acls +30:55.349 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:55.349 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:55.357 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:55.380 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:55.380 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:55.380 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:55.380 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 +30:55.380 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-daba52f5-46f6-7489-a6ed-c22fb04ae340.dc1 +30:55.384 [INFO] TestAgent.server: Updating LAN server: server="Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +30:55.384 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340.dc1 area=wan +30:55.444 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:55.508 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:55.508 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:55.508 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:55.508 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:55.508 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:55.508 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 +30:55.508 [INFO] TestAgent.server: member joined, marking health alive: member=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 +30:55.528 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-daba52f5-46f6-7489-a6ed-c22fb04ae340 +30:55.544 [INFO] TestAgent.server: federation state anti-entropy synced +30:55.544 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:55.576 [INFO] TestAgent: Synced node info +30:55.584 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44442 latency=19.644555ms +30:55.622 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44446 latency=29.8008ms +30:55.640 [INFO] TestAgent: Requesting shutdown +30:55.648 [INFO] TestAgent.server: shutting down server +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:55.648 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:55.648 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:55.648 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:55.648 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:55.657 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:55.658Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:55.658Z [INFO] agent.router.manager: shutting down +30:55.658 [INFO] TestAgent: consul server down +30:55.658 [INFO] TestAgent: shutdown complete +30:55.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +30:55.658 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +30:55.658 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +30:56.159 [INFO] TestAgent: Waiting for endpoints to shut down +30:56.159 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleCreateCommand_JSON (1.05s) +2021-10-05T16:30:55.179Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:55.197Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:55.221 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c924c50f-b9e1-80ac-c2a1-4db36f32a8ba Address:127.0.0.1:14097}]" +30:55.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba.dc1 127.0.0.1 +30:55.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba 127.0.0.1 +2021-10-05T16:30:55.222Z [INFO] agent.router: Initializing LAN area manager +30:55.228 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= +30:55.233 [INFO] TestAgent.server: Adding LAN server: server="Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +30:55.233 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba.dc1 area=wan +30:55.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp +30:55.234 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp +30:55.235 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp +30:55.235 [INFO] TestAgent: started state syncer +30:55.240 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp +30:55.284 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:55.296 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:55.296 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 +30:55.305 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:55.305 [DEBUG] TestAgent.server.raft: vote granted: from=c924c50f-b9e1-80ac-c2a1-4db36f32a8ba term=2 tally=1 +30:55.305 [INFO] TestAgent.server.raft: election won: tally=1 +30:55.305 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" +30:55.305 [INFO] TestAgent.server: cluster leadership acquired +30:55.305 [INFO] TestAgent.server: New leader elected: payload=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba +30:55.311 [INFO] TestAgent.server: initializing acls +30:55.316 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:55.316 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:55.325 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:55.345 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:55.345 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:55.345 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:55.345 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba +30:55.345 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba.dc1 +30:55.345 [INFO] TestAgent.server: Updating LAN server: server="Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +30:55.345 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba.dc1 area=wan +30:55.360 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:55.416 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:55.416 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:55.416 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:55.416 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:55.416 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:55.416 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba +30:55.416 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba +30:55.481 [INFO] TestAgent.server: federation state anti-entropy synced +30:55.484 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c924c50f-b9e1-80ac-c2a1-4db36f32a8ba +30:55.524 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:55.544 [INFO] TestAgent: Synced node info +30:55.544 [DEBUG] TestAgent: Node info in sync +30:55.587 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:59764 latency=13.849974ms +30:55.597 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59768 latency=5.348469ms +30:55.647 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59770 latency=28.869241ms +30:55.667 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59772 latency=14.258961ms +30:55.674 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59774 latency=4.979876ms +30:55.696 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:59776 latency=12.694039ms +30:55.699 [INFO] TestAgent: Requesting shutdown +30:55.699 [INFO] TestAgent.server: shutting down server +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:55.699 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:55.699 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:55.699 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:55.699 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:55.704 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:55.705Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:55.705Z [INFO] agent.router.manager: shutting down +30:55.705 [INFO] TestAgent: consul server down +30:55.705 [INFO] TestAgent: shutdown complete +30:55.705 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp +30:55.705 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp +30:55.705 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp +30:56.206 [INFO] TestAgent: Waiting for endpoints to shut down +30:56.206 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleCreateCommand_Pretty (1.09s) --- PASS: TestRoleCreateCommand_Pretty/policy-name (0.01s) - --- PASS: TestRoleCreateCommand_Pretty/policy-id (0.01s) - --- PASS: TestRoleCreateCommand_Pretty/service-identity (0.01s) + --- PASS: TestRoleCreateCommand_Pretty/policy-id (0.05s) + --- PASS: TestRoleCreateCommand_Pretty/service-identity (0.02s) --- PASS: TestRoleCreateCommand_Pretty/dc-scoped-service-identity (0.01s) - --- PASS: TestRoleCreateCommand_Pretty/node-identity (0.01s) + --- PASS: TestRoleCreateCommand_Pretty/node-identity (0.02s) PASS -ok github.com/hashicorp/consul/command/acl/role/create 0.985s +ok github.com/hashicorp/consul/command/acl/role/create 1.466s === RUN TestRoleDeleteCommand_noTabs === PAUSE TestRoleDeleteCommand_noTabs === RUN TestRoleDeleteCommand @@ -75907,90 +76175,92 @@ === RUN TestRoleDeleteCommand/id_or_name_required === RUN TestRoleDeleteCommand/delete_works === RUN TestRoleDeleteCommand/delete_works_via_prefixes -2022-11-07T22:45:14.281Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:14.292Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:14.321 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b87471a7-bd4e-f975-357f-4ad72f63acec Address:127.0.0.1:16132}]" -45:14.321 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:16132 [Follower]" leader= -45:14.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b87471a7-bd4e-f975-357f-4ad72f63acec.dc1 127.0.0.1 -45:14.322 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b87471a7-bd4e-f975-357f-4ad72f63acec 127.0.0.1 -2022-11-07T22:45:14.323Z [INFO] agent.router: Initializing LAN area manager -45:14.323 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b87471a7-bd4e-f975-357f-4ad72f63acec.dc1 area=wan -45:14.323 [INFO] TestAgent.server: Adding LAN server: server="Node-b87471a7-bd4e-f975-357f-4ad72f63acec (Addr: tcp/127.0.0.1:16132) (DC: dc1)" -45:14.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=udp -45:14.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=tcp -45:14.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:16128 network=tcp -45:14.323 [INFO] TestAgent: started state syncer -45:14.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:16133 network=tcp -45:14.369 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:14.369 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:16132 [Candidate]" term=2 -45:14.373 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:14.383 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:14.383 [DEBUG] TestAgent.server.raft: vote granted: from=b87471a7-bd4e-f975-357f-4ad72f63acec term=2 tally=1 -45:14.383 [INFO] TestAgent.server.raft: election won: tally=1 -45:14.383 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:16132 [Leader]" -45:14.383 [INFO] TestAgent.server: cluster leadership acquired -45:14.383 [INFO] TestAgent.server: New leader elected: payload=Node-b87471a7-bd4e-f975-357f-4ad72f63acec -45:14.391 [INFO] TestAgent.server: initializing acls -45:14.397 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:14.397 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:14.403 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:14.416 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:14.416 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:14.416 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:14.417 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b87471a7-bd4e-f975-357f-4ad72f63acec -45:14.417 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b87471a7-bd4e-f975-357f-4ad72f63acec.dc1 -45:14.417 [INFO] TestAgent.server: Updating LAN server: server="Node-b87471a7-bd4e-f975-357f-4ad72f63acec (Addr: tcp/127.0.0.1:16132) (DC: dc1)" -45:14.417 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b87471a7-bd4e-f975-357f-4ad72f63acec.dc1 area=wan -45:14.428 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:14.455 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:14.455 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:14.455 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:14.455 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:14.455 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:14.455 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b87471a7-bd4e-f975-357f-4ad72f63acec -45:14.455 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b87471a7-bd4e-f975-357f-4ad72f63acec -45:14.469 [INFO] TestAgent.server: federation state anti-entropy synced -45:14.469 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b87471a7-bd4e-f975-357f-4ad72f63acec -45:14.483 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:33120 latency=9.087945ms -45:14.488 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/930c9d16-dd5b-366f-5a74-ae184bffca81 from=127.0.0.1:33122 latency=4.508028ms -45:14.489 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/930c9d16-dd5b-366f-5a74-ae184bffca81 from=127.0.0.1:33120 latency=77.95µs -45:14.495 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:33120 latency=5.68229ms -45:14.496 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:33124 latency=208.231µs -45:14.501 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/30de7ee8-5099-456b-8936-34b352464ac1 from=127.0.0.1:33124 latency=4.774079ms -45:14.502 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/30de7ee8-5099-456b-8936-34b352464ac1 from=127.0.0.1:33120 latency=44.147µs -45:14.502 [INFO] TestAgent: Requesting shutdown -45:14.502 [INFO] TestAgent.server: shutting down server -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:14.502 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:14.502 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:14.502 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:14.502 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:14.502 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:14.506 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:14.510Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:14.510Z [INFO] agent.router.manager: shutting down -45:14.510 [INFO] TestAgent: consul server down -45:14.510 [INFO] TestAgent: shutdown complete -45:14.510 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=tcp -45:14.510 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=udp -45:14.510 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:16128 network=tcp -45:15.010 [INFO] TestAgent: Waiting for endpoints to shut down -45:15.010 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleDeleteCommand (0.77s) +2021-10-05T16:30:57.261Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:57.293Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:57.327 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c Address:127.0.0.1:13069}]" +30:57.328 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13069 [Follower]" leader= +30:57.328 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c.dc1 127.0.0.1 +30:57.329 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c 127.0.0.1 +2021-10-05T16:30:57.330Z [INFO] agent.router: Initializing LAN area manager +30:57.331 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=udp +30:57.331 [INFO] TestAgent.server: Adding LAN server: server="Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c (Addr: tcp/127.0.0.1:13069) (DC: dc1)" +30:57.331 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c.dc1 area=wan +30:57.331 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=tcp +30:57.340 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13065 network=tcp +30:57.340 [INFO] TestAgent: started state syncer +30:57.340 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13070 network=tcp +30:57.382 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:57.386 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:57.386 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13069 [Candidate]" term=2 +30:57.405 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:57.405 [DEBUG] TestAgent.server.raft: vote granted: from=30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c term=2 tally=1 +30:57.405 [INFO] TestAgent.server.raft: election won: tally=1 +30:57.405 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13069 [Leader]" +30:57.407 [INFO] TestAgent.server: cluster leadership acquired +30:57.407 [INFO] TestAgent.server: New leader elected: payload=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c +30:57.411 [INFO] TestAgent.server: initializing acls +30:57.414 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:57.414 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:57.421 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:57.440 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:57.440 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:57.440 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:57.440 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c +30:57.440 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c.dc1 +30:57.441 [INFO] TestAgent.server: Updating LAN server: server="Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c (Addr: tcp/127.0.0.1:13069) (DC: dc1)" +30:57.441 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c.dc1 area=wan +30:57.469 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:57.489 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:57.489 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:57.489 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:57.489 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:57.489 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:57.489 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c +30:57.489 [INFO] TestAgent.server: member joined, marking health alive: member=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c +30:57.505 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-30c3e1cc-6695-5a4b-64f2-e9e1f05b5a5c +30:57.512 [INFO] TestAgent.server: federation state anti-entropy synced +30:57.637 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:57.646 [INFO] TestAgent: Synced node info +30:57.646 [DEBUG] TestAgent: Node info in sync +30:57.667 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:49752 latency=5.536997ms +30:57.693 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/726dbf5c-e9bf-b83c-f02b-0ac1a7d6fd3b from=127.0.0.1:49754 latency=16.776006ms +30:57.706 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/726dbf5c-e9bf-b83c-f02b-0ac1a7d6fd3b from=127.0.0.1:49752 latency=141.154µs +30:57.715 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:49752 latency=8.59584ms +30:57.721 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:49756 latency=1.495966ms +30:57.737 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/role/91aad024-4c16-206b-09d6-7b00d9a79813 from=127.0.0.1:49756 latency=12.519653ms +30:57.740 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/91aad024-4c16-206b-09d6-7b00d9a79813 from=127.0.0.1:49752 latency=119.228µs +30:57.744 [INFO] TestAgent: Requesting shutdown +30:57.744 [INFO] TestAgent.server: shutting down server +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:57.744 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:57.744 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:57.744 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:57.744 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:57.752 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:57.762Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:57.762Z [INFO] agent.router.manager: shutting down +30:57.762 [INFO] TestAgent: consul server down +30:57.762 [INFO] TestAgent: shutdown complete +30:57.762 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=tcp +30:57.762 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=udp +30:57.762 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13065 network=tcp +30:58.264 [INFO] TestAgent: Waiting for endpoints to shut down +30:58.264 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleDeleteCommand (1.12s) --- PASS: TestRoleDeleteCommand/id_or_name_required (0.00s) - --- PASS: TestRoleDeleteCommand/delete_works (0.02s) - --- PASS: TestRoleDeleteCommand/delete_works_via_prefixes (0.01s) + --- PASS: TestRoleDeleteCommand/delete_works (0.05s) + --- PASS: TestRoleDeleteCommand/delete_works_via_prefixes (0.04s) PASS -ok github.com/hashicorp/consul/command/acl/role/delete 0.793s +ok github.com/hashicorp/consul/command/acl/role/delete 1.223s === RUN TestRoleListCommand_noTabs === PAUSE TestRoleListCommand_noTabs === RUN TestRoleListCommand @@ -75998,180 +76268,181 @@ === RUN TestRoleListCommand_JSON === PAUSE TestRoleListCommand_JSON === CONT TestRoleListCommand_noTabs -=== CONT TestRoleListCommand_JSON -=== CONT TestRoleListCommand --- PASS: TestRoleListCommand_noTabs (0.00s) +=== CONT TestRoleListCommand_JSON [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestRoleListCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:16.660Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:16.670Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:16.693 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dedbb3c1-23e4-6718-bc09-243dee93db23 Address:127.0.0.1:10006}]" -45:16.693 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -45:16.693 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dedbb3c1-23e4-6718-bc09-243dee93db23.dc1 127.0.0.1 -45:16.694 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dedbb3c1-23e4-6718-bc09-243dee93db23 127.0.0.1 -2022-11-07T22:45:16.694Z [INFO] agent.router: Initializing LAN area manager -45:16.695 [INFO] TestAgent.server: Adding LAN server: server="Node-dedbb3c1-23e4-6718-bc09-243dee93db23 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:16.695 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dedbb3c1-23e4-6718-bc09-243dee93db23.dc1 area=wan -45:16.695 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -45:16.696 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -45:16.696 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -45:16.696 [INFO] TestAgent: started state syncer -45:16.696 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -45:16.743 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:16.743 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -45:16.745 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:16.752 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:16.752 [DEBUG] TestAgent.server.raft: vote granted: from=dedbb3c1-23e4-6718-bc09-243dee93db23 term=2 tally=1 -45:16.752 [INFO] TestAgent.server.raft: election won: tally=1 -45:16.752 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -45:16.752 [INFO] TestAgent.server: cluster leadership acquired -45:16.752 [INFO] TestAgent.server: New leader elected: payload=Node-dedbb3c1-23e4-6718-bc09-243dee93db23 -45:16.758 [INFO] TestAgent.server: initializing acls -45:16.761 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:16.762 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:16.768 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:16.776 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:16.776 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:16.776 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:16.776 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-dedbb3c1-23e4-6718-bc09-243dee93db23 -45:16.776 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-dedbb3c1-23e4-6718-bc09-243dee93db23.dc1 -45:16.777 [INFO] TestAgent.server: Updating LAN server: server="Node-dedbb3c1-23e4-6718-bc09-243dee93db23 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:16.777 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-dedbb3c1-23e4-6718-bc09-243dee93db23.dc1 area=wan -45:16.786 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:16.801 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:16.801 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:16.801 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:16.801 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:16.801 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:16.801 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dedbb3c1-23e4-6718-bc09-243dee93db23 -45:16.801 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dedbb3c1-23e4-6718-bc09-243dee93db23 -45:16.809 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dedbb3c1-23e4-6718-bc09-243dee93db23 -45:16.814 [INFO] TestAgent.server: federation state anti-entropy synced -45:16.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36610 latency=6.958339ms -45:16.830 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36610 latency=4.675242ms -45:16.834 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36610 latency=3.237621ms -45:16.838 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36610 latency=3.712132ms -45:16.842 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36610 latency=3.29007ms -45:16.843 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:36614 latency=801.441µs -45:16.844 [INFO] TestAgent: Requesting shutdown -45:16.844 [INFO] TestAgent.server: shutting down server -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:16.844 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:16.844 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:16.844 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:16.844 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:16.844 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:16.848 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:16.851Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:16.851Z [INFO] agent.router.manager: shutting down -45:16.851 [INFO] TestAgent: consul server down -45:16.851 [INFO] TestAgent: shutdown complete -45:16.851 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -45:16.851 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -45:16.851 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -45:17.352 [INFO] TestAgent: Waiting for endpoints to shut down -45:17.352 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleListCommand_JSON (0.73s) -2022-11-07T22:45:16.661Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:16.671Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:16.696 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c39a99b-4c98-b899-4833-8880bcbb7634 Address:127.0.0.1:10013}]" -45:16.696 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= -45:16.696 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 127.0.0.1 -45:16.697 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c39a99b-4c98-b899-4833-8880bcbb7634 127.0.0.1 -2022-11-07T22:45:16.697Z [INFO] agent.router: Initializing LAN area manager -45:16.697 [INFO] TestAgent.server: Adding LAN server: server="Node-0c39a99b-4c98-b899-4833-8880bcbb7634 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:16.697 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 area=wan -45:16.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp -45:16.697 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp -45:16.697 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp -45:16.697 [INFO] TestAgent: started state syncer -45:16.697 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp -45:16.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:16.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 -45:16.745 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:16.745 [DEBUG] TestAgent.server.raft: vote granted: from=0c39a99b-4c98-b899-4833-8880bcbb7634 term=2 tally=1 -45:16.745 [INFO] TestAgent.server.raft: election won: tally=1 -45:16.745 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" -45:16.745 [INFO] TestAgent.server: cluster leadership acquired -45:16.745 [INFO] TestAgent.server: New leader elected: payload=Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.747 [INFO] TestAgent.server: initializing acls -45:16.752 [INFO] TestAgent.server: initializing acls -45:16.752 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:16.752 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:16.757 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:16.757 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:16.766 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:16.772 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:16.772 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:16.772 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:16.772 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:16.772 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -45:16.772 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.772 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 -45:16.772 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 area=wan -45:16.772 [INFO] TestAgent.server: Updating LAN server: server="Node-0c39a99b-4c98-b899-4833-8880bcbb7634 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:16.776 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:16.776 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.776 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 -45:16.777 [INFO] TestAgent.server: Updating LAN server: server="Node-0c39a99b-4c98-b899-4833-8880bcbb7634 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:16.777 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-0c39a99b-4c98-b899-4833-8880bcbb7634.dc1 area=wan -45:16.787 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:16.802 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:16.802 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:16.802 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:16.802 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:16.802 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:16.802 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.802 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.810 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.810 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c39a99b-4c98-b899-4833-8880bcbb7634 -45:16.815 [INFO] TestAgent.server: federation state anti-entropy synced -45:16.838 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44830 latency=5.771242ms -45:16.843 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44830 latency=3.588178ms -45:16.846 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44830 latency=2.855966ms -45:16.849 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44830 latency=2.60834ms -45:16.853 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:44830 latency=3.027727ms -45:16.857 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:44834 latency=182.007µs -45:16.858 [INFO] TestAgent: Requesting shutdown -45:16.858 [INFO] TestAgent.server: shutting down server -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:16.858 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:16.858 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:16.858 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:16.858 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:16.858 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:16.862 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:16.865Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:16.865Z [INFO] agent.router.manager: shutting down -45:16.865 [INFO] TestAgent: consul server down -45:16.865 [INFO] TestAgent: shutdown complete -45:16.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp -45:16.865 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp -45:16.865 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp -45:17.365 [INFO] TestAgent: Waiting for endpoints to shut down -45:17.365 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleListCommand (0.74s) +2021-10-05T16:30:58.527Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:58.551Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:58.599 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc Address:127.0.0.1:18174}]" +30:58.600 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 127.0.0.1 +30:58.601 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc 127.0.0.1 +2021-10-05T16:30:58.601Z [INFO] agent.router: Initializing LAN area manager +30:58.601 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp +30:58.601 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= +30:58.601 [INFO] TestAgent.server: Adding LAN server: server="Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +30:58.602 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 area=wan +30:58.602 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp +30:58.602 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp +30:58.602 [INFO] TestAgent: started state syncer +30:58.603 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp +30:58.652 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:58.663 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:58.663 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 +30:58.682 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:58.682 [DEBUG] TestAgent.server.raft: vote granted: from=04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc term=2 tally=1 +30:58.682 [INFO] TestAgent.server.raft: election won: tally=1 +30:58.682 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" +30:58.684 [INFO] TestAgent.server: cluster leadership acquired +30:58.684 [INFO] TestAgent.server: New leader elected: payload=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.716 [INFO] TestAgent.server: initializing acls +30:58.725 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:58.725 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:58.737 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:58.752 [INFO] TestAgent.server: initializing acls +30:58.752 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:58.760 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:58.760 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:58.760 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:58.760 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.761 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 +30:58.761 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 area=wan +30:58.761 [INFO] TestAgent.server: Updating LAN server: server="Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +30:58.776 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:58.776 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +30:58.776 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.776 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 +30:58.776 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc.dc1 area=wan +30:58.777 [INFO] TestAgent.server: Updating LAN server: server="Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +30:58.793 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:58.815 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:58.815 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:58.815 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:58.815 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:58.815 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:58.815 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.815 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.819 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.819 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04ca7e8d-166b-dce4-13bc-fbbbd8ed40cc +30:58.832 [INFO] TestAgent.server: federation state anti-entropy synced +30:58.908 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +30:58.913 [INFO] TestAgent: Synced node info +30:59.007 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52894 latency=14.269454ms +30:59.011 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52894 latency=4.134212ms +30:59.024 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52894 latency=11.578257ms +30:59.048 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52894 latency=22.685379ms +30:59.055 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52894 latency=6.445151ms +30:59.056 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:52898 latency=344.891µs +30:59.057 [INFO] TestAgent: Requesting shutdown +30:59.057 [INFO] TestAgent.server: shutting down server +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:59.057 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:59.057 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:59.057 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:59.057 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:59.061 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:59.067Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:59.067Z [INFO] agent.router.manager: shutting down +30:59.067 [INFO] TestAgent: consul server down +30:59.067 [INFO] TestAgent: shutdown complete +30:59.067 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp +30:59.067 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp +30:59.067 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp +30:59.568 [INFO] TestAgent: Waiting for endpoints to shut down +30:59.568 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleListCommand_JSON (1.12s) +2021-10-05T16:30:58.530Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:30:58.557Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +30:58.608 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:06be176c-995a-975e-9b39-45eb14b5be0e Address:127.0.0.1:18181}]" +30:58.609 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-06be176c-995a-975e-9b39-45eb14b5be0e.dc1 127.0.0.1 +30:58.609 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-06be176c-995a-975e-9b39-45eb14b5be0e 127.0.0.1 +2021-10-05T16:30:58.609Z [INFO] agent.router: Initializing LAN area manager +30:58.610 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=udp +30:58.610 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18181 [Follower]" leader= +30:58.610 [INFO] TestAgent.server: Adding LAN server: server="Node-06be176c-995a-975e-9b39-45eb14b5be0e (Addr: tcp/127.0.0.1:18181) (DC: dc1)" +30:58.610 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-06be176c-995a-975e-9b39-45eb14b5be0e.dc1 area=wan +30:58.610 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=tcp +30:58.611 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18177 network=tcp +30:58.611 [INFO] TestAgent: started state syncer +30:58.611 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18182 network=tcp +30:58.652 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +30:58.652 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18181 [Candidate]" term=2 +30:58.663 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +30:58.681 [DEBUG] TestAgent.server.raft: votes: needed=1 +30:58.681 [DEBUG] TestAgent.server.raft: vote granted: from=06be176c-995a-975e-9b39-45eb14b5be0e term=2 tally=1 +30:58.681 [INFO] TestAgent.server.raft: election won: tally=1 +30:58.681 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18181 [Leader]" +30:58.684 [INFO] TestAgent.server: cluster leadership acquired +30:58.684 [INFO] TestAgent.server: New leader elected: payload=Node-06be176c-995a-975e-9b39-45eb14b5be0e +30:58.696 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:58.712 [INFO] TestAgent.server: initializing acls +30:58.713 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +30:58.716 [INFO] TestAgent.server: Created ACL 'global-management' policy +30:58.716 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +30:58.725 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +30:58.750 [INFO] TestAgent.server: Created ACL anonymous token from configuration +30:58.750 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +30:58.750 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +30:58.750 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-06be176c-995a-975e-9b39-45eb14b5be0e +30:58.750 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-06be176c-995a-975e-9b39-45eb14b5be0e.dc1 +30:58.751 [INFO] TestAgent.server: Updating LAN server: server="Node-06be176c-995a-975e-9b39-45eb14b5be0e (Addr: tcp/127.0.0.1:18181) (DC: dc1)" +30:58.751 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-06be176c-995a-975e-9b39-45eb14b5be0e.dc1 area=wan +30:58.767 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +30:58.788 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +30:58.788 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +30:58.788 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +30:58.788 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +30:58.788 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +30:58.788 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06be176c-995a-975e-9b39-45eb14b5be0e +30:58.788 [INFO] TestAgent.server: member joined, marking health alive: member=Node-06be176c-995a-975e-9b39-45eb14b5be0e +30:58.796 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-06be176c-995a-975e-9b39-45eb14b5be0e +30:58.812 [INFO] TestAgent.server: federation state anti-entropy synced +30:59.054 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:58464 latency=9.538874ms +30:59.072 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:58464 latency=16.180514ms +30:59.089 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:58464 latency=15.596078ms +30:59.112 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:58464 latency=16.329688ms +30:59.136 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:58464 latency=16.375233ms +30:59.141 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:58468 latency=736.828µs +30:59.142 [INFO] TestAgent: Requesting shutdown +30:59.142 [INFO] TestAgent.server: shutting down server +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +30:59.142 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +30:59.142 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +30:59.142 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +30:59.142 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +30:59.149 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:30:59.160Z [INFO] agent.router.manager: shutting down +2021-10-05T16:30:59.160Z [INFO] agent.router.manager: shutting down +30:59.160 [INFO] TestAgent: consul server down +30:59.160 [INFO] TestAgent: shutdown complete +30:59.160 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=tcp +30:59.160 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=udp +30:59.160 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18177 network=tcp +30:59.664 [INFO] TestAgent: Waiting for endpoints to shut down +30:59.664 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleListCommand (1.21s) PASS -ok github.com/hashicorp/consul/command/acl/role/list 0.771s +ok github.com/hashicorp/consul/command/acl/role/list 1.323s === RUN TestRoleReadCommand_noTabs === PAUSE TestRoleReadCommand_noTabs === RUN TestRoleReadCommand @@ -76180,202 +76451,194 @@ === PAUSE TestRoleReadCommand_JSON === CONT TestRoleReadCommand_noTabs === CONT TestRoleReadCommand_JSON +--- PASS: TestRoleReadCommand_noTabs (0.00s) === CONT TestRoleReadCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestRoleReadCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== RUN TestRoleReadCommand_JSON/read_by_id === RUN TestRoleReadCommand/id_or_name_required === RUN TestRoleReadCommand/read_by_id_not_found === RUN TestRoleReadCommand/read_by_name_not_found === RUN TestRoleReadCommand/read_by_id === RUN TestRoleReadCommand/read_by_id_prefix === RUN TestRoleReadCommand/read_by_name -=== RUN TestRoleReadCommand_JSON/read_by_id -2022-11-07T22:45:19.082Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.094Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.117 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b1d08e22-ecd7-37a3-2d91-db2924e53028 Address:127.0.0.1:30426}]" -45:19.117 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= -45:19.120 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 127.0.0.1 -45:19.121 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 127.0.0.1 -2022-11-07T22:45:19.121Z [INFO] agent.router: Initializing LAN area manager -45:19.121 [INFO] TestAgent.server: Adding LAN server: server="Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:19.121 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 area=wan -45:19.121 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp -45:19.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp -45:19.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp -45:19.122 [INFO] TestAgent: started state syncer -45:19.122 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp -45:19.159 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.159 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 -45:19.169 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.169 [DEBUG] TestAgent.server.raft: vote granted: from=b1d08e22-ecd7-37a3-2d91-db2924e53028 term=2 tally=1 -45:19.169 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.169 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" -45:19.169 [INFO] TestAgent.server: cluster leadership acquired -45:19.169 [INFO] TestAgent.server: New leader elected: payload=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.172 [INFO] TestAgent.server: initializing acls -45:19.176 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:19.176 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.176 [INFO] TestAgent.server: initializing acls -45:19.176 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.182 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.195 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.195 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.195 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:19.195 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:19.195 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -45:19.196 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.196 [INFO] TestAgent.server: Updating LAN server: server="Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:19.196 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 -45:19.196 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 area=wan -45:19.196 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.196 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 -45:19.196 [INFO] TestAgent.server: Updating LAN server: server="Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:19.196 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028.dc1 area=wan -45:19.204 [INFO] TestAgent: Synced node info -45:19.204 [DEBUG] TestAgent: Node info in sync -45:19.211 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.227 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.227 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.227 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.227 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.227 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.227 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.227 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.230 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b1d08e22-ecd7-37a3-2d91-db2924e53028 -45:19.385 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/507f9021-240b-9448-890c-c14c4cd03217 from=127.0.0.1:35030 latency=380.587µs -45:19.386 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/blah from=127.0.0.1:35032 latency=105.564µs -45:19.392 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:35034 latency=5.289114ms -45:19.394 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1d74bb6c-fbb7-9e49-c70c-fac5f4ac787c from=127.0.0.1:35036 latency=1.060514ms -45:19.398 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:35034 latency=3.311916ms -45:19.400 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:35038 latency=846.617µs -45:19.400 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/9b2c219e-38c1-37b2-811f-8a4d5ee1be9d from=127.0.0.1:35038 latency=161.433µs -45:19.404 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:35034 latency=3.13727ms -45:19.405 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/test-role-by-name from=127.0.0.1:35040 latency=195.35µs -45:19.405 [INFO] TestAgent: Requesting shutdown -45:19.405 [INFO] TestAgent.server: shutting down server -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.405 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.405 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.409 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.413Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.413Z [INFO] agent.router.manager: shutting down -45:19.413 [INFO] TestAgent: consul server down -45:19.413 [INFO] TestAgent: shutdown complete -45:19.413 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp -45:19.413 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp -45:19.413 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp -45:19.913 [INFO] TestAgent: Waiting for endpoints to shut down -45:19.913 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleReadCommand (0.87s) +2021-10-05T16:31:05.702Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:05.762Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:05.793 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd4d2b2d-6dd0-066c-474e-6cf07e5559ea Address:127.0.0.1:19202}]" +31:05.794 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 127.0.0.1 +31:05.795 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19202 [Follower]" leader= +31:05.795 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea 127.0.0.1 +2021-10-05T16:31:05.795Z [INFO] agent.router: Initializing LAN area manager +31:05.796 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 area=wan +31:05.796 [INFO] TestAgent.server: Adding LAN server: server="Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea (Addr: tcp/127.0.0.1:19202) (DC: dc1)" +31:05.796 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=udp +31:05.796 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=tcp +31:05.797 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19198 network=tcp +31:05.797 [INFO] TestAgent: started state syncer +31:05.797 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19203 network=tcp +31:05.848 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:05.864 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:05.864 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19202 [Candidate]" term=2 +31:05.912 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:05.912 [DEBUG] TestAgent.server.raft: vote granted: from=cd4d2b2d-6dd0-066c-474e-6cf07e5559ea term=2 tally=1 +31:05.912 [INFO] TestAgent.server.raft: election won: tally=1 +31:05.912 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19202 [Leader]" +31:05.912 [INFO] TestAgent.server: cluster leadership acquired +31:05.912 [INFO] TestAgent.server: New leader elected: payload=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:05.929 [INFO] TestAgent.server: initializing acls +31:05.935 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:05.935 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:05.943 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:05.948 [INFO] TestAgent.server: initializing acls +31:05.948 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:05.953 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:05.953 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:05.953 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:05.954 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:05.954 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 +31:05.954 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 area=wan +31:05.954 [INFO] TestAgent.server: Updating LAN server: server="Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea (Addr: tcp/127.0.0.1:19202) (DC: dc1)" +31:05.961 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:05.961 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:05.962 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:05.962 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 +31:05.962 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea.dc1 area=wan +31:05.962 [INFO] TestAgent.server: Updating LAN server: server="Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea (Addr: tcp/127.0.0.1:19202) (DC: dc1)" +31:05.986 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:06.029 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:06.029 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:06.029 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:06.029 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:06.029 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:06.029 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:06.029 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:06.035 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:06.035 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd4d2b2d-6dd0-066c-474e-6cf07e5559ea +31:06.041 [INFO] TestAgent.server: federation state anti-entropy synced +31:06.100 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:06.132 [INFO] TestAgent: Synced node info +31:06.141 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:52614 latency=52.883144ms +31:06.173 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/0b982207-7fc0-ac6e-c880-ece8988cda75 from=127.0.0.1:52626 latency=260.967µs +31:06.176 [INFO] TestAgent: Requesting shutdown +31:06.176 [INFO] TestAgent.server: shutting down server +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:06.176 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:06.176 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:06.176 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:06.176 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:06.185 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:06.196Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:06.196Z [INFO] agent.router.manager: shutting down +31:06.196 [INFO] TestAgent: consul server down +31:06.196 [INFO] TestAgent: shutdown complete +31:06.196 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=tcp +31:06.196 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=udp +31:06.196 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19198 network=tcp +31:06.700 [INFO] TestAgent: Waiting for endpoints to shut down +31:06.700 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleReadCommand_JSON (1.15s) + --- PASS: TestRoleReadCommand_JSON/read_by_id (0.12s) +2021-10-05T16:31:05.696Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:05.713Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:05.750 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:caa3a85b-d91e-ed36-268b-cdb294f6e08e Address:127.0.0.1:19195}]" +31:05.751 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e.dc1 127.0.0.1 +31:05.760 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= +31:05.761 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e 127.0.0.1 +2021-10-05T16:31:05.761Z [INFO] agent.router: Initializing LAN area manager +31:05.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=udp +31:05.761 [INFO] TestAgent.server: Adding LAN server: server="Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e (Addr: tcp/127.0.0.1:19195) (DC: dc1)" +31:05.761 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e.dc1 area=wan +31:05.761 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=tcp +31:05.762 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19191 network=tcp +31:05.762 [INFO] TestAgent: started state syncer +31:05.762 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19196 network=tcp +31:05.808 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:05.808 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 +31:05.812 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:05.832 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:05.832 [DEBUG] TestAgent.server.raft: vote granted: from=caa3a85b-d91e-ed36-268b-cdb294f6e08e term=2 tally=1 +31:05.832 [INFO] TestAgent.server.raft: election won: tally=1 +31:05.832 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" +31:05.836 [INFO] TestAgent.server: cluster leadership acquired +31:05.836 [INFO] TestAgent.server: New leader elected: payload=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e +31:05.844 [INFO] TestAgent.server: initializing acls +31:05.846 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:05.846 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:05.868 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:05.892 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:05.892 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:05.892 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:05.892 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e +31:05.893 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e.dc1 +31:05.896 [INFO] TestAgent.server: Updating LAN server: server="Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e (Addr: tcp/127.0.0.1:19195) (DC: dc1)" +31:05.896 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e.dc1 area=wan +31:05.928 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:05.955 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:05.955 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:05.955 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:05.955 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:05.955 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:05.955 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e +31:05.955 [INFO] TestAgent.server: member joined, marking health alive: member=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e +31:05.965 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-caa3a85b-d91e-ed36-268b-cdb294f6e08e +31:05.987 [INFO] TestAgent.server: federation state anti-entropy synced +31:06.046 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:06.055 [INFO] TestAgent: Synced node info +31:06.056 [DEBUG] TestAgent: Node info in sync +31:06.090 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/46a8b2e7-9bb5-d595-9ddd-56f6b8162f69 from=127.0.0.1:49736 latency=464.538µs +31:06.096 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/blah from=127.0.0.1:49738 latency=160.819µs +31:06.116 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:49740 latency=19.563465ms +31:06.125 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/b67e32d0-c609-8e54-5984-3a2c0a794995 from=127.0.0.1:49742 latency=5.857189ms +31:06.144 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:49740 latency=10.13788ms +31:06.152 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/roles from=127.0.0.1:49744 latency=574.051µs +31:06.172 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/db15b7dd-213e-416e-b0e6-3b875da86206 from=127.0.0.1:49744 latency=334.275µs +31:06.192 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:49740 latency=18.182784ms +31:06.201 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/name/test-role-by-name from=127.0.0.1:49748 latency=1.264097ms +31:06.202 [INFO] TestAgent: Requesting shutdown +31:06.202 [INFO] TestAgent.server: shutting down server +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:06.202 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:06.202 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:06.202 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:06.202 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:06.216 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:06.224Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:06.224Z [INFO] agent.router.manager: shutting down +31:06.224 [INFO] TestAgent: consul server down +31:06.224 [INFO] TestAgent: shutdown complete +31:06.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp +31:06.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp +31:06.224 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp +31:06.728 [INFO] TestAgent: Waiting for endpoints to shut down +31:06.728 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleReadCommand (1.18s) --- PASS: TestRoleReadCommand/id_or_name_required (0.00s) - --- PASS: TestRoleReadCommand/read_by_id_not_found (0.00s) - --- PASS: TestRoleReadCommand/read_by_name_not_found (0.00s) - --- PASS: TestRoleReadCommand/read_by_id (0.01s) - --- PASS: TestRoleReadCommand/read_by_id_prefix (0.01s) - --- PASS: TestRoleReadCommand/read_by_name (0.00s) -2022-11-07T22:45:19.084Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.094Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.119 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:beacf32f-c990-7b5f-9683-c0d2d62a2cde Address:127.0.0.1:30433}]" -45:19.119 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30433 [Follower]" leader= -45:19.120 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 127.0.0.1 -45:19.121 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde 127.0.0.1 -2022-11-07T22:45:19.121Z [INFO] agent.router: Initializing LAN area manager -45:19.121 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 area=wan -45:19.121 [INFO] TestAgent.server: Adding LAN server: server="Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:19.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=udp -45:19.122 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=tcp -45:19.122 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30429 network=tcp -45:19.122 [INFO] TestAgent: started state syncer -45:19.122 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30434 network=tcp -45:19.161 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.161 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30433 [Candidate]" term=2 -45:19.171 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.171 [DEBUG] TestAgent.server.raft: vote granted: from=beacf32f-c990-7b5f-9683-c0d2d62a2cde term=2 tally=1 -45:19.171 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.171 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30433 [Leader]" -45:19.171 [INFO] TestAgent.server: cluster leadership acquired -45:19.171 [INFO] TestAgent.server: initializing acls -45:19.172 [INFO] TestAgent.server: New leader elected: payload=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.177 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:19.177 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.177 [INFO] TestAgent.server: initializing acls -45:19.177 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.194 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.194 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.203 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.203 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:19.203 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:19.203 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.203 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -45:19.203 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.203 [INFO] TestAgent.server: Updating LAN server: server="Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:19.203 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 -45:19.203 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.203 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 -45:19.203 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 area=wan -45:19.203 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde.dc1 area=wan -45:19.203 [INFO] TestAgent.server: Updating LAN server: server="Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:19.212 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.227 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.227 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.227 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.227 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.227 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.227 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.227 [INFO] TestAgent.server: member joined, marking health alive: member=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.235 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.235 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-beacf32f-c990-7b5f-9683-c0d2d62a2cde -45:19.335 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:19.339 [INFO] TestAgent: Synced node info -45:19.339 [DEBUG] TestAgent: Node info in sync -45:19.415 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36950 latency=3.216278ms -45:19.416 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9233601-393b-101e-13a3-0d20e070941e from=127.0.0.1:36952 latency=156.293µs -45:19.416 [INFO] TestAgent: Requesting shutdown -45:19.416 [INFO] TestAgent.server: shutting down server -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:19.416 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:19.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:19.416 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:19.420 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.423Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.423Z [INFO] agent.router.manager: shutting down -45:19.423 [INFO] TestAgent: consul server down -45:19.423 [INFO] TestAgent: shutdown complete -45:19.423 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=tcp -45:19.423 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=udp -45:19.423 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30429 network=tcp -45:19.923 [INFO] TestAgent: Waiting for endpoints to shut down -45:19.924 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleReadCommand_JSON (0.88s) - --- PASS: TestRoleReadCommand_JSON/read_by_id (0.00s) + --- PASS: TestRoleReadCommand/read_by_id_not_found (0.04s) + --- PASS: TestRoleReadCommand/read_by_name_not_found (0.01s) + --- PASS: TestRoleReadCommand/read_by_id (0.04s) + --- PASS: TestRoleReadCommand/read_by_id_prefix (0.04s) + --- PASS: TestRoleReadCommand/read_by_name (0.03s) PASS -ok github.com/hashicorp/consul/command/acl/role/read 0.906s +ok github.com/hashicorp/consul/command/acl/role/read 1.305s === RUN TestRoleUpdateCommand_noTabs === PAUSE TestRoleUpdateCommand_noTabs === RUN TestRoleUpdateCommand @@ -76385,448 +76648,465 @@ === RUN TestRoleUpdateCommand_noMerge === PAUSE TestRoleUpdateCommand_noMerge === CONT TestRoleUpdateCommand_noTabs +--- PASS: TestRoleUpdateCommand_noTabs (0.00s) === CONT TestRoleUpdateCommand_noMerge -=== CONT TestRoleUpdateCommand_JSON -=== CONT TestRoleUpdateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestRoleUpdateCommand +=== CONT TestRoleUpdateCommand_JSON [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestRoleUpdateCommand_noTabs (0.00s) +=== RUN TestRoleUpdateCommand_noMerge/update_a_role_that_does_not_exist +=== RUN TestRoleUpdateCommand_noMerge/update_with_policy_by_name === RUN TestRoleUpdateCommand_JSON/update_a_role_that_does_not_exist +=== RUN TestRoleUpdateCommand_noMerge/update_with_policy_by_id === RUN TestRoleUpdateCommand_JSON/update_with_policy_by_name +=== RUN TestRoleUpdateCommand_noMerge/update_with_service_identity +=== RUN TestRoleUpdateCommand_noMerge/update_with_service_identity_scoped_to_2_DCs === RUN TestRoleUpdateCommand/update_a_role_that_does_not_exist === RUN TestRoleUpdateCommand/update_with_policy_by_name === RUN TestRoleUpdateCommand/update_with_policy_by_id === RUN TestRoleUpdateCommand/update_with_service_identity === RUN TestRoleUpdateCommand/update_with_service_identity_scoped_to_2_DCs === RUN TestRoleUpdateCommand/update_with_node_identity -=== RUN TestRoleUpdateCommand_noMerge/update_a_role_that_does_not_exist -=== RUN TestRoleUpdateCommand_noMerge/update_with_policy_by_name -=== RUN TestRoleUpdateCommand_noMerge/update_with_policy_by_id -=== RUN TestRoleUpdateCommand_noMerge/update_with_service_identity -=== RUN TestRoleUpdateCommand_noMerge/update_with_service_identity_scoped_to_2_DCs -2022-11-07T22:45:19.347Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.359Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.394 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c55ad3dc-61d3-9af6-d565-275da64012a9 Address:127.0.0.1:21244}]" -45:19.394 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= -45:19.394 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c55ad3dc-61d3-9af6-d565-275da64012a9.dc1 127.0.0.1 -45:19.395 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c55ad3dc-61d3-9af6-d565-275da64012a9 127.0.0.1 -2022-11-07T22:45:19.395Z [INFO] agent.router: Initializing LAN area manager -45:19.395 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp -45:19.395 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c55ad3dc-61d3-9af6-d565-275da64012a9.dc1 area=wan -45:19.395 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp -45:19.396 [INFO] TestAgent.server: Adding LAN server: server="Node-c55ad3dc-61d3-9af6-d565-275da64012a9 (Addr: tcp/127.0.0.1:21244) (DC: dc1)" -45:19.396 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp -45:19.396 [INFO] TestAgent: started state syncer -45:19.396 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp -45:19.442 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.442 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 -45:19.445 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:19.450 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.450 [DEBUG] TestAgent.server.raft: vote granted: from=c55ad3dc-61d3-9af6-d565-275da64012a9 term=2 tally=1 -45:19.450 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.450 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" -45:19.450 [INFO] TestAgent.server: cluster leadership acquired -45:19.450 [INFO] TestAgent.server: New leader elected: payload=Node-c55ad3dc-61d3-9af6-d565-275da64012a9 -45:19.455 [INFO] TestAgent.server: initializing acls -45:19.460 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:19.460 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.468 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.470 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -45:19.479 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.479 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:19.479 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:19.479 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c55ad3dc-61d3-9af6-d565-275da64012a9 -45:19.480 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c55ad3dc-61d3-9af6-d565-275da64012a9.dc1 -45:19.480 [INFO] TestAgent.server: Updating LAN server: server="Node-c55ad3dc-61d3-9af6-d565-275da64012a9 (Addr: tcp/127.0.0.1:21244) (DC: dc1)" -45:19.480 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c55ad3dc-61d3-9af6-d565-275da64012a9.dc1 area=wan -45:19.491 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.511 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.511 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.511 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.511 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.511 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.512 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c55ad3dc-61d3-9af6-d565-275da64012a9 -45:19.512 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c55ad3dc-61d3-9af6-d565-275da64012a9 -45:19.515 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.523 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c55ad3dc-61d3-9af6-d565-275da64012a9 -45:19.536 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:38408 latency=7.16848ms -45:19.542 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:38408 latency=3.653365ms -45:19.543 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/e9b492a4-0fa2-d90b-138a-d33c60976cac from=127.0.0.1:38410 latency=127.903µs -45:19.545 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/d826ddfa-119b-6962-8897-9b7948579289 from=127.0.0.1:38412 latency=836.778µs -45:19.549 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/d826ddfa-119b-6962-8897-9b7948579289 from=127.0.0.1:38412 latency=3.287979ms -45:19.549 [INFO] TestAgent: Requesting shutdown -45:19.549 [INFO] TestAgent.server: shutting down server -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.549 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.549 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.549 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.549 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.552 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.555Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.555Z [INFO] agent.router.manager: shutting down -45:19.555 [INFO] TestAgent: consul server down -45:19.555 [INFO] TestAgent: shutdown complete -45:19.555 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp -45:19.555 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp -45:19.555 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp -45:20.056 [INFO] TestAgent: Waiting for endpoints to shut down -45:20.056 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleUpdateCommand_JSON (0.75s) +2021-10-05T16:31:07.668Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:07.694Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:07.746 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 Address:127.0.0.1:12062}]" +31:07.747 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 127.0.0.1 +31:07.748 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 127.0.0.1 +2021-10-05T16:31:07.748Z [INFO] agent.router: Initializing LAN area manager +31:07.748 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12062 [Follower]" leader= +31:07.748 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12057 network=tcp +31:07.759 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12057 network=udp +31:07.748 [INFO] TestAgent.server: Adding LAN server: server="Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 (Addr: tcp/127.0.0.1:12062) (DC: dc1)" +31:07.758 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 area=wan +31:07.760 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12058 network=tcp +31:07.760 [INFO] TestAgent: started state syncer +31:07.760 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12063 network=tcp +31:07.798 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:07.816 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:07.816 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12062 [Candidate]" term=2 +31:07.855 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:07.855 [DEBUG] TestAgent.server.raft: vote granted: from=d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 term=2 tally=1 +31:07.855 [INFO] TestAgent.server.raft: election won: tally=1 +31:07.855 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12062 [Leader]" +31:07.855 [INFO] TestAgent.server: cluster leadership acquired +31:07.855 [INFO] TestAgent.server: New leader elected: payload=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.869 [INFO] TestAgent.server: initializing acls +31:07.873 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:07.873 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:07.880 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:07.892 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:07.909 [INFO] TestAgent.server: initializing acls +31:07.909 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:07.938 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:07.938 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:07.938 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:07.938 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:07.938 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:07.938 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.938 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 +31:07.939 [INFO] TestAgent.server: Updating LAN server: server="Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 (Addr: tcp/127.0.0.1:12062) (DC: dc1)" +31:07.939 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.939 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 +31:07.939 [INFO] TestAgent.server: Updating LAN server: server="Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 (Addr: tcp/127.0.0.1:12062) (DC: dc1)" +31:07.939 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 area=wan +31:07.939 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752.dc1 area=wan +31:07.954 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:07.984 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:07.984 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:07.984 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:07.984 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:07.984 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:07.984 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.984 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.988 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.988 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d6694c6b-4c25-2c8d-43d1-24b3d4e9d752 +31:07.998 [INFO] TestAgent.server: federation state anti-entropy synced +31:08.055 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:60192 latency=5.875297ms +31:08.060 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:60192 latency=5.177628ms +31:08.065 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/e6c93c64-57a9-e9e6-5d27-266e2076adb9 from=127.0.0.1:60194 latency=158.91µs +31:08.066 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/29242578-e5be-5e58-0813-9afc27203402 from=127.0.0.1:60196 latency=342.902µs +31:08.072 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/29242578-e5be-5e58-0813-9afc27203402 from=127.0.0.1:60196 latency=5.375559ms +31:08.073 [INFO] TestAgent: Requesting shutdown +31:08.073 [INFO] TestAgent.server: shutting down server +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:08.073 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:08.073 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:08.073 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:08.073 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:08.078 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:08.081Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:08.081Z [INFO] agent.router.manager: shutting down +31:08.081 [INFO] TestAgent: consul server down +31:08.081 [INFO] TestAgent: shutdown complete +31:08.081 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12057 network=tcp +31:08.081 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12057 network=udp +31:08.081 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12058 network=tcp +31:08.584 [INFO] TestAgent: Waiting for endpoints to shut down +31:08.584 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleUpdateCommand_JSON (1.03s) --- PASS: TestRoleUpdateCommand_JSON/update_a_role_that_does_not_exist (0.00s) --- PASS: TestRoleUpdateCommand_JSON/update_with_policy_by_name (0.01s) -2022-11-07T22:45:19.346Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.363Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.394 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:073de2fb-2134-fbf9-917f-e268cac891fb Address:127.0.0.1:21251}]" -45:19.394 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21251 [Follower]" leader= -45:19.394 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-073de2fb-2134-fbf9-917f-e268cac891fb.dc1 127.0.0.1 -45:19.395 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-073de2fb-2134-fbf9-917f-e268cac891fb 127.0.0.1 -2022-11-07T22:45:19.395Z [INFO] agent.router: Initializing LAN area manager -45:19.395 [INFO] TestAgent.server: Adding LAN server: server="Node-073de2fb-2134-fbf9-917f-e268cac891fb (Addr: tcp/127.0.0.1:21251) (DC: dc1)" -45:19.395 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=tcp -45:19.395 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-073de2fb-2134-fbf9-917f-e268cac891fb.dc1 area=wan -45:19.395 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=udp -45:19.397 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21247 network=tcp -45:19.397 [INFO] TestAgent: started state syncer -45:19.397 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21252 network=tcp -45:19.445 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:19.454 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.454 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21251 [Candidate]" term=2 -45:19.463 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.463 [DEBUG] TestAgent.server.raft: vote granted: from=073de2fb-2134-fbf9-917f-e268cac891fb term=2 tally=1 -45:19.463 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.463 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21251 [Leader]" -45:19.463 [INFO] TestAgent.server: cluster leadership acquired -45:19.463 [INFO] TestAgent.server: New leader elected: payload=Node-073de2fb-2134-fbf9-917f-e268cac891fb -45:19.471 [INFO] TestAgent.server: initializing acls -45:19.476 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:19.476 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.483 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.493 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.493 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:19.493 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:19.493 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-073de2fb-2134-fbf9-917f-e268cac891fb -45:19.493 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-073de2fb-2134-fbf9-917f-e268cac891fb.dc1 -45:19.493 [INFO] TestAgent.server: Updating LAN server: server="Node-073de2fb-2134-fbf9-917f-e268cac891fb (Addr: tcp/127.0.0.1:21251) (DC: dc1)" -45:19.493 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-073de2fb-2134-fbf9-917f-e268cac891fb.dc1 area=wan -45:19.502 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.520 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.520 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.520 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.520 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.520 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.520 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-073de2fb-2134-fbf9-917f-e268cac891fb -45:19.520 [INFO] TestAgent.server: member joined, marking health alive: member=Node-073de2fb-2134-fbf9-917f-e268cac891fb -45:19.528 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-073de2fb-2134-fbf9-917f-e268cac891fb -45:19.534 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.575 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36706 latency=5.166352ms -45:19.578 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:36706 latency=3.007486ms -45:19.583 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:36706 latency=3.121189ms -45:19.584 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/e5ba9fc4-b9e7-33ec-89e9-768bee2604b8 from=127.0.0.1:36708 latency=86.521µs -45:19.586 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36710 latency=184.697µs -45:19.591 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36710 latency=5.15559ms -45:19.593 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36706 latency=318.56µs -45:19.595 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36712 latency=291.638µs -45:19.601 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36712 latency=5.802346ms -45:19.602 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36706 latency=113.051µs -45:19.603 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36714 latency=120.056µs -45:19.607 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36714 latency=3.924681ms -45:19.608 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36706 latency=121.109µs -45:19.609 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36716 latency=152.006µs -45:19.613 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36716 latency=3.219983ms -45:19.613 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36706 latency=110.523µs -45:19.614 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36718 latency=134.268µs -45:19.618 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36718 latency=3.002621ms -45:19.618 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/1440860a-1047-8152-8ef1-41da60a9be0a from=127.0.0.1:36706 latency=126.994µs -45:19.619 [INFO] TestAgent: Requesting shutdown -45:19.619 [INFO] TestAgent.server: shutting down server -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.619 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.619 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.619 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.619 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.619 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.622 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.626Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.626Z [INFO] agent.router.manager: shutting down -45:19.626 [INFO] TestAgent: consul server down -45:19.626 [INFO] TestAgent: shutdown complete -45:19.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=tcp -45:19.626 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=udp -45:19.626 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21247 network=tcp -45:20.126 [INFO] TestAgent: Waiting for endpoints to shut down -45:20.126 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleUpdateCommand (0.82s) +2021-10-05T16:31:07.647Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:07.693Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:07.772 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8d6efbb5-60c1-97de-e213-bb72eab58956 Address:127.0.0.1:12048}]" +31:07.773 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8d6efbb5-60c1-97de-e213-bb72eab58956.dc1 127.0.0.1 +31:07.773 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8d6efbb5-60c1-97de-e213-bb72eab58956 127.0.0.1 +2021-10-05T16:31:07.773Z [INFO] agent.router: Initializing LAN area manager +31:07.776 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= +31:07.776 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8d6efbb5-60c1-97de-e213-bb72eab58956.dc1 area=wan +31:07.776 [INFO] TestAgent.server: Adding LAN server: server="Node-8d6efbb5-60c1-97de-e213-bb72eab58956 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +31:07.808 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp +31:07.808 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp +31:07.809 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp +31:07.809 [INFO] TestAgent: started state syncer +31:07.809 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp +31:07.821 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:07.821 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 +31:07.828 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:07.836 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:07.836 [DEBUG] TestAgent.server.raft: vote granted: from=8d6efbb5-60c1-97de-e213-bb72eab58956 term=2 tally=1 +31:07.836 [INFO] TestAgent.server.raft: election won: tally=1 +31:07.836 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" +31:07.837 [INFO] TestAgent.server: cluster leadership acquired +31:07.837 [INFO] TestAgent.server: New leader elected: payload=Node-8d6efbb5-60c1-97de-e213-bb72eab58956 +31:07.842 [INFO] TestAgent.server: initializing acls +31:07.847 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:07.847 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:07.856 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:07.870 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:07.870 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:07.870 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:07.870 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8d6efbb5-60c1-97de-e213-bb72eab58956 +31:07.870 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8d6efbb5-60c1-97de-e213-bb72eab58956.dc1 +31:07.871 [INFO] TestAgent.server: Updating LAN server: server="Node-8d6efbb5-60c1-97de-e213-bb72eab58956 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +31:07.871 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8d6efbb5-60c1-97de-e213-bb72eab58956.dc1 area=wan +31:07.885 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:07.949 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:07.949 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:07.949 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:07.949 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:07.949 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:07.949 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d6efbb5-60c1-97de-e213-bb72eab58956 +31:07.949 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8d6efbb5-60c1-97de-e213-bb72eab58956 +31:07.966 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d6efbb5-60c1-97de-e213-bb72eab58956 +31:07.972 [INFO] TestAgent.server: federation state anti-entropy synced +31:08.002 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:57928 latency=12.800899ms +31:08.015 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:57928 latency=10.29144ms +31:08.030 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:57928 latency=9.292764ms +31:08.033 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/4f95804d-7ade-8b7b-0b8c-dcacf0c4325f from=127.0.0.1:57932 latency=207.187µs +31:08.038 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:57928 latency=4.728111ms +31:08.039 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/ed5f69c4-588a-126b-712f-704716ddc009 from=127.0.0.1:57934 latency=244.609µs +31:08.060 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/ed5f69c4-588a-126b-712f-704716ddc009 from=127.0.0.1:57934 latency=16.257569ms +31:08.064 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/ed5f69c4-588a-126b-712f-704716ddc009 from=127.0.0.1:57928 latency=284.569µs +31:08.070 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:57928 latency=5.358408ms +31:08.072 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/3555c7f7-c2ba-4f23-a84c-c45f17e281a4 from=127.0.0.1:57942 latency=365.01µs +31:08.077 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/3555c7f7-c2ba-4f23-a84c-c45f17e281a4 from=127.0.0.1:57942 latency=3.558393ms +31:08.081 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/3555c7f7-c2ba-4f23-a84c-c45f17e281a4 from=127.0.0.1:57928 latency=537.928µs +31:08.090 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:57928 latency=5.718367ms +31:08.092 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/e4c96b43-abba-693f-d951-339c06559510 from=127.0.0.1:57944 latency=483.937µs +31:08.095 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/e4c96b43-abba-693f-d951-339c06559510 from=127.0.0.1:57944 latency=3.052874ms +31:08.100 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/e4c96b43-abba-693f-d951-339c06559510 from=127.0.0.1:57928 latency=3.254028ms +31:08.136 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:57928 latency=16.204536ms +31:08.142 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/c4ce46f9-71fe-437f-bafe-0608d3b210ae from=127.0.0.1:57946 latency=604.809µs +31:08.149 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/c4ce46f9-71fe-437f-bafe-0608d3b210ae from=127.0.0.1:57946 latency=6.549313ms +31:08.152 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/c4ce46f9-71fe-437f-bafe-0608d3b210ae from=127.0.0.1:57928 latency=226.998µs +31:08.153 [INFO] TestAgent: Requesting shutdown +31:08.153 [INFO] TestAgent.server: shutting down server +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:08.153 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:08.153 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:08.153 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:08.153 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:08.153 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:08.160 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:08.168Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:08.168Z [INFO] agent.router.manager: shutting down +31:08.168 [INFO] TestAgent: consul server down +31:08.168 [INFO] TestAgent: shutdown complete +31:08.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp +31:08.168 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp +31:08.168 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp +31:08.668 [INFO] TestAgent: Waiting for endpoints to shut down +31:08.668 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleUpdateCommand_noMerge (1.11s) + --- PASS: TestRoleUpdateCommand_noMerge/update_a_role_that_does_not_exist (0.00s) + --- PASS: TestRoleUpdateCommand_noMerge/update_with_policy_by_name (0.03s) + --- PASS: TestRoleUpdateCommand_noMerge/update_with_policy_by_id (0.02s) + --- PASS: TestRoleUpdateCommand_noMerge/update_with_service_identity (0.02s) + --- PASS: TestRoleUpdateCommand_noMerge/update_with_service_identity_scoped_to_2_DCs (0.05s) +2021-10-05T16:31:07.654Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:07.691Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:07.772 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ecab27ea-c306-4def-13eb-349b0f58d120 Address:127.0.0.1:12055}]" +31:07.773 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 127.0.0.1 +31:07.774 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12055 [Follower]" leader= +31:07.809 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ecab27ea-c306-4def-13eb-349b0f58d120 127.0.0.1 +2021-10-05T16:31:07.809Z [INFO] agent.router: Initializing LAN area manager +31:07.809 [INFO] TestAgent.server: Adding LAN server: server="Node-ecab27ea-c306-4def-13eb-349b0f58d120 (Addr: tcp/127.0.0.1:12055) (DC: dc1)" +31:07.809 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 area=wan +31:07.809 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=tcp +31:07.809 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=udp +31:07.810 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12051 network=tcp +31:07.810 [INFO] TestAgent: started state syncer +31:07.810 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12056 network=tcp +31:07.841 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:07.841 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12055 [Candidate]" term=2 +31:07.858 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:07.858 [DEBUG] TestAgent.server.raft: vote granted: from=ecab27ea-c306-4def-13eb-349b0f58d120 term=2 tally=1 +31:07.858 [INFO] TestAgent.server.raft: election won: tally=1 +31:07.858 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12055 [Leader]" +31:07.859 [INFO] TestAgent.server: cluster leadership acquired +31:07.859 [INFO] TestAgent.server: New leader elected: payload=Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:07.860 [INFO] TestAgent.server: initializing acls +31:07.872 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:07.872 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:07.872 [INFO] TestAgent.server: initializing acls +31:07.872 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:07.913 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:07.913 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:07.923 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:07.923 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:07.923 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:07.923 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:07.923 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:07.924 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 +31:07.924 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 area=wan +31:07.924 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:07.924 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:07.924 [INFO] TestAgent.server: Updating LAN server: server="Node-ecab27ea-c306-4def-13eb-349b0f58d120 (Addr: tcp/127.0.0.1:12055) (DC: dc1)" +31:07.924 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 +31:07.924 [INFO] TestAgent.server: Updating LAN server: server="Node-ecab27ea-c306-4def-13eb-349b0f58d120 (Addr: tcp/127.0.0.1:12055) (DC: dc1)" +31:07.924 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ecab27ea-c306-4def-13eb-349b0f58d120.dc1 area=wan +31:07.964 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:07.965 [INFO] TestAgent: Synced node info +31:08.003 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:08.003 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:08.003 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:08.003 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:08.003 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:08.003 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:08.003 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:08.025 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:08.025 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ecab27ea-c306-4def-13eb-349b0f58d120 +31:08.025 [INFO] TestAgent.server: federation state anti-entropy synced +31:08.177 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47510 latency=4.379386ms +31:08.200 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47510 latency=16.460325ms +31:08.221 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:47510 latency=16.364414ms +31:08.228 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/7069e023-a311-75de-1fb2-9e3a36505b69 from=127.0.0.1:47512 latency=151.782µs +31:08.230 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47514 latency=231.007µs +31:08.248 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47514 latency=18.120786ms +31:08.260 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47510 latency=626.965µs +31:08.266 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47518 latency=614.397µs +31:08.288 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47518 latency=22.006599ms +31:08.292 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47510 latency=318.451µs +31:08.295 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47520 latency=435.791µs +31:08.312 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47520 latency=17.133261ms +31:08.317 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47510 latency=313.875µs +31:08.319 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47522 latency=247.888µs +31:08.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47522 latency=7.866849ms +31:08.328 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47510 latency=269.327µs +31:08.330 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47524 latency=238.379µs +31:08.338 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47524 latency=7.427992ms +31:08.339 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/f9df8fe7-e947-44b8-9c34-5ab067bff058 from=127.0.0.1:47510 latency=264.637µs +31:08.339 [INFO] TestAgent: Requesting shutdown +31:08.339 [INFO] TestAgent.server: shutting down server +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:08.340 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:08.340 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:08.340 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:08.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:08.352 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:08.357Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:08.357Z [INFO] agent.router.manager: shutting down +31:08.357 [INFO] TestAgent: consul server down +31:08.357 [INFO] TestAgent: shutdown complete +31:08.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=tcp +31:08.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=udp +31:08.357 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12051 network=tcp +31:08.860 [INFO] TestAgent: Waiting for endpoints to shut down +31:08.860 [INFO] TestAgent: Endpoints down +--- PASS: TestRoleUpdateCommand (1.30s) --- PASS: TestRoleUpdateCommand/update_a_role_that_does_not_exist (0.00s) - --- PASS: TestRoleUpdateCommand/update_with_policy_by_name (0.01s) - --- PASS: TestRoleUpdateCommand/update_with_policy_by_id (0.01s) - --- PASS: TestRoleUpdateCommand/update_with_service_identity (0.01s) + --- PASS: TestRoleUpdateCommand/update_with_policy_by_name (0.04s) + --- PASS: TestRoleUpdateCommand/update_with_policy_by_id (0.03s) + --- PASS: TestRoleUpdateCommand/update_with_service_identity (0.02s) --- PASS: TestRoleUpdateCommand/update_with_service_identity_scoped_to_2_DCs (0.01s) --- PASS: TestRoleUpdateCommand/update_with_node_identity (0.01s) -2022-11-07T22:45:19.343Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:19.359Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:19.391 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d5b2bbba-32cb-f3de-e9ea-9424818c9d22 Address:127.0.0.1:21237}]" -45:19.391 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= -45:19.391 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22.dc1 127.0.0.1 -45:19.392 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 127.0.0.1 -2022-11-07T22:45:19.392Z [INFO] agent.router: Initializing LAN area manager -45:19.392 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22.dc1 area=wan -45:19.392 [INFO] TestAgent.server: Adding LAN server: server="Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -45:19.392 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp -45:19.392 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp -45:19.392 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp -45:19.392 [INFO] TestAgent: started state syncer -45:19.392 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp -45:19.442 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:19.444 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:19.444 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 -45:19.451 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:19.451 [DEBUG] TestAgent.server.raft: vote granted: from=d5b2bbba-32cb-f3de-e9ea-9424818c9d22 term=2 tally=1 -45:19.451 [INFO] TestAgent.server.raft: election won: tally=1 -45:19.451 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" -45:19.451 [INFO] TestAgent.server: cluster leadership acquired -45:19.451 [INFO] TestAgent.server: New leader elected: payload=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 -45:19.456 [INFO] TestAgent.server: initializing acls -45:19.460 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:19.460 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:19.468 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -45:19.469 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:19.482 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:19.482 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:19.482 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:19.482 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 -45:19.482 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22.dc1 -45:19.482 [INFO] TestAgent.server: Updating LAN server: server="Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -45:19.482 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22.dc1 area=wan -45:19.490 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:19.510 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:19.510 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:19.510 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:19.510 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:19.510 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:19.510 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 -45:19.510 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 -45:19.518 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d5b2bbba-32cb-f3de-e9ea-9424818c9d22 -45:19.526 [INFO] TestAgent.server: federation state anti-entropy synced -45:19.679 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:40084 latency=6.594188ms -45:19.684 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:40084 latency=5.057235ms -45:19.688 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:40084 latency=3.473395ms -45:19.689 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/5a4526d1-ef43-524b-def5-6aa840725591 from=127.0.0.1:40086 latency=122.725µs -45:19.692 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:40084 latency=2.880767ms -45:19.694 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/042ade8c-e57a-4e4d-5d8a-45e949df2abd from=127.0.0.1:40088 latency=107.39µs -45:19.699 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/042ade8c-e57a-4e4d-5d8a-45e949df2abd from=127.0.0.1:40088 latency=4.160138ms -45:19.699 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/042ade8c-e57a-4e4d-5d8a-45e949df2abd from=127.0.0.1:40084 latency=197.833µs -45:19.706 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:40084 latency=4.858728ms -45:19.708 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/243e85bc-f984-fe43-08a4-67b2843ecef2 from=127.0.0.1:40090 latency=157.961µs -45:19.712 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/243e85bc-f984-fe43-08a4-67b2843ecef2 from=127.0.0.1:40090 latency=3.929565ms -45:19.712 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/243e85bc-f984-fe43-08a4-67b2843ecef2 from=127.0.0.1:40084 latency=144.863µs -45:19.716 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:40084 latency=3.44641ms -45:19.717 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/7343aa2f-9002-ab99-475f-6662c38c6736 from=127.0.0.1:40092 latency=135.495µs -45:19.722 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/7343aa2f-9002-ab99-475f-6662c38c6736 from=127.0.0.1:40092 latency=3.933809ms -45:19.722 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/7343aa2f-9002-ab99-475f-6662c38c6736 from=127.0.0.1:40084 latency=156.622µs -45:19.727 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role from=127.0.0.1:40084 latency=4.162382ms -45:19.728 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/93b3c692-226a-cea2-698d-90bc05c5a7eb from=127.0.0.1:40094 latency=169.914µs -45:19.731 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/role/93b3c692-226a-cea2-698d-90bc05c5a7eb from=127.0.0.1:40094 latency=3.148965ms -45:19.732 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/role/93b3c692-226a-cea2-698d-90bc05c5a7eb from=127.0.0.1:40084 latency=207.509µs -45:19.732 [INFO] TestAgent: Requesting shutdown -45:19.732 [INFO] TestAgent.server: shutting down server -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:19.732 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:19.732 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:19.732 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:19.732 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:19.736 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:19.740Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:19.740Z [INFO] agent.router.manager: shutting down -45:19.740 [INFO] TestAgent: consul server down -45:19.740 [INFO] TestAgent: shutdown complete -45:19.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp -45:19.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp -45:19.740 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp -45:20.240 [INFO] TestAgent: Waiting for endpoints to shut down -45:20.240 [INFO] TestAgent: Endpoints down ---- PASS: TestRoleUpdateCommand_noMerge (0.93s) - --- PASS: TestRoleUpdateCommand_noMerge/update_a_role_that_does_not_exist (0.00s) - --- PASS: TestRoleUpdateCommand_noMerge/update_with_policy_by_name (0.01s) - --- PASS: TestRoleUpdateCommand_noMerge/update_with_policy_by_id (0.01s) - --- PASS: TestRoleUpdateCommand_noMerge/update_with_service_identity (0.01s) - --- PASS: TestRoleUpdateCommand_noMerge/update_with_service_identity_scoped_to_2_DCs (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/role/update 0.960s +ok github.com/hashicorp/consul/command/acl/role/update 1.414s === RUN TestRulesTranslateCommand_noTabs === PAUSE TestRulesTranslateCommand_noTabs === RUN TestRulesTranslateCommand === PAUSE TestRulesTranslateCommand === CONT TestRulesTranslateCommand_noTabs +--- PASS: TestRulesTranslateCommand_noTabs (0.00s) === CONT TestRulesTranslateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestRulesTranslateCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestRulesTranslateCommand/file === RUN TestRulesTranslateCommand/stdin === RUN TestRulesTranslateCommand/arg === RUN TestRulesTranslateCommand/exclusive-options -2022-11-07T22:45:21.418Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:21.429Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:21.452 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:21a8ff47-b03e-b2d3-4e03-269815934db9 Address:127.0.0.1:26342}]" -45:21.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= -45:21.453 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-21a8ff47-b03e-b2d3-4e03-269815934db9.dc1 127.0.0.1 -45:21.453 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-21a8ff47-b03e-b2d3-4e03-269815934db9 127.0.0.1 -2022-11-07T22:45:21.453Z [INFO] agent.router: Initializing LAN area manager -45:21.454 [INFO] TestAgent.server: Adding LAN server: server="Node-21a8ff47-b03e-b2d3-4e03-269815934db9 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:21.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp -45:21.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp -45:21.454 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-21a8ff47-b03e-b2d3-4e03-269815934db9.dc1 area=wan -45:21.454 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp -45:21.454 [INFO] TestAgent: started state syncer -45:21.454 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp -45:21.504 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:21.512 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:21.512 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 -45:21.520 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:21.520 [DEBUG] TestAgent.server.raft: vote granted: from=21a8ff47-b03e-b2d3-4e03-269815934db9 term=2 tally=1 -45:21.520 [INFO] TestAgent.server.raft: election won: tally=1 -45:21.520 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" -45:21.520 [INFO] TestAgent.server: cluster leadership acquired -45:21.520 [INFO] TestAgent.server: New leader elected: payload=Node-21a8ff47-b03e-b2d3-4e03-269815934db9 -45:21.525 [INFO] TestAgent.server: initializing acls -45:21.529 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:21.529 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:21.535 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:21.542 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:21.542 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:21.542 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:21.543 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-21a8ff47-b03e-b2d3-4e03-269815934db9 -45:21.543 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-21a8ff47-b03e-b2d3-4e03-269815934db9.dc1 -45:21.543 [INFO] TestAgent.server: Updating LAN server: server="Node-21a8ff47-b03e-b2d3-4e03-269815934db9 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:21.543 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-21a8ff47-b03e-b2d3-4e03-269815934db9.dc1 area=wan -45:21.552 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:21.569 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:21.569 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:21.569 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:21.569 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:21.569 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:21.569 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-21a8ff47-b03e-b2d3-4e03-269815934db9 -45:21.569 [INFO] TestAgent.server: member joined, marking health alive: member=Node-21a8ff47-b03e-b2d3-4e03-269815934db9 -45:21.572 [INFO] TestAgent.server: federation state anti-entropy synced -45:21.577 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-21a8ff47-b03e-b2d3-4e03-269815934db9 -45:21.687 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:21.692 [INFO] TestAgent: Synced node info -45:21.806 [INFO] TestAgent: Requesting shutdown -45:21.806 [INFO] TestAgent.server: shutting down server -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:21.806 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:21.806 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:21.806 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:21.806 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:21.810 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:21.813Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:21.813Z [INFO] agent.router.manager: shutting down -45:21.813 [INFO] TestAgent: consul server down -45:21.813 [INFO] TestAgent: shutdown complete -45:21.813 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp -45:21.813 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp -45:21.813 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp -45:22.313 [INFO] TestAgent: Waiting for endpoints to shut down -45:22.313 [INFO] TestAgent: Endpoints down ---- PASS: TestRulesTranslateCommand (0.93s) +2021-10-05T16:31:10.692Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:10.731Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:10.788 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:efff4665-c598-2675-d6fd-24af24184367 Address:127.0.0.1:22258}]" +31:10.789 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-efff4665-c598-2675-d6fd-24af24184367.dc1 127.0.0.1 +31:10.790 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-efff4665-c598-2675-d6fd-24af24184367 127.0.0.1 +2021-10-05T16:31:10.790Z [INFO] agent.router: Initializing LAN area manager +31:10.790 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp +31:10.790 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= +31:10.791 [INFO] TestAgent.server: Adding LAN server: server="Node-efff4665-c598-2675-d6fd-24af24184367 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +31:10.791 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-efff4665-c598-2675-d6fd-24af24184367.dc1 area=wan +31:10.791 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp +31:10.791 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp +31:10.791 [INFO] TestAgent: started state syncer +31:10.791 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp +31:10.837 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:10.837 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 +31:10.844 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:10.855 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:10.855 [DEBUG] TestAgent.server.raft: vote granted: from=efff4665-c598-2675-d6fd-24af24184367 term=2 tally=1 +31:10.855 [INFO] TestAgent.server.raft: election won: tally=1 +31:10.855 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" +31:10.855 [INFO] TestAgent.server: cluster leadership acquired +31:10.856 [INFO] TestAgent.server: New leader elected: payload=Node-efff4665-c598-2675-d6fd-24af24184367 +31:10.862 [INFO] TestAgent.server: initializing acls +31:10.868 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:10.868 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:10.888 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:10.911 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:10.911 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:10.911 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:10.911 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-efff4665-c598-2675-d6fd-24af24184367 +31:10.911 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-efff4665-c598-2675-d6fd-24af24184367.dc1 +31:10.918 [INFO] TestAgent.server: Updating LAN server: server="Node-efff4665-c598-2675-d6fd-24af24184367 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +31:10.918 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-efff4665-c598-2675-d6fd-24af24184367.dc1 area=wan +31:10.936 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:11.004 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:11.004 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:11.004 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:11.004 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:11.004 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:11.004 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-efff4665-c598-2675-d6fd-24af24184367 +31:11.004 [INFO] TestAgent.server: member joined, marking health alive: member=Node-efff4665-c598-2675-d6fd-24af24184367 +31:11.013 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-efff4665-c598-2675-d6fd-24af24184367 +31:11.028 [INFO] TestAgent.server: federation state anti-entropy synced +31:11.039 [INFO] TestAgent: Requesting shutdown +31:11.039 [INFO] TestAgent.server: shutting down server +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:11.039 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:11.039 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:11.039 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:11.039 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:11.039 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:11.052 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:11.058Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:11.058Z [INFO] agent.router.manager: shutting down +31:11.058 [INFO] TestAgent: consul server down +31:11.058 [INFO] TestAgent: shutdown complete +31:11.058 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp +31:11.059 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp +31:11.059 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp +31:11.560 [INFO] TestAgent: Waiting for endpoints to shut down +31:11.560 [INFO] TestAgent: Endpoints down +--- PASS: TestRulesTranslateCommand (1.00s) --- PASS: TestRulesTranslateCommand/file (0.00s) --- PASS: TestRulesTranslateCommand/stdin (0.00s) --- PASS: TestRulesTranslateCommand/arg (0.00s) --- PASS: TestRulesTranslateCommand/exclusive-options (0.00s) PASS -ok github.com/hashicorp/consul/command/acl/rules 0.958s +ok github.com/hashicorp/consul/command/acl/rules 1.111s === RUN TestFormatToken === RUN TestFormatToken/legacy -=== RUN TestFormatToken/legacy/json === RUN TestFormatToken/legacy/pretty === RUN TestFormatToken/legacy/pretty-meta +=== RUN TestFormatToken/legacy/json === RUN TestFormatToken/basic +=== RUN TestFormatToken/basic/pretty === RUN TestFormatToken/basic/pretty-meta === RUN TestFormatToken/basic/json -=== RUN TestFormatToken/basic/pretty === RUN TestFormatToken/complex === RUN TestFormatToken/complex/pretty === RUN TestFormatToken/complex/pretty-meta === RUN TestFormatToken/complex/json ---- PASS: TestFormatToken (0.00s) - --- PASS: TestFormatToken/legacy (0.00s) - --- PASS: TestFormatToken/legacy/json (0.00s) - --- PASS: TestFormatToken/legacy/pretty (0.00s) +--- PASS: TestFormatToken (0.04s) + --- PASS: TestFormatToken/legacy (0.01s) + --- PASS: TestFormatToken/legacy/pretty (0.01s) --- PASS: TestFormatToken/legacy/pretty-meta (0.00s) - --- PASS: TestFormatToken/basic (0.00s) + --- PASS: TestFormatToken/legacy/json (0.00s) + --- PASS: TestFormatToken/basic (0.01s) + --- PASS: TestFormatToken/basic/pretty (0.00s) --- PASS: TestFormatToken/basic/pretty-meta (0.00s) --- PASS: TestFormatToken/basic/json (0.00s) - --- PASS: TestFormatToken/basic/pretty (0.00s) - --- PASS: TestFormatToken/complex (0.00s) + --- PASS: TestFormatToken/complex (0.01s) --- PASS: TestFormatToken/complex/pretty (0.00s) --- PASS: TestFormatToken/complex/pretty-meta (0.00s) --- PASS: TestFormatToken/complex/json (0.00s) === RUN TestFormatTokenList -=== RUN TestFormatTokenList/basic -=== RUN TestFormatTokenList/basic/pretty -=== RUN TestFormatTokenList/basic/pretty-meta -=== RUN TestFormatTokenList/basic/json === RUN TestFormatTokenList/legacy === RUN TestFormatTokenList/legacy/pretty === RUN TestFormatTokenList/legacy/pretty-meta === RUN TestFormatTokenList/legacy/json === RUN TestFormatTokenList/complex -=== RUN TestFormatTokenList/complex/pretty === RUN TestFormatTokenList/complex/pretty-meta === RUN TestFormatTokenList/complex/json ---- PASS: TestFormatTokenList (0.00s) - --- PASS: TestFormatTokenList/basic (0.00s) - --- PASS: TestFormatTokenList/basic/pretty (0.00s) - --- PASS: TestFormatTokenList/basic/pretty-meta (0.00s) - --- PASS: TestFormatTokenList/basic/json (0.00s) - --- PASS: TestFormatTokenList/legacy (0.00s) +=== RUN TestFormatTokenList/complex/pretty +=== RUN TestFormatTokenList/basic +=== RUN TestFormatTokenList/basic/pretty +=== RUN TestFormatTokenList/basic/pretty-meta +=== RUN TestFormatTokenList/basic/json +--- PASS: TestFormatTokenList (0.04s) + --- PASS: TestFormatTokenList/legacy (0.01s) --- PASS: TestFormatTokenList/legacy/pretty (0.00s) --- PASS: TestFormatTokenList/legacy/pretty-meta (0.00s) --- PASS: TestFormatTokenList/legacy/json (0.00s) - --- PASS: TestFormatTokenList/complex (0.00s) - --- PASS: TestFormatTokenList/complex/pretty (0.00s) + --- PASS: TestFormatTokenList/complex (0.01s) --- PASS: TestFormatTokenList/complex/pretty-meta (0.00s) --- PASS: TestFormatTokenList/complex/json (0.00s) + --- PASS: TestFormatTokenList/complex/pretty (0.00s) + --- PASS: TestFormatTokenList/basic (0.01s) + --- PASS: TestFormatTokenList/basic/pretty (0.00s) + --- PASS: TestFormatTokenList/basic/pretty-meta (0.00s) + --- PASS: TestFormatTokenList/basic/json (0.00s) PASS -ok github.com/hashicorp/consul/command/acl/token 0.011s +ok github.com/hashicorp/consul/command/acl/token 0.117s === RUN TestTokenCloneCommand_noTabs === PAUSE TestTokenCloneCommand_noTabs === RUN TestTokenCloneCommand_Pretty @@ -76834,188 +77114,188 @@ === RUN TestTokenCloneCommand_JSON === PAUSE TestTokenCloneCommand_JSON === CONT TestTokenCloneCommand_noTabs +--- PASS: TestTokenCloneCommand_noTabs (0.00s) === CONT TestTokenCloneCommand_JSON -=== CONT TestTokenCloneCommand_Pretty [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestTokenCloneCommand_noTabs (0.00s) +=== CONT TestTokenCloneCommand_Pretty [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== RUN TestTokenCloneCommand_Pretty/Description -=== RUN TestTokenCloneCommand_Pretty/Without_Description === RUN TestTokenCloneCommand_JSON/Description +=== RUN TestTokenCloneCommand_Pretty/Description === RUN TestTokenCloneCommand_JSON/Without_Description -2022-11-07T22:45:24.281Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:24.292Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:24.318 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 Address:127.0.0.1:26342}]" -45:24.319 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= -45:24.319 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2.dc1 127.0.0.1 -45:24.320 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 127.0.0.1 -2022-11-07T22:45:24.320Z [INFO] agent.router: Initializing LAN area manager -45:24.320 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp -45:24.320 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2.dc1 area=wan -45:24.320 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp -45:24.320 [INFO] TestAgent.server: Adding LAN server: server="Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:24.323 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp -45:24.323 [INFO] TestAgent: started state syncer -45:24.323 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp -45:24.371 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:24.371 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:24.371 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 -45:24.379 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:24.379 [DEBUG] TestAgent.server.raft: vote granted: from=63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 term=2 tally=1 -45:24.379 [INFO] TestAgent.server.raft: election won: tally=1 -45:24.379 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" -45:24.379 [INFO] TestAgent.server: cluster leadership acquired -45:24.379 [INFO] TestAgent.server: New leader elected: payload=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 -45:24.385 [INFO] TestAgent.server: initializing acls -45:24.389 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:24.389 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:24.400 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:24.411 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:24.411 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:24.411 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:24.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 -45:24.411 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2.dc1 -45:24.412 [INFO] TestAgent.server: Updating LAN server: server="Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -45:24.412 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2.dc1 area=wan -45:24.422 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:24.440 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:24.440 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:24.440 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:24.440 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:24.440 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:24.440 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 -45:24.440 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 -45:24.446 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63a04869-ded5-c5f8-8dd2-fabbfb53d8c2 -45:24.450 [INFO] TestAgent.server: federation state anti-entropy synced -45:24.462 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:24.468 [INFO] TestAgent: Synced node info -45:24.468 [DEBUG] TestAgent: Node info in sync -45:24.470 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:56642 latency=10.812035ms -45:24.475 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:56642 latency=4.264054ms -45:24.480 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/e20f0755-1ebc-ab4c-882f-87dbe1abd9e2/clone from=127.0.0.1:56644 latency=3.583181ms -45:24.480 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/a559c536-843c-a98f-b881-7c46256b852a from=127.0.0.1:56642 latency=106.172µs -45:24.486 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/e20f0755-1ebc-ab4c-882f-87dbe1abd9e2/clone from=127.0.0.1:56646 latency=5.06719ms -45:24.487 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/749569a6-1509-4ae3-1546-3707d0e10c8c from=127.0.0.1:56642 latency=111.23µs -45:24.487 [INFO] TestAgent: Requesting shutdown -45:24.487 [INFO] TestAgent.server: shutting down server -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:24.487 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:24.487 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:24.487 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:24.487 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:24.493 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:24.498Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:24.498Z [INFO] agent.router.manager: shutting down -45:24.498 [INFO] TestAgent: consul server down -45:24.498 [INFO] TestAgent: shutdown complete -45:24.498 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp -45:24.498 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp -45:24.498 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp -45:24.998 [INFO] TestAgent: Waiting for endpoints to shut down -45:24.998 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenCloneCommand_Pretty (0.75s) - --- PASS: TestTokenCloneCommand_Pretty/Description (0.01s) - --- PASS: TestTokenCloneCommand_Pretty/Without_Description (0.01s) -2022-11-07T22:45:24.281Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:24.294Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:24.320 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b8506c93-c70e-86a2-5000-ede5aa5313d6 Address:127.0.0.1:26349}]" -45:24.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= -45:24.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 127.0.0.1 -45:24.324 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 127.0.0.1 -2022-11-07T22:45:24.324Z [INFO] agent.router: Initializing LAN area manager -45:24.324 [INFO] TestAgent.server: Adding LAN server: server="Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -45:24.324 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp -45:24.324 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp -45:24.324 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 area=wan -45:24.325 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp -45:24.325 [INFO] TestAgent: started state syncer -45:24.325 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp -45:24.359 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:24.359 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 -45:24.367 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:24.367 [DEBUG] TestAgent.server.raft: vote granted: from=b8506c93-c70e-86a2-5000-ede5aa5313d6 term=2 tally=1 -45:24.367 [INFO] TestAgent.server.raft: election won: tally=1 -45:24.367 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" -45:24.367 [INFO] TestAgent.server: cluster leadership acquired -45:24.368 [INFO] TestAgent.server: New leader elected: payload=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.373 [INFO] TestAgent.server: initializing acls -45:24.375 [INFO] TestAgent.server: initializing acls -45:24.376 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:24.376 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:24.386 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:24.386 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:24.403 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:24.403 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:24.410 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:24.410 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:24.410 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:24.410 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:24.410 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -45:24.410 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.411 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 -45:24.411 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 area=wan -45:24.411 [INFO] TestAgent.server: Updating LAN server: server="Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -45:24.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.411 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 -45:24.411 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6.dc1 area=wan -45:24.411 [INFO] TestAgent.server: Updating LAN server: server="Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -45:24.422 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:24.440 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:24.440 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:24.440 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:24.440 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:24.440 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:24.440 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.440 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.450 [INFO] TestAgent.server: federation state anti-entropy synced -45:24.450 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.450 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b8506c93-c70e-86a2-5000-ede5aa5313d6 -45:24.657 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:54696 latency=5.650891ms -45:24.661 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54696 latency=3.513169ms -45:24.665 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/39504555-e74b-c59c-bab8-8c58f8be98b7/clone from=127.0.0.1:54698 latency=3.418853ms -45:24.672 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/39504555-e74b-c59c-bab8-8c58f8be98b7/clone from=127.0.0.1:54700 latency=3.945413ms -45:24.673 [INFO] TestAgent: Requesting shutdown -45:24.673 [INFO] TestAgent.server: shutting down server -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:24.673 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:24.673 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:24.673 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:24.673 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:24.673 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:24.676 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:24.679Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:24.679Z [INFO] agent.router.manager: shutting down -45:24.679 [INFO] TestAgent: consul server down -45:24.679 [INFO] TestAgent: shutdown complete -45:24.679 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp -45:24.679 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp -45:24.679 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp -45:25.179 [INFO] TestAgent: Waiting for endpoints to shut down -45:25.179 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenCloneCommand_JSON (0.93s) - --- PASS: TestTokenCloneCommand_JSON/Description (0.01s) - --- PASS: TestTokenCloneCommand_JSON/Without_Description (0.01s) +=== RUN TestTokenCloneCommand_Pretty/Without_Description +2021-10-05T16:31:19.984Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.063Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.154 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f8b88bf8-07de-bc66-b8ba-10cd02cc277f Address:127.0.0.1:14097}]" +31:20.154 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= +31:20.154 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f.dc1 127.0.0.1 +31:20.155 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f 127.0.0.1 +2021-10-05T16:31:20.155Z [INFO] agent.router: Initializing LAN area manager +31:20.155 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp +31:20.156 [INFO] TestAgent.server: Adding LAN server: server="Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +31:20.156 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f.dc1 area=wan +31:20.156 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp +31:20.156 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp +31:20.156 [INFO] TestAgent: started state syncer +31:20.156 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp +31:20.190 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.190 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 +31:20.207 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.224 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.224 [DEBUG] TestAgent.server.raft: vote granted: from=f8b88bf8-07de-bc66-b8ba-10cd02cc277f term=2 tally=1 +31:20.224 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.224 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" +31:20.224 [INFO] TestAgent.server: cluster leadership acquired +31:20.224 [INFO] TestAgent.server: New leader elected: payload=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f +31:20.243 [INFO] TestAgent.server: initializing acls +31:20.251 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.251 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.266 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:20.297 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.297 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.297 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.297 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f +31:20.297 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f.dc1 +31:20.299 [INFO] TestAgent.server: Updating LAN server: server="Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +31:20.299 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f.dc1 area=wan +31:20.332 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.382 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.382 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.382 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.382 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.382 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.382 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f +31:20.382 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f +31:20.400 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f8b88bf8-07de-bc66-b8ba-10cd02cc277f +31:20.410 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.512 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.531 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:59920 latency=26.847641ms +31:20.550 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:59920 latency=19.109765ms +31:20.551 [INFO] TestAgent: Synced node info +31:20.551 [DEBUG] TestAgent: Node info in sync +31:20.564 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/abcc1f00-33d8-79ad-b362-4df71a7456a8/clone from=127.0.0.1:59966 latency=10.36737ms +31:20.568 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/9e42682a-73c8-edb5-90af-c46f3da070a6 from=127.0.0.1:59920 latency=352.184µs +31:20.583 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/abcc1f00-33d8-79ad-b362-4df71a7456a8/clone from=127.0.0.1:59970 latency=11.036905ms +31:20.585 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/8f0bbb39-e15d-9384-768e-4547cf22f725 from=127.0.0.1:59920 latency=310.407µs +31:20.588 [INFO] TestAgent: Requesting shutdown +31:20.588 [INFO] TestAgent.server: shutting down server +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.588 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.588 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.588 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.588 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.600 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.609Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.609Z [INFO] agent.router.manager: shutting down +31:20.609 [INFO] TestAgent: consul server down +31:20.609 [INFO] TestAgent: shutdown complete +31:20.609 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp +31:20.609 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp +31:20.609 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp +31:21.110 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.110 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenCloneCommand_Pretty (1.23s) + --- PASS: TestTokenCloneCommand_Pretty/Description (0.02s) + --- PASS: TestTokenCloneCommand_Pretty/Without_Description (0.02s) +2021-10-05T16:31:19.994Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.051Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.093 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:51d80ce4-e6a8-192d-4b9d-3de4a25065f9 Address:127.0.0.1:14090}]" +31:20.094 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +31:20.094 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 127.0.0.1 +31:20.095 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 127.0.0.1 +2021-10-05T16:31:20.095Z [INFO] agent.router: Initializing LAN area manager +31:20.096 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +31:20.096 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 area=wan +31:20.096 [INFO] TestAgent.server: Adding LAN server: server="Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +31:20.096 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +31:20.101 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +31:20.101 [INFO] TestAgent: started state syncer +31:20.102 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +31:20.148 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.148 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +31:20.148 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.203 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.203 [DEBUG] TestAgent.server.raft: vote granted: from=51d80ce4-e6a8-192d-4b9d-3de4a25065f9 term=2 tally=1 +31:20.203 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.203 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +31:20.204 [INFO] TestAgent.server: cluster leadership acquired +31:20.204 [INFO] TestAgent.server: New leader elected: payload=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.220 [INFO] TestAgent.server: initializing acls +31:20.234 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.234 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.248 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:20.249 [INFO] TestAgent.server: initializing acls +31:20.249 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.289 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.289 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.289 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.289 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:20.289 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.289 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 +31:20.289 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 area=wan +31:20.289 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.289 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.289 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 +31:20.292 [INFO] TestAgent.server: Updating LAN server: server="Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +31:20.292 [INFO] TestAgent.server: Updating LAN server: server="Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +31:20.292 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9.dc1 area=wan +31:20.320 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.392 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.392 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.392 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.392 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.392 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.392 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.392 [INFO] TestAgent.server: member joined, marking health alive: member=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.404 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.404 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-51d80ce4-e6a8-192d-4b9d-3de4a25065f9 +31:20.416 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.464 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:20.473 [INFO] TestAgent: Synced node info +31:20.529 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:44620 latency=16.156348ms +31:20.536 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:44620 latency=5.630818ms +31:20.565 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/55864ec5-fa8f-6a67-922f-504522a31853/clone from=127.0.0.1:44642 latency=13.235511ms +31:20.581 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/55864ec5-fa8f-6a67-922f-504522a31853/clone from=127.0.0.1:44648 latency=8.549621ms +31:20.584 [INFO] TestAgent: Requesting shutdown +31:20.584 [INFO] TestAgent.server: shutting down server +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.584 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.584 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.584 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.596 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.620Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.620Z [INFO] agent.router.manager: shutting down +31:20.620 [INFO] TestAgent: consul server down +31:20.620 [INFO] TestAgent: shutdown complete +31:20.620 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +31:20.620 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +31:20.620 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +31:21.122 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.122 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenCloneCommand_JSON (1.25s) + --- PASS: TestTokenCloneCommand_JSON/Description (0.03s) + --- PASS: TestTokenCloneCommand_JSON/Without_Description (0.02s) PASS -ok github.com/hashicorp/consul/command/acl/token/clone 0.962s +ok github.com/hashicorp/consul/command/acl/token/clone 1.520s === RUN TestTokenCreateCommand_noTabs === PAUSE TestTokenCreateCommand_noTabs === RUN TestTokenCreateCommand_Pretty @@ -77025,263 +77305,278 @@ === CONT TestTokenCreateCommand_noTabs === CONT TestTokenCreateCommand_JSON === CONT TestTokenCreateCommand_Pretty -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... --- PASS: TestTokenCreateCommand_noTabs (0.00s) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestTokenCreateCommand_Pretty/policy-name === RUN TestTokenCreateCommand_Pretty/policy-id === RUN TestTokenCreateCommand_Pretty/node-identity === RUN TestTokenCreateCommand_Pretty/predefined-ids -2022-11-07T22:45:24.361Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:24.374Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:24.407 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a33f110f-27a2-733b-7d83-78b95f1f9c26 Address:127.0.0.1:24307}]" -45:24.407 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24307 [Follower]" leader= -45:24.408 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a33f110f-27a2-733b-7d83-78b95f1f9c26.dc1 127.0.0.1 -45:24.409 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 127.0.0.1 -2022-11-07T22:45:24.409Z [INFO] agent.router: Initializing LAN area manager -45:24.410 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26.dc1 area=wan -45:24.410 [INFO] TestAgent.server: Adding LAN server: server="Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:24.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=udp -45:24.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=tcp -45:24.411 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24303 network=tcp -45:24.411 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24308 network=tcp -45:24.411 [INFO] TestAgent: started state syncer -45:24.461 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:24.461 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:24.461 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24307 [Candidate]" term=2 -45:24.472 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:24.472 [DEBUG] TestAgent.server.raft: vote granted: from=a33f110f-27a2-733b-7d83-78b95f1f9c26 term=2 tally=1 -45:24.472 [INFO] TestAgent.server.raft: election won: tally=1 -45:24.472 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24307 [Leader]" -45:24.472 [INFO] TestAgent.server: cluster leadership acquired -45:24.472 [INFO] TestAgent.server: New leader elected: payload=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 -45:24.479 [INFO] TestAgent.server: initializing acls -45:24.482 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:24.482 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:24.488 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:24.500 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:24.500 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:24.500 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:24.500 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 -45:24.501 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a33f110f-27a2-733b-7d83-78b95f1f9c26.dc1 -45:24.501 [INFO] TestAgent.server: Updating LAN server: server="Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:24.501 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26.dc1 area=wan -45:24.511 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:24.529 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:24.530 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:24.530 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:24.530 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:24.530 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:24.530 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 -45:24.530 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 -45:24.540 [INFO] TestAgent.server: federation state anti-entropy synced -45:24.540 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a33f110f-27a2-733b-7d83-78b95f1f9c26 -45:24.620 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47880 latency=8.638389ms -45:24.626 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:47882 latency=4.05013ms -45:24.627 [INFO] TestAgent: Requesting shutdown -45:24.627 [INFO] TestAgent.server: shutting down server -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:24.627 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:24.627 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:24.627 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:24.627 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:24.627 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:24.630 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:24.635Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:24.635Z [INFO] agent.router.manager: shutting down -45:24.635 [INFO] TestAgent: consul server down -45:24.635 [INFO] TestAgent: shutdown complete -45:24.635 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=tcp -45:24.635 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=udp -45:24.635 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24303 network=tcp -45:25.136 [INFO] TestAgent: Waiting for endpoints to shut down -45:25.136 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenCreateCommand_JSON (0.81s) -2022-11-07T22:45:24.361Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:24.374Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:24.408 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:739e6180-26a9-9b62-f3e6-2f7ee7d381dc Address:127.0.0.1:24300}]" -45:24.409 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24300 [Follower]" leader= -45:24.409 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 -45:24.411 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 -2022-11-07T22:45:24.411Z [INFO] agent.router: Initializing LAN area manager -45:24.411 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan -45:24.411 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:24.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=udp -45:24.411 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=tcp -45:24.412 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24296 network=tcp -45:24.412 [INFO] TestAgent: started state syncer -45:24.412 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24301 network=tcp -45:24.461 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:24.464 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:24.464 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24300 [Candidate]" term=2 -45:24.475 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:24.475 [DEBUG] TestAgent.server.raft: vote granted: from=739e6180-26a9-9b62-f3e6-2f7ee7d381dc term=2 tally=1 -45:24.475 [INFO] TestAgent.server.raft: election won: tally=1 -45:24.475 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24300 [Leader]" -45:24.475 [INFO] TestAgent.server: cluster leadership acquired -45:24.475 [INFO] TestAgent.server: New leader elected: payload=test-node -45:24.481 [INFO] TestAgent.server: initializing acls -45:24.486 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:24.486 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:24.493 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:24.505 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:24.505 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:24.505 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:24.505 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: test-node -45:24.505 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: test-node.dc1 -45:24.505 [INFO] TestAgent.server: Updating LAN server: server="test-node (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:24.505 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=test-node.dc1 area=wan -45:24.518 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:24.538 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:24.538 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:24.538 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:24.538 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:24.538 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:24.538 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -45:24.538 [INFO] TestAgent.server: member joined, marking health alive: member=test-node -45:24.548 [INFO] TestAgent.server: federation state anti-entropy synced -45:24.548 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node -45:24.559 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:24.563 [INFO] TestAgent: Synced node info -45:24.563 [DEBUG] TestAgent: Node info in sync -45:24.721 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:46550 latency=4.027053ms -45:24.726 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46552 latency=3.58378ms -45:24.730 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46554 latency=3.575667ms -45:24.736 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46556 latency=4.034717ms -45:24.738 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:46558 latency=341.489µs -45:24.743 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46560 latency=3.419735ms -45:24.743 [INFO] TestAgent: Requesting shutdown -45:24.743 [INFO] TestAgent.server: shutting down server -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:24.743 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:24.743 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:24.743 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:24.743 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:24.747 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:24.751Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:24.751Z [INFO] agent.router.manager: shutting down -45:24.751 [INFO] TestAgent: consul server down -45:24.751 [INFO] TestAgent: shutdown complete -45:24.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=tcp -45:24.751 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=udp -45:24.751 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24296 network=tcp -45:25.251 [INFO] TestAgent: Waiting for endpoints to shut down -45:25.251 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenCreateCommand_Pretty (0.93s) - --- PASS: TestTokenCreateCommand_Pretty/policy-name (0.00s) - --- PASS: TestTokenCreateCommand_Pretty/policy-id (0.00s) - --- PASS: TestTokenCreateCommand_Pretty/node-identity (0.01s) - --- PASS: TestTokenCreateCommand_Pretty/predefined-ids (0.00s) +2021-10-05T16:31:20.014Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.123Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.221 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c8910cb2-88c1-9e25-6eda-23f3e2c75935 Address:127.0.0.1:25328}]" +31:20.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 127.0.0.1 +31:20.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 127.0.0.1 +2021-10-05T16:31:20.222Z [INFO] agent.router: Initializing LAN area manager +31:20.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=udp +31:20.222 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25328 [Follower]" leader= +31:20.228 [INFO] TestAgent.server: Adding LAN server: server="Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +31:20.230 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 area=wan +31:20.230 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=tcp +31:20.231 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25324 network=tcp +31:20.231 [INFO] TestAgent: started state syncer +31:20.231 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25329 network=tcp +31:20.260 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.260 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25328 [Candidate]" term=2 +31:20.280 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.324 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.324 [DEBUG] TestAgent.server.raft: vote granted: from=c8910cb2-88c1-9e25-6eda-23f3e2c75935 term=2 tally=1 +31:20.324 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.324 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25328 [Leader]" +31:20.324 [INFO] TestAgent.server: cluster leadership acquired +31:20.325 [INFO] TestAgent.server: New leader elected: payload=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.356 [INFO] TestAgent.server: initializing acls +31:20.364 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.364 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.376 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:20.382 [INFO] TestAgent.server: initializing acls +31:20.382 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.408 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.408 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.408 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.408 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:20.409 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.409 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 +31:20.409 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 area=wan +31:20.409 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.409 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.409 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 +31:20.409 [INFO] TestAgent.server: Updating LAN server: server="Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +31:20.409 [INFO] TestAgent.server: Updating LAN server: server="Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" +31:20.409 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935.dc1 area=wan +31:20.433 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.479 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.479 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.479 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.479 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.479 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.479 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.479 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.488 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.488 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c8910cb2-88c1-9e25-6eda-23f3e2c75935 +31:20.493 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.510 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:37426 latency=7.628162ms +31:20.525 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37438 latency=11.77541ms +31:20.528 [INFO] TestAgent: Requesting shutdown +31:20.528 [INFO] TestAgent.server: shutting down server +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.528 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.528 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.528 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.528 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.528 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:20.528 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.528 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.528 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.529 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.529 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.539 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.547Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.547Z [INFO] agent.router.manager: shutting down +31:20.547 [INFO] TestAgent: consul server down +31:20.547 [INFO] TestAgent: shutdown complete +31:20.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=tcp +31:20.547 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=udp +31:20.547 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25324 network=tcp +31:21.048 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.048 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenCreateCommand_JSON (1.21s) +2021-10-05T16:31:19.997Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:20.122Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:20.182 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88cd809f-ff14-205f-6f94-7b27b05a01a2 Address:127.0.0.1:25321}]" +31:20.183 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: test-node.dc1 127.0.0.1 +31:20.183 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: test-node 127.0.0.1 +2021-10-05T16:31:20.183Z [INFO] agent.router: Initializing LAN area manager +31:20.184 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= +31:20.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp +31:20.184 [INFO] TestAgent.server: Adding LAN server: server="test-node (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +31:20.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=test-node.dc1 area=wan +31:20.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp +31:20.185 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp +31:20.185 [INFO] TestAgent: started state syncer +31:20.185 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp +31:20.233 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:20.233 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 +31:20.236 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:20.250 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:20.250 [DEBUG] TestAgent.server.raft: vote granted: from=88cd809f-ff14-205f-6f94-7b27b05a01a2 term=2 tally=1 +31:20.250 [INFO] TestAgent.server.raft: election won: tally=1 +31:20.250 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" +31:20.250 [INFO] TestAgent.server: cluster leadership acquired +31:20.250 [INFO] TestAgent.server: New leader elected: payload=test-node +31:20.264 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" +31:20.288 [INFO] TestAgent.server: initializing acls +31:20.304 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:20.304 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.313 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:20.336 [INFO] TestAgent.server: initializing acls +31:20.336 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:20.350 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.350 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:20.350 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:20.350 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:20.351 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: test-node +31:20.351 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: test-node.dc1 +31:20.351 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=test-node.dc1 area=wan +31:20.351 [INFO] TestAgent.server: Updating LAN server: server="test-node (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +31:20.351 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:20.351 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: test-node +31:20.351 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: test-node.dc1 +31:20.352 [INFO] TestAgent.server: Updating LAN server: server="test-node (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +31:20.352 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=test-node.dc1 area=wan +31:20.380 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:20.430 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:20.430 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:20.430 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:20.430 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:20.430 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:20.430 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:20.430 [INFO] TestAgent.server: member joined, marking health alive: member=test-node +31:20.438 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:20.438 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=test-node +31:20.445 [INFO] TestAgent.server: federation state anti-entropy synced +31:20.524 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:54244 latency=24.081234ms +31:20.546 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54298 latency=14.581164ms +31:20.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54302 latency=16.441743ms +31:20.584 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54310 latency=13.148434ms +31:20.589 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:54312 latency=626.469µs +31:20.606 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54314 latency=6.676327ms +31:20.606 [INFO] TestAgent: Requesting shutdown +31:20.606 [INFO] TestAgent.server: shutting down server +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:20.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:20.606 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:20.606 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:20.606 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:20.613 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:20.617Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:20.617Z [INFO] agent.router.manager: shutting down +31:20.617 [INFO] TestAgent: consul server down +31:20.617 [INFO] TestAgent: shutdown complete +31:20.617 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp +31:20.617 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp +31:20.617 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp +31:21.120 [INFO] TestAgent: Waiting for endpoints to shut down +31:21.120 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenCreateCommand_Pretty (1.28s) + --- PASS: TestTokenCreateCommand_Pretty/policy-name (0.02s) + --- PASS: TestTokenCreateCommand_Pretty/policy-id (0.02s) + --- PASS: TestTokenCreateCommand_Pretty/node-identity (0.02s) + --- PASS: TestTokenCreateCommand_Pretty/predefined-ids (0.01s) PASS -ok github.com/hashicorp/consul/command/acl/token/create 0.960s +ok github.com/hashicorp/consul/command/acl/token/create 1.515s === RUN TestTokenDeleteCommand_noTabs === PAUSE TestTokenDeleteCommand_noTabs === RUN TestTokenDeleteCommand === PAUSE TestTokenDeleteCommand === CONT TestTokenDeleteCommand_noTabs -=== CONT TestTokenDeleteCommand --- PASS: TestTokenDeleteCommand_noTabs (0.00s) +=== CONT TestTokenDeleteCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:26.365Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:26.376Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:26.397 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8e10ece6-6b25-d71e-c662-ff684c7968c1 Address:127.0.0.1:19195}]" -45:26.397 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= -45:26.397 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8e10ece6-6b25-d71e-c662-ff684c7968c1.dc1 127.0.0.1 -45:26.398 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 127.0.0.1 -2022-11-07T22:45:26.398Z [INFO] agent.router: Initializing LAN area manager -45:26.398 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1.dc1 area=wan -45:26.398 [INFO] TestAgent.server: Adding LAN server: server="Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 (Addr: tcp/127.0.0.1:19195) (DC: dc1)" -45:26.398 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=tcp -45:26.398 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=udp -45:26.399 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19191 network=tcp -45:26.399 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19196 network=tcp -45:26.399 [INFO] TestAgent: started state syncer -45:26.448 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:26.456 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:26.456 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 -45:26.464 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:26.464 [DEBUG] TestAgent.server.raft: vote granted: from=8e10ece6-6b25-d71e-c662-ff684c7968c1 term=2 tally=1 -45:26.464 [INFO] TestAgent.server.raft: election won: tally=1 -45:26.464 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" -45:26.464 [INFO] TestAgent.server: cluster leadership acquired -45:26.464 [INFO] TestAgent.server: New leader elected: payload=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 -45:26.469 [INFO] TestAgent.server: initializing acls -45:26.472 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:26.472 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:26.478 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:26.486 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:26.486 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:26.486 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:26.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 -45:26.486 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8e10ece6-6b25-d71e-c662-ff684c7968c1.dc1 -45:26.486 [INFO] TestAgent.server: Updating LAN server: server="Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 (Addr: tcp/127.0.0.1:19195) (DC: dc1)" -45:26.486 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1.dc1 area=wan -45:26.494 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:26.508 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:26.508 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:26.508 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:26.508 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:26.508 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:26.508 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 -45:26.508 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 -45:26.516 [INFO] TestAgent.server: federation state anti-entropy synced -45:26.516 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8e10ece6-6b25-d71e-c662-ff684c7968c1 -45:26.609 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:26.614 [INFO] TestAgent: Synced node info -45:26.748 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:53080 latency=4.081567ms -45:26.754 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/51199544-aee6-7517-904f-ea27499829d0 from=127.0.0.1:53082 latency=5.374355ms -45:26.756 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/51199544-aee6-7517-904f-ea27499829d0 from=127.0.0.1:53080 error="ACL not found" -45:26.756 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/51199544-aee6-7517-904f-ea27499829d0 from=127.0.0.1:53080 latency=290.208µs -45:26.756 [INFO] TestAgent: Requesting shutdown -45:26.756 [INFO] TestAgent.server: shutting down server -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:26.756 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:26.756 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:26.756 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:26.756 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:26.760 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:26.764Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:26.764Z [INFO] agent.router.manager: shutting down -45:26.764 [INFO] TestAgent: consul server down -45:26.764 [INFO] TestAgent: shutdown complete -45:26.764 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp -45:26.764 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp -45:26.764 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp -45:27.264 [INFO] TestAgent: Waiting for endpoints to shut down -45:27.264 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenDeleteCommand (0.94s) +2021-10-05T16:31:22.114Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:22.130Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:22.166 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ea5950a8-02d5-b2aa-3b13-0acfb343b6ca Address:127.0.0.1:30426}]" +31:22.168 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca.dc1 127.0.0.1 +31:22.169 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca 127.0.0.1 +2021-10-05T16:31:22.169Z [INFO] agent.router: Initializing LAN area manager +31:22.172 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= +31:22.175 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp +31:22.176 [INFO] TestAgent.server: Adding LAN server: server="Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca (Addr: tcp/127.0.0.1:30426) (DC: dc1)" +31:22.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca.dc1 area=wan +31:22.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp +31:22.185 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp +31:22.185 [INFO] TestAgent: started state syncer +31:22.185 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp +31:22.231 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:22.236 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:22.236 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 +31:22.250 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:22.250 [DEBUG] TestAgent.server.raft: vote granted: from=ea5950a8-02d5-b2aa-3b13-0acfb343b6ca term=2 tally=1 +31:22.250 [INFO] TestAgent.server.raft: election won: tally=1 +31:22.250 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" +31:22.250 [INFO] TestAgent.server: cluster leadership acquired +31:22.250 [INFO] TestAgent.server: New leader elected: payload=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca +31:22.260 [INFO] TestAgent.server: initializing acls +31:22.265 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:22.265 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:22.271 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:22.281 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:22.281 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:22.281 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:22.281 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca +31:22.281 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca.dc1 +31:22.284 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca.dc1 area=wan +31:22.284 [INFO] TestAgent.server: Updating LAN server: server="Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca (Addr: tcp/127.0.0.1:30426) (DC: dc1)" +31:22.303 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:22.360 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:22.360 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:22.360 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:22.360 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:22.360 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:22.360 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca +31:22.360 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca +31:22.407 [INFO] TestAgent: Synced node info +31:22.408 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ea5950a8-02d5-b2aa-3b13-0acfb343b6ca +31:22.422 [INFO] TestAgent.server: federation state anti-entropy synced +31:22.441 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:54620 latency=19.322627ms +31:22.452 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/acl/token/95005098-638d-536e-8e5e-df15166c4c5a from=127.0.0.1:54622 latency=4.895034ms +31:22.454 [ERROR] TestAgent.http: Request error: method=GET url=/v1/acl/token/95005098-638d-536e-8e5e-df15166c4c5a from=127.0.0.1:54620 error="ACL not found" +31:22.455 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/95005098-638d-536e-8e5e-df15166c4c5a from=127.0.0.1:54620 latency=370.822µs +31:22.464 [INFO] TestAgent: Requesting shutdown +31:22.464 [INFO] TestAgent.server: shutting down server +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:22.464 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:22.464 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:22.464 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:22.464 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:22.476 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:22.483Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:22.483Z [INFO] agent.router.manager: shutting down +31:22.483 [INFO] TestAgent: consul server down +31:22.483 [INFO] TestAgent: shutdown complete +31:22.483 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp +31:22.484 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp +31:22.484 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp +31:22.984 [INFO] TestAgent: Waiting for endpoints to shut down +31:22.984 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenDeleteCommand (0.93s) PASS -ok github.com/hashicorp/consul/command/acl/token/delete 0.971s +ok github.com/hashicorp/consul/command/acl/token/delete 1.144s === RUN TestTokenListCommand_noTabs === PAUSE TestTokenListCommand_noTabs === RUN TestTokenListCommand_Pretty @@ -77290,179 +77585,180 @@ === PAUSE TestTokenListCommand_JSON === CONT TestTokenListCommand_noTabs === CONT TestTokenListCommand_JSON +--- PASS: TestTokenListCommand_noTabs (0.01s) === CONT TestTokenListCommand_Pretty [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestTokenListCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:29.218Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:29.235Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:29.270 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5e9e8718-3b79-7f0d-0433-c536cab568eb Address:127.0.0.1:30426}]" -45:29.271 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= -45:29.271 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 127.0.0.1 -45:29.272 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb 127.0.0.1 -2022-11-07T22:45:29.272Z [INFO] agent.router: Initializing LAN area manager -45:29.272 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 area=wan -45:29.272 [INFO] TestAgent.server: Adding LAN server: server="Node-5e9e8718-3b79-7f0d-0433-c536cab568eb (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:29.272 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp -45:29.272 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp -45:29.272 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp -45:29.272 [INFO] TestAgent: started state syncer -45:29.272 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp -45:29.312 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:29.312 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 -45:29.320 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:29.320 [DEBUG] TestAgent.server.raft: vote granted: from=5e9e8718-3b79-7f0d-0433-c536cab568eb term=2 tally=1 -45:29.320 [INFO] TestAgent.server.raft: election won: tally=1 -45:29.320 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" -45:29.320 [INFO] TestAgent.server: cluster leadership acquired -45:29.320 [INFO] TestAgent.server: New leader elected: payload=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.322 [INFO] TestAgent.server: initializing acls -45:29.326 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:29.326 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:29.326 [INFO] TestAgent.server: initializing acls -45:29.326 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:29.331 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:29.342 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:29.342 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:29.342 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:29.342 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:29.342 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -45:29.342 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.342 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 -45:29.342 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 area=wan -45:29.342 [INFO] TestAgent.server: Updating LAN server: server="Node-5e9e8718-3b79-7f0d-0433-c536cab568eb (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:29.342 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.342 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 -45:29.342 [INFO] TestAgent.server: Updating LAN server: server="Node-5e9e8718-3b79-7f0d-0433-c536cab568eb (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:29.342 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb.dc1 area=wan -45:29.351 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:29.373 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:29.373 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:29.373 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:29.373 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:29.373 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:29.373 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.373 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.382 [INFO] TestAgent.server: federation state anti-entropy synced -45:29.382 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.382 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5e9e8718-3b79-7f0d-0433-c536cab568eb -45:29.411 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:35112 latency=3.976924ms -45:29.415 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:35112 latency=3.940099ms -45:29.421 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:35112 latency=4.944688ms -45:29.424 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:35112 latency=3.247575ms -45:29.430 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:35112 latency=5.320927ms -45:29.431 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:35114 latency=826.2µs -45:29.433 [INFO] TestAgent: Requesting shutdown -45:29.433 [INFO] TestAgent.server: shutting down server -45:29.433 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:29.433 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:29.433 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:29.434 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:29.434 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:29.434 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:29.434 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:29.434 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:29.434 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:29.434 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:29.438 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:29.442Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:29.442Z [INFO] agent.router.manager: shutting down -45:29.442 [INFO] TestAgent: consul server down -45:29.442 [INFO] TestAgent: shutdown complete -45:29.442 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp -45:29.442 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp -45:29.442 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp -45:29.942 [INFO] TestAgent: Waiting for endpoints to shut down -45:29.942 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenListCommand_JSON (0.76s) -2022-11-07T22:45:29.218Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:29.232Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:29.266 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:82263d4c-cb23-5c3e-f291-3a04dd807927 Address:127.0.0.1:30433}]" -45:29.266 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30433 [Follower]" leader= -45:29.266 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-82263d4c-cb23-5c3e-f291-3a04dd807927.dc1 127.0.0.1 -45:29.267 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-82263d4c-cb23-5c3e-f291-3a04dd807927 127.0.0.1 -2022-11-07T22:45:29.267Z [INFO] agent.router: Initializing LAN area manager -45:29.267 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-82263d4c-cb23-5c3e-f291-3a04dd807927.dc1 area=wan -45:29.268 [INFO] TestAgent.server: Adding LAN server: server="Node-82263d4c-cb23-5c3e-f291-3a04dd807927 (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:29.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=udp -45:29.268 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=tcp -45:29.269 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30429 network=tcp -45:29.269 [INFO] TestAgent: started state syncer -45:29.269 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30434 network=tcp -45:29.315 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:29.315 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30433 [Candidate]" term=2 -45:29.317 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:29.323 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:29.323 [DEBUG] TestAgent.server.raft: vote granted: from=82263d4c-cb23-5c3e-f291-3a04dd807927 term=2 tally=1 -45:29.323 [INFO] TestAgent.server.raft: election won: tally=1 -45:29.323 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30433 [Leader]" -45:29.323 [INFO] TestAgent.server: cluster leadership acquired -45:29.323 [INFO] TestAgent.server: New leader elected: payload=Node-82263d4c-cb23-5c3e-f291-3a04dd807927 -45:29.329 [INFO] TestAgent.server: initializing acls -45:29.333 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:29.333 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:29.341 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:29.349 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:29.349 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:29.349 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:29.349 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-82263d4c-cb23-5c3e-f291-3a04dd807927 -45:29.349 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-82263d4c-cb23-5c3e-f291-3a04dd807927.dc1 -45:29.349 [INFO] TestAgent.server: Updating LAN server: server="Node-82263d4c-cb23-5c3e-f291-3a04dd807927 (Addr: tcp/127.0.0.1:30433) (DC: dc1)" -45:29.349 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-82263d4c-cb23-5c3e-f291-3a04dd807927.dc1 area=wan -45:29.358 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:29.377 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:29.377 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:29.377 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:29.377 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:29.377 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:29.377 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-82263d4c-cb23-5c3e-f291-3a04dd807927 -45:29.377 [INFO] TestAgent.server: member joined, marking health alive: member=Node-82263d4c-cb23-5c3e-f291-3a04dd807927 -45:29.385 [INFO] TestAgent.server: federation state anti-entropy synced -45:29.386 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-82263d4c-cb23-5c3e-f291-3a04dd807927 -45:29.546 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:29.551 [INFO] TestAgent: Synced node info -45:29.551 [DEBUG] TestAgent: Node info in sync -45:29.582 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37030 latency=3.319564ms -45:29.587 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37030 latency=3.661477ms -45:29.590 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37030 latency=2.969303ms -45:29.595 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37030 latency=4.116204ms -45:29.598 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:37030 latency=2.749868ms -45:29.599 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:37034 latency=856.249µs -45:29.599 [INFO] TestAgent: Requesting shutdown -45:29.599 [INFO] TestAgent.server: shutting down server -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:29.599 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:29.599 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:29.600 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:29.600 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:29.603 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:29.606Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:29.606Z [INFO] agent.router.manager: shutting down -45:29.606 [INFO] TestAgent: consul server down -45:29.606 [INFO] TestAgent: shutdown complete -45:29.606 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=tcp -45:29.606 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=udp -45:29.606 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30429 network=tcp -45:30.107 [INFO] TestAgent: Waiting for endpoints to shut down -45:30.107 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenListCommand_Pretty (0.93s) +2021-10-05T16:31:32.038Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.073Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.115 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e3686018-8cb7-c3c7-2159-87c868f909f6 Address:127.0.0.1:10013}]" +31:32.116 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= +31:32.116 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 127.0.0.1 +31:32.117 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e3686018-8cb7-c3c7-2159-87c868f909f6 127.0.0.1 +2021-10-05T16:31:32.117Z [INFO] agent.router: Initializing LAN area manager +31:32.117 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp +31:32.117 [INFO] TestAgent.server: Adding LAN server: server="Node-e3686018-8cb7-c3c7-2159-87c868f909f6 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +31:32.117 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 area=wan +31:32.118 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp +31:32.118 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp +31:32.118 [INFO] TestAgent: started state syncer +31:32.118 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp +31:32.169 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:32.176 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.176 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 +31:32.220 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.220 [DEBUG] TestAgent.server.raft: vote granted: from=e3686018-8cb7-c3c7-2159-87c868f909f6 term=2 tally=1 +31:32.220 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.220 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" +31:32.224 [INFO] TestAgent.server: cluster leadership acquired +31:32.224 [INFO] TestAgent.server: New leader elected: payload=Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.238 [INFO] TestAgent.server: initializing acls +31:32.243 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:32.243 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:32.256 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:32.271 [INFO] TestAgent.server: initializing acls +31:32.271 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:32.294 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:32.294 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:32.294 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:32.294 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +31:32.294 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.294 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 +31:32.294 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 area=wan +31:32.294 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:32.294 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.294 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 +31:32.296 [INFO] TestAgent.server: Updating LAN server: server="Node-e3686018-8cb7-c3c7-2159-87c868f909f6 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +31:32.296 [INFO] TestAgent.server: Updating LAN server: server="Node-e3686018-8cb7-c3c7-2159-87c868f909f6 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" +31:32.296 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e3686018-8cb7-c3c7-2159-87c868f909f6.dc1 area=wan +31:32.331 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.376 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.376 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.376 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.376 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.376 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.376 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.376 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e3686018-8cb7-c3c7-2159-87c868f909f6 +31:32.404 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.448 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:32.464 [INFO] TestAgent: Synced node info +31:32.502 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:49280 latency=21.887698ms +31:32.520 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:49280 latency=11.917005ms +31:32.548 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:49280 latency=20.350825ms +31:32.573 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:49280 latency=17.435766ms +31:32.592 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:49280 latency=17.957178ms +31:32.601 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:49284 latency=525.879µs +31:32.612 [INFO] TestAgent: Requesting shutdown +31:32.617 [INFO] TestAgent.server: shutting down server +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.617 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:32.617 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.617 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.617 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.618 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:32.618 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:32.618 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.618 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.618 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.636 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.644Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.644Z [INFO] agent.router.manager: shutting down +31:32.645 [INFO] TestAgent: consul server down +31:32.645 [INFO] TestAgent: shutdown complete +31:32.645 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp +31:32.645 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp +31:32.645 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp +31:33.152 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.152 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenListCommand_JSON (1.23s) +2021-10-05T16:31:32.029Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.052Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.094 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:58b31c3b-9e62-53e4-f6b8-c393bd7bac8c Address:127.0.0.1:10006}]" +31:32.095 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c.dc1 127.0.0.1 +31:32.096 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c 127.0.0.1 +2021-10-05T16:31:32.096Z [INFO] agent.router: Initializing LAN area manager +31:32.096 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= +31:32.096 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp +31:32.096 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c.dc1 area=wan +31:32.096 [INFO] TestAgent.server: Adding LAN server: server="Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c (Addr: tcp/127.0.0.1:10006) (DC: dc1)" +31:32.096 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp +31:32.097 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp +31:32.097 [INFO] TestAgent: started state syncer +31:32.097 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp +31:32.132 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.132 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 +31:32.147 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:32.172 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.172 [DEBUG] TestAgent.server.raft: vote granted: from=58b31c3b-9e62-53e4-f6b8-c393bd7bac8c term=2 tally=1 +31:32.172 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.172 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" +31:32.172 [INFO] TestAgent.server: cluster leadership acquired +31:32.172 [INFO] TestAgent.server: New leader elected: payload=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c +31:32.195 [INFO] TestAgent.server: initializing acls +31:32.215 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:32.215 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:32.222 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:32.231 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:32.231 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:32.231 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:32.232 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c +31:32.232 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c.dc1 +31:32.232 [INFO] TestAgent.server: Updating LAN server: server="Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c (Addr: tcp/127.0.0.1:10006) (DC: dc1)" +31:32.232 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c.dc1 area=wan +31:32.244 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.292 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.292 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.292 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.292 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.292 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.292 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c +31:32.292 [INFO] TestAgent.server: member joined, marking health alive: member=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c +31:32.321 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-58b31c3b-9e62-53e4-f6b8-c393bd7bac8c +31:32.329 [INFO] TestAgent.server: federation state anti-entropy synced +31:32.551 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:32.560 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:48974 latency=19.527287ms +31:32.572 [INFO] TestAgent: Synced node info +31:32.582 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:48974 latency=17.897397ms +31:32.595 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:48974 latency=11.32332ms +31:32.624 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:48974 latency=22.960764ms +31:32.634 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:48974 latency=9.030814ms +31:32.635 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/tokens from=127.0.0.1:48978 latency=554.204µs +31:32.636 [INFO] TestAgent: Requesting shutdown +31:32.636 [INFO] TestAgent.server: shutting down server +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:32.636 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:32.636 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:32.636 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:32.636 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:32.643 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:32.652Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:32.652Z [INFO] agent.router.manager: shutting down +31:32.652 [INFO] TestAgent: consul server down +31:32.652 [INFO] TestAgent: shutdown complete +31:32.652 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp +31:32.652 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp +31:32.652 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp +31:33.154 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.154 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenListCommand_Pretty (1.22s) PASS -ok github.com/hashicorp/consul/command/acl/token/list 0.956s +ok github.com/hashicorp/consul/command/acl/token/list 1.447s === RUN TestTokenReadCommand_noTabs === PAUSE TestTokenReadCommand_noTabs === RUN TestTokenReadCommand_Pretty @@ -77470,162 +77766,164 @@ === RUN TestTokenReadCommand_JSON === PAUSE TestTokenReadCommand_JSON === CONT TestTokenReadCommand_noTabs +--- PASS: TestTokenReadCommand_noTabs (0.00s) === CONT TestTokenReadCommand_JSON [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestTokenReadCommand_Pretty ---- PASS: TestTokenReadCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:29.335Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:29.350Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:29.378 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd50e092-6fc9-441c-8728-145be9747a6d Address:127.0.0.1:19195}]" -45:29.378 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= -45:29.379 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd50e092-6fc9-441c-8728-145be9747a6d.dc1 127.0.0.1 -45:29.380 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd50e092-6fc9-441c-8728-145be9747a6d 127.0.0.1 -2022-11-07T22:45:29.380Z [INFO] agent.router: Initializing LAN area manager -45:29.380 [INFO] TestAgent.server: Adding LAN server: server="Node-cd50e092-6fc9-441c-8728-145be9747a6d (Addr: tcp/127.0.0.1:19195) (DC: dc1)" -45:29.380 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd50e092-6fc9-441c-8728-145be9747a6d.dc1 area=wan -45:29.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=tcp -45:29.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=udp -45:29.380 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19191 network=tcp -45:29.380 [INFO] TestAgent: started state syncer -45:29.380 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19196 network=tcp -45:29.423 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:29.423 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 -45:29.430 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:29.433 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:29.433 [DEBUG] TestAgent.server.raft: vote granted: from=cd50e092-6fc9-441c-8728-145be9747a6d term=2 tally=1 -45:29.433 [INFO] TestAgent.server.raft: election won: tally=1 -45:29.433 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" -45:29.434 [INFO] TestAgent.server: cluster leadership acquired -45:29.434 [INFO] TestAgent.server: New leader elected: payload=Node-cd50e092-6fc9-441c-8728-145be9747a6d -45:29.441 [INFO] TestAgent.server: initializing acls -45:29.444 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:29.444 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:29.450 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:29.461 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:29.461 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:29.461 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:29.461 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-cd50e092-6fc9-441c-8728-145be9747a6d -45:29.461 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-cd50e092-6fc9-441c-8728-145be9747a6d.dc1 -45:29.461 [INFO] TestAgent.server: Updating LAN server: server="Node-cd50e092-6fc9-441c-8728-145be9747a6d (Addr: tcp/127.0.0.1:19195) (DC: dc1)" -45:29.461 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-cd50e092-6fc9-441c-8728-145be9747a6d.dc1 area=wan -45:29.470 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:29.487 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:29.487 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:29.487 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:29.487 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:29.487 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:29.487 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd50e092-6fc9-441c-8728-145be9747a6d -45:29.487 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd50e092-6fc9-441c-8728-145be9747a6d -45:29.490 [INFO] TestAgent.server: federation state anti-entropy synced -45:29.496 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd50e092-6fc9-441c-8728-145be9747a6d -45:29.506 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:53090 latency=6.221974ms -45:29.508 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/b5c8e9fc-3bdd-77cf-483e-e943bae3536e from=127.0.0.1:53092 latency=1.018556ms -45:29.508 [INFO] TestAgent: Requesting shutdown -45:29.508 [INFO] TestAgent.server: shutting down server -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:29.508 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:29.508 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:29.508 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:29.508 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:29.508 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:29.512 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:29.516Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:29.516Z [INFO] agent.router.manager: shutting down -45:29.516 [INFO] TestAgent: consul server down -45:29.516 [INFO] TestAgent: shutdown complete -45:29.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp -45:29.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp -45:29.516 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp -45:30.016 [INFO] TestAgent: Waiting for endpoints to shut down -45:30.016 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenReadCommand_JSON (0.72s) -2022-11-07T22:45:29.335Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:29.350Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:29.376 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:602ddd0a-67f8-0501-af6d-fd565566857c Address:127.0.0.1:19202}]" -45:29.376 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19202 [Follower]" leader= -45:29.377 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-602ddd0a-67f8-0501-af6d-fd565566857c.dc1 127.0.0.1 -45:29.377 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-602ddd0a-67f8-0501-af6d-fd565566857c 127.0.0.1 -2022-11-07T22:45:29.377Z [INFO] agent.router: Initializing LAN area manager -45:29.378 [INFO] TestAgent.server: Adding LAN server: server="Node-602ddd0a-67f8-0501-af6d-fd565566857c (Addr: tcp/127.0.0.1:19202) (DC: dc1)" -45:29.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=udp -45:29.378 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-602ddd0a-67f8-0501-af6d-fd565566857c.dc1 area=wan -45:29.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=tcp -45:29.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19198 network=tcp -45:29.380 [INFO] TestAgent: started state syncer -45:29.380 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19203 network=tcp -45:29.421 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:29.421 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19202 [Candidate]" term=2 -45:29.428 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:29.432 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:29.432 [DEBUG] TestAgent.server.raft: vote granted: from=602ddd0a-67f8-0501-af6d-fd565566857c term=2 tally=1 -45:29.432 [INFO] TestAgent.server.raft: election won: tally=1 -45:29.432 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19202 [Leader]" -45:29.432 [INFO] TestAgent.server: cluster leadership acquired -45:29.432 [INFO] TestAgent.server: New leader elected: payload=Node-602ddd0a-67f8-0501-af6d-fd565566857c -45:29.440 [INFO] TestAgent.server: initializing acls -45:29.443 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:29.444 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:29.449 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:29.460 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:29.460 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:29.460 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:29.460 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-602ddd0a-67f8-0501-af6d-fd565566857c -45:29.460 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-602ddd0a-67f8-0501-af6d-fd565566857c.dc1 -45:29.460 [INFO] TestAgent.server: Updating LAN server: server="Node-602ddd0a-67f8-0501-af6d-fd565566857c (Addr: tcp/127.0.0.1:19202) (DC: dc1)" -45:29.460 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-602ddd0a-67f8-0501-af6d-fd565566857c.dc1 area=wan -45:29.469 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:29.486 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:29.486 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:29.486 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:29.486 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:29.486 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:29.486 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-602ddd0a-67f8-0501-af6d-fd565566857c -45:29.486 [INFO] TestAgent.server: member joined, marking health alive: member=Node-602ddd0a-67f8-0501-af6d-fd565566857c -45:29.491 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-602ddd0a-67f8-0501-af6d-fd565566857c -45:29.496 [INFO] TestAgent.server: federation state anti-entropy synced -45:29.579 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:57462 latency=4.725093ms -45:29.580 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/2b9b8b8f-6d35-4bed-e2e5-a5f2c79a7c1f from=127.0.0.1:57466 latency=163.054µs -45:29.580 [INFO] TestAgent: Requesting shutdown -45:29.581 [INFO] TestAgent.server: shutting down server -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:29.581 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:29.581 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:29.581 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:29.581 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:29.581 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:29.584 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:29.591Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:29.591Z [INFO] agent.router.manager: shutting down -45:29.591 [INFO] TestAgent: consul server down -45:29.591 [INFO] TestAgent: shutdown complete -45:29.591 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=tcp -45:29.591 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=udp -45:29.591 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19198 network=tcp -45:30.091 [INFO] TestAgent: Waiting for endpoints to shut down -45:30.091 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenReadCommand_Pretty (0.79s) +2021-10-05T16:31:32.523Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.569Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.657 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8d1ec6f4-8ae2-b951-7799-2990fd11ada3 Address:127.0.0.1:20223}]" +31:32.657 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20223 [Follower]" leader= +31:32.658 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3.dc1 127.0.0.1 +31:32.659 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 127.0.0.1 +2021-10-05T16:31:32.659Z [INFO] agent.router: Initializing LAN area manager +31:32.659 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=udp +31:32.660 [INFO] TestAgent.server: Adding LAN server: server="Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 (Addr: tcp/127.0.0.1:20223) (DC: dc1)" +31:32.660 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3.dc1 area=wan +31:32.660 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=tcp +31:32.661 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20219 network=tcp +31:32.661 [INFO] TestAgent: started state syncer +31:32.661 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20224 network=tcp +31:32.699 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.699 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20223 [Candidate]" term=2 +31:32.710 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:32.720 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.720 [DEBUG] TestAgent.server.raft: vote granted: from=8d1ec6f4-8ae2-b951-7799-2990fd11ada3 term=2 tally=1 +31:32.720 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.720 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20223 [Leader]" +31:32.724 [INFO] TestAgent.server: cluster leadership acquired +31:32.724 [INFO] TestAgent.server: New leader elected: payload=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 +31:32.733 [INFO] TestAgent.server: initializing acls +31:32.742 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:32.742 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:32.760 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:32.784 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:32.784 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:32.784 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:32.784 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 +31:32.784 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3.dc1 +31:32.784 [INFO] TestAgent.server: Updating LAN server: server="Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 (Addr: tcp/127.0.0.1:20223) (DC: dc1)" +31:32.784 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3.dc1 area=wan +31:32.800 [INFO] TestAgent: Synced node info +31:32.810 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.838 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.838 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.838 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.838 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.838 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.838 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 +31:32.838 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 +31:32.851 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8d1ec6f4-8ae2-b951-7799-2990fd11ada3 +31:32.870 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.002 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46244 latency=18.076371ms +31:33.004 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/cae8a006-e448-3501-b4c7-a5f66499ca89 from=127.0.0.1:46246 latency=498.294µs +31:33.005 [INFO] TestAgent: Requesting shutdown +31:33.005 [INFO] TestAgent.server: shutting down server +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.005 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.005 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.005 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.005 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.010 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.019Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.019Z [INFO] agent.router.manager: shutting down +31:33.019 [INFO] TestAgent: consul server down +31:33.019 [INFO] TestAgent: shutdown complete +31:33.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=tcp +31:33.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=udp +31:33.019 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20219 network=tcp +31:33.520 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.520 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenReadCommand_Pretty (1.11s) +2021-10-05T16:31:32.526Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:32.585Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:32.660 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:74abd4b2-9417-7e9b-1e69-4a29ca18e808 Address:127.0.0.1:20216}]" +31:32.660 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= +31:32.661 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808.dc1 127.0.0.1 +31:32.661 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 127.0.0.1 +2021-10-05T16:31:32.661Z [INFO] agent.router: Initializing LAN area manager +31:32.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp +31:32.662 [INFO] TestAgent.server: Adding LAN server: server="Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" +31:32.662 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808.dc1 area=wan +31:32.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp +31:32.662 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp +31:32.662 [INFO] TestAgent: started state syncer +31:32.662 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp +31:32.711 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:32.711 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 +31:32.713 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:32.731 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:32.731 [DEBUG] TestAgent.server.raft: vote granted: from=74abd4b2-9417-7e9b-1e69-4a29ca18e808 term=2 tally=1 +31:32.731 [INFO] TestAgent.server.raft: election won: tally=1 +31:32.731 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" +31:32.732 [INFO] TestAgent.server: cluster leadership acquired +31:32.732 [INFO] TestAgent.server: New leader elected: payload=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 +31:32.750 [INFO] TestAgent.server: initializing acls +31:32.757 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:32.757 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:32.773 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:32.797 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:32.797 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:32.797 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:32.797 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 +31:32.798 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808.dc1 +31:32.800 [INFO] TestAgent.server: Updating LAN server: server="Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" +31:32.800 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808.dc1 area=wan +31:32.842 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:32.892 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:32.892 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:32.892 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:32.892 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:32.892 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:32.892 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 +31:32.892 [INFO] TestAgent.server: member joined, marking health alive: member=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 +31:32.908 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-74abd4b2-9417-7e9b-1e69-4a29ca18e808 +31:32.932 [INFO] TestAgent.server: federation state anti-entropy synced +31:33.020 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:33.028 [INFO] TestAgent: Synced node info +31:33.029 [DEBUG] TestAgent: Node info in sync +31:33.110 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:33454 latency=5.808409ms +31:33.111 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/7114bb8c-e119-7b89-5243-5f2da4e89d01 from=127.0.0.1:33456 latency=510.695µs +31:33.112 [INFO] TestAgent: Requesting shutdown +31:33.112 [INFO] TestAgent.server: shutting down server +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:33.112 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:33.112 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:33.112 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:33.112 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:33.116 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:33.125Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:33.125Z [INFO] agent.router.manager: shutting down +31:33.126 [INFO] TestAgent: consul server down +31:33.126 [INFO] TestAgent: shutdown complete +31:33.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp +31:33.126 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp +31:33.126 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp +31:33.628 [INFO] TestAgent: Waiting for endpoints to shut down +31:33.628 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenReadCommand_JSON (1.22s) PASS -ok github.com/hashicorp/consul/command/acl/token/read 0.829s +ok github.com/hashicorp/consul/command/acl/token/read 1.490s === RUN TestTokenUpdateCommand_noTabs === PAUSE TestTokenUpdateCommand_noTabs === RUN TestTokenUpdateCommand @@ -77633,374 +77931,373 @@ === RUN TestTokenUpdateCommand_JSON === PAUSE TestTokenUpdateCommand_JSON === CONT TestTokenUpdateCommand_noTabs +--- PASS: TestTokenUpdateCommand_noTabs (0.00s) === CONT TestTokenUpdateCommand_JSON -=== CONT TestTokenUpdateCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestTokenUpdateCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestTokenUpdateCommand_noTabs (0.00s) -=== RUN TestTokenUpdateCommand_JSON/update_with_policy_by_name === RUN TestTokenUpdateCommand/node-identity === RUN TestTokenUpdateCommand/node-identity-merge +=== RUN TestTokenUpdateCommand_JSON/update_with_policy_by_name === RUN TestTokenUpdateCommand/policy-name === RUN TestTokenUpdateCommand/policy-id === RUN TestTokenUpdateCommand/merge-description -2022-11-07T22:45:31.291Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:31.303Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:31.350 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 Address:127.0.0.1:24300}]" -45:31.350 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24300 [Follower]" leader= -45:31.351 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822.dc1 127.0.0.1 -45:31.351 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 127.0.0.1 -2022-11-07T22:45:31.351Z [INFO] agent.router: Initializing LAN area manager -45:31.351 [INFO] TestAgent.server: Adding LAN server: server="Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:31.351 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822.dc1 area=wan -45:31.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=tcp -45:31.351 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=udp -45:31.353 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24296 network=tcp -45:31.353 [INFO] TestAgent: started state syncer -45:31.353 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24301 network=tcp -45:31.401 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:31.404 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:31.404 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24300 [Candidate]" term=2 -45:31.412 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:31.412 [DEBUG] TestAgent.server.raft: vote granted: from=2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 term=2 tally=1 -45:31.412 [INFO] TestAgent.server.raft: election won: tally=1 -45:31.412 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24300 [Leader]" -45:31.412 [INFO] TestAgent.server: cluster leadership acquired -45:31.412 [INFO] TestAgent.server: New leader elected: payload=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 -45:31.417 [INFO] TestAgent.server: initializing acls -45:31.421 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:31.421 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:31.427 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:31.435 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:31.435 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:31.435 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:31.435 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 -45:31.435 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822.dc1 -45:31.435 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822.dc1 area=wan -45:31.435 [INFO] TestAgent.server: Updating LAN server: server="Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:31.446 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:31.461 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:31.461 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:31.461 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:31.461 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:31.461 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:31.461 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 -45:31.461 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 -45:31.467 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fc0410f-b3fb-65a9-5fbb-3bc89ad8d822 -45:31.472 [INFO] TestAgent.server: federation state anti-entropy synced -45:31.475 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:31.480 [INFO] TestAgent: Synced node info -45:31.480 [DEBUG] TestAgent: Node info in sync -45:31.600 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:46588 latency=5.711809ms -45:31.606 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:46588 latency=3.474437ms -45:31.608 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/f8d61f74-fd33-3843-d327-7ab59d81b085 from=127.0.0.1:46590 latency=386.167µs -45:31.611 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/f8d61f74-fd33-3843-d327-7ab59d81b085 from=127.0.0.1:46590 latency=2.792682ms -45:31.611 [INFO] TestAgent: Requesting shutdown -45:31.612 [INFO] TestAgent.server: shutting down server -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:31.612 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:31.612 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:31.612 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:31.612 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:31.615 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:31.618Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:31.618Z [INFO] agent.router.manager: shutting down -45:31.618 [INFO] TestAgent: consul server down -45:31.618 [INFO] TestAgent: shutdown complete -45:31.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=tcp -45:31.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=udp -45:31.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24296 network=tcp -45:32.118 [INFO] TestAgent: Waiting for endpoints to shut down -45:32.118 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenUpdateCommand_JSON (0.86s) - --- PASS: TestTokenUpdateCommand_JSON/update_with_policy_by_name (0.00s) +2021-10-05T16:31:34.311Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.339Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.393 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:49ac9b20-fb06-f13d-447d-4b90d4599a00 Address:127.0.0.1:12048}]" +31:34.393 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= +31:34.395 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49ac9b20-fb06-f13d-447d-4b90d4599a00.dc1 127.0.0.1 +31:34.395 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 127.0.0.1 +2021-10-05T16:31:34.395Z [INFO] agent.router: Initializing LAN area manager +31:34.396 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp +31:34.396 [INFO] TestAgent.server: Adding LAN server: server="Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +31:34.396 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00.dc1 area=wan +31:34.396 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp +31:34.397 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp +31:34.397 [INFO] TestAgent: started state syncer +31:34.397 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp +31:34.443 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.443 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 +31:34.448 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:34.456 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.456 [DEBUG] TestAgent.server.raft: vote granted: from=49ac9b20-fb06-f13d-447d-4b90d4599a00 term=2 tally=1 +31:34.456 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.456 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" +31:34.460 [INFO] TestAgent.server: cluster leadership acquired +31:34.460 [INFO] TestAgent.server: New leader elected: payload=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 +31:34.468 [INFO] TestAgent.server: initializing acls +31:34.475 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:34.475 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:34.491 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:34.515 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:34.515 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:34.515 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:34.516 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 +31:34.516 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-49ac9b20-fb06-f13d-447d-4b90d4599a00.dc1 +31:34.516 [INFO] TestAgent.server: Updating LAN server: server="Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +31:34.516 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00.dc1 area=wan +31:34.545 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.586 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.586 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.586 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.586 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.586 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.592 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 +31:34.592 [INFO] TestAgent.server: member joined, marking health alive: member=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 +31:34.609 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-49ac9b20-fb06-f13d-447d-4b90d4599a00 +31:34.609 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.622 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:34.641 [INFO] TestAgent: Synced node info +31:34.833 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:58132 latency=10.348815ms +31:34.845 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:58132 latency=4.880356ms +31:34.851 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/2eaa63b8-237e-216c-10b6-2eb70a978a17 from=127.0.0.1:58138 latency=1.277246ms +31:34.869 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/2eaa63b8-237e-216c-10b6-2eb70a978a17 from=127.0.0.1:58138 latency=17.291492ms +31:34.873 [INFO] TestAgent: Requesting shutdown +31:34.873 [INFO] TestAgent.server: shutting down server +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:34.873 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:34.873 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:34.873 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:34.873 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:34.888 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:34.896Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:34.896Z [INFO] agent.router.manager: shutting down +31:34.896 [INFO] TestAgent: consul server down +31:34.896 [INFO] TestAgent: shutdown complete +31:34.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp +31:34.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp +31:34.896 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp +31:35.400 [INFO] TestAgent: Waiting for endpoints to shut down +31:35.400 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenUpdateCommand_JSON (1.20s) + --- PASS: TestTokenUpdateCommand_JSON/update_with_policy_by_name (0.02s) === RUN TestTokenUpdateCommand/legacy-upgrade -2022-11-07T22:45:31.292Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:31.303Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:31.352 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:161b5f55-0318-e032-1189-5c222c9588cd Address:127.0.0.1:24307}]" -45:31.353 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24307 [Follower]" leader= -45:31.354 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-161b5f55-0318-e032-1189-5c222c9588cd.dc1 127.0.0.1 -45:31.354 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-161b5f55-0318-e032-1189-5c222c9588cd 127.0.0.1 -2022-11-07T22:45:31.354Z [INFO] agent.router: Initializing LAN area manager -45:31.354 [INFO] TestAgent.server: Adding LAN server: server="Node-161b5f55-0318-e032-1189-5c222c9588cd (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:31.354 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-161b5f55-0318-e032-1189-5c222c9588cd.dc1 area=wan -45:31.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=udp -45:31.355 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24302 network=tcp -45:31.355 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24303 network=tcp -45:31.355 [INFO] TestAgent: started state syncer -45:31.355 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24308 network=tcp -45:31.404 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -45:31.413 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:31.413 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24307 [Candidate]" term=2 -45:31.421 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:31.421 [DEBUG] TestAgent.server.raft: vote granted: from=161b5f55-0318-e032-1189-5c222c9588cd term=2 tally=1 -45:31.421 [INFO] TestAgent.server.raft: election won: tally=1 -45:31.421 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24307 [Leader]" -45:31.421 [INFO] TestAgent.server: cluster leadership acquired -45:31.421 [INFO] TestAgent.server: New leader elected: payload=Node-161b5f55-0318-e032-1189-5c222c9588cd -45:31.427 [INFO] TestAgent.server: initializing acls -45:31.431 [INFO] TestAgent.server: Created ACL 'global-management' policy -45:31.431 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -45:31.438 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -45:31.447 [INFO] TestAgent.server: Created ACL anonymous token from configuration -45:31.447 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -45:31.447 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -45:31.447 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-161b5f55-0318-e032-1189-5c222c9588cd -45:31.447 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-161b5f55-0318-e032-1189-5c222c9588cd.dc1 -45:31.447 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-161b5f55-0318-e032-1189-5c222c9588cd.dc1 area=wan -45:31.447 [INFO] TestAgent.server: Updating LAN server: server="Node-161b5f55-0318-e032-1189-5c222c9588cd (Addr: tcp/127.0.0.1:24307) (DC: dc1)" -45:31.457 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:31.475 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:31.475 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:31.475 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:31.475 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:31.475 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:31.475 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-161b5f55-0318-e032-1189-5c222c9588cd -45:31.475 [INFO] TestAgent.server: member joined, marking health alive: member=Node-161b5f55-0318-e032-1189-5c222c9588cd -45:31.478 [INFO] TestAgent.server: federation state anti-entropy synced -45:31.484 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-161b5f55-0318-e032-1189-5c222c9588cd -45:31.730 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:31.735 [INFO] TestAgent: Synced node info -45:31.756 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47932 latency=3.721672ms -45:31.759 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:47932 latency=2.939636ms -45:31.763 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/create from=127.0.0.1:47932 latency=3.728264ms -45:31.765 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47934 latency=135.098µs -45:31.769 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47934 latency=3.707442ms -45:31.772 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47936 latency=584.781µs -45:31.776 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47936 latency=3.270567ms -45:31.778 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47938 latency=405.359µs -45:31.781 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47938 latency=3.158768ms -45:31.783 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47940 latency=204.191µs -45:31.786 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47940 latency=3.304856ms -45:31.788 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47942 latency=333.081µs -45:31.792 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/58a5937f-7f63-0794-383a-20ac9f2c3068 from=127.0.0.1:47942 latency=2.82825ms -45:31.792 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=110.887µs -45:31.818 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=146.612µs -45:31.844 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=185.461µs -45:31.870 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=202.793µs -45:31.896 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=149.521µs -45:31.922 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=395.315µs -45:31.949 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=204.556µs -45:31.975 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=495.981µs -45:32.001 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=206.313µs -45:32.027 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=183.961µs -45:32.053 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=210.886µs -45:32.079 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=189.282µs -45:32.105 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=253.29µs -45:32.132 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=177.997µs -45:32.158 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=209.86µs -45:32.184 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=199.029µs -45:32.210 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=227.189µs -45:32.236 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=199.409µs -45:32.262 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=174.22µs -45:32.289 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=216.628µs -45:32.315 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=188.993µs -45:32.341 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=241.603µs -45:32.367 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=178.021µs -45:32.393 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=189.245µs -45:32.419 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=164.227µs -45:32.445 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=196.787µs -45:32.471 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47932 latency=207.638µs -45:32.473 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/3358322b-6d47-7a98-b37b-03f0b203923b from=127.0.0.1:47946 latency=282.918µs -45:32.481 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/3358322b-6d47-7a98-b37b-03f0b203923b from=127.0.0.1:47946 latency=6.285346ms -45:32.481 [INFO] TestAgent: Requesting shutdown -45:32.481 [INFO] TestAgent.server: shutting down server -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:32.481 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:32.481 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:32.481 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -45:32.481 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:32.485 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:32.488Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:32.488Z [INFO] agent.router.manager: shutting down -45:32.488 [INFO] TestAgent: consul server down -45:32.488 [INFO] TestAgent: shutdown complete -45:32.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=tcp -45:32.488 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24302 network=udp -45:32.488 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24303 network=tcp -45:32.988 [INFO] TestAgent: Waiting for endpoints to shut down -45:32.988 [INFO] TestAgent: Endpoints down ---- PASS: TestTokenUpdateCommand (1.73s) - --- PASS: TestTokenUpdateCommand/node-identity (0.01s) - --- PASS: TestTokenUpdateCommand/node-identity-merge (0.01s) - --- PASS: TestTokenUpdateCommand/policy-name (0.01s) - --- PASS: TestTokenUpdateCommand/policy-id (0.01s) - --- PASS: TestTokenUpdateCommand/merge-description (0.00s) - --- PASS: TestTokenUpdateCommand/legacy-upgrade (0.01s) +2021-10-05T16:31:34.326Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:34.375Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:34.466 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a0a20b5-351c-1c5a-a25c-934b2cd39c66 Address:127.0.0.1:12055}]" +31:34.467 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66.dc1 127.0.0.1 +31:34.467 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 127.0.0.1 +2021-10-05T16:31:34.467Z [INFO] agent.router: Initializing LAN area manager +31:34.467 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=udp +31:34.467 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12055 [Follower]" leader= +31:34.468 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66.dc1 area=wan +31:34.468 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12050 network=tcp +31:34.469 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12051 network=tcp +31:34.470 [INFO] TestAgent.server: Adding LAN server: server="Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 (Addr: tcp/127.0.0.1:12055) (DC: dc1)" +31:34.470 [INFO] TestAgent: started state syncer +31:34.474 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12056 network=tcp +31:34.506 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:34.506 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12055 [Candidate]" term=2 +31:34.518 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +31:34.530 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:34.530 [DEBUG] TestAgent.server.raft: vote granted: from=4a0a20b5-351c-1c5a-a25c-934b2cd39c66 term=2 tally=1 +31:34.530 [INFO] TestAgent.server.raft: election won: tally=1 +31:34.530 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12055 [Leader]" +31:34.530 [INFO] TestAgent.server: cluster leadership acquired +31:34.531 [INFO] TestAgent.server: New leader elected: payload=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 +31:34.549 [INFO] TestAgent.server: initializing acls +31:34.553 [INFO] TestAgent.server: Created ACL 'global-management' policy +31:34.553 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +31:34.570 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +31:34.596 [INFO] TestAgent.server: Created ACL anonymous token from configuration +31:34.596 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +31:34.596 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +31:34.596 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 +31:34.596 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66.dc1 +31:34.597 [INFO] TestAgent.server: Updating LAN server: server="Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 (Addr: tcp/127.0.0.1:12055) (DC: dc1)" +31:34.597 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66.dc1 area=wan +31:34.641 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:34.708 [INFO] TestAgent: Synced node info +31:34.732 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:34.732 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:34.732 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:34.732 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:34.732 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:34.732 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 +31:34.732 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 +31:34.765 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/policy from=127.0.0.1:47692 latency=20.648153ms +31:34.765 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a0a20b5-351c-1c5a-a25c-934b2cd39c66 +31:34.780 [INFO] TestAgent.server: federation state anti-entropy synced +31:34.802 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:47692 latency=20.722882ms +31:34.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/create from=127.0.0.1:47692 latency=15.074531ms +31:34.830 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47696 latency=3.249505ms +31:34.838 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47696 latency=7.413206ms +31:34.842 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47698 latency=351.954µs +31:34.855 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47698 latency=6.882148ms +31:34.869 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47702 latency=597.758µs +31:34.893 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47702 latency=16.672412ms +31:34.900 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47704 latency=428.204µs +31:34.924 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47704 latency=20.043527ms +31:34.928 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47706 latency=376.104µs +31:34.937 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/0e3c6864-e6e5-fd9a-4eff-ee41e7500cc9 from=127.0.0.1:47706 latency=4.204207ms +31:34.944 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=264.136µs +31:34.973 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=1.203311ms +31:35.002 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=299.784µs +31:35.028 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=340.41µs +31:35.059 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=400.478µs +31:35.087 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=384.457µs +31:35.115 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=473.937µs +31:35.148 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=665.313µs +31:35.176 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=273.103µs +31:35.207 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=221.804µs +31:35.236 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=208.636µs +31:35.262 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=219.103µs +31:35.289 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=250.497µs +31:35.320 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=299.704µs +31:35.352 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=3.850988ms +31:35.384 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=271.22µs +31:35.412 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=248.562µs +31:35.444 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=333.551µs +31:35.472 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=327.273µs +31:35.500 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=254.835µs +31:35.532 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=241.238µs +31:35.564 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=219.879µs +31:35.596 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=263.552µs +31:35.628 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/self from=127.0.0.1:47692 latency=491.966µs +31:35.634 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/acl/token/d06d6404-c5f4-2556-a906-261bce62e03f from=127.0.0.1:47710 latency=734.723µs +31:35.652 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token/d06d6404-c5f4-2556-a906-261bce62e03f from=127.0.0.1:47710 latency=17.313047ms +31:35.654 [INFO] TestAgent: Requesting shutdown +31:35.654 [INFO] TestAgent.server: shutting down server +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:35.654 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:35.654 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:35.654 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:35.654 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:35.668 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:35.676Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:35.676Z [INFO] agent.router.manager: shutting down +31:35.676 [INFO] TestAgent: consul server down +31:35.676 [INFO] TestAgent: shutdown complete +31:35.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=tcp +31:35.676 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12050 network=udp +31:35.676 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12051 network=tcp +31:36.181 [INFO] TestAgent: Waiting for endpoints to shut down +31:36.181 [INFO] TestAgent: Endpoints down +--- PASS: TestTokenUpdateCommand (1.97s) + --- PASS: TestTokenUpdateCommand/node-identity (0.02s) + --- PASS: TestTokenUpdateCommand/node-identity-merge (0.02s) + --- PASS: TestTokenUpdateCommand/policy-name (0.04s) + --- PASS: TestTokenUpdateCommand/policy-id (0.03s) + --- PASS: TestTokenUpdateCommand/merge-description (0.02s) + --- PASS: TestTokenUpdateCommand/legacy-upgrade (0.02s) PASS -ok github.com/hashicorp/consul/command/acl/token/update 1.762s +ok github.com/hashicorp/consul/command/acl/token/update 2.300s === RUN TestConfigFail === PAUSE TestConfigFail === RUN TestRetryJoin [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:34.408Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:34.418Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:34.437 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:86eebcb8-616b-22f3-428e-dc82ceebf650 Address:127.0.0.1:14097}]" -45:34.437 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= -45:34.439 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-86eebcb8-616b-22f3-428e-dc82ceebf650.dc1 127.0.0.1 -45:34.440 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-86eebcb8-616b-22f3-428e-dc82ceebf650 127.0.0.1 -2022-11-07T22:45:34.440Z [INFO] agent.router: Initializing LAN area manager -45:34.440 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp -45:34.440 [INFO] TestAgent.server: Adding LAN server: server="Node-86eebcb8-616b-22f3-428e-dc82ceebf650 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" -45:34.440 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp -45:34.440 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-86eebcb8-616b-22f3-428e-dc82ceebf650.dc1 area=wan -45:34.441 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp -45:34.441 [INFO] TestAgent: started state syncer -45:34.441 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" -45:34.441 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" -45:34.441 [INFO] TestAgent: Joining cluster...: cluster=WAN -45:34.441 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:14089] -45:34.441 [INFO] TestAgent: Joining cluster...: cluster=LAN -45:34.441 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:14088] -45:34.441 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:14089 -45:34.441 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp -45:34.441 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:14088 -45:34.442 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e989306-4b36-d145-03e0-4cdad6b739ac.dc1 127.0.0.1 -45:34.442 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -45:34.442 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 -45:34.442 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e989306-4b36-d145-03e0-4cdad6b739ac.dc1 area=wan -45:34.442 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e989306-4b36-d145-03e0-4cdad6b739ac 127.0.0.1 -45:34.442 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -45:34.442 [DEBUG] TestAgent: systemd notify failed: error="No socket" -45:34.442 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 -45:34.442 [INFO] TestAgent.server: Adding LAN server: server="Node-1e989306-4b36-d145-03e0-4cdad6b739ac (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -45:34.503 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:34.503 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 -45:34.511 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:34.511 [DEBUG] TestAgent.server.raft: vote granted: from=86eebcb8-616b-22f3-428e-dc82ceebf650 term=2 tally=1 -45:34.511 [INFO] TestAgent.server.raft: election won: tally=1 -45:34.511 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" -45:34.511 [INFO] TestAgent.server: cluster leadership acquired -45:34.511 [INFO] TestAgent.server: New leader elected: payload=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:34.517 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14097 -45:34.526 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:34.546 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:34.546 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:34.546 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:34.546 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:34.546 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:34.546 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 other=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.546 [INFO] TestAgent.server: member joined, marking health alive: member=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:34.552 [INFO] TestAgent.server: federation state anti-entropy synced -45:34.559 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-1e989306-4b36-d145-03e0-4cdad6b739ac other=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:34.559 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.596 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader -45:34.598 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:34.604 [INFO] TestAgent: Synced node info -45:34.604 [DEBUG] TestAgent: Node info in sync -45:34.737 [INFO] TestAgent: Requesting shutdown -45:34.737 [INFO] TestAgent.server: shutting down server -45:34.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:34.737 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:34.737 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:34.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:34.737 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:34.737 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:34.737 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:34.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:34.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:34.737 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:34.741 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:34.745Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:34.745Z [INFO] agent.router.manager: shutting down -45:34.745 [INFO] TestAgent: consul server down -45:34.745 [INFO] TestAgent: shutdown complete -45:34.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp -45:34.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp -45:34.745 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp -45:35.245 [INFO] TestAgent: Waiting for endpoints to shut down -45:35.245 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:34.162Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:34.173Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:34.194 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e989306-4b36-d145-03e0-4cdad6b739ac Address:127.0.0.1:14090}]" -45:34.194 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= -45:34.195 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e989306-4b36-d145-03e0-4cdad6b739ac.dc1 127.0.0.1 -45:34.195 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e989306-4b36-d145-03e0-4cdad6b739ac 127.0.0.1 -2022-11-07T22:45:34.195Z [INFO] agent.router: Initializing LAN area manager -45:34.195 [INFO] TestAgent.server: Adding LAN server: server="Node-1e989306-4b36-d145-03e0-4cdad6b739ac (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -45:34.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp -45:34.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp -45:34.196 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e989306-4b36-d145-03e0-4cdad6b739ac.dc1 area=wan -45:34.196 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp -45:34.196 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp -45:34.196 [INFO] TestAgent: started state syncer -45:34.243 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:34.243 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 -45:34.251 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:34.252 [DEBUG] TestAgent.server.raft: vote granted: from=1e989306-4b36-d145-03e0-4cdad6b739ac term=2 tally=1 -45:34.252 [INFO] TestAgent.server.raft: election won: tally=1 -45:34.252 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" -45:34.252 [INFO] TestAgent.server: cluster leadership acquired -45:34.252 [INFO] TestAgent.server: New leader elected: payload=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.257 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 -45:34.268 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:34.285 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:34.285 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:34.285 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:34.285 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:34.285 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:34.285 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.285 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.297 [INFO] TestAgent.server: federation state anti-entropy synced -45:34.326 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:34.329 [INFO] TestAgent: Synced node info -45:34.329 [DEBUG] TestAgent: Node info in sync -45:34.441 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:53280 -45:34.441 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:33582 -45:34.442 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-86eebcb8-616b-22f3-428e-dc82ceebf650.dc1 127.0.0.1 -45:34.442 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-86eebcb8-616b-22f3-428e-dc82ceebf650.dc1 area=wan -45:34.442 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-86eebcb8-616b-22f3-428e-dc82ceebf650 127.0.0.1 -45:34.442 [INFO] TestAgent.server: Adding LAN server: server="Node-86eebcb8-616b-22f3-428e-dc82ceebf650 (Addr: tcp/127.0.0.1:14097) (DC: dc1)" -45:34.442 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 other=Node-1e989306-4b36-d145-03e0-4cdad6b739ac -45:34.442 [INFO] TestAgent.server: member joined, marking health alive: member=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:34.640 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader -45:34.640 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:34.641 [INFO] TestAgent.server: New leader elected: payload=Node-86eebcb8-616b-22f3-428e-dc82ceebf650 -45:35.245 [INFO] TestAgent: Requesting shutdown -45:35.245 [INFO] TestAgent.server: shutting down server -45:35.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:35.245 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:35.245 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:35.245 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:35.245 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:35.245 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:35.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:35.245 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:35.245 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:35.245 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:35.250 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:35.255Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:35.255Z [INFO] agent.router.manager: shutting down -45:35.255 [INFO] TestAgent: consul server down -45:35.255 [INFO] TestAgent: shutdown complete -45:35.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp -45:35.255 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp -45:35.255 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp -45:35.696 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-86eebcb8-616b-22f3-428e-dc82ceebf650 (timeout reached) -45:35.755 [INFO] TestAgent: Waiting for endpoints to shut down -45:35.755 [INFO] TestAgent: Endpoints down ---- PASS: TestRetryJoin (1.63s) +2021-10-05T16:31:45.105Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.131Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.180 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:54399593-f2c5-35c2-9402-9739ff77248a Address:127.0.0.1:21244}]" +31:45.180 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= +31:45.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-54399593-f2c5-35c2-9402-9739ff77248a.dc1 127.0.0.1 +31:45.188 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-54399593-f2c5-35c2-9402-9739ff77248a 127.0.0.1 +2021-10-05T16:31:45.188Z [INFO] agent.router: Initializing LAN area manager +31:45.189 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp +31:45.189 [INFO] TestAgent.server: Adding LAN server: server="Node-54399593-f2c5-35c2-9402-9739ff77248a (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +31:45.189 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-54399593-f2c5-35c2-9402-9739ff77248a.dc1 area=wan +31:45.189 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp +31:45.190 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp +31:45.190 [INFO] TestAgent: started state syncer +31:45.190 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp +31:45.190 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws gce mdns os packet" +31:45.190 [INFO] TestAgent: Joining cluster...: cluster=LAN +31:45.190 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:21235] +31:45.190 [INFO] TestAgent: Retry join is supported for the following discovery methods: cluster=WAN discovery_methods="aliyun aws gce mdns os packet" +31:45.190 [INFO] TestAgent: Joining cluster...: cluster=WAN +31:45.190 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:21236] +31:45.192 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:21235 +31:45.225 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:21236 +31:45.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a66d227-00e8-bbd9-83b2-416b734a84dd.dc1 127.0.0.1 +31:45.233 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +31:45.233 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=WAN num_agents=1 +31:45.233 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd.dc1 area=wan +31:45.243 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.243 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 +31:45.260 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a66d227-00e8-bbd9-83b2-416b734a84dd 127.0.0.1 +31:45.261 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +31:45.261 [DEBUG] TestAgent: systemd notify failed: error="No socket" +31:45.261 [INFO] TestAgent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 +31:45.261 [INFO] TestAgent.server: Adding LAN server: server="Node-4a66d227-00e8-bbd9-83b2-416b734a84dd (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +31:45.281 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.281 [DEBUG] TestAgent.server.raft: vote granted: from=54399593-f2c5-35c2-9402-9739ff77248a term=2 tally=1 +31:45.281 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.281 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" +31:45.281 [INFO] TestAgent.server: cluster leadership acquired +31:45.281 [INFO] TestAgent.server: New leader elected: payload=Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.304 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21244 +31:45.352 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.389 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.389 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.389 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.389 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.389 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.390 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-54399593-f2c5-35c2-9402-9739ff77248a other=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:45.390 [INFO] TestAgent.server: member joined, marking health alive: member=Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.398 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd other=Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.398 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:45.404 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.404 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.404 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.421 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:45.424 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.430 [INFO] TestAgent: Synced node info +31:45.605 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.605 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.605 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.625 [INFO] TestAgent: Requesting shutdown +31:45.626 [INFO] TestAgent.server: shutting down server +31:45.626 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.626 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.626 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.626 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.626 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.626 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:45.626 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.626 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.626 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.626 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.636 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.646Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.646Z [INFO] agent.router.manager: shutting down +31:45.646 [INFO] TestAgent: consul server down +31:45.646 [INFO] TestAgent: shutdown complete +31:45.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp +31:45.646 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp +31:45.646 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp +31:46.147 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.147 [INFO] TestAgent: Endpoints down +2021-10-05T16:31:44.679Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:44.699Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:44.798 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a66d227-00e8-bbd9-83b2-416b734a84dd Address:127.0.0.1:21237}]" +31:44.799 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a66d227-00e8-bbd9-83b2-416b734a84dd.dc1 127.0.0.1 +31:44.800 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= +31:44.801 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a66d227-00e8-bbd9-83b2-416b734a84dd 127.0.0.1 +2021-10-05T16:31:44.801Z [INFO] agent.router: Initializing LAN area manager +31:44.801 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp +31:44.801 [INFO] TestAgent.server: Adding LAN server: server="Node-4a66d227-00e8-bbd9-83b2-416b734a84dd (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +31:44.802 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd.dc1 area=wan +31:44.802 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp +31:44.803 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp +31:44.803 [INFO] TestAgent: started state syncer +31:44.803 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp +31:44.872 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:44.872 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 +31:44.920 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:44.920 [DEBUG] TestAgent.server.raft: vote granted: from=4a66d227-00e8-bbd9-83b2-416b734a84dd term=2 tally=1 +31:44.920 [INFO] TestAgent.server.raft: election won: tally=1 +31:44.920 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" +31:44.924 [INFO] TestAgent.server: cluster leadership acquired +31:44.924 [INFO] TestAgent.server: New leader elected: payload=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:44.960 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21237 +31:44.988 [INFO] TestAgent: Synced node info +31:45.020 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.049 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.049 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.049 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.049 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.049 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.049 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:45.049 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:45.068 [INFO] TestAgent.server: federation state anti-entropy synced +31:45.225 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:60436 +31:45.233 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-54399593-f2c5-35c2-9402-9739ff77248a.dc1 127.0.0.1 +31:45.233 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-54399593-f2c5-35c2-9402-9739ff77248a.dc1 area=wan +31:45.239 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:48086 +31:45.240 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-54399593-f2c5-35c2-9402-9739ff77248a 127.0.0.1 +31:45.240 [INFO] TestAgent.server: Adding LAN server: server="Node-54399593-f2c5-35c2-9402-9739ff77248a (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +31:45.240 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-54399593-f2c5-35c2-9402-9739ff77248a other=Node-4a66d227-00e8-bbd9-83b2-416b734a84dd +31:45.240 [INFO] TestAgent.server: member joined, marking health alive: member=Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.391 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.391 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.391 [INFO] TestAgent.server: New leader elected: payload=Node-54399593-f2c5-35c2-9402-9739ff77248a +31:45.597 [DEBUG] TestAgent.server.serf.lan: serf: messageUserEventType: consul:new-leader +31:45.597 [DEBUG] TestAgent.server.serf.lan: serf: messageJoinType: Node-54399593-f2c5-35c2-9402-9739ff77248a +31:46.147 [INFO] TestAgent: Requesting shutdown +31:46.147 [INFO] TestAgent.server: shutting down server +31:46.147 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:46.147 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:46.147 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:46.147 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:46.147 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:46.147 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:46.147 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:46.147 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:46.147 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:46.147 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:46.156 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:46.163Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:46.163Z [INFO] agent.router.manager: shutting down +31:46.163 [INFO] TestAgent: consul server down +31:46.163 [INFO] TestAgent: shutdown complete +31:46.163 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp +31:46.163 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp +31:46.163 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp +31:46.304 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-54399593-f2c5-35c2-9402-9739ff77248a (timeout reached) +31:46.664 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.664 [INFO] TestAgent: Endpoints down +--- PASS: TestRetryJoin (2.04s) === RUN TestRetryJoinFail === PAUSE TestRetryJoinFail === RUN TestRetryJoinWanFail @@ -78010,41 +78307,41 @@ === RUN TestBadDataDirPermissions === PAUSE TestBadDataDirPermissions === CONT TestConfigFail -=== CONT TestBadDataDirPermissions -=== CONT TestRetryJoinWanFail === RUN TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter= === CONT TestProtectDataDir +=== CONT TestBadDataDirPermissions +=== CONT TestRetryJoinWanFail === CONT TestRetryJoinFail ---- PASS: TestBadDataDirPermissions (0.02s) ---- PASS: TestProtectDataDir (0.02s) +--- PASS: TestProtectDataDir (0.05s) +--- PASS: TestBadDataDirPermissions (0.05s) === RUN TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter=foo_some-other-arg +--- PASS: TestRetryJoinWanFail (0.13s) === RUN TestConfigFail/agent_-server_-bind=10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_0.0.0.0_-bind_10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_::_-bind_10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_[::]_-bind_10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_0.0.0.0_-bind_10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_::_-bind_10.0.0.1 -=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_[::]_-bind_10.0.0.1 ---- PASS: TestConfigFail (0.34s) - --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter= (0.05s) - --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter=foo_some-other-arg (0.03s) - --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1 (0.04s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_0.0.0.0_-bind_10.0.0.1 (0.04s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_::_-bind_10.0.0.1 (0.04s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise_[::]_-bind_10.0.0.1 (0.04s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_0.0.0.0_-bind_10.0.0.1 (0.03s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_::_-bind_10.0.0.1 (0.04s) - --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul668731631_-advertise-wan_[::]_-bind_10.0.0.1 (0.04s) ---- PASS: TestRetryJoinFail (0.54s) ---- PASS: TestRetryJoinWanFail (0.57s) +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_0.0.0.0_-bind_10.0.0.1 +--- PASS: TestRetryJoinFail (0.64s) +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_::_-bind_10.0.0.1 +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_[::]_-bind_10.0.0.1 +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_0.0.0.0_-bind_10.0.0.1 +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_::_-bind_10.0.0.1 +=== RUN TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_[::]_-bind_10.0.0.1 +--- PASS: TestConfigFail (1.51s) + --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter= (0.12s) + --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1_-datacenter=foo_some-other-arg (0.15s) + --- PASS: TestConfigFail/agent_-server_-bind=10.0.0.1 (0.14s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_0.0.0.0_-bind_10.0.0.1 (0.36s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_::_-bind_10.0.0.1 (0.14s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise_[::]_-bind_10.0.0.1 (0.10s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_0.0.0.0_-bind_10.0.0.1 (0.08s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_::_-bind_10.0.0.1 (0.18s) + --- PASS: TestConfigFail/agent_-server_-data-dir_/tmp/consul-test/TestConfigFail-consul987952697_-advertise-wan_[::]_-bind_10.0.0.1 (0.23s) PASS -ok github.com/hashicorp/consul/command/agent 2.227s +ok github.com/hashicorp/consul/command/agent 3.643s === RUN TestCatalogCommand_noTabs === PAUSE TestCatalogCommand_noTabs === CONT TestCatalogCommand_noTabs --- PASS: TestCatalogCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/catalog 0.005s +ok github.com/hashicorp/consul/command/catalog 0.059s === RUN TestCatalogListDatacentersCommand_noTabs === PAUSE TestCatalogListDatacentersCommand_noTabs === RUN TestCatalogListDatacentersCommand_Validation @@ -78052,73 +78349,72 @@ === RUN TestCatalogListDatacentersCommand === PAUSE TestCatalogListDatacentersCommand === CONT TestCatalogListDatacentersCommand_noTabs +--- PASS: TestCatalogListDatacentersCommand_noTabs (0.00s) === CONT TestCatalogListDatacentersCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestCatalogListDatacentersCommand_Validation ---- PASS: TestCatalogListDatacentersCommand_noTabs (0.00s) --- PASS: TestCatalogListDatacentersCommand_Validation (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:34.489Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:34.500Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:34.519 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6317a518-42c8-602d-9b93-3bd5aaa158af Address:127.0.0.1:24300}]" -45:34.519 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24300 [Follower]" leader= -45:34.519 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6317a518-42c8-602d-9b93-3bd5aaa158af.dc1 127.0.0.1 -45:34.520 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6317a518-42c8-602d-9b93-3bd5aaa158af 127.0.0.1 -2022-11-07T22:45:34.520Z [INFO] agent.router: Initializing LAN area manager -45:34.520 [INFO] TestAgent.server: Adding LAN server: server="Node-6317a518-42c8-602d-9b93-3bd5aaa158af (Addr: tcp/127.0.0.1:24300) (DC: dc1)" -45:34.520 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6317a518-42c8-602d-9b93-3bd5aaa158af.dc1 area=wan -45:34.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=udp -45:34.520 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=tcp -45:34.520 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24296 network=tcp -45:34.521 [INFO] TestAgent: started state syncer -45:34.521 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24301 network=tcp -45:34.588 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:34.588 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24300 [Candidate]" term=2 -45:34.597 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:34.597 [DEBUG] TestAgent.server.raft: vote granted: from=6317a518-42c8-602d-9b93-3bd5aaa158af term=2 tally=1 -45:34.597 [INFO] TestAgent.server.raft: election won: tally=1 -45:34.597 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24300 [Leader]" -45:34.597 [INFO] TestAgent.server: cluster leadership acquired -45:34.597 [INFO] TestAgent.server: New leader elected: payload=Node-6317a518-42c8-602d-9b93-3bd5aaa158af -45:34.603 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:24300 -45:34.617 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:34.636 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:34.636 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:34.636 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:34.636 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:34.636 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:34.636 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6317a518-42c8-602d-9b93-3bd5aaa158af -45:34.636 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6317a518-42c8-602d-9b93-3bd5aaa158af -45:34.645 [INFO] TestAgent.server: federation state anti-entropy synced -45:34.650 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:34.653 [INFO] TestAgent: Synced node info -45:34.758 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/datacenters from=127.0.0.1:46616 latency=401.759µs -45:34.758 [INFO] TestAgent: Requesting shutdown -45:34.758 [INFO] TestAgent.server: shutting down server -45:34.758 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:34.758 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:34.758 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:34.758 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:34.758 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:34.758 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:34.758 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:34.758 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:34.758 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:34.758 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:34.761 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:34.764Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:34.764Z [INFO] agent.router.manager: shutting down -45:34.764 [INFO] TestAgent: consul server down -45:34.764 [INFO] TestAgent: shutdown complete -45:34.764 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=tcp -45:34.764 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=udp -45:34.764 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24296 network=tcp -45:35.264 [INFO] TestAgent: Waiting for endpoints to shut down -45:35.264 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogListDatacentersCommand (0.81s) +2021-10-05T16:31:45.621Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:45.657Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:45.707 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9b398136-6ae9-8c63-e1fc-2b1239a12779 Address:127.0.0.1:31447}]" +31:45.708 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9b398136-6ae9-8c63-e1fc-2b1239a12779.dc1 127.0.0.1 +31:45.708 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= +31:45.712 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 127.0.0.1 +2021-10-05T16:31:45.712Z [INFO] agent.router: Initializing LAN area manager +31:45.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp +31:45.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp +31:45.713 [INFO] TestAgent.server: Adding LAN server: server="Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" +31:45.714 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp +31:45.714 [INFO] TestAgent: started state syncer +31:45.714 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp +31:45.714 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9b398136-6ae9-8c63-e1fc-2b1239a12779.dc1 area=wan +31:45.760 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:45.760 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 +31:45.788 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:45.788 [DEBUG] TestAgent.server.raft: vote granted: from=9b398136-6ae9-8c63-e1fc-2b1239a12779 term=2 tally=1 +31:45.788 [INFO] TestAgent.server.raft: election won: tally=1 +31:45.788 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" +31:45.790 [INFO] TestAgent.server: cluster leadership acquired +31:45.790 [INFO] TestAgent.server: New leader elected: payload=Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 +31:45.823 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 +31:45.848 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:45.924 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/datacenters from=127.0.0.1:45670 latency=448.949µs +31:45.928 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:45.928 [INFO] TestAgent: Requesting shutdown +31:45.928 [INFO] TestAgent.server: shutting down server +31:45.928 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:45.928 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:45.928 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:45.928 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:45.932 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:45.932 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 +31:45.932 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 +31:45.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:45.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:45.936 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:45.936 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:45.928 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +31:45.936 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:45.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:45.936 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:45.936 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:45.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:45.940 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:45.945Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:45.945Z [INFO] agent.router.manager: shutting down +31:45.956 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-9b398136-6ae9-8c63-e1fc-2b1239a12779 127.0.0.1 31445 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:9b398136-6ae9-8c63-e1fc-2b1239a12779 port:31447 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:31446] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +31:45.956 [INFO] TestAgent: consul server down +31:45.956 [INFO] TestAgent: shutdown complete +31:45.956 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp +31:45.956 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp +31:45.956 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp +31:46.460 [INFO] TestAgent: Waiting for endpoints to shut down +31:46.460 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogListDatacentersCommand (0.90s) PASS -ok github.com/hashicorp/consul/command/catalog/list/dc 0.838s +ok github.com/hashicorp/consul/command/catalog/list/dc 1.016s === RUN TestCatalogListNodesCommand_noTabs === PAUSE TestCatalogListNodesCommand_noTabs === RUN TestCatalogListNodesCommand_Validation @@ -78130,11 +78426,11 @@ --- SKIP: TestCatalogListNodesCommand_verticalBar (0.00s) === CONT TestCatalogListNodesCommand_noTabs === CONT TestCatalogListNodesCommand -=== CONT TestCatalogListNodesCommand_Validation [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +--- PASS: TestCatalogListNodesCommand_noTabs (0.00s) +=== CONT TestCatalogListNodesCommand_Validation --- PASS: TestCatalogListNodesCommand_Validation (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] ---- PASS: TestCatalogListNodesCommand_noTabs (0.00s) [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestCatalogListNodesCommand/simple === RUN TestCatalogListNodesCommand/detailed @@ -78143,79 +78439,79 @@ === RUN TestCatalogListNodesCommand/near === RUN TestCatalogListNodesCommand/service_present === RUN TestCatalogListNodesCommand/service_missing -2022-11-07T22:45:37.042Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:37.053Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:37.080 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4a793672-83dd-9b20-5e48-bbad18e24b3f Address:127.0.0.1:30426}]" -45:37.080 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= -45:37.081 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4a793672-83dd-9b20-5e48-bbad18e24b3f.dc1 127.0.0.1 -45:37.081 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4a793672-83dd-9b20-5e48-bbad18e24b3f 127.0.0.1 -2022-11-07T22:45:37.081Z [INFO] agent.router: Initializing LAN area manager -45:37.081 [INFO] TestAgent.server: Adding LAN server: server="Node-4a793672-83dd-9b20-5e48-bbad18e24b3f (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:37.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp -45:37.081 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp -45:37.081 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4a793672-83dd-9b20-5e48-bbad18e24b3f.dc1 area=wan -45:37.082 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp -45:37.082 [INFO] TestAgent: started state syncer -45:37.082 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp -45:37.129 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:37.129 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 -45:37.138 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:37.138 [DEBUG] TestAgent.server.raft: vote granted: from=4a793672-83dd-9b20-5e48-bbad18e24b3f term=2 tally=1 -45:37.138 [INFO] TestAgent.server.raft: election won: tally=1 -45:37.138 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" -45:37.138 [INFO] TestAgent.server: cluster leadership acquired -45:37.138 [INFO] TestAgent.server: New leader elected: payload=Node-4a793672-83dd-9b20-5e48-bbad18e24b3f -45:37.144 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30426 -45:37.149 [INFO] TestAgent: Synced node info -45:37.149 [DEBUG] TestAgent: Node info in sync -45:37.161 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:37.174 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:37.174 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:37.174 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:37.174 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:37.174 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:37.174 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4a793672-83dd-9b20-5e48-bbad18e24b3f -45:37.174 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4a793672-83dd-9b20-5e48-bbad18e24b3f -45:37.179 [INFO] TestAgent.server: federation state anti-entropy synced -45:37.280 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:35174 latency=439.673µs -45:37.282 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:35176 latency=1.215882ms -45:37.284 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?node-meta=foo%3Abar from=127.0.0.1:35178 latency=273.299µs -45:37.286 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?filter=Meta.foo+%3D%3D+bar from=127.0.0.1:35180 latency=547.551µs -45:37.288 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?near=_agent from=127.0.0.1:35182 latency=214.885µs -45:37.290 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/consul from=127.0.0.1:35184 latency=615.285µs -45:37.291 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/this-service-will-literally-never-exist from=127.0.0.1:35186 latency=176.011µs -45:37.292 [INFO] TestAgent: Requesting shutdown -45:37.292 [INFO] TestAgent.server: shutting down server -45:37.292 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:37.292 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:37.292 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:37.292 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:37.292 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:37.292 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:37.292 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:37.292 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:37.292 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:37.292 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:37.295 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:37.299Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:37.299Z [INFO] agent.router.manager: shutting down -45:37.299 [INFO] TestAgent: consul server down -45:37.299 [INFO] TestAgent: shutdown complete -45:37.299 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp -45:37.299 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp -45:37.299 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp -45:37.799 [INFO] TestAgent: Waiting for endpoints to shut down -45:37.799 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogListNodesCommand (0.79s) - --- PASS: TestCatalogListNodesCommand/simple (0.00s) - --- PASS: TestCatalogListNodesCommand/detailed (0.00s) - --- PASS: TestCatalogListNodesCommand/node-meta (0.00s) - --- PASS: TestCatalogListNodesCommand/filter (0.00s) - --- PASS: TestCatalogListNodesCommand/near (0.00s) +2021-10-05T16:31:46.824Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:46.850Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:46.902 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b98eb076-3092-203a-628d-da0c02b94895 Address:127.0.0.1:25321}]" +31:46.903 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= +31:46.903 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b98eb076-3092-203a-628d-da0c02b94895.dc1 127.0.0.1 +31:46.905 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b98eb076-3092-203a-628d-da0c02b94895 127.0.0.1 +2021-10-05T16:31:46.905Z [INFO] agent.router: Initializing LAN area manager +31:46.906 [INFO] TestAgent.server: Adding LAN server: server="Node-b98eb076-3092-203a-628d-da0c02b94895 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +31:46.906 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b98eb076-3092-203a-628d-da0c02b94895.dc1 area=wan +31:46.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp +31:46.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp +31:46.907 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp +31:46.907 [INFO] TestAgent: started state syncer +31:46.907 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp +31:46.960 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:46.960 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 +31:47.008 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:47.008 [DEBUG] TestAgent.server.raft: vote granted: from=b98eb076-3092-203a-628d-da0c02b94895 term=2 tally=1 +31:47.008 [INFO] TestAgent.server.raft: election won: tally=1 +31:47.008 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" +31:47.012 [INFO] TestAgent.server: cluster leadership acquired +31:47.012 [INFO] TestAgent.server: New leader elected: payload=Node-b98eb076-3092-203a-628d-da0c02b94895 +31:47.036 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25321 +31:47.081 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:47.106 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:47.106 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:47.106 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:47.106 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:47.106 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:47.106 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b98eb076-3092-203a-628d-da0c02b94895 +31:47.106 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b98eb076-3092-203a-628d-da0c02b94895 +31:47.149 [INFO] TestAgent: Synced node info +31:47.149 [DEBUG] TestAgent: Node info in sync +31:47.153 [INFO] TestAgent.server: federation state anti-entropy synced +31:47.187 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:54508 latency=2.74624ms +31:47.202 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:54510 latency=439.895µs +31:47.208 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?node-meta=foo%3Abar from=127.0.0.1:54512 latency=396.498µs +31:47.225 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?filter=Meta.foo+%3D%3D+bar from=127.0.0.1:54514 latency=6.851981ms +31:47.245 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes?near=_agent from=127.0.0.1:54516 latency=399.354µs +31:47.251 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/consul from=127.0.0.1:54518 latency=801.316µs +31:47.281 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/this-service-will-literally-never-exist from=127.0.0.1:54520 latency=446.044µs +31:47.281 [INFO] TestAgent: Requesting shutdown +31:47.281 [INFO] TestAgent.server: shutting down server +31:47.281 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:47.281 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:47.281 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:47.281 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:47.281 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:47.281 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:47.281 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:47.281 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:47.281 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:47.281 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:47.291 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:47.297Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:47.297Z [INFO] agent.router.manager: shutting down +31:47.297 [INFO] TestAgent: consul server down +31:47.297 [INFO] TestAgent: shutdown complete +31:47.297 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp +31:47.297 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp +31:47.297 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp +31:47.800 [INFO] TestAgent: Waiting for endpoints to shut down +31:47.800 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogListNodesCommand (1.14s) + --- PASS: TestCatalogListNodesCommand/simple (0.01s) + --- PASS: TestCatalogListNodesCommand/detailed (0.01s) + --- PASS: TestCatalogListNodesCommand/node-meta (0.01s) + --- PASS: TestCatalogListNodesCommand/filter (0.01s) + --- PASS: TestCatalogListNodesCommand/near (0.02s) --- PASS: TestCatalogListNodesCommand/service_present (0.00s) - --- PASS: TestCatalogListNodesCommand/service_missing (0.00s) + --- PASS: TestCatalogListNodesCommand/service_missing (0.03s) PASS -ok github.com/hashicorp/consul/command/catalog/list/nodes 0.820s +ok github.com/hashicorp/consul/command/catalog/list/nodes 1.248s === RUN TestCatalogListServicesCommand_noTabs === PAUSE TestCatalogListServicesCommand_noTabs === RUN TestCatalogListServicesCommand_Validation @@ -78223,91 +78519,92 @@ === RUN TestCatalogListServicesCommand === PAUSE TestCatalogListServicesCommand === CONT TestCatalogListServicesCommand_noTabs +--- PASS: TestCatalogListServicesCommand_noTabs (0.00s) === CONT TestCatalogListServicesCommand -=== CONT TestCatalogListServicesCommand_Validation [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -[INFO] freeport: detected ephemeral port range of [32768, 60999] +=== CONT TestCatalogListServicesCommand_Validation --- PASS: TestCatalogListServicesCommand_Validation (0.00s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCatalogListServicesCommand_noTabs (0.00s) === RUN TestCatalogListServicesCommand/simple === RUN TestCatalogListServicesCommand/tags === RUN TestCatalogListServicesCommand/node_missing === RUN TestCatalogListServicesCommand/node_present === RUN TestCatalogListServicesCommand/node-meta -2022-11-07T22:45:39.337Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:39.350Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:39.379 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e429a70-7db7-767a-362f-79e8432f76df Address:127.0.0.1:14090}]" -45:39.379 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= -45:39.380 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2e429a70-7db7-767a-362f-79e8432f76df.dc1 127.0.0.1 -45:39.381 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2e429a70-7db7-767a-362f-79e8432f76df 127.0.0.1 -2022-11-07T22:45:39.381Z [INFO] agent.router: Initializing LAN area manager -45:39.381 [INFO] TestAgent.server: Adding LAN server: server="Node-2e429a70-7db7-767a-362f-79e8432f76df (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -45:39.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp -45:39.381 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2e429a70-7db7-767a-362f-79e8432f76df.dc1 area=wan -45:39.381 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp -45:39.382 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp -45:39.382 [INFO] TestAgent: started state syncer -45:39.382 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp -45:39.417 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:39.418 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 -45:39.428 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:39.428 [DEBUG] TestAgent.server.raft: vote granted: from=2e429a70-7db7-767a-362f-79e8432f76df term=2 tally=1 -45:39.428 [INFO] TestAgent.server.raft: election won: tally=1 -45:39.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" -45:39.428 [INFO] TestAgent.server: cluster leadership acquired -45:39.428 [INFO] TestAgent.server: New leader elected: payload=Node-2e429a70-7db7-767a-362f-79e8432f76df -45:39.434 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 -45:39.445 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:39.463 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:39.463 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:39.463 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:39.463 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:39.463 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:39.463 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2e429a70-7db7-767a-362f-79e8432f76df -45:39.463 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2e429a70-7db7-767a-362f-79e8432f76df -45:39.468 [INFO] TestAgent.server: federation state anti-entropy synced -45:39.516 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:39.519 [INFO] TestAgent: Synced node info -45:39.522 [INFO] TestAgent: Synced service: service=testing -45:39.522 [DEBUG] TestAgent: Node info in sync -45:39.522 [DEBUG] TestAgent: Service in sync: service=testing -45:39.522 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:56350 latency=8.948519ms -45:39.523 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:56352 latency=240.065µs -45:39.525 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:56354 latency=209.303µs -45:39.526 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/not-a-real-node from=127.0.0.1:56356 latency=794.704µs -45:39.530 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/Node-2e429a70-7db7-767a-362f-79e8432f76df from=127.0.0.1:56358 latency=280.737µs -45:39.531 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services?node-meta=foo%3Abar from=127.0.0.1:56360 latency=184.765µs -45:39.531 [INFO] TestAgent: Requesting shutdown -45:39.531 [INFO] TestAgent.server: shutting down server -45:39.531 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:39.531 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:39.531 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:39.531 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:39.531 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:39.532 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:39.532 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:39.532 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:39.532 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:39.532 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:39.535 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:39.538Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:39.538Z [INFO] agent.router.manager: shutting down -45:39.538 [INFO] TestAgent: consul server down -45:39.538 [INFO] TestAgent: shutdown complete -45:39.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp -45:39.538 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp -45:39.538 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp -45:40.038 [INFO] TestAgent: Waiting for endpoints to shut down -45:40.038 [INFO] TestAgent: Endpoints down ---- PASS: TestCatalogListServicesCommand (0.74s) - --- PASS: TestCatalogListServicesCommand/simple (0.00s) +2021-10-05T16:31:57.090Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.138Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.196 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 Address:127.0.0.1:22258}]" +31:57.196 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= +31:57.197 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03.dc1 127.0.0.1 +31:57.198 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 127.0.0.1 +2021-10-05T16:31:57.198Z [INFO] agent.router: Initializing LAN area manager +31:57.198 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp +31:57.198 [INFO] TestAgent.server: Adding LAN server: server="Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +31:57.198 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03.dc1 area=wan +31:57.198 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp +31:57.199 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp +31:57.199 [INFO] TestAgent: started state syncer +31:57.199 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp +31:57.244 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.244 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 +31:57.284 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.284 [DEBUG] TestAgent.server.raft: vote granted: from=ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 term=2 tally=1 +31:57.284 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.284 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" +31:57.287 [INFO] TestAgent.server: cluster leadership acquired +31:57.288 [INFO] TestAgent.server: New leader elected: payload=Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 +31:57.292 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 +31:57.313 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.396 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.396 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.396 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.396 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.396 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.396 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 +31:57.396 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 +31:57.422 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.580 [INFO] TestAgent: Synced node info +31:57.588 [INFO] TestAgent: Synced service: service=testing +31:57.588 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:45738 latency=19.083978ms +31:57.588 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:57.595 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:45740 latency=2.575165ms +31:57.597 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services from=127.0.0.1:45742 latency=276.471µs +31:57.601 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/not-a-real-node from=127.0.0.1:45744 latency=2.756581ms +31:57.606 [INFO] TestAgent: Synced node info +31:57.612 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/node/Node-ca20fdc1-db73-a1c1-fa3b-233b52c8bf03 from=127.0.0.1:45746 latency=470.978µs +31:57.614 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/services?node-meta=foo%3Abar from=127.0.0.1:45748 latency=262.903µs +31:57.614 [INFO] TestAgent: Requesting shutdown +31:57.614 [INFO] TestAgent.server: shutting down server +31:57.614 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.614 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.614 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.614 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.614 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.614 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.614 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.614 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.614 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.614 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.620 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.628Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.628Z [INFO] agent.router.manager: shutting down +31:57.629 [WARN] TestAgent: Syncing service failed.: service=testing error="leadership lost while committing log" +31:57.629 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="leadership lost while committing log" +31:57.629 [INFO] TestAgent: consul server down +31:57.629 [INFO] TestAgent: shutdown complete +31:57.629 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp +31:57.629 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp +31:57.629 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp +31:58.132 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.132 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogListServicesCommand (1.17s) + --- PASS: TestCatalogListServicesCommand/simple (0.01s) --- PASS: TestCatalogListServicesCommand/tags (0.00s) - --- PASS: TestCatalogListServicesCommand/node_missing (0.00s) + --- PASS: TestCatalogListServicesCommand/node_missing (0.01s) --- PASS: TestCatalogListServicesCommand/node_present (0.00s) --- PASS: TestCatalogListServicesCommand/node-meta (0.00s) PASS -ok github.com/hashicorp/consul/command/catalog/list/services 0.768s +ok github.com/hashicorp/consul/command/catalog/list/services 1.277s ? github.com/hashicorp/consul/command/config [no test files] === RUN TestConfigDelete_noTabs === PAUSE TestConfigDelete_noTabs @@ -78316,146 +78613,147 @@ === RUN TestConfigDelete_InvalidArgs === PAUSE TestConfigDelete_InvalidArgs === CONT TestConfigDelete_noTabs +--- PASS: TestConfigDelete_noTabs (0.00s) === CONT TestConfigDelete_InvalidArgs === RUN TestConfigDelete_InvalidArgs/no_kind === CONT TestConfigDelete -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestConfigDelete_noTabs (0.00s) === RUN TestConfigDelete_InvalidArgs/no_name ---- PASS: TestConfigDelete_InvalidArgs (0.00s) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +--- PASS: TestConfigDelete_InvalidArgs (0.01s) --- PASS: TestConfigDelete_InvalidArgs/no_kind (0.00s) - --- PASS: TestConfigDelete_InvalidArgs/no_name (0.00s) + --- PASS: TestConfigDelete_InvalidArgs/no_name (0.01s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:40.344Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:40.355Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:40.377 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa Address:127.0.0.1:18174}]" -45:40.377 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -45:40.378 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa.dc1 127.0.0.1 -45:40.378 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa 127.0.0.1 -2022-11-07T22:45:40.379Z [INFO] agent.router: Initializing LAN area manager -45:40.379 [INFO] TestAgent.server: Adding LAN server: server="Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -45:40.379 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa.dc1 area=wan -45:40.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -45:40.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -45:40.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -45:40.379 [INFO] TestAgent: started state syncer -45:40.379 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -45:40.416 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:40.416 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -45:40.424 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:40.424 [DEBUG] TestAgent.server.raft: vote granted: from=e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa term=2 tally=1 -45:40.424 [INFO] TestAgent.server.raft: election won: tally=1 -45:40.424 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -45:40.424 [INFO] TestAgent.server: cluster leadership acquired -45:40.424 [INFO] TestAgent.server: New leader elected: payload=Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa -45:40.429 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18174 -45:40.442 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:40.463 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:40.463 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:40.463 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:40.463 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:40.463 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:40.464 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa -45:40.464 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e9f58fe2-66b7-9b0c-3042-1c9d2a3eb0aa -45:40.468 [INFO] TestAgent.server: federation state anti-entropy synced -45:40.473 [INFO] TestAgent: Synced node info -45:40.473 [DEBUG] TestAgent: Node info in sync -45:40.540 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:47216 latency=4.418953ms -45:40.544 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config/service-defaults/web from=127.0.0.1:47218 latency=3.833888ms -45:40.547 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:47216 error="Config entry not found for "service-defaults" / "web"" -45:40.547 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:47216 latency=536.568µs -45:40.548 [INFO] TestAgent: Requesting shutdown -45:40.548 [INFO] TestAgent.server: shutting down server -45:40.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:40.548 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:40.548 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:40.548 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:40.548 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:40.549 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:40.549 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:40.553 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:40.558Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:40.558Z [INFO] agent.router.manager: shutting down -45:40.558 [INFO] TestAgent: consul server down -45:40.558 [INFO] TestAgent: shutdown complete -45:40.558 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -45:40.558 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -45:40.559 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -45:41.059 [INFO] TestAgent: Waiting for endpoints to shut down -45:41.059 [INFO] TestAgent: Endpoints down ---- PASS: TestConfigDelete (0.75s) +2021-10-05T16:31:57.438Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:57.462Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:57.506 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2a02b37f-b5c3-c32e-5175-26d7d94d9add Address:127.0.0.1:15111}]" +31:57.507 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +31:57.507 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add.dc1 127.0.0.1 +31:57.508 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add 127.0.0.1 +2021-10-05T16:31:57.508Z [INFO] agent.router: Initializing LAN area manager +31:57.509 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +31:57.509 [INFO] TestAgent.server: Adding LAN server: server="Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +31:57.509 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add.dc1 area=wan +31:57.509 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +31:57.509 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +31:57.509 [INFO] TestAgent: started state syncer +31:57.510 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +31:57.568 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:57.568 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +31:57.589 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:57.589 [DEBUG] TestAgent.server.raft: vote granted: from=2a02b37f-b5c3-c32e-5175-26d7d94d9add term=2 tally=1 +31:57.589 [INFO] TestAgent.server.raft: election won: tally=1 +31:57.589 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +31:57.589 [INFO] TestAgent.server: cluster leadership acquired +31:57.589 [INFO] TestAgent.server: New leader elected: payload=Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add +31:57.620 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 +31:57.645 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:57.696 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:57.696 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:57.696 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:57.697 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:57.697 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:57.697 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add +31:57.697 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2a02b37f-b5c3-c32e-5175-26d7d94d9add +31:57.710 [INFO] TestAgent.server: federation state anti-entropy synced +31:57.817 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +31:57.823 [INFO] TestAgent: Synced node info +31:57.823 [DEBUG] TestAgent: Node info in sync +31:57.858 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:56084 latency=6.882421ms +31:57.879 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/config/service-defaults/web from=127.0.0.1:56086 latency=9.400505ms +31:57.890 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:56084 error="Config entry not found for "service-defaults" / "web"" +31:57.892 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:56084 latency=2.69191ms +31:57.896 [INFO] TestAgent: Requesting shutdown +31:57.896 [INFO] TestAgent.server: shutting down server +31:57.896 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:57.896 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:57.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:57.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:57.896 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:57.896 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:57.896 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:57.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:57.896 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:57.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:57.908 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:57.920Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:57.920Z [INFO] agent.router.manager: shutting down +31:57.920 [INFO] TestAgent: consul server down +31:57.920 [INFO] TestAgent: shutdown complete +31:57.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +31:57.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +31:57.920 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +31:58.424 [INFO] TestAgent: Waiting for endpoints to shut down +31:58.424 [INFO] TestAgent: Endpoints down +--- PASS: TestConfigDelete (1.04s) PASS -ok github.com/hashicorp/consul/command/config/delete 0.780s +ok github.com/hashicorp/consul/command/config/delete 1.153s === RUN TestConfigList_noTabs === PAUSE TestConfigList_noTabs === RUN TestConfigList [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:41.860Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:41.869Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:41.894 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1323a8c5-4e21-0da8-02d9-beb3a9550dbc Address:127.0.0.1:22258}]" -45:41.894 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= -45:41.895 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc.dc1 127.0.0.1 -45:41.895 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc 127.0.0.1 -2022-11-07T22:45:41.896Z [INFO] agent.router: Initializing LAN area manager -45:41.896 [INFO] TestAgent.server: Adding LAN server: server="Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc (Addr: tcp/127.0.0.1:22258) (DC: dc1)" -45:41.896 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc.dc1 area=wan -45:41.896 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp -45:41.896 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp -45:41.896 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp -45:41.896 [INFO] TestAgent: started state syncer -45:41.896 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp -45:41.961 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:41.961 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 -45:41.970 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:41.970 [DEBUG] TestAgent.server.raft: vote granted: from=1323a8c5-4e21-0da8-02d9-beb3a9550dbc term=2 tally=1 -45:41.970 [INFO] TestAgent.server.raft: election won: tally=1 -45:41.970 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" -45:41.970 [INFO] TestAgent.server: cluster leadership acquired -45:41.970 [INFO] TestAgent.server: New leader elected: payload=Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc -45:41.977 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 -45:41.988 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:42.005 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:42.005 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:42.005 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:42.005 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:42.005 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:42.005 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc -45:42.005 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1323a8c5-4e21-0da8-02d9-beb3a9550dbc -45:42.014 [INFO] TestAgent.server: federation state anti-entropy synced -45:42.175 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:39016 latency=4.40803ms -45:42.182 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:39016 latency=3.242422ms -45:42.186 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:39016 latency=3.218247ms -45:42.187 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults from=127.0.0.1:39018 latency=877.598µs -45:42.188 [INFO] TestAgent: Requesting shutdown -45:42.188 [INFO] TestAgent.server: shutting down server -45:42.188 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:42.188 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:42.188 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:42.188 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:42.188 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:42.188 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:42.188 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:42.188 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:42.188 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:42.188 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:42.188 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:42.192 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:42.195Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:42.195Z [INFO] agent.router.manager: shutting down -45:42.195 [INFO] TestAgent: consul server down -45:42.195 [INFO] TestAgent: shutdown complete -45:42.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp -45:42.195 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp -45:42.195 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp -45:42.695 [INFO] TestAgent: Waiting for endpoints to shut down -45:42.695 [INFO] TestAgent: Endpoints down ---- PASS: TestConfigList (0.88s) +2021-10-05T16:31:58.573Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:31:58.594Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +31:58.638 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:141278bc-1a7d-93a4-4dd3-b5bebbc3c79e Address:127.0.0.1:27363}]" +31:58.639 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e.dc1 127.0.0.1 +31:58.639 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e 127.0.0.1 +2021-10-05T16:31:58.639Z [INFO] agent.router: Initializing LAN area manager +31:58.640 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +31:58.640 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +31:58.640 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +31:58.640 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e.dc1 area=wan +31:58.640 [INFO] TestAgent.server: Adding LAN server: server="Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +31:58.641 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +31:58.641 [INFO] TestAgent: started state syncer +31:58.641 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +31:58.680 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +31:58.680 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +31:58.694 [DEBUG] TestAgent.server.raft: votes: needed=1 +31:58.694 [DEBUG] TestAgent.server.raft: vote granted: from=141278bc-1a7d-93a4-4dd3-b5bebbc3c79e term=2 tally=1 +31:58.694 [INFO] TestAgent.server.raft: election won: tally=1 +31:58.694 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +31:58.694 [INFO] TestAgent.server: cluster leadership acquired +31:58.694 [INFO] TestAgent.server: New leader elected: payload=Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e +31:58.713 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +31:58.732 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +31:58.773 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +31:58.773 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +31:58.773 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +31:58.773 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +31:58.773 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +31:58.773 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e +31:58.773 [INFO] TestAgent.server: member joined, marking health alive: member=Node-141278bc-1a7d-93a4-4dd3-b5bebbc3c79e +31:58.790 [INFO] TestAgent.server: federation state anti-entropy synced +31:58.835 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:59274 latency=11.949369ms +31:58.852 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:59274 latency=10.394039ms +31:58.873 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:59274 latency=17.121869ms +31:58.878 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults from=127.0.0.1:59276 latency=1.249803ms +31:58.880 [INFO] TestAgent: Requesting shutdown +31:58.880 [INFO] TestAgent.server: shutting down server +31:58.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +31:58.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +31:58.880 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +31:58.880 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +31:58.880 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +31:58.880 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +31:58.880 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +31:58.880 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +31:58.880 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +31:58.880 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +31:58.880 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +31:58.887 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:31:58.891Z [INFO] agent.router.manager: shutting down +2021-10-05T16:31:58.891Z [INFO] agent.router.manager: shutting down +31:58.891 [INFO] TestAgent: consul server down +31:58.891 [INFO] TestAgent: shutdown complete +31:58.891 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +31:58.891 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +31:58.891 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +31:59.392 [INFO] TestAgent: Waiting for endpoints to shut down +31:59.392 [INFO] TestAgent: Endpoints down +--- PASS: TestConfigList (0.90s) === RUN TestConfigList_InvalidArgs === PAUSE TestConfigList_InvalidArgs === CONT TestConfigList_noTabs @@ -78465,7 +78763,7 @@ --- PASS: TestConfigList_InvalidArgs (0.00s) --- PASS: TestConfigList_InvalidArgs/no_kind (0.00s) PASS -ok github.com/hashicorp/consul/command/config/list 0.907s +ok github.com/hashicorp/consul/command/config/list 0.988s === RUN TestConfigRead_noTabs === PAUSE TestConfigRead_noTabs === RUN TestConfigRead @@ -78473,77 +78771,78 @@ === RUN TestConfigRead_InvalidArgs === PAUSE TestConfigRead_InvalidArgs === CONT TestConfigRead_noTabs +--- PASS: TestConfigRead_noTabs (0.00s) === CONT TestConfigRead_InvalidArgs === RUN TestConfigRead_InvalidArgs/no_name === CONT TestConfigRead ---- PASS: TestConfigRead_noTabs (0.00s) -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestConfigRead_InvalidArgs/no_kind -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... --- PASS: TestConfigRead_InvalidArgs (0.00s) --- PASS: TestConfigRead_InvalidArgs/no_name (0.00s) --- PASS: TestConfigRead_InvalidArgs/no_kind (0.00s) -2022-11-07T22:45:44.130Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:44.140Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:44.160 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3c3a8133-26c2-0852-a55d-5cf88061fc82 Address:127.0.0.1:18174}]" -45:44.160 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -45:44.161 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3c3a8133-26c2-0852-a55d-5cf88061fc82.dc1 127.0.0.1 -45:44.161 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3c3a8133-26c2-0852-a55d-5cf88061fc82 127.0.0.1 -2022-11-07T22:45:44.161Z [INFO] agent.router: Initializing LAN area manager -45:44.162 [INFO] TestAgent.server: Adding LAN server: server="Node-3c3a8133-26c2-0852-a55d-5cf88061fc82 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -45:44.162 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3c3a8133-26c2-0852-a55d-5cf88061fc82.dc1 area=wan -45:44.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -45:44.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -45:44.163 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -45:44.163 [INFO] TestAgent: started state syncer -45:44.163 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -45:44.196 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:44.196 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -45:44.204 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:44.204 [DEBUG] TestAgent.server.raft: vote granted: from=3c3a8133-26c2-0852-a55d-5cf88061fc82 term=2 tally=1 -45:44.204 [INFO] TestAgent.server.raft: election won: tally=1 -45:44.204 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -45:44.204 [INFO] TestAgent.server: cluster leadership acquired -45:44.204 [INFO] TestAgent.server: New leader elected: payload=Node-3c3a8133-26c2-0852-a55d-5cf88061fc82 -45:44.209 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18174 -45:44.221 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:44.240 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:44.240 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:44.240 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:44.240 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:44.240 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:44.240 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3c3a8133-26c2-0852-a55d-5cf88061fc82 -45:44.240 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3c3a8133-26c2-0852-a55d-5cf88061fc82 -45:44.246 [INFO] TestAgent.server: federation state anti-entropy synced -45:44.398 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:48144 latency=3.788669ms -45:44.400 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:48146 latency=1.610077ms -45:44.403 [INFO] TestAgent: Requesting shutdown -45:44.403 [INFO] TestAgent.server: shutting down server -45:44.403 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:44.403 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:44.403 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:44.403 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:44.403 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:44.403 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:44.403 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:44.403 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:44.403 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:44.403 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:44.403 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:44.407 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:44.410Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:44.410Z [INFO] agent.router.manager: shutting down -45:44.410 [INFO] TestAgent: consul server down -45:44.410 [INFO] TestAgent: shutdown complete -45:44.410 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -45:44.410 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -45:44.410 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -45:44.910 [INFO] TestAgent: Waiting for endpoints to shut down -45:44.910 [INFO] TestAgent: Endpoints down ---- PASS: TestConfigRead (0.81s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:32:09.460Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.473Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.533 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e80a50ac-a338-9356-8141-1a317cf79da1 Address:127.0.0.1:18174}]" +32:09.533 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= +32:09.541 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e80a50ac-a338-9356-8141-1a317cf79da1.dc1 127.0.0.1 +32:09.546 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e80a50ac-a338-9356-8141-1a317cf79da1 127.0.0.1 +2021-10-05T16:32:09.546Z [INFO] agent.router: Initializing LAN area manager +32:09.548 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e80a50ac-a338-9356-8141-1a317cf79da1.dc1 area=wan +32:09.548 [INFO] TestAgent.server: Adding LAN server: server="Node-e80a50ac-a338-9356-8141-1a317cf79da1 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" +32:09.555 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp +32:09.555 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp +32:09.555 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp +32:09.556 [INFO] TestAgent: started state syncer +32:09.556 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp +32:09.602 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.602 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 +32:09.617 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.617 [DEBUG] TestAgent.server.raft: vote granted: from=e80a50ac-a338-9356-8141-1a317cf79da1 term=2 tally=1 +32:09.617 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.617 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" +32:09.617 [INFO] TestAgent.server: cluster leadership acquired +32:09.617 [INFO] TestAgent.server: New leader elected: payload=Node-e80a50ac-a338-9356-8141-1a317cf79da1 +32:09.636 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18174 +32:09.676 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.736 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.736 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.736 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.736 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.736 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.736 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e80a50ac-a338-9356-8141-1a317cf79da1 +32:09.736 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e80a50ac-a338-9356-8141-1a317cf79da1 +32:09.799 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.907 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:09.918 [INFO] TestAgent: Synced node info +32:09.937 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:53458 latency=12.139162ms +32:09.938 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:53460 latency=322.249µs +32:09.940 [INFO] TestAgent: Requesting shutdown +32:09.940 [INFO] TestAgent.server: shutting down server +32:09.940 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.940 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.940 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.940 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.940 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.940 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.940 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.940 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.940 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.940 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.948 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.952Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.952Z [INFO] agent.router.manager: shutting down +32:09.952 [INFO] TestAgent: consul server down +32:09.952 [INFO] TestAgent: shutdown complete +32:09.952 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp +32:09.952 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp +32:09.952 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp +32:10.453 [INFO] TestAgent: Waiting for endpoints to shut down +32:10.453 [INFO] TestAgent: Endpoints down +--- PASS: TestConfigRead (1.07s) PASS -ok github.com/hashicorp/consul/command/config/read 0.845s +ok github.com/hashicorp/consul/command/config/read 1.248s === RUN TestConfigWrite_noTabs === PAUSE TestConfigWrite_noTabs === RUN TestConfigWrite @@ -78551,19 +78850,17 @@ === RUN TestParseConfigEntry === PAUSE TestParseConfigEntry === CONT TestConfigWrite_noTabs +--- PASS: TestConfigWrite_noTabs (0.00s) === CONT TestParseConfigEntry === RUN TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(hcl_snake_case) === RUN TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(hcl_camel_case) === RUN TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(json_snake_case) -=== CONT TestConfigWrite ---- PASS: TestConfigWrite_noTabs (0.00s) === RUN TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(json_camel_case) === RUN TestParseConfigEntry/proxy-defaults_(hcl_snake_case) -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestConfigWrite === RUN TestParseConfigEntry/proxy-defaults_(hcl_camel_case) +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestParseConfigEntry/proxy-defaults_(json_snake_case) -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestParseConfigEntry/proxy-defaults_(json_camel_case) === RUN TestParseConfigEntry/terminating-gateway_(hcl_snake_case) === RUN TestParseConfigEntry/terminating-gateway_(hcl_camel_case) @@ -78575,6 +78872,8 @@ === RUN TestParseConfigEntry/service-defaults_(json_camel_case) === RUN TestParseConfigEntry/service-router:_kitchen_sink_(hcl_snake_case) === RUN TestParseConfigEntry/service-router:_kitchen_sink_(hcl_camel_case) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestParseConfigEntry/service-router:_kitchen_sink_(json_snake_case) === RUN TestParseConfigEntry/service-router:_kitchen_sink_(json_camel_case) === RUN TestParseConfigEntry/service-splitter:_kitchen_sink_(hcl_snake_case) @@ -78605,7 +78904,7 @@ === RUN TestParseConfigEntry/ingress-gateway:_kitchen_sink_(hcl_camel_case) === RUN TestParseConfigEntry/ingress-gateway:_kitchen_sink_(json_snake_case) === RUN TestParseConfigEntry/ingress-gateway:_kitchen_sink_(json_camel_case) ---- PASS: TestParseConfigEntry (0.02s) +--- PASS: TestParseConfigEntry (0.06s) --- PASS: TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(hcl_snake_case) (0.00s) --- PASS: TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(hcl_camel_case) (0.00s) --- PASS: TestParseConfigEntry/proxy-defaults:_extra_fields_or_typo_(json_snake_case) (0.00s) @@ -78623,9 +78922,9 @@ --- PASS: TestParseConfigEntry/service-defaults_(json_snake_case) (0.00s) --- PASS: TestParseConfigEntry/service-defaults_(json_camel_case) (0.00s) --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(hcl_snake_case) (0.00s) - --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(hcl_camel_case) (0.00s) + --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(hcl_camel_case) (0.01s) --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(json_snake_case) (0.00s) - --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(json_camel_case) (0.00s) + --- PASS: TestParseConfigEntry/service-router:_kitchen_sink_(json_camel_case) (0.01s) --- PASS: TestParseConfigEntry/service-splitter:_kitchen_sink_(hcl_snake_case) (0.00s) --- PASS: TestParseConfigEntry/service-splitter:_kitchen_sink_(hcl_camel_case) (0.00s) --- PASS: TestParseConfigEntry/service-splitter:_kitchen_sink_(json_snake_case) (0.00s) @@ -78636,7 +78935,7 @@ --- PASS: TestParseConfigEntry/service-resolver:_subsets_with_failover_(json_camel_case) (0.00s) --- PASS: TestParseConfigEntry/service-resolver:_redirect_(hcl_snake_case) (0.00s) --- PASS: TestParseConfigEntry/service-resolver:_redirect_(hcl_camel_case) (0.00s) - --- PASS: TestParseConfigEntry/service-resolver:_redirect_(json_snake_case) (0.00s) + --- PASS: TestParseConfigEntry/service-resolver:_redirect_(json_snake_case) (0.01s) --- PASS: TestParseConfigEntry/service-resolver:_redirect_(json_camel_case) (0.00s) --- PASS: TestParseConfigEntry/service-resolver:_default_(hcl_snake_case) (0.00s) --- PASS: TestParseConfigEntry/service-resolver:_default_(hcl_camel_case) (0.00s) @@ -78657,224 +78956,226 @@ === RUN TestConfigWrite/File === RUN TestConfigWrite/Stdin === RUN TestConfigWrite/No_config -2022-11-07T22:45:45.150Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:45.159Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:45.179 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:15e0c264-e990-01e7-41ea-de7789e411f8 Address:127.0.0.1:30426}]" -45:45.179 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= -45:45.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-15e0c264-e990-01e7-41ea-de7789e411f8.dc1 127.0.0.1 -45:45.182 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-15e0c264-e990-01e7-41ea-de7789e411f8 127.0.0.1 -2022-11-07T22:45:45.182Z [INFO] agent.router: Initializing LAN area manager -45:45.182 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-15e0c264-e990-01e7-41ea-de7789e411f8.dc1 area=wan -45:45.182 [INFO] TestAgent.server: Adding LAN server: server="Node-15e0c264-e990-01e7-41ea-de7789e411f8 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" -45:45.182 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp -45:45.182 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp -45:45.182 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp -45:45.182 [INFO] TestAgent: started state syncer -45:45.182 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp -45:45.233 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:45.233 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 -45:45.242 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:45.242 [DEBUG] TestAgent.server.raft: vote granted: from=15e0c264-e990-01e7-41ea-de7789e411f8 term=2 tally=1 -45:45.242 [INFO] TestAgent.server.raft: election won: tally=1 -45:45.242 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" -45:45.242 [INFO] TestAgent.server: cluster leadership acquired -45:45.242 [INFO] TestAgent.server: New leader elected: payload=Node-15e0c264-e990-01e7-41ea-de7789e411f8 -45:45.247 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30426 -45:45.257 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:45.276 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:45.276 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:45.276 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:45.276 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:45.276 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:45.276 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-15e0c264-e990-01e7-41ea-de7789e411f8 -45:45.276 [INFO] TestAgent.server: member joined, marking health alive: member=Node-15e0c264-e990-01e7-41ea-de7789e411f8 -45:45.281 [INFO] TestAgent.server: federation state anti-entropy synced -45:45.435 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:36134 latency=4.126735ms -45:45.438 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:36136 latency=337.479µs -45:45.444 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:36138 latency=3.559383ms -45:45.444 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/proxy-defaults/global from=127.0.0.1:36136 latency=154.235µs -45:45.445 [INFO] TestAgent: Requesting shutdown -45:45.445 [INFO] TestAgent.server: shutting down server -45:45.445 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:45.445 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:45.445 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:45.445 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:45.445 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:45.445 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:45.445 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:45.445 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:45.445 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:45.445 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:45.445 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:45.449 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:45.452Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:45.452Z [INFO] agent.router.manager: shutting down -45:45.452 [INFO] TestAgent: consul server down -45:45.452 [INFO] TestAgent: shutdown complete -45:45.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp -45:45.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp -45:45.452 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp -45:45.953 [INFO] TestAgent: Waiting for endpoints to shut down -45:45.953 [INFO] TestAgent: Endpoints down ---- PASS: TestConfigWrite (0.84s) - --- PASS: TestConfigWrite/File (0.01s) - --- PASS: TestConfigWrite/Stdin (0.01s) +2021-10-05T16:32:09.450Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:09.490Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:09.538 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4e74e839-a2df-6553-d37e-157a0e1df1e6 Address:127.0.0.1:14090}]" +32:09.539 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4e74e839-a2df-6553-d37e-157a0e1df1e6.dc1 127.0.0.1 +32:09.540 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4e74e839-a2df-6553-d37e-157a0e1df1e6 127.0.0.1 +2021-10-05T16:32:09.540Z [INFO] agent.router: Initializing LAN area manager +32:09.540 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +32:09.540 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +32:09.540 [INFO] TestAgent.server: Adding LAN server: server="Node-4e74e839-a2df-6553-d37e-157a0e1df1e6 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +32:09.540 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4e74e839-a2df-6553-d37e-157a0e1df1e6.dc1 area=wan +32:09.541 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +32:09.541 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +32:09.541 [INFO] TestAgent: started state syncer +32:09.541 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +32:09.589 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:09.589 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +32:09.605 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:09.605 [DEBUG] TestAgent.server.raft: vote granted: from=4e74e839-a2df-6553-d37e-157a0e1df1e6 term=2 tally=1 +32:09.605 [INFO] TestAgent.server.raft: election won: tally=1 +32:09.605 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +32:09.608 [INFO] TestAgent.server: cluster leadership acquired +32:09.608 [INFO] TestAgent.server: New leader elected: payload=Node-4e74e839-a2df-6553-d37e-157a0e1df1e6 +32:09.616 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 +32:09.653 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:09.724 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:09.724 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:09.724 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:09.724 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:09.724 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:09.724 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4e74e839-a2df-6553-d37e-157a0e1df1e6 +32:09.724 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4e74e839-a2df-6553-d37e-157a0e1df1e6 +32:09.734 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:45030 latency=33.499287ms +32:09.743 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/service-defaults/web from=127.0.0.1:45032 latency=2.323496ms +32:09.775 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:45034 latency=13.788068ms +32:09.775 [INFO] TestAgent.server: federation state anti-entropy synced +32:09.776 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/proxy-defaults/global from=127.0.0.1:45032 latency=303.124µs +32:09.777 [INFO] TestAgent: Requesting shutdown +32:09.777 [INFO] TestAgent.server: shutting down server +32:09.777 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:09.777 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:09.777 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:09.777 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:09.777 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:09.777 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:09.777 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:09.777 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:09.777 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:09.777 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:09.777 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:09.804 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:09.808Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:09.808Z [INFO] agent.router.manager: shutting down +32:09.808 [INFO] TestAgent: consul server down +32:09.808 [INFO] TestAgent: shutdown complete +32:09.808 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +32:09.808 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +32:09.808 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +32:10.308 [INFO] TestAgent: Waiting for endpoints to shut down +32:10.308 [INFO] TestAgent: Endpoints down +--- PASS: TestConfigWrite (0.99s) + --- PASS: TestConfigWrite/File (0.06s) + --- PASS: TestConfigWrite/Stdin (0.02s) --- PASS: TestConfigWrite/No_config (0.00s) PASS -ok github.com/hashicorp/consul/command/config/write 0.869s +ok github.com/hashicorp/consul/command/config/write 1.094s === RUN TestConnectCommand_noTabs === PAUSE TestConnectCommand_noTabs === CONT TestConnectCommand_noTabs --- PASS: TestConnectCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/connect 0.005s +ok github.com/hashicorp/consul/command/connect 0.028s === RUN TestCatalogCommand_noTabs === PAUSE TestCatalogCommand_noTabs === CONT TestCatalogCommand_noTabs --- PASS: TestCatalogCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/connect/ca 0.005s +ok github.com/hashicorp/consul/command/connect/ca 0.051s === RUN TestConnectCAGetConfigCommand_noTabs === PAUSE TestConnectCAGetConfigCommand_noTabs === RUN TestConnectCAGetConfigCommand === PAUSE TestConnectCAGetConfigCommand === CONT TestConnectCAGetConfigCommand_noTabs +--- PASS: TestConnectCAGetConfigCommand_noTabs (0.00s) === CONT TestConnectCAGetConfigCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestConnectCAGetConfigCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:47.418Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:47.429Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:47.453 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c1da46a1-d48b-f033-11c0-b34a4be60e6b Address:127.0.0.1:31447}]" -45:47.453 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= -45:47.454 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b.dc1 127.0.0.1 -45:47.455 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b 127.0.0.1 -2022-11-07T22:45:47.455Z [INFO] agent.router: Initializing LAN area manager -45:47.455 [INFO] TestAgent.server: Adding LAN server: server="Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b (Addr: tcp/127.0.0.1:31447) (DC: dc1)" -45:47.455 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b.dc1 area=wan -45:47.455 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp -45:47.455 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp -45:47.456 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp -45:47.456 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp -45:47.456 [INFO] TestAgent: started state syncer -45:47.497 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:47.497 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 -45:47.506 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:47.506 [DEBUG] TestAgent.server.raft: vote granted: from=c1da46a1-d48b-f033-11c0-b34a4be60e6b term=2 tally=1 -45:47.506 [INFO] TestAgent.server.raft: election won: tally=1 -45:47.506 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" -45:47.506 [INFO] TestAgent.server: cluster leadership acquired -45:47.506 [INFO] TestAgent.server: New leader elected: payload=Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b -45:47.511 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 -45:47.524 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:47.542 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:47.542 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:47.543 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:47.543 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:47.543 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:47.543 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b -45:47.543 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c1da46a1-d48b-f033-11c0-b34a4be60e6b -45:47.552 [INFO] TestAgent.server: federation state anti-entropy synced -45:47.588 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:47.590 [INFO] TestAgent: Synced node info -45:47.662 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:57058 latency=607.944µs -45:47.663 [INFO] TestAgent: Requesting shutdown -45:47.663 [INFO] TestAgent.server: shutting down server -45:47.663 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:47.663 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:47.663 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:47.663 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:47.663 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:47.663 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:47.663 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:47.663 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:47.663 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:47.663 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:47.666 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:47.669Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:47.669Z [INFO] agent.router.manager: shutting down -45:47.669 [INFO] TestAgent: consul server down -45:47.669 [INFO] TestAgent: shutdown complete -45:47.669 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp -45:47.669 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp -45:47.669 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp -45:48.170 [INFO] TestAgent: Waiting for endpoints to shut down -45:48.170 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCAGetConfigCommand (0.79s) +2021-10-05T16:32:12.655Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:12.673Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:12.736 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9701f7d8-14cf-b4a8-c155-78c19acecd78 Address:127.0.0.1:24300}]" +32:12.737 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9701f7d8-14cf-b4a8-c155-78c19acecd78.dc1 127.0.0.1 +32:12.738 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9701f7d8-14cf-b4a8-c155-78c19acecd78 127.0.0.1 +2021-10-05T16:32:12.738Z [INFO] agent.router: Initializing LAN area manager +32:12.739 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:24300 [Follower]" leader= +32:12.739 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=tcp +32:12.739 [INFO] TestAgent: Started DNS server: address=127.0.0.1:24295 network=udp +32:12.740 [INFO] TestAgent.server: Adding LAN server: server="Node-9701f7d8-14cf-b4a8-c155-78c19acecd78 (Addr: tcp/127.0.0.1:24300) (DC: dc1)" +32:12.740 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9701f7d8-14cf-b4a8-c155-78c19acecd78.dc1 area=wan +32:12.741 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:24296 network=tcp +32:12.741 [INFO] TestAgent: started state syncer +32:12.741 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:24301 network=tcp +32:12.800 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:12.800 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:24300 [Candidate]" term=2 +32:12.836 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:12.836 [DEBUG] TestAgent.server.raft: vote granted: from=9701f7d8-14cf-b4a8-c155-78c19acecd78 term=2 tally=1 +32:12.836 [INFO] TestAgent.server.raft: election won: tally=1 +32:12.836 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:24300 [Leader]" +32:12.840 [INFO] TestAgent.server: cluster leadership acquired +32:12.840 [INFO] TestAgent.server: New leader elected: payload=Node-9701f7d8-14cf-b4a8-c155-78c19acecd78 +32:12.859 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:24300 +32:12.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:12.981 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:12.981 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:12.981 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:12.981 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:12.981 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:12.981 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9701f7d8-14cf-b4a8-c155-78c19acecd78 +32:12.981 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9701f7d8-14cf-b4a8-c155-78c19acecd78 +32:13.032 [INFO] TestAgent.server: federation state anti-entropy synced +32:13.135 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:13.148 [INFO] TestAgent: Synced node info +32:13.154 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/ca/configuration from=127.0.0.1:53786 latency=773.1µs +32:13.155 [INFO] TestAgent: Requesting shutdown +32:13.155 [INFO] TestAgent.server: shutting down server +32:13.155 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:13.155 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:13.155 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:13.155 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:13.155 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:13.155 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:13.155 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:13.156 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:13.156 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:13.156 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:13.162 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:13.178Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:13.178Z [INFO] agent.router.manager: shutting down +32:13.178 [INFO] TestAgent: consul server down +32:13.178 [INFO] TestAgent: shutdown complete +32:13.178 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=tcp +32:13.178 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:24295 network=udp +32:13.178 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:24296 network=tcp +32:13.680 [INFO] TestAgent: Waiting for endpoints to shut down +32:13.680 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCAGetConfigCommand (1.09s) PASS -ok github.com/hashicorp/consul/command/connect/ca/get 0.813s +ok github.com/hashicorp/consul/command/connect/ca/get 1.245s === RUN TestConnectCASetConfigCommand_noTabs === PAUSE TestConnectCASetConfigCommand_noTabs === RUN TestConnectCASetConfigCommand === PAUSE TestConnectCASetConfigCommand === CONT TestConnectCASetConfigCommand_noTabs +--- PASS: TestConnectCASetConfigCommand_noTabs (0.00s) === CONT TestConnectCASetConfigCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestConnectCASetConfigCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:48.973Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:48.984Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:49.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:052109de-0902-b92b-a1f4-8db59fd8370c Address:127.0.0.1:14090}]" -45:49.018 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= -45:49.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-052109de-0902-b92b-a1f4-8db59fd8370c.dc1 127.0.0.1 -45:49.019 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-052109de-0902-b92b-a1f4-8db59fd8370c 127.0.0.1 -2022-11-07T22:45:49.019Z [INFO] agent.router: Initializing LAN area manager -45:49.020 [INFO] TestAgent.server: Adding LAN server: server="Node-052109de-0902-b92b-a1f4-8db59fd8370c (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -45:49.020 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp -45:49.020 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp -45:49.020 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-052109de-0902-b92b-a1f4-8db59fd8370c.dc1 area=wan -45:49.020 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp -45:49.020 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp -45:49.020 [INFO] TestAgent: started state syncer -45:49.071 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:49.071 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 -45:49.080 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:49.080 [DEBUG] TestAgent.server.raft: vote granted: from=052109de-0902-b92b-a1f4-8db59fd8370c term=2 tally=1 -45:49.080 [INFO] TestAgent.server.raft: election won: tally=1 -45:49.080 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" -45:49.080 [INFO] TestAgent.server: cluster leadership acquired -45:49.080 [INFO] TestAgent.server: New leader elected: payload=Node-052109de-0902-b92b-a1f4-8db59fd8370c -45:49.085 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 -45:49.096 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:49.119 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:49.119 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:49.119 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:49.119 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:49.119 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:49.119 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-052109de-0902-b92b-a1f4-8db59fd8370c -45:49.119 [INFO] TestAgent.server: member joined, marking health alive: member=Node-052109de-0902-b92b-a1f4-8db59fd8370c -45:49.131 [INFO] TestAgent.server: federation state anti-entropy synced -45:49.236 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:49.240 [INFO] TestAgent: Synced node info -45:49.454 [INFO] TestAgent.server.connect: CA provider config updated -45:49.454 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:57302 latency=3.696053ms -45:49.454 [INFO] TestAgent: Requesting shutdown -45:49.454 [INFO] TestAgent.server: shutting down server -45:49.454 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:49.454 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:49.454 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:49.454 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:49.454 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:49.454 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:49.454 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:49.454 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:49.454 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:49.454 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:49.458 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:49.461Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:49.461Z [INFO] agent.router.manager: shutting down -45:49.461 [INFO] TestAgent: consul server down -45:49.461 [INFO] TestAgent: shutdown complete -45:49.461 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp -45:49.461 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp -45:49.461 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp -45:49.961 [INFO] TestAgent: Waiting for endpoints to shut down -45:49.961 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectCASetConfigCommand (1.03s) +2021-10-05T16:32:21.353Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:21.386Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:21.472 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6fc4ae79-1e6d-d258-8664-b2f190dda7c5 Address:127.0.0.1:27363}]" +32:21.473 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5.dc1 127.0.0.1 +32:21.473 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5 127.0.0.1 +2021-10-05T16:32:21.473Z [INFO] agent.router: Initializing LAN area manager +32:21.474 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +32:21.474 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +32:21.488 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5.dc1 area=wan +32:21.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +32:21.488 [INFO] TestAgent.server: Adding LAN server: server="Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +32:21.493 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +32:21.493 [INFO] TestAgent: started state syncer +32:21.497 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +32:21.540 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:21.540 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +32:21.558 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:21.558 [DEBUG] TestAgent.server.raft: vote granted: from=6fc4ae79-1e6d-d258-8664-b2f190dda7c5 term=2 tally=1 +32:21.558 [INFO] TestAgent.server.raft: election won: tally=1 +32:21.558 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +32:21.558 [INFO] TestAgent.server: cluster leadership acquired +32:21.558 [INFO] TestAgent.server: New leader elected: payload=Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5 +32:21.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +32:21.579 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:21.609 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:21.609 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:21.609 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:21.609 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:21.609 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:21.609 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5 +32:21.609 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6fc4ae79-1e6d-d258-8664-b2f190dda7c5 +32:21.629 [INFO] TestAgent.server: federation state anti-entropy synced +32:21.643 [INFO] TestAgent.server.connect: CA provider config updated +32:21.643 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/ca/configuration from=127.0.0.1:59438 latency=4.268075ms +32:21.643 [INFO] TestAgent: Requesting shutdown +32:21.643 [INFO] TestAgent.server: shutting down server +32:21.643 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:21.643 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:21.643 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:21.643 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:21.643 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:21.644 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:21.644 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:21.644 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:21.644 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:21.644 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:21.644 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:21.645 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:21.656 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +32:21.656 [INFO] TestAgent: Synced node info +32:21.656 [DEBUG] TestAgent: Node info in sync +2021-10-05T16:32:21.668Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:21.668Z [INFO] agent.router.manager: shutting down +32:21.668 [INFO] TestAgent: consul server down +32:21.668 [INFO] TestAgent: shutdown complete +32:21.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +32:21.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +32:21.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +32:22.172 [INFO] TestAgent: Waiting for endpoints to shut down +32:22.172 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectCASetConfigCommand (0.93s) PASS -ok github.com/hashicorp/consul/command/connect/ca/set 1.065s +ok github.com/hashicorp/consul/command/connect/ca/set 1.184s === RUN TestBootstrapConfig_ConfigureArgs === RUN TestBootstrapConfig_ConfigureArgs/defaults === RUN TestBootstrapConfig_ConfigureArgs/extra-stats-sinks @@ -78899,8 +79200,8 @@ === RUN TestBootstrapConfig_ConfigureArgs/ready-bind-addr === RUN TestBootstrapConfig_ConfigureArgs/ready-bind-addr-with-overrides === RUN TestBootstrapConfig_ConfigureArgs/ready-bind-addr-and-prometheus-and-stats ---- PASS: TestBootstrapConfig_ConfigureArgs (0.01s) - --- PASS: TestBootstrapConfig_ConfigureArgs/defaults (0.00s) +--- PASS: TestBootstrapConfig_ConfigureArgs (0.03s) + --- PASS: TestBootstrapConfig_ConfigureArgs/defaults (0.01s) --- PASS: TestBootstrapConfig_ConfigureArgs/extra-stats-sinks (0.00s) --- PASS: TestBootstrapConfig_ConfigureArgs/simple-statsd-sink (0.00s) --- PASS: TestBootstrapConfig_ConfigureArgs/simple-statsd-sink-plus-extra (0.00s) @@ -81938,33 +82239,33 @@ ] } } ---- PASS: TestGenerateConfig (0.10s) - --- PASS: TestGenerateConfig/no-args (0.00s) - --- PASS: TestGenerateConfig/defaults (0.01s) - --- PASS: TestGenerateConfig/token-arg (0.00s) - --- PASS: TestGenerateConfig/token-env (0.00s) - --- PASS: TestGenerateConfig/token-file-arg (0.00s) - --- PASS: TestGenerateConfig/token-file-env (0.00s) +--- PASS: TestGenerateConfig (0.44s) + --- PASS: TestGenerateConfig/no-args (0.01s) + --- PASS: TestGenerateConfig/defaults (0.02s) + --- PASS: TestGenerateConfig/token-arg (0.04s) + --- PASS: TestGenerateConfig/token-env (0.01s) + --- PASS: TestGenerateConfig/token-file-arg (0.02s) + --- PASS: TestGenerateConfig/token-file-env (0.01s) --- PASS: TestGenerateConfig/grpc-addr-flag (0.01s) - --- PASS: TestGenerateConfig/grpc-addr-env (0.00s) - --- PASS: TestGenerateConfig/grpc-addr-unix (0.00s) - --- PASS: TestGenerateConfig/grpc-addr-config (0.00s) - --- PASS: TestGenerateConfig/access-log-path (0.00s) + --- PASS: TestGenerateConfig/grpc-addr-env (0.02s) + --- PASS: TestGenerateConfig/grpc-addr-unix (0.01s) + --- PASS: TestGenerateConfig/grpc-addr-config (0.05s) + --- PASS: TestGenerateConfig/access-log-path (0.02s) --- PASS: TestGenerateConfig/missing-ca-file (0.00s) - --- PASS: TestGenerateConfig/existing-ca-file (0.00s) + --- PASS: TestGenerateConfig/existing-ca-file (0.01s) --- PASS: TestGenerateConfig/missing-ca-path (0.00s) - --- PASS: TestGenerateConfig/existing-ca-path (0.01s) - --- PASS: TestGenerateConfig/custom-bootstrap (0.00s) - --- PASS: TestGenerateConfig/extra_-single (0.00s) - --- PASS: TestGenerateConfig/extra_-multiple (0.00s) - --- PASS: TestGenerateConfig/stats-config-override (0.00s) - --- PASS: TestGenerateConfig/zipkin-tracing-config (0.00s) - --- PASS: TestGenerateConfig/CONSUL_HTTP_ADDR-with-https-scheme-enables-tls (0.00s) - --- PASS: TestGenerateConfig/ingress-gateway (0.00s) - --- PASS: TestGenerateConfig/ingress-gateway-address-specified (0.00s) - --- PASS: TestGenerateConfig/ingress-gateway-register-with-service-without-proxy-id (0.01s) - --- PASS: TestGenerateConfig/ingress-gateway-register-with-service-and-proxy-id (0.00s) - --- PASS: TestGenerateConfig/ingress-gateway-no-auto-register (0.00s) + --- PASS: TestGenerateConfig/existing-ca-path (0.02s) + --- PASS: TestGenerateConfig/custom-bootstrap (0.03s) + --- PASS: TestGenerateConfig/extra_-single (0.01s) + --- PASS: TestGenerateConfig/extra_-multiple (0.02s) + --- PASS: TestGenerateConfig/stats-config-override (0.03s) + --- PASS: TestGenerateConfig/zipkin-tracing-config (0.01s) + --- PASS: TestGenerateConfig/CONSUL_HTTP_ADDR-with-https-scheme-enables-tls (0.01s) + --- PASS: TestGenerateConfig/ingress-gateway (0.01s) + --- PASS: TestGenerateConfig/ingress-gateway-address-specified (0.01s) + --- PASS: TestGenerateConfig/ingress-gateway-register-with-service-without-proxy-id (0.05s) + --- PASS: TestGenerateConfig/ingress-gateway-register-with-service-and-proxy-id (0.03s) + --- PASS: TestGenerateConfig/ingress-gateway-no-auto-register (0.01s) === RUN TestEnvoy_GatewayRegistration === PAUSE TestEnvoy_GatewayRegistration === RUN TestEnvoyCommand_canBindInternal @@ -81975,12 +82276,12 @@ === RUN TestExecEnvoy/hot-restart-version === RUN TestExecEnvoy/hot-restart-version#01 === RUN TestExecEnvoy/hot-restart-version#02 ---- PASS: TestExecEnvoy (0.28s) - --- PASS: TestExecEnvoy/default (0.06s) - --- PASS: TestExecEnvoy/hot-restart-epoch (0.05s) - --- PASS: TestExecEnvoy/hot-restart-version (0.05s) - --- PASS: TestExecEnvoy/hot-restart-version#01 (0.05s) - --- PASS: TestExecEnvoy/hot-restart-version#02 (0.06s) +--- PASS: TestExecEnvoy (2.20s) + --- PASS: TestExecEnvoy/default (0.52s) + --- PASS: TestExecEnvoy/hot-restart-epoch (0.34s) + --- PASS: TestExecEnvoy/hot-restart-version (0.72s) + --- PASS: TestExecEnvoy/hot-restart-version#01 (0.26s) + --- PASS: TestExecEnvoy/hot-restart-version#02 (0.36s) === RUN TestHelperProcess --- PASS: TestHelperProcess (0.00s) === RUN TestServiceAddressValue_Value @@ -82014,22 +82315,22 @@ --- PASS: TestServiceAddressValue_Set/invalid_port (0.00s) --- PASS: TestServiceAddressValue_Set/invalid_address_format (0.00s) === CONT TestEnvoyCommand_noTabs +--- PASS: TestEnvoyCommand_noTabs (0.01s) === CONT TestEnvoyCommand_canBindInternal -=== CONT TestEnvoy_GatewayRegistration === RUN TestEnvoyCommand_canBindInternal/IPNet +=== CONT TestEnvoy_GatewayRegistration === RUN TestEnvoyCommand_canBindInternal/IPNet/ipv4 === RUN TestEnvoyCommand_canBindInternal/IPNet/secondary_ipv4 === RUN TestEnvoyCommand_canBindInternal/IPNet/ipv6 === RUN TestEnvoyCommand_canBindInternal/IPNet/ipv4_not_found === RUN TestEnvoyCommand_canBindInternal/IPNet/ipv6_not_found === RUN TestEnvoyCommand_canBindInternal/IPAddr -=== RUN TestEnvoyCommand_canBindInternal/IPAddr/ipv6_not_found -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestEnvoyCommand_canBindInternal/IPAddr/ipv4 === RUN TestEnvoyCommand_canBindInternal/IPAddr/secondary_ipv4 === RUN TestEnvoyCommand_canBindInternal/IPAddr/ipv6 === RUN TestEnvoyCommand_canBindInternal/IPAddr/ipv4_not_found ---- PASS: TestEnvoyCommand_canBindInternal (0.00s) +=== RUN TestEnvoyCommand_canBindInternal/IPAddr/ipv6_not_found +--- PASS: TestEnvoyCommand_canBindInternal (0.01s) --- PASS: TestEnvoyCommand_canBindInternal/IPNet (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPNet/ipv4 (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPNet/secondary_ipv4 (0.00s) @@ -82037,20 +82338,20 @@ --- PASS: TestEnvoyCommand_canBindInternal/IPNet/ipv4_not_found (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPNet/ipv6_not_found (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPAddr (0.00s) - --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/ipv6_not_found (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/ipv4 (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/secondary_ipv4 (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/ipv6 (0.00s) --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/ipv4_not_found (0.00s) + --- PASS: TestEnvoyCommand_canBindInternal/IPAddr/ipv6_not_found (0.00s) === CONT TestEnvoyGateway_Validation === RUN TestEnvoyGateway_Validation/-register_for_non-gateway ---- PASS: TestEnvoyCommand_noTabs (0.00s) -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestEnvoyGateway_Validation/-mesh-gateway_and_-gateway_cannot_be_combined === RUN TestEnvoyGateway_Validation/no_proxy_registration_specified_nor_discovered ---- PASS: TestEnvoyGateway_Validation (0.01s) - --- PASS: TestEnvoyGateway_Validation/-register_for_non-gateway (0.00s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +--- PASS: TestEnvoyGateway_Validation (0.02s) + --- PASS: TestEnvoyGateway_Validation/-register_for_non-gateway (0.01s) --- PASS: TestEnvoyGateway_Validation/-mesh-gateway_and_-gateway_cannot_be_combined (0.00s) --- PASS: TestEnvoyGateway_Validation/no_proxy_registration_specified_nor_discovered (0.00s) === RUN TestEnvoy_GatewayRegistration/register_gateway_with_proxy-id_and_name @@ -82083,7 +82384,7 @@ { "socket_address": { "address": "127.0.0.1", - "port_value": 29406 + "port_value": 27364 } } ] @@ -82273,7 +82574,7 @@ { "socket_address": { "address": "127.0.0.1", - "port_value": 29406 + "port_value": 27364 } } ] @@ -82463,7 +82764,7 @@ { "socket_address": { "address": "127.0.0.1", - "port_value": 29406 + "port_value": 27364 } } ] @@ -82653,7 +82954,7 @@ { "socket_address": { "address": "127.0.0.1", - "port_value": 29406 + "port_value": 27364 } } ] @@ -82813,121 +83114,124 @@ ] } } -2022-11-07T22:45:50.546Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:50.556Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:50.614 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9be02b7b-c2a2-20fd-911c-b8b916cf1b54 Address:127.0.0.1:29405}]" -45:50.615 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= -45:50.615 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54.dc1 127.0.0.1 -45:50.615 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54 127.0.0.1 -2022-11-07T22:45:50.616Z [INFO] agent.router: Initializing LAN area manager -45:50.616 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54.dc1 area=wan -45:50.616 [INFO] TestAgent.server: Adding LAN server: server="Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" -45:50.616 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp -45:50.616 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp -45:50.618 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp -45:50.618 [INFO] TestAgent: started state syncer -45:50.618 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp -45:50.667 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:50.667 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 -45:50.675 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:50.675 [DEBUG] TestAgent.server.raft: vote granted: from=9be02b7b-c2a2-20fd-911c-b8b916cf1b54 term=2 tally=1 -45:50.675 [INFO] TestAgent.server.raft: election won: tally=1 -45:50.675 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" -45:50.675 [INFO] TestAgent.server: cluster leadership acquired -45:50.675 [INFO] TestAgent.server: New leader elected: payload=Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54 -45:50.680 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 -45:50.691 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:50.718 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:50.718 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:50.718 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:50.718 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:50.718 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:50.718 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54 -45:50.718 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9be02b7b-c2a2-20fd-911c-b8b916cf1b54 -45:50.723 [INFO] TestAgent.server: federation state anti-entropy synced -45:50.943 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:50.947 [INFO] TestAgent: Synced node info -45:50.947 [DEBUG] TestAgent: Node info in sync -45:51.028 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=us-ingress -45:51.028 [ERROR] TestAgent.proxycfg: watch error: service_id=us-ingress-1 id=gateway-config error="invalid type for config entry: " -45:51.028 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=us-ingress -45:51.038 [DEBUG] TestAgent: Node info in sync -45:51.044 [INFO] TestAgent: Synced service: service=us-ingress-1 -45:51.044 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 -45:51.044 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54852 latency=16.703482ms -45:51.045 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:54852 latency=961.385µs -45:51.048 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:54852 latency=244.945µs -45:51.051 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:54854 latency=418.319µs -45:51.054 [ERROR] TestAgent.proxycfg: watch error: service_id=us-ingress id=gateway-config error="invalid type for config entry: " -45:51.061 [DEBUG] TestAgent: Node info in sync -45:51.065 [INFO] TestAgent: Synced service: service=us-ingress -45:51.065 [DEBUG] TestAgent: Service in sync: service=us-ingress-1 -45:51.065 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 -45:51.065 [DEBUG] TestAgent: Check in sync: check=service:us-ingress -45:51.065 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54856 latency=11.955138ms -45:51.066 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:54856 latency=1.002303ms -45:51.069 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress from=127.0.0.1:54856 latency=942.066µs -45:51.070 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress from=127.0.0.1:54854 latency=97.556µs -45:51.072 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=ingress-gateway -45:51.072 [ERROR] TestAgent.proxycfg: watch error: service_id=ingress-gateway id=gateway-config error="invalid type for config entry: " -45:51.072 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=ingress-gateway -45:51.084 [DEBUG] TestAgent: Node info in sync -45:51.084 [DEBUG] TestAgent: Service in sync: service=us-ingress-1 -45:51.084 [DEBUG] TestAgent: Service in sync: service=us-ingress -45:51.090 [INFO] TestAgent: Synced service: service=ingress-gateway -45:51.090 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 -45:51.090 [DEBUG] TestAgent: Check in sync: check=service:us-ingress -45:51.090 [DEBUG] TestAgent: Check in sync: check=service:ingress-gateway -45:51.090 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54858 latency=18.737152ms -45:51.091 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:54858 latency=914.774µs -45:51.093 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ingress-gateway from=127.0.0.1:54858 latency=163.891µs -45:51.097 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ingress-gateway from=127.0.0.1:54854 latency=191.149µs -45:51.105 [DEBUG] TestAgent: Node info in sync -45:51.109 [INFO] TestAgent: Synced service: service=us-ingress-1 -45:51.109 [DEBUG] TestAgent: Service in sync: service=us-ingress -45:51.109 [DEBUG] TestAgent: Service in sync: service=ingress-gateway -45:51.109 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 -45:51.109 [DEBUG] TestAgent: Check in sync: check=service:us-ingress -45:51.109 [DEBUG] TestAgent: Check in sync: check=service:ingress-gateway -45:51.109 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54860 latency=11.09536ms -45:51.111 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:54860 latency=880.256µs -45:51.112 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:54860 latency=143.549µs -45:51.113 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:54854 latency=246.007µs -45:51.113 [INFO] TestAgent: Requesting shutdown -45:51.113 [INFO] TestAgent.server: shutting down server -45:51.113 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:51.113 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:51.113 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:51.113 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:51.113 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:51.114 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:51.114 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:51.114 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:51.114 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:51.114 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:51.118 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:51.121Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:51.121Z [INFO] agent.router.manager: shutting down -45:51.121 [INFO] TestAgent: consul server down -45:51.121 [INFO] TestAgent: shutdown complete -45:51.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp -45:51.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp -45:51.121 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp -45:51.621 [INFO] TestAgent: Waiting for endpoints to shut down -45:51.621 [INFO] TestAgent: Endpoints down ---- PASS: TestEnvoy_GatewayRegistration (1.11s) - --- PASS: TestEnvoy_GatewayRegistration/register_gateway_with_proxy-id_and_name (0.03s) - --- PASS: TestEnvoy_GatewayRegistration/register_gateway_without_proxy-id_with_name (0.02s) - --- PASS: TestEnvoy_GatewayRegistration/register_gateway_without_proxy-id_and_without_name (0.03s) - --- PASS: TestEnvoy_GatewayRegistration/register_gateway_with_proxy-id_without_name (0.02s) +2021-10-05T16:32:25.213Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:25.227Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:25.284 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:631b5d6a-a435-620c-4e50-ef13b044b3d2 Address:127.0.0.1:27363}]" +32:25.285 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-631b5d6a-a435-620c-4e50-ef13b044b3d2.dc1 127.0.0.1 +32:25.285 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-631b5d6a-a435-620c-4e50-ef13b044b3d2 127.0.0.1 +2021-10-05T16:32:25.285Z [INFO] agent.router: Initializing LAN area manager +32:25.300 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +32:25.300 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +32:25.301 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-631b5d6a-a435-620c-4e50-ef13b044b3d2.dc1 area=wan +32:25.301 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +32:25.301 [INFO] TestAgent.server: Adding LAN server: server="Node-631b5d6a-a435-620c-4e50-ef13b044b3d2 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +32:25.332 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +32:25.332 [INFO] TestAgent: started state syncer +32:25.332 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +32:25.356 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:25.356 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +32:25.399 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:25.399 [DEBUG] TestAgent.server.raft: vote granted: from=631b5d6a-a435-620c-4e50-ef13b044b3d2 term=2 tally=1 +32:25.399 [INFO] TestAgent.server.raft: election won: tally=1 +32:25.399 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +32:25.400 [INFO] TestAgent.server: cluster leadership acquired +32:25.400 [INFO] TestAgent.server: New leader elected: payload=Node-631b5d6a-a435-620c-4e50-ef13b044b3d2 +32:25.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +32:25.448 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:25.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:25.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:25.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:25.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:25.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:25.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-631b5d6a-a435-620c-4e50-ef13b044b3d2 +32:25.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-631b5d6a-a435-620c-4e50-ef13b044b3d2 +32:25.524 [INFO] TestAgent.server: federation state anti-entropy synced +32:25.670 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=us-ingress +32:25.670 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=us-ingress +32:25.670 [ERROR] TestAgent.proxycfg: watch error: service_id=us-ingress-1 id=gateway-config error="invalid type for config entry: " +32:25.688 [INFO] TestAgent: Synced node info +32:25.695 [INFO] TestAgent: Synced service: service=us-ingress-1 +32:25.695 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 +32:25.695 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59512 latency=26.985595ms +32:25.700 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59512 latency=5.137056ms +32:25.710 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:59512 latency=325.051µs +32:25.714 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:59516 latency=1.226643ms +32:25.721 [ERROR] TestAgent.proxycfg: watch error: service_id=us-ingress id=gateway-config error="invalid type for config entry: " +32:25.724 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:25.724 [DEBUG] TestAgent: Node info in sync +32:25.724 [DEBUG] TestAgent: Service in sync: service=us-ingress-1 +32:25.729 [INFO] TestAgent: Synced service: service=us-ingress +32:25.729 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 +32:25.729 [DEBUG] TestAgent: Check in sync: check=service:us-ingress +32:25.748 [DEBUG] TestAgent: Node info in sync +32:25.748 [DEBUG] TestAgent: Service in sync: service=us-ingress-1 +32:25.748 [DEBUG] TestAgent: Service in sync: service=us-ingress +32:25.748 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 +32:25.748 [DEBUG] TestAgent: Check in sync: check=service:us-ingress +32:25.748 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59518 latency=27.825608ms +32:25.750 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59518 latency=1.632844ms +32:25.761 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress from=127.0.0.1:59518 latency=2.227104ms +32:25.764 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress from=127.0.0.1:59516 latency=207.774µs +32:25.782 [ERROR] TestAgent.proxycfg: watch error: service_id=ingress-gateway id=gateway-config error="invalid type for config entry: " +32:25.782 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=ingress-gateway +32:25.782 [WARN] TestAgent.server.catalog: no terminating-gateway or ingress-gateway associated with this gateway: gateway=ingress-gateway +32:25.795 [DEBUG] TestAgent: Node info in sync +32:25.795 [DEBUG] TestAgent: Service in sync: service=us-ingress-1 +32:25.795 [DEBUG] TestAgent: Service in sync: service=us-ingress +32:25.800 [INFO] TestAgent: Synced service: service=ingress-gateway +32:25.800 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 +32:25.800 [DEBUG] TestAgent: Check in sync: check=service:us-ingress +32:25.800 [DEBUG] TestAgent: Check in sync: check=service:ingress-gateway +32:25.800 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59522 latency=20.137887ms +32:25.802 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59522 latency=1.149667ms +32:25.816 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ingress-gateway from=127.0.0.1:59522 latency=569.17µs +32:25.824 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/ingress-gateway from=127.0.0.1:59516 latency=493.842µs +32:25.844 [DEBUG] TestAgent: Node info in sync +32:25.865 [INFO] TestAgent: Synced service: service=us-ingress-1 +32:25.865 [DEBUG] TestAgent: Service in sync: service=us-ingress +32:25.865 [DEBUG] TestAgent: Service in sync: service=ingress-gateway +32:25.865 [DEBUG] TestAgent: Check in sync: check=service:us-ingress-1 +32:25.865 [DEBUG] TestAgent: Check in sync: check=service:us-ingress +32:25.865 [DEBUG] TestAgent: Check in sync: check=service:ingress-gateway +32:25.865 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59526 latency=38.340547ms +32:25.875 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59526 latency=3.598763ms +32:25.890 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:59526 latency=217.664µs +32:25.892 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/us-ingress-1 from=127.0.0.1:59516 latency=929.037µs +32:25.893 [INFO] TestAgent: Requesting shutdown +32:25.893 [INFO] TestAgent.server: shutting down server +32:25.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:25.893 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:25.893 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:25.893 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:25.893 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:25.893 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:25.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:25.893 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:25.893 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:25.894 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:25.908 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:25.916Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:25.916Z [INFO] agent.router.manager: shutting down +32:25.916 [INFO] TestAgent: consul server down +32:25.916 [INFO] TestAgent: shutdown complete +32:25.916 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +32:25.916 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +32:25.916 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +32:26.420 [INFO] TestAgent: Waiting for endpoints to shut down +32:26.420 [INFO] TestAgent: Endpoints down +--- PASS: TestEnvoy_GatewayRegistration (1.32s) + --- PASS: TestEnvoy_GatewayRegistration/register_gateway_with_proxy-id_and_name (0.05s) + --- PASS: TestEnvoy_GatewayRegistration/register_gateway_without_proxy-id_with_name (0.05s) + --- PASS: TestEnvoy_GatewayRegistration/register_gateway_without_proxy-id_and_without_name (0.05s) + --- PASS: TestEnvoy_GatewayRegistration/register_gateway_with_proxy-id_without_name (0.07s) PASS -ok github.com/hashicorp/consul/command/connect/envoy 1.523s +ok github.com/hashicorp/consul/command/connect/envoy 4.097s === RUN TestConnectEnvoyPipeBootstrapCommand_noTabs === PAUSE TestConnectEnvoyPipeBootstrapCommand_noTabs === CONT TestConnectEnvoyPipeBootstrapCommand_noTabs --- PASS: TestConnectEnvoyPipeBootstrapCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap 0.005s +ok github.com/hashicorp/consul/command/connect/envoy/pipe-bootstrap 0.062s === RUN TestConnectExpose === PAUSE TestConnectExpose === RUN TestConnectExpose_invalidFlags @@ -82935,219 +83239,220 @@ === RUN TestConnectExpose_existingConfig === PAUSE TestConnectExpose_existingConfig === CONT TestConnectExpose +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestConnectExpose_existingConfig === CONT TestConnectExpose_invalidFlags -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestConnectExpose_invalidFlags/missing_service === RUN TestConnectExpose_invalidFlags/missing_gateway === RUN TestConnectExpose_invalidFlags/missing_port -2022-11-07T22:45:52.525Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:52.540Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:52.567 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b65aa9ca-18c7-17c0-582f-682ba62a6ae5 Address:127.0.0.1:10013}]" -45:52.567 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= -45:52.567 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5.dc1 127.0.0.1 -45:52.568 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5 127.0.0.1 -2022-11-07T22:45:52.568Z [INFO] agent.router: Initializing LAN area manager -45:52.568 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5.dc1 area=wan -45:52.568 [INFO] TestAgent.server: Adding LAN server: server="Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -45:52.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp -45:52.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp -45:52.569 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp -45:52.569 [INFO] TestAgent: started state syncer -45:52.569 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp -45:52.615 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:52.615 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 -45:52.623 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:52.623 [DEBUG] TestAgent.server.raft: vote granted: from=b65aa9ca-18c7-17c0-582f-682ba62a6ae5 term=2 tally=1 -45:52.623 [INFO] TestAgent.server.raft: election won: tally=1 -45:52.623 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" -45:52.624 [INFO] TestAgent.server: cluster leadership acquired -45:52.624 [INFO] TestAgent.server: New leader elected: payload=Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5 -45:52.629 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10013 -45:52.643 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:52.667 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:52.667 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:52.667 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:52.667 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:52.667 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:52.667 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5 -45:52.667 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b65aa9ca-18c7-17c0-582f-682ba62a6ae5 -45:52.675 [INFO] TestAgent.server: federation state anti-entropy synced -45:52.686 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:46014 latency=4.910058ms -45:52.695 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:46014 latency=8.010118ms -45:52.700 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:46014 latency=4.039725ms -45:52.701 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46016 latency=252.325µs -45:52.706 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=14 from=127.0.0.1:46016 latency=4.606072ms -45:52.706 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46016 latency=153.576µs -45:52.712 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46016 latency=5.154115ms -45:52.712 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46014 latency=74.8µs -45:52.713 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46018 latency=91.576µs -45:52.718 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=15 from=127.0.0.1:46018 latency=4.329408ms -45:52.718 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46018 latency=145.401µs -45:52.723 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46018 latency=4.614622ms -45:52.724 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46014 latency=89.109µs -45:52.725 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46020 latency=183.3µs -45:52.731 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=17 from=127.0.0.1:46020 latency=5.776495ms -45:52.732 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46020 latency=113.489µs -45:52.740 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46020 latency=7.014115ms -45:52.740 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46014 latency=103.407µs -45:52.741 [INFO] TestAgent: Requesting shutdown -45:52.741 [INFO] TestAgent.server: shutting down server -45:52.741 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:52.741 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:52.741 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:52.741 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:52.741 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:52.741 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:52.741 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:52.741 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:52.741 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:52.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:52.741 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:52.744 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:52.748Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:52.748Z [INFO] agent.router.manager: shutting down -45:52.748 [INFO] TestAgent: consul server down -45:52.748 [INFO] TestAgent: shutdown complete -45:52.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp -45:52.748 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp -45:52.748 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp -45:53.248 [INFO] TestAgent: Waiting for endpoints to shut down -45:53.248 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectExpose_existingConfig (0.76s) -2022-11-07T22:45:52.530Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:52.544Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:52.569 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7992df74-a777-619b-a337-04b20f3545cb Address:127.0.0.1:10006}]" -45:52.570 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -45:52.570 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7992df74-a777-619b-a337-04b20f3545cb.dc1 127.0.0.1 -45:52.570 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7992df74-a777-619b-a337-04b20f3545cb 127.0.0.1 -2022-11-07T22:45:52.570Z [INFO] agent.router: Initializing LAN area manager -45:52.570 [INFO] TestAgent.server: Adding LAN server: server="Node-7992df74-a777-619b-a337-04b20f3545cb (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:52.570 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7992df74-a777-619b-a337-04b20f3545cb.dc1 area=wan -45:52.570 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -45:52.571 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -45:52.571 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -45:52.571 [INFO] TestAgent: started state syncer -45:52.571 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -45:52.635 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:52.635 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -45:52.646 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:52.646 [DEBUG] TestAgent.server.raft: vote granted: from=7992df74-a777-619b-a337-04b20f3545cb term=2 tally=1 -45:52.646 [INFO] TestAgent.server.raft: election won: tally=1 -45:52.646 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -45:52.646 [INFO] TestAgent.server: cluster leadership acquired -45:52.646 [INFO] TestAgent.server: New leader elected: payload=Node-7992df74-a777-619b-a337-04b20f3545cb -45:52.653 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10006 -45:52.668 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:52.695 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:52.695 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:52.695 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:52.695 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:52.695 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:52.695 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7992df74-a777-619b-a337-04b20f3545cb -45:52.695 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7992df74-a777-619b-a337-04b20f3545cb -45:52.707 [INFO] TestAgent.server: federation state anti-entropy synced -45:52.726 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37804 error="Config entry not found for "ingress-gateway" / "ingress"" -45:52.726 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37804 latency=137.447µs -45:52.730 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=0 from=127.0.0.1:37804 latency=3.314168ms -45:52.730 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37804 latency=66.072µs -45:52.734 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:37804 latency=3.093727ms -45:52.734 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37806 latency=118.391µs -45:52.735 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37806 latency=108.302µs -45:52.740 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37808 latency=547.559µs -45:52.744 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=12 from=127.0.0.1:37808 latency=3.883571ms -45:52.744 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37808 latency=86.169µs -45:52.745 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37806 latency=87.814µs -45:52.745 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37806 latency=125.434µs -45:52.746 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37810 latency=87.359µs -45:52.747 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:37806 latency=75.754µs -45:52.747 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37806 latency=112.457µs -45:52.747 [INFO] TestAgent: Requesting shutdown -45:52.748 [INFO] TestAgent.server: shutting down server -45:52.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:52.748 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:52.748 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:52.748 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:52.748 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:52.748 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:52.748 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:52.748 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:52.748 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:52.748 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:52.748 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:52.751 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:52.754Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:52.754Z [INFO] agent.router.manager: shutting down -45:52.754 [INFO] TestAgent: consul server down -45:52.754 [INFO] TestAgent: shutdown complete -45:52.754 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -45:52.754 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -45:52.754 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -45:53.254 [INFO] TestAgent: Waiting for endpoints to shut down -45:53.254 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectExpose (0.76s) -2022-11-07T22:45:52.529Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:52.540Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:52.565 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:84774862-9460-a398-d276-27f147ef56e6 Address:127.0.0.1:10020}]" -45:52.565 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10020 [Follower]" leader= -45:52.565 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-84774862-9460-a398-d276-27f147ef56e6.dc1 127.0.0.1 -45:52.566 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-84774862-9460-a398-d276-27f147ef56e6 127.0.0.1 -2022-11-07T22:45:52.566Z [INFO] agent.router: Initializing LAN area manager -45:52.566 [INFO] TestAgent.server: Adding LAN server: server="Node-84774862-9460-a398-d276-27f147ef56e6 (Addr: tcp/127.0.0.1:10020) (DC: dc1)" -45:52.566 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-84774862-9460-a398-d276-27f147ef56e6.dc1 area=wan -45:52.566 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10015 network=tcp -45:52.566 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10015 network=udp -45:52.567 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10016 network=tcp -45:52.567 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10021 network=tcp -45:52.567 [INFO] TestAgent: started state syncer -45:52.630 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:52.630 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10020 [Candidate]" term=2 -45:52.641 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:52.642 [DEBUG] TestAgent.server.raft: vote granted: from=84774862-9460-a398-d276-27f147ef56e6 term=2 tally=1 -45:52.642 [INFO] TestAgent.server.raft: election won: tally=1 -45:52.642 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10020 [Leader]" -45:52.642 [INFO] TestAgent.server: cluster leadership acquired -45:52.642 [INFO] TestAgent.server: New leader elected: payload=Node-84774862-9460-a398-d276-27f147ef56e6 -45:52.650 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10020 -45:52.666 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:52.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:52.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:52.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:52.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:52.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:52.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-84774862-9460-a398-d276-27f147ef56e6 -45:52.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-84774862-9460-a398-d276-27f147ef56e6 -45:52.700 [INFO] TestAgent.server: federation state anti-entropy synced -45:52.822 [INFO] TestAgent: Requesting shutdown -45:52.822 [INFO] TestAgent.server: shutting down server -45:52.822 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:52.822 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:52.822 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:52.822 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:52.822 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:52.822 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:52.822 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:52.822 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:52.822 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:52.822 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:52.826 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:52.829Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:52.829Z [INFO] agent.router.manager: shutting down -45:52.829 [INFO] TestAgent: consul server down -45:52.829 [INFO] TestAgent: shutdown complete -45:52.829 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10015 network=tcp -45:52.829 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10015 network=udp -45:52.829 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10016 network=tcp -45:53.329 [INFO] TestAgent: Waiting for endpoints to shut down -45:53.329 [INFO] TestAgent: Endpoints down ---- PASS: TestConnectExpose_invalidFlags (0.84s) +2021-10-05T16:32:27.293Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:27.309Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:27.353 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:90d2b7d8-5d40-e582-f3b6-ed353e831bdd Address:127.0.0.1:22272}]" +32:27.354 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22272 [Follower]" leader= +32:27.355 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd.dc1 127.0.0.1 +32:27.385 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd 127.0.0.1 +2021-10-05T16:32:27.385Z [INFO] agent.router: Initializing LAN area manager +32:27.385 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22267 network=udp +32:27.385 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd.dc1 area=wan +32:27.385 [INFO] TestAgent.server: Adding LAN server: server="Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd (Addr: tcp/127.0.0.1:22272) (DC: dc1)" +32:27.385 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22267 network=tcp +32:27.386 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22268 network=tcp +32:27.386 [INFO] TestAgent: started state syncer +32:27.386 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22273 network=tcp +32:27.399 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:27.399 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22272 [Candidate]" term=2 +32:27.413 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:27.413 [DEBUG] TestAgent.server.raft: vote granted: from=90d2b7d8-5d40-e582-f3b6-ed353e831bdd term=2 tally=1 +32:27.413 [INFO] TestAgent.server.raft: election won: tally=1 +32:27.413 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22272 [Leader]" +32:27.413 [INFO] TestAgent.server: cluster leadership acquired +32:27.414 [INFO] TestAgent.server: New leader elected: payload=Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd +32:27.429 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22272 +32:27.456 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:27.499 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:27.499 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:27.499 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:27.499 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:27.499 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:27.499 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd +32:27.499 [INFO] TestAgent.server: member joined, marking health alive: member=Node-90d2b7d8-5d40-e582-f3b6-ed353e831bdd +32:27.528 [INFO] TestAgent.server: federation state anti-entropy synced +32:27.539 [INFO] TestAgent: Requesting shutdown +32:27.539 [INFO] TestAgent.server: shutting down server +32:27.539 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:27.539 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:27.539 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:27.539 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:27.539 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:27.539 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:27.539 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:27.539 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:27.539 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:27.539 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:27.539 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:27.548 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:27.560Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:27.560Z [INFO] agent.router.manager: shutting down +32:27.560 [INFO] TestAgent: consul server down +32:27.560 [INFO] TestAgent: shutdown complete +32:27.560 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22267 network=tcp +32:27.560 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22267 network=udp +32:27.560 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22268 network=tcp +32:28.064 [INFO] TestAgent: Waiting for endpoints to shut down +32:28.064 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectExpose_invalidFlags (0.92s) --- PASS: TestConnectExpose_invalidFlags/missing_service (0.00s) --- PASS: TestConnectExpose_invalidFlags/missing_gateway (0.00s) --- PASS: TestConnectExpose_invalidFlags/missing_port (0.00s) +2021-10-05T16:32:27.235Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:27.299Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:27.353 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a789fdd9-aed1-4129-d893-cac42fb55520 Address:127.0.0.1:22258}]" +32:27.354 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= +32:27.354 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a789fdd9-aed1-4129-d893-cac42fb55520.dc1 127.0.0.1 +32:27.355 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a789fdd9-aed1-4129-d893-cac42fb55520 127.0.0.1 +2021-10-05T16:32:27.355Z [INFO] agent.router: Initializing LAN area manager +32:27.359 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a789fdd9-aed1-4129-d893-cac42fb55520.dc1 area=wan +32:27.359 [INFO] TestAgent.server: Adding LAN server: server="Node-a789fdd9-aed1-4129-d893-cac42fb55520 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +32:27.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp +32:27.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp +32:27.385 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp +32:27.385 [INFO] TestAgent: started state syncer +32:27.385 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp +32:27.396 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:27.396 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 +32:27.412 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:27.412 [DEBUG] TestAgent.server.raft: vote granted: from=a789fdd9-aed1-4129-d893-cac42fb55520 term=2 tally=1 +32:27.412 [INFO] TestAgent.server.raft: election won: tally=1 +32:27.412 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" +32:27.413 [INFO] TestAgent.server: cluster leadership acquired +32:27.413 [INFO] TestAgent.server: New leader elected: payload=Node-a789fdd9-aed1-4129-d893-cac42fb55520 +32:27.431 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 +32:27.447 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:27.512 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:27.512 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:27.512 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:27.512 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:27.512 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:27.512 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a789fdd9-aed1-4129-d893-cac42fb55520 +32:27.512 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a789fdd9-aed1-4129-d893-cac42fb55520 +32:27.526 [ERROR] TestAgent.http: Request error: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46026 error="Config entry not found for "ingress-gateway" / "ingress"" +32:27.527 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46026 latency=369.077µs +32:27.565 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=0 from=127.0.0.1:46026 latency=37.48587ms +32:27.565 [INFO] TestAgent.server: federation state anti-entropy synced +32:27.566 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46026 latency=554.176µs +32:27.579 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46026 latency=12.447123ms +32:27.581 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46030 latency=343.534µs +32:27.583 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46030 latency=300.013µs +32:27.590 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46032 latency=285.679µs +32:27.599 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=12 from=127.0.0.1:46032 latency=8.456789ms +32:27.600 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46032 latency=617.667µs +32:27.603 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46030 latency=428.177µs +32:27.606 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46030 latency=472.516µs +32:27.611 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46034 latency=333.266µs +32:27.612 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:46030 latency=204.165µs +32:27.613 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:46030 latency=157.333µs +32:27.613 [INFO] TestAgent: Requesting shutdown +32:27.613 [INFO] TestAgent.server: shutting down server +32:27.613 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:27.613 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:27.613 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:27.613 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:27.613 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:27.614 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:27.614 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:27.614 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:27.614 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:27.614 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:27.614 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:27.624 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:27.631Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:27.631Z [INFO] agent.router.manager: shutting down +32:27.631 [INFO] TestAgent: consul server down +32:27.631 [INFO] TestAgent: shutdown complete +32:27.631 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp +32:27.631 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp +32:27.631 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp +32:28.132 [INFO] TestAgent: Waiting for endpoints to shut down +32:28.132 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectExpose (0.99s) +2021-10-05T16:32:27.260Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:27.272Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:27.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:186ee0f1-907f-8fe7-9460-b3d78abbeaed Address:127.0.0.1:22265}]" +32:27.324 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22265 [Follower]" leader= +32:27.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed.dc1 127.0.0.1 +32:27.326 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed 127.0.0.1 +2021-10-05T16:32:27.326Z [INFO] agent.router: Initializing LAN area manager +32:27.332 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed.dc1 area=wan +32:27.332 [INFO] TestAgent.server: Adding LAN server: server="Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed (Addr: tcp/127.0.0.1:22265) (DC: dc1)" +32:27.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22260 network=udp +32:27.332 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22260 network=tcp +32:27.336 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22261 network=tcp +32:27.336 [INFO] TestAgent: started state syncer +32:27.337 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22266 network=tcp +32:27.376 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:27.376 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22265 [Candidate]" term=2 +32:27.406 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:27.406 [DEBUG] TestAgent.server.raft: vote granted: from=186ee0f1-907f-8fe7-9460-b3d78abbeaed term=2 tally=1 +32:27.406 [INFO] TestAgent.server.raft: election won: tally=1 +32:27.406 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22265 [Leader]" +32:27.406 [INFO] TestAgent.server: cluster leadership acquired +32:27.406 [INFO] TestAgent.server: New leader elected: payload=Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed +32:27.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22265 +32:27.421 [INFO] TestAgent: Synced node info +32:27.458 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:27.465 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:55874 latency=22.577218ms +32:27.479 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:55874 latency=12.990672ms +32:27.503 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:55874 latency=22.220621ms +32:27.526 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:27.526 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:27.526 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:27.526 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:27.526 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:27.526 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed +32:27.526 [INFO] TestAgent.server: member joined, marking health alive: member=Node-186ee0f1-907f-8fe7-9460-b3d78abbeaed +32:27.547 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55878 latency=2.252572ms +32:27.556 [INFO] TestAgent.server: federation state anti-entropy synced +32:27.578 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=12 from=127.0.0.1:55878 latency=17.646826ms +32:27.581 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:55878 latency=362.843µs +32:27.599 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:55878 latency=15.564522ms +32:27.600 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55874 latency=171.802µs +32:27.611 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55886 latency=258.82µs +32:27.629 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=16 from=127.0.0.1:55886 latency=17.121507ms +32:27.630 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:55886 latency=268.875µs +32:27.635 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:55886 latency=5.048259ms +32:27.636 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55874 latency=442.619µs +32:27.638 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55888 latency=294.827µs +32:27.652 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/config?cas=18 from=127.0.0.1:55888 latency=13.64226ms +32:27.656 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:55888 latency=209.856µs +32:27.667 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:55888 latency=10.437673ms +32:27.676 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/config/ingress-gateway/ingress from=127.0.0.1:55874 latency=281.511µs +32:27.677 [INFO] TestAgent: Requesting shutdown +32:27.677 [INFO] TestAgent.server: shutting down server +32:27.677 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:27.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:27.677 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:27.677 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:27.677 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:27.677 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:27.677 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:27.677 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:27.677 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:27.677 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:27.685 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:27.685Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:27.685Z [INFO] agent.router.manager: shutting down +32:27.685 [INFO] TestAgent: consul server down +32:27.685 [INFO] TestAgent: shutdown complete +32:27.685 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22260 network=tcp +32:27.685 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22260 network=udp +32:27.685 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22261 network=tcp +32:28.188 [INFO] TestAgent: Waiting for endpoints to shut down +32:28.188 [INFO] TestAgent: Endpoints down +--- PASS: TestConnectExpose_existingConfig (1.04s) PASS -ok github.com/hashicorp/consul/command/connect/expose 0.870s +ok github.com/hashicorp/consul/command/connect/expose 1.163s === RUN TestFlagUpstreams_impl --- PASS: TestFlagUpstreams_impl (0.00s) === RUN TestFlagUpstreams @@ -83184,741 +83489,711 @@ [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCatalogCommand_noTabs (0.00s) -2022-11-07T22:45:54.114Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.132Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.164 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:db143838-eb8a-4f57-c550-b5ae87ad53ed Address:127.0.0.1:21237}]" -45:54.164 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= -45:54.164 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-db143838-eb8a-4f57-c550-b5ae87ad53ed.dc1 127.0.0.1 -45:54.164 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-db143838-eb8a-4f57-c550-b5ae87ad53ed 127.0.0.1 -2022-11-07T22:45:54.164Z [INFO] agent.router: Initializing LAN area manager -45:54.164 [INFO] TestAgent.server: Adding LAN server: server="Node-db143838-eb8a-4f57-c550-b5ae87ad53ed (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -45:54.165 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-db143838-eb8a-4f57-c550-b5ae87ad53ed.dc1 area=wan -45:54.165 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp -45:54.165 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp -45:54.165 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp -45:54.165 [INFO] TestAgent: started state syncer -45:54.165 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp -45:54.232 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:54.232 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 -45:54.243 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:54.243 [DEBUG] TestAgent.server.raft: vote granted: from=db143838-eb8a-4f57-c550-b5ae87ad53ed term=2 tally=1 -45:54.243 [INFO] TestAgent.server.raft: election won: tally=1 -45:54.243 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" -45:54.243 [INFO] TestAgent.server: cluster leadership acquired -45:54.243 [INFO] TestAgent.server: New leader elected: payload=Node-db143838-eb8a-4f57-c550-b5ae87ad53ed -45:54.250 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21237 -45:54.273 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:54.273 [INFO] TestAgent: Synced node info -45:54.277 [INFO] TestAgent: Synced service: service=no-sidecar -45:54.283 [INFO] TestAgent: Synced service: service=one-sidecar -45:54.290 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy -45:54.300 [INFO] TestAgent: Synced service: service=two-sidecars -45:54.306 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:54.306 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:54.306 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:54.306 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:54.306 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:54.306 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-db143838-eb8a-4f57-c550-b5ae87ad53ed -45:54.306 [INFO] TestAgent.server: member joined, marking health alive: member=Node-db143838-eb8a-4f57-c550-b5ae87ad53ed -45:54.310 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy -45:54.327 [INFO] TestAgent.server: federation state anti-entropy synced -45:54.327 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:54.327 [DEBUG] TestAgent: Node info in sync -45:54.327 [DEBUG] TestAgent: Service in sync: service=other-sidecar-for-two-sidecars -45:54.327 [DEBUG] TestAgent: Service in sync: service=no-sidecar -45:54.327 [DEBUG] TestAgent: Service in sync: service=one-sidecar -45:54.327 [DEBUG] TestAgent: Service in sync: service=one-sidecar-sidecar-proxy -45:54.327 [DEBUG] TestAgent: Service in sync: service=two-sidecars -45:54.327 [DEBUG] TestAgent: Service in sync: service=two-sidecars-sidecar-proxy -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:54.327 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:54.376 [INFO] TestAgent: Requesting shutdown -45:54.377 [INFO] TestAgent.server: shutting down server -45:54.377 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:54.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:54.377 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:54.377 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:54.377 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:54.377 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:54.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:54.377 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:54.377 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:54.377 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:54.380 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:54.384Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:54.384Z [INFO] agent.router.manager: shutting down -45:54.384 [INFO] TestAgent: consul server down -45:54.384 [INFO] TestAgent: shutdown complete -45:54.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp -45:54.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp -45:54.384 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp -45:54.884 [INFO] TestAgent: Waiting for endpoints to shut down -45:54.884 [INFO] TestAgent: Endpoints down +--- PASS: TestCatalogCommand_noTabs (0.01s) +2021-10-05T16:32:34.123Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:34.166Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:34.219 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b8460f2-9e03-03b3-fea3-fe6571a22d22 Address:127.0.0.1:21237}]" +32:34.219 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= +32:34.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22.dc1 127.0.0.1 +32:34.235 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22 127.0.0.1 +2021-10-05T16:32:34.236Z [INFO] agent.router: Initializing LAN area manager +32:34.237 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp +32:34.237 [INFO] TestAgent.server: Adding LAN server: server="Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +32:34.237 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22.dc1 area=wan +32:34.238 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp +32:34.239 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp +32:34.239 [INFO] TestAgent: started state syncer +32:34.252 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp +32:34.281 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:34.281 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 +32:34.291 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:34.291 [DEBUG] TestAgent.server.raft: vote granted: from=3b8460f2-9e03-03b3-fea3-fe6571a22d22 term=2 tally=1 +32:34.291 [INFO] TestAgent.server.raft: election won: tally=1 +32:34.291 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" +32:34.293 [INFO] TestAgent.server: cluster leadership acquired +32:34.293 [INFO] TestAgent.server: New leader elected: payload=Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22 +32:34.298 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21237 +32:34.319 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:34.392 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:34.392 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:34.392 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:34.392 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:34.392 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:34.392 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22 +32:34.392 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3b8460f2-9e03-03b3-fea3-fe6571a22d22 +32:34.422 [INFO] TestAgent: Requesting shutdown +32:34.423 [INFO] TestAgent.server: shutting down server +32:34.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:34.423 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:34.423 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:34.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:34.423 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:34.423 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:34.423 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:34.423 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:34.423 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:34.428 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:34.428 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:34.434 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:34.444Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:34.444Z [INFO] agent.router.manager: shutting down +32:34.444 [INFO] TestAgent: consul server down +32:34.444 [INFO] TestAgent: shutdown complete +32:34.444 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp +32:34.444 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp +32:34.444 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp +32:34.946 [INFO] TestAgent: Waiting for endpoints to shut down +32:34.946 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-service_flag_with_upstreams -2022-11-07T22:45:54.437Z [INFO] TestRegisterMonitor_good: registered Consul service: service=foo-proxy -2022-11-07T22:45:54.442Z [INFO] TestRegisterMonitor_good: stop request received, deregistering -2022-11-07T22:45:54.111Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.128Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.161 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:48f0685a-742e-c022-c0d6-12c2aca33075 Address:127.0.0.1:21251}]" -45:54.161 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21251 [Follower]" leader= -45:54.161 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-48f0685a-742e-c022-c0d6-12c2aca33075.dc1 127.0.0.1 -45:54.161 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-48f0685a-742e-c022-c0d6-12c2aca33075 127.0.0.1 -2022-11-07T22:45:54.161Z [INFO] agent.router: Initializing LAN area manager -45:54.162 [INFO] TestAgent.server: Adding LAN server: server="Node-48f0685a-742e-c022-c0d6-12c2aca33075 (Addr: tcp/127.0.0.1:21251) (DC: dc1)" -45:54.162 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-48f0685a-742e-c022-c0d6-12c2aca33075.dc1 area=wan -45:54.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=udp -45:54.162 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=tcp -45:54.162 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21247 network=tcp -45:54.162 [INFO] TestAgent: started state syncer -45:54.162 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21252 network=tcp -45:54.201 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:54.201 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21251 [Candidate]" term=2 -45:54.209 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:54.209 [DEBUG] TestAgent.server.raft: vote granted: from=48f0685a-742e-c022-c0d6-12c2aca33075 term=2 tally=1 -45:54.209 [INFO] TestAgent.server.raft: election won: tally=1 -45:54.209 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21251 [Leader]" -45:54.209 [INFO] TestAgent.server: cluster leadership acquired -45:54.209 [INFO] TestAgent.server: New leader elected: payload=Node-48f0685a-742e-c022-c0d6-12c2aca33075 -45:54.215 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21251 -45:54.228 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:54.248 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:54.248 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:54.248 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:54.248 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:54.248 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:54.248 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-48f0685a-742e-c022-c0d6-12c2aca33075 -45:54.248 [INFO] TestAgent.server: member joined, marking health alive: member=Node-48f0685a-742e-c022-c0d6-12c2aca33075 -45:54.261 [INFO] TestAgent.server: federation state anti-entropy synced -45:54.266 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:54.270 [INFO] TestAgent: Synced node info -45:54.416 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:37874 latency=830.812µs -45:54.416 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/foo-proxy?stale= from=127.0.0.1:37876 latency=763.236µs -45:54.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.427 [DEBUG] TestAgent: Node info in sync -45:54.437 [INFO] TestAgent: Synced service: service=foo-proxy -45:54.437 [DEBUG] TestAgent: Check in sync: check=foo-proxy-ttl -45:54.437 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37876 latency=20.772852ms -45:54.442 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:37876 latency=151.903µs -45:54.443 [DEBUG] TestAgent: removed check: check=foo-proxy-ttl -45:54.443 [DEBUG] TestAgent: removed service: service=foo-proxy -45:54.443 [DEBUG] TestAgent: Node info in sync -45:54.445 [INFO] TestAgent: Deregistered service: service=foo-proxy -45:54.445 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/foo-proxy from=127.0.0.1:37876 latency=3.011466ms -45:54.446 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:37876 latency=775.961µs -45:54.447 [INFO] TestAgent: Requesting shutdown -45:54.447 [INFO] TestAgent.server: shutting down server -45:54.447 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:54.447 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:54.447 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:54.447 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:54.447 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:54.447 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:54.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:54.447 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:54.447 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:54.447 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:54.450 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:54.453Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:54.453Z [INFO] agent.router.manager: shutting down -45:54.453 [INFO] TestAgent: consul server down -45:54.453 [INFO] TestAgent: shutdown complete -45:54.453 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=tcp -45:54.453 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=udp -45:54.453 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21247 network=tcp -45:54.954 [INFO] TestAgent: Waiting for endpoints to shut down -45:54.954 [INFO] TestAgent: Endpoints down ---- PASS: TestRegisterMonitor_good (0.88s) -2022-11-07T22:45:54.572Z [INFO] TestRegisterMonitor_heartbeat: registered Consul service: service=foo-proxy -2022-11-07T22:45:54.664Z [INFO] TestRegisterMonitor_heartbeat: stop request received, deregistering -2022-11-07T22:45:54.116Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.124Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.157 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:62ded285-aed3-b1c9-ee25-065eeaddb6aa Address:127.0.0.1:21244}]" -45:54.157 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= -45:54.158 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa.dc1 127.0.0.1 -45:54.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa 127.0.0.1 -2022-11-07T22:45:54.160Z [INFO] agent.router: Initializing LAN area manager -45:54.160 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa.dc1 area=wan -45:54.160 [INFO] TestAgent.server: Adding LAN server: server="Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa (Addr: tcp/127.0.0.1:21244) (DC: dc1)" -45:54.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp -45:54.160 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp -45:54.161 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp -45:54.161 [INFO] TestAgent: started state syncer -45:54.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp -45:54.216 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:54.216 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 -45:54.225 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:54.225 [DEBUG] TestAgent.server.raft: vote granted: from=62ded285-aed3-b1c9-ee25-065eeaddb6aa term=2 tally=1 -45:54.225 [INFO] TestAgent.server.raft: election won: tally=1 -45:54.225 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" -45:54.225 [INFO] TestAgent.server: cluster leadership acquired -45:54.225 [INFO] TestAgent.server: New leader elected: payload=Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa -45:54.231 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21244 -45:54.244 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:54.272 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:54.272 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:54.272 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:54.272 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:54.272 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:54.272 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa -45:54.272 [INFO] TestAgent.server: member joined, marking health alive: member=Node-62ded285-aed3-b1c9-ee25-065eeaddb6aa -45:54.278 [INFO] TestAgent.server: federation state anti-entropy synced -45:54.388 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:54.392 [INFO] TestAgent: Synced node info -45:54.554 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:39588 latency=113.409µs -45:54.554 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/foo-proxy?stale= from=127.0.0.1:39590 latency=318.866µs -45:54.555 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.555 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.555 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.555 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -45:54.564 [DEBUG] TestAgent: Node info in sync -45:54.571 [INFO] TestAgent: Synced service: service=foo-proxy -45:54.571 [DEBUG] TestAgent: Check in sync: check=foo-proxy-ttl -45:54.571 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:39590 latency=17.3014ms -45:54.579 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:39590 latency=181.243µs -45:54.580 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:39590 latency=166.794µs -45:54.581 [DEBUG] TestAgent: Check status updated: check=foo-proxy-ttl status=critical -45:54.581 [DEBUG] TestAgent: Node info in sync -45:54.581 [DEBUG] TestAgent: Service in sync: service=foo-proxy -45:54.586 [INFO] TestAgent: Synced check: check=foo-proxy-ttl -45:54.586 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/fail/foo-proxy-ttl?note= from=127.0.0.1:39590 latency=5.380686ms -45:54.586 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:39590 latency=139.788µs -45:54.612 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:39590 latency=178.194µs -45:54.638 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:39590 latency=128.943µs -45:54.638 [DEBUG] TestAgent: Check status updated: check=foo-proxy-ttl status=passing -45:54.639 [DEBUG] TestAgent: Node info in sync -45:54.639 [DEBUG] TestAgent: Service in sync: service=foo-proxy -45:54.642 [INFO] TestAgent: Synced check: check=foo-proxy-ttl -45:54.642 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/pass/foo-proxy-ttl?note= from=127.0.0.1:39590 latency=3.350551ms -45:54.664 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:39590 latency=168.677µs -45:54.664 [DEBUG] TestAgent: removed check: check=foo-proxy-ttl -45:54.664 [DEBUG] TestAgent: removed service: service=foo-proxy -45:54.664 [DEBUG] TestAgent: Node info in sync -45:54.670 [INFO] TestAgent: Deregistered service: service=foo-proxy -45:54.670 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/foo-proxy from=127.0.0.1:39590 latency=5.617824ms -45:54.670 [INFO] TestAgent: Requesting shutdown -45:54.670 [INFO] TestAgent.server: shutting down server -45:54.670 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:54.670 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:54.670 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:54.670 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:54.670 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:54.670 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:54.670 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:54.670 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:54.670 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:54.670 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:54.673 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:54.677Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:54.677Z [INFO] agent.router.manager: shutting down -45:54.677 [INFO] TestAgent: consul server down -45:54.677 [INFO] TestAgent: shutdown complete -45:54.677 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp -45:54.677 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp -45:54.677 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp -45:55.177 [INFO] TestAgent: Waiting for endpoints to shut down -45:55.177 [INFO] TestAgent: Endpoints down ---- PASS: TestRegisterMonitor_heartbeat (1.10s) -2022-11-07T22:45:54.898Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:54.908Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:54.929 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b04a2033-11d8-dc45-c365-c2b53af6ea97 Address:127.0.0.1:21258}]" -45:54.929 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21258 [Follower]" leader= -45:54.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b04a2033-11d8-dc45-c365-c2b53af6ea97.dc1 127.0.0.1 -45:54.930 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b04a2033-11d8-dc45-c365-c2b53af6ea97 127.0.0.1 -2022-11-07T22:45:54.930Z [INFO] agent.router: Initializing LAN area manager -45:54.930 [INFO] TestAgent.server: Adding LAN server: server="Node-b04a2033-11d8-dc45-c365-c2b53af6ea97 (Addr: tcp/127.0.0.1:21258) (DC: dc1)" -45:54.930 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b04a2033-11d8-dc45-c365-c2b53af6ea97.dc1 area=wan -45:54.930 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=udp -45:54.930 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=tcp -45:54.931 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21254 network=tcp -45:54.931 [INFO] TestAgent: started state syncer -45:54.931 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21259 network=tcp -45:54.997 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:54.997 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21258 [Candidate]" term=2 -45:55.005 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:55.005 [DEBUG] TestAgent.server.raft: vote granted: from=b04a2033-11d8-dc45-c365-c2b53af6ea97 term=2 tally=1 -45:55.005 [INFO] TestAgent.server.raft: election won: tally=1 -45:55.005 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21258 [Leader]" -45:55.005 [INFO] TestAgent.server: cluster leadership acquired -45:55.005 [INFO] TestAgent.server: New leader elected: payload=Node-b04a2033-11d8-dc45-c365-c2b53af6ea97 -45:55.011 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21258 -45:55.015 [INFO] TestAgent: Requesting shutdown -45:55.015 [INFO] TestAgent.server: shutting down server -45:55.015 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:55.015 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:55.019 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:55.022Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:55.022Z [INFO] agent.router.manager: shutting down -45:55.026 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" -45:55.026 [INFO] TestAgent: consul server down -45:55.026 [INFO] TestAgent: shutdown complete -45:55.026 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=tcp -45:55.026 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=udp -45:55.026 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21254 network=tcp -45:55.526 [INFO] TestAgent: Waiting for endpoints to shut down -45:55.526 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:34.471Z [INFO] TestRegisterMonitor_heartbeat: registered Consul service: service=foo-proxy +2021-10-05T16:32:34.572Z [INFO] TestRegisterMonitor_heartbeat: stop request received, deregistering +2021-10-05T16:32:34.126Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:34.158Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:34.220 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:61bfbfd7-ccd6-c09a-1244-07a2859ef66b Address:127.0.0.1:21244}]" +32:34.220 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= +32:34.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b.dc1 127.0.0.1 +32:34.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b 127.0.0.1 +2021-10-05T16:32:34.222Z [INFO] agent.router: Initializing LAN area manager +32:34.236 [INFO] TestAgent.server: Adding LAN server: server="Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +32:34.236 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b.dc1 area=wan +32:34.237 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp +32:34.270 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:34.270 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 +32:34.279 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:34.279 [DEBUG] TestAgent.server.raft: vote granted: from=61bfbfd7-ccd6-c09a-1244-07a2859ef66b term=2 tally=1 +32:34.279 [INFO] TestAgent.server.raft: election won: tally=1 +32:34.279 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" +32:34.279 [INFO] TestAgent.server: cluster leadership acquired +32:34.280 [INFO] TestAgent.server: New leader elected: payload=Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b +32:34.285 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21244 +32:34.295 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp +32:34.295 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp +32:34.295 [INFO] TestAgent: started state syncer +32:34.295 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:34.296 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp +32:34.312 [INFO] TestAgent: Synced node info +32:34.312 [DEBUG] TestAgent: Node info in sync +32:34.324 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:34.324 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:34.324 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:34.324 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:34.324 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:34.324 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b +32:34.324 [INFO] TestAgent.server: member joined, marking health alive: member=Node-61bfbfd7-ccd6-c09a-1244-07a2859ef66b +32:34.366 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/foo-proxy?stale= from=127.0.0.1:36844 latency=15.964475ms +32:34.371 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:36842 latency=1.300881ms +32:34.378 [INFO] TestAgent.server: federation state anti-entropy synced +32:34.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.421 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.422 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.422 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.424 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:36842 latency=266.283µs +32:34.430 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36842 latency=883.289µs +32:34.431 [DEBUG] TestAgent: Node info in sync +32:34.431 [DEBUG] TestAgent: Check status updated: check=foo-proxy-ttl status=critical +32:34.471 [INFO] TestAgent: Synced service: service=foo-proxy +32:34.471 [DEBUG] TestAgent: Check in sync: check=foo-proxy-ttl +32:34.471 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:36844 latency=93.086226ms +32:34.471 [DEBUG] TestAgent: Node info in sync +32:34.471 [DEBUG] TestAgent: Service in sync: service=foo-proxy +32:34.476 [INFO] TestAgent: Synced check: check=foo-proxy-ttl +32:34.476 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/fail/foo-proxy-ttl?note= from=127.0.0.1:36842 latency=44.640868ms +32:34.477 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36842 latency=285.367µs +32:34.503 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36842 latency=183.651µs +32:34.537 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36842 latency=6.003708ms +32:34.541 [DEBUG] TestAgent: Check status updated: check=foo-proxy-ttl status=passing +32:34.542 [DEBUG] TestAgent: Node info in sync +32:34.542 [DEBUG] TestAgent: Service in sync: service=foo-proxy +32:34.553 [INFO] TestAgent: Synced check: check=foo-proxy-ttl +32:34.553 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/check/pass/foo-proxy-ttl?note= from=127.0.0.1:36844 latency=11.904238ms +32:34.572 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:36844 latency=267.383µs +32:34.573 [DEBUG] TestAgent: removed check: check=foo-proxy-ttl +32:34.573 [DEBUG] TestAgent: removed service: service=foo-proxy +32:34.573 [DEBUG] TestAgent: Node info in sync +32:34.587 [INFO] TestAgent: Deregistered service: service=foo-proxy +32:34.587 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/foo-proxy from=127.0.0.1:36844 latency=14.795585ms +32:34.588 [INFO] TestAgent: Requesting shutdown +32:34.588 [INFO] TestAgent.server: shutting down server +32:34.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:34.588 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:34.588 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:34.588 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:34.588 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:34.588 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:34.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:34.588 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:34.588 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:34.588 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:34.596 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:34.608Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:34.608Z [INFO] agent.router.manager: shutting down +32:34.608 [INFO] TestAgent: consul server down +32:34.608 [INFO] TestAgent: shutdown complete +32:34.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp +32:34.608 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp +32:34.608 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp +32:35.108 [INFO] TestAgent: Waiting for endpoints to shut down +32:35.108 [INFO] TestAgent: Endpoints down +--- PASS: TestRegisterMonitor_heartbeat (1.09s) +2021-10-05T16:32:34.786Z [INFO] TestRegisterMonitor_good: registered Consul service: service=foo-proxy +2021-10-05T16:32:34.786Z [INFO] TestRegisterMonitor_good: stop request received, deregistering +2021-10-05T16:32:34.130Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:34.177Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:34.252 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4713e837-0231-c2f6-5fa8-2f874f857056 Address:127.0.0.1:21251}]" +32:34.252 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21251 [Follower]" leader= +32:34.300 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4713e837-0231-c2f6-5fa8-2f874f857056.dc1 127.0.0.1 +32:34.301 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4713e837-0231-c2f6-5fa8-2f874f857056 127.0.0.1 +2021-10-05T16:32:34.301Z [INFO] agent.router: Initializing LAN area manager +32:34.302 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=udp +32:34.302 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4713e837-0231-c2f6-5fa8-2f874f857056.dc1 area=wan +32:34.302 [INFO] TestAgent.server: Adding LAN server: server="Node-4713e837-0231-c2f6-5fa8-2f874f857056 (Addr: tcp/127.0.0.1:21251) (DC: dc1)" +32:34.302 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=tcp +32:34.302 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21247 network=tcp +32:34.302 [INFO] TestAgent: started state syncer +32:34.303 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21252 network=tcp +32:34.308 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:34.312 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21251 [Candidate]" term=2 +32:34.335 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:34.335 [DEBUG] TestAgent.server.raft: vote granted: from=4713e837-0231-c2f6-5fa8-2f874f857056 term=2 tally=1 +32:34.335 [INFO] TestAgent.server.raft: election won: tally=1 +32:34.335 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21251 [Leader]" +32:34.336 [INFO] TestAgent.server: cluster leadership acquired +32:34.336 [INFO] TestAgent.server: New leader elected: payload=Node-4713e837-0231-c2f6-5fa8-2f874f857056 +32:34.372 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21251 +32:34.441 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:34.501 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:34.501 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:34.501 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:34.501 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:34.501 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:34.501 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4713e837-0231-c2f6-5fa8-2f874f857056 +32:34.501 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4713e837-0231-c2f6-5fa8-2f874f857056 +32:34.514 [INFO] TestAgent.server: federation state anti-entropy synced +32:34.638 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:34.643 [INFO] TestAgent: Synced node info +32:34.749 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/service/foo-proxy?stale= from=127.0.0.1:33216 latency=563.416µs +32:34.749 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:33214 latency=85.905µs +32:34.752 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.752 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.752 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.752 [ERROR] TestAgent.proxycfg: watch error: service_id=foo-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +32:34.764 [DEBUG] TestAgent: Node info in sync +32:34.784 [INFO] TestAgent: Synced service: service=foo-proxy +32:34.784 [DEBUG] TestAgent: Check in sync: check=foo-proxy-ttl +32:34.784 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:33216 latency=34.261818ms +32:34.785 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:33214 latency=9.332787ms +32:34.789 [DEBUG] TestAgent: removed check: check=foo-proxy-ttl +32:34.789 [DEBUG] TestAgent: removed service: service=foo-proxy +32:34.789 [DEBUG] TestAgent: Node info in sync +32:34.792 [INFO] TestAgent: Deregistered service: service=foo-proxy +32:34.792 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/foo-proxy from=127.0.0.1:33214 latency=3.448483ms +32:34.793 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:33214 latency=142.246µs +32:34.796 [INFO] TestAgent: Requesting shutdown +32:34.796 [INFO] TestAgent.server: shutting down server +32:34.796 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:34.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:34.796 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:34.796 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:34.796 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:34.796 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:34.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:34.796 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:34.796 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:34.796 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:34.804 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:34.811Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:34.811Z [INFO] agent.router.manager: shutting down +32:34.811 [INFO] TestAgent: consul server down +32:34.811 [INFO] TestAgent: shutdown complete +32:34.811 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=tcp +32:34.811 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=udp +32:34.811 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21247 network=tcp +32:35.316 [INFO] TestAgent: Waiting for endpoints to shut down +32:35.316 [INFO] TestAgent: Endpoints down +--- PASS: TestRegisterMonitor_good (1.29s) +2021-10-05T16:32:34.978Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:34.993Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:35.048 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2860544c-c1ab-a129-15f3-9f3e13a85d76 Address:127.0.0.1:21258}]" +32:35.049 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2860544c-c1ab-a129-15f3-9f3e13a85d76.dc1 127.0.0.1 +32:35.049 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2860544c-c1ab-a129-15f3-9f3e13a85d76 127.0.0.1 +2021-10-05T16:32:35.049Z [INFO] agent.router: Initializing LAN area manager +32:35.050 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=udp +32:35.050 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21258 [Follower]" leader= +32:35.050 [INFO] TestAgent.server: Adding LAN server: server="Node-2860544c-c1ab-a129-15f3-9f3e13a85d76 (Addr: tcp/127.0.0.1:21258) (DC: dc1)" +32:35.050 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2860544c-c1ab-a129-15f3-9f3e13a85d76.dc1 area=wan +32:35.051 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=tcp +32:35.051 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21254 network=tcp +32:35.051 [INFO] TestAgent: started state syncer +32:35.052 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21259 network=tcp +32:35.114 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:35.114 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21258 [Candidate]" term=2 +32:35.158 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:35.158 [DEBUG] TestAgent.server.raft: vote granted: from=2860544c-c1ab-a129-15f3-9f3e13a85d76 term=2 tally=1 +32:35.158 [INFO] TestAgent.server.raft: election won: tally=1 +32:35.158 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21258 [Leader]" +32:35.158 [INFO] TestAgent.server: cluster leadership acquired +32:35.158 [INFO] TestAgent.server: New leader elected: payload=Node-2860544c-c1ab-a129-15f3-9f3e13a85d76 +32:35.167 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21258 +32:35.208 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:35.260 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:35.260 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:35.260 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:35.260 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:35.260 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:35.260 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2860544c-c1ab-a129-15f3-9f3e13a85d76 +32:35.260 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2860544c-c1ab-a129-15f3-9f3e13a85d76 +32:35.320 [INFO] TestAgent.server: federation state anti-entropy synced +32:35.335 [INFO] TestAgent: Requesting shutdown +32:35.338 [INFO] TestAgent.server: shutting down server +32:35.338 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:35.338 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:35.338 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:35.338 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:35.338 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:35.338 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:35.338 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:35.338 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:35.338 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:35.338 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:35.338 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:35.347 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:35.350Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:35.350Z [INFO] agent.router.manager: shutting down +32:35.350 [INFO] TestAgent: consul server down +32:35.350 [INFO] TestAgent: shutdown complete +32:35.350 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=tcp +32:35.350 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=udp +32:35.351 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21254 network=tcp +32:35.852 [INFO] TestAgent: Waiting for endpoints to shut down +32:35.852 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-service_flag_with_-service-addr -2022-11-07T22:45:55.539Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:55.549Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:55.573 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9b1a16f4-49a6-d008-0710-0e36111f8f46 Address:127.0.0.1:21265}]" -45:55.573 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21265 [Follower]" leader= -45:55.573 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9b1a16f4-49a6-d008-0710-0e36111f8f46.dc1 127.0.0.1 -45:55.574 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9b1a16f4-49a6-d008-0710-0e36111f8f46 127.0.0.1 -2022-11-07T22:45:55.574Z [INFO] agent.router: Initializing LAN area manager -45:55.574 [INFO] TestAgent.server: Adding LAN server: server="Node-9b1a16f4-49a6-d008-0710-0e36111f8f46 (Addr: tcp/127.0.0.1:21265) (DC: dc1)" -45:55.574 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9b1a16f4-49a6-d008-0710-0e36111f8f46.dc1 area=wan -45:55.574 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=udp -45:55.574 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=tcp -45:55.575 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21261 network=tcp -45:55.575 [INFO] TestAgent: started state syncer -45:55.575 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21266 network=tcp -45:55.641 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:55.641 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21265 [Candidate]" term=2 -45:55.650 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:55.650 [DEBUG] TestAgent.server.raft: vote granted: from=9b1a16f4-49a6-d008-0710-0e36111f8f46 term=2 tally=1 -45:55.650 [INFO] TestAgent.server.raft: election won: tally=1 -45:55.650 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21265 [Leader]" -45:55.650 [INFO] TestAgent.server: cluster leadership acquired -45:55.650 [INFO] TestAgent.server: New leader elected: payload=Node-9b1a16f4-49a6-d008-0710-0e36111f8f46 -45:55.655 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21265 -45:55.666 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:55.667 [INFO] TestAgent: Requesting shutdown -45:55.667 [INFO] TestAgent.server: shutting down server -45:55.667 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:55.667 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:55.671 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:55.676Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:55.676Z [INFO] agent.router.manager: shutting down -45:55.680 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -45:55.680 [INFO] TestAgent: consul server down -45:55.680 [INFO] TestAgent: shutdown complete -45:55.680 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=tcp -45:55.680 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=udp -45:55.680 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21261 network=tcp -45:56.180 [INFO] TestAgent: Waiting for endpoints to shut down -45:56.180 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:35.890Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:35.926Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:36.009 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:768009a7-4bac-b0e1-ba72-fdb1a760a6f3 Address:127.0.0.1:21265}]" +32:36.010 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-768009a7-4bac-b0e1-ba72-fdb1a760a6f3.dc1 127.0.0.1 +32:36.010 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21265 [Follower]" leader= +32:36.010 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-768009a7-4bac-b0e1-ba72-fdb1a760a6f3 127.0.0.1 +2021-10-05T16:32:36.010Z [INFO] agent.router: Initializing LAN area manager +32:36.011 [INFO] TestAgent.server: Adding LAN server: server="Node-768009a7-4bac-b0e1-ba72-fdb1a760a6f3 (Addr: tcp/127.0.0.1:21265) (DC: dc1)" +32:36.011 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-768009a7-4bac-b0e1-ba72-fdb1a760a6f3.dc1 area=wan +32:36.011 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=udp +32:36.011 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=tcp +32:36.012 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21261 network=tcp +32:36.012 [INFO] TestAgent: started state syncer +32:36.012 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21266 network=tcp +32:36.054 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:36.054 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21265 [Candidate]" term=2 +32:36.062 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:36.062 [DEBUG] TestAgent.server.raft: vote granted: from=768009a7-4bac-b0e1-ba72-fdb1a760a6f3 term=2 tally=1 +32:36.062 [INFO] TestAgent.server.raft: election won: tally=1 +32:36.062 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21265 [Leader]" +32:36.063 [INFO] TestAgent.server: cluster leadership acquired +32:36.063 [INFO] TestAgent.server: New leader elected: payload=Node-768009a7-4bac-b0e1-ba72-fdb1a760a6f3 +32:36.069 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21265 +32:36.070 [INFO] TestAgent: Requesting shutdown +32:36.070 [INFO] TestAgent.server: shutting down server +32:36.070 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:36.070 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:36.073 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:36.078Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:36.078Z [INFO] agent.router.manager: shutting down +32:36.080 [INFO] TestAgent: consul server down +32:36.080 [INFO] TestAgent: shutdown complete +32:36.080 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=tcp +32:36.080 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=udp +32:36.080 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21261 network=tcp +32:36.080 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +32:36.080 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:36.584 [INFO] TestAgent: Waiting for endpoints to shut down +32:36.584 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-service,_-service-addr,_-listen -2022-11-07T22:45:56.190Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:56.200Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:56.221 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c0cd65ea-721c-a677-16af-c325b14464e2 Address:127.0.0.1:21272}]" -45:56.221 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21272 [Follower]" leader= -45:56.221 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c0cd65ea-721c-a677-16af-c325b14464e2.dc1 127.0.0.1 -45:56.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c0cd65ea-721c-a677-16af-c325b14464e2 127.0.0.1 -2022-11-07T22:45:56.222Z [INFO] agent.router: Initializing LAN area manager -45:56.222 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c0cd65ea-721c-a677-16af-c325b14464e2.dc1 area=wan -45:56.222 [INFO] TestAgent.server: Adding LAN server: server="Node-c0cd65ea-721c-a677-16af-c325b14464e2 (Addr: tcp/127.0.0.1:21272) (DC: dc1)" -45:56.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=udp -45:56.222 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=tcp -45:56.223 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21268 network=tcp -45:56.223 [INFO] TestAgent: started state syncer -45:56.223 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21273 network=tcp -45:56.273 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:56.273 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21272 [Candidate]" term=2 -45:56.281 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:56.281 [DEBUG] TestAgent.server.raft: vote granted: from=c0cd65ea-721c-a677-16af-c325b14464e2 term=2 tally=1 -45:56.281 [INFO] TestAgent.server.raft: election won: tally=1 -45:56.281 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21272 [Leader]" -45:56.281 [INFO] TestAgent.server: cluster leadership acquired -45:56.282 [INFO] TestAgent.server: New leader elected: payload=Node-c0cd65ea-721c-a677-16af-c325b14464e2 -45:56.287 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21272 -45:56.298 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:56.315 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:56.315 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:56.315 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:56.315 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:56.315 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:56.315 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c0cd65ea-721c-a677-16af-c325b14464e2 -45:56.315 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c0cd65ea-721c-a677-16af-c325b14464e2 -45:56.319 [INFO] TestAgent.server: federation state anti-entropy synced -45:56.437 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:56.441 [INFO] TestAgent: Synced node info -45:56.447 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy -45:56.452 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars -45:56.457 [INFO] TestAgent: Synced service: service=no-sidecar -45:56.461 [INFO] TestAgent: Synced service: service=one-sidecar -45:56.469 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy -45:56.473 [INFO] TestAgent: Synced service: service=two-sidecars -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:56.473 [DEBUG] TestAgent: Node info in sync -45:56.473 [DEBUG] TestAgent: Service in sync: service=two-sidecars-sidecar-proxy -45:56.473 [DEBUG] TestAgent: Service in sync: service=other-sidecar-for-two-sidecars -45:56.473 [DEBUG] TestAgent: Service in sync: service=no-sidecar -45:56.473 [DEBUG] TestAgent: Service in sync: service=one-sidecar -45:56.473 [DEBUG] TestAgent: Service in sync: service=one-sidecar-sidecar-proxy -45:56.473 [DEBUG] TestAgent: Service in sync: service=two-sidecars -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:56.473 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:56.474 [INFO] TestAgent: Requesting shutdown -45:56.474 [INFO] TestAgent.server: shutting down server -45:56.474 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:56.474 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:56.474 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:56.474 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:56.474 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:56.474 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:56.474 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:56.474 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:56.474 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:56.474 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:56.477 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:56.481Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:56.481Z [INFO] agent.router.manager: shutting down -45:56.481 [INFO] TestAgent: consul server down -45:56.481 [INFO] TestAgent: shutdown complete -45:56.481 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=tcp -45:56.481 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=udp -45:56.481 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21268 network=tcp -45:56.981 [INFO] TestAgent: Waiting for endpoints to shut down -45:56.981 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:36.615Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:36.662Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:36.701 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:944c3ed4-0b35-bfd8-5393-d565ff65e3f9 Address:127.0.0.1:21272}]" +32:36.702 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9.dc1 127.0.0.1 +32:36.703 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9 127.0.0.1 +2021-10-05T16:32:36.703Z [INFO] agent.router: Initializing LAN area manager +32:36.703 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=udp +32:36.703 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21272 [Follower]" leader= +32:36.704 [INFO] TestAgent.server: Adding LAN server: server="Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9 (Addr: tcp/127.0.0.1:21272) (DC: dc1)" +32:36.704 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9.dc1 area=wan +32:36.704 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=tcp +32:36.704 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21268 network=tcp +32:36.704 [INFO] TestAgent: started state syncer +32:36.704 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21273 network=tcp +32:36.764 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:36.764 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21272 [Candidate]" term=2 +32:36.803 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:36.803 [DEBUG] TestAgent.server.raft: vote granted: from=944c3ed4-0b35-bfd8-5393-d565ff65e3f9 term=2 tally=1 +32:36.803 [INFO] TestAgent.server.raft: election won: tally=1 +32:36.803 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21272 [Leader]" +32:36.803 [INFO] TestAgent.server: cluster leadership acquired +32:36.803 [INFO] TestAgent.server: New leader elected: payload=Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9 +32:36.835 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21272 +32:36.855 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:36.884 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:36.884 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:36.884 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:36.884 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:36.884 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:36.884 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9 +32:36.884 [INFO] TestAgent.server: member joined, marking health alive: member=Node-944c3ed4-0b35-bfd8-5393-d565ff65e3f9 +32:36.932 [INFO] TestAgent.server: federation state anti-entropy synced +32:36.960 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:36.975 [INFO] TestAgent: Synced node info +32:36.977 [INFO] TestAgent: Requesting shutdown +32:36.992 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy +32:37.012 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars +32:37.027 [INFO] TestAgent: Synced service: service=no-sidecar +32:37.031 [INFO] TestAgent: Synced service: service=one-sidecar +32:37.043 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy +32:37.047 [INFO] TestAgent: Synced service: service=two-sidecars +32:37.047 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 +32:37.047 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 +32:37.047 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 +32:37.047 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 +32:37.047 [INFO] TestAgent.server: shutting down server +32:37.047 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:37.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:37.047 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:37.047 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:37.047 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:37.048 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:37.048 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:37.048 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:37.051 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +32:37.051 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:37.052 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:37.052Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:37.052Z [INFO] agent.router.manager: shutting down +32:37.052 [INFO] TestAgent: consul server down +32:37.052 [INFO] TestAgent: shutdown complete +32:37.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=tcp +32:37.052 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=udp +32:37.052 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21268 network=tcp +32:37.555 [INFO] TestAgent: Waiting for endpoints to shut down +32:37.555 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-sidecar-for,_no_sidecar -2022-11-07T22:45:56.996Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.005Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.025 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c533e6f7-c443-1870-099b-da4dbf4a89f3 Address:127.0.0.1:21279}]" -45:57.025 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21279 [Follower]" leader= -45:57.025 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c533e6f7-c443-1870-099b-da4dbf4a89f3.dc1 127.0.0.1 -45:57.025 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c533e6f7-c443-1870-099b-da4dbf4a89f3 127.0.0.1 -2022-11-07T22:45:57.026Z [INFO] agent.router: Initializing LAN area manager -45:57.026 [INFO] TestAgent.server: Adding LAN server: server="Node-c533e6f7-c443-1870-099b-da4dbf4a89f3 (Addr: tcp/127.0.0.1:21279) (DC: dc1)" -45:57.026 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c533e6f7-c443-1870-099b-da4dbf4a89f3.dc1 area=wan -45:57.026 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=udp -45:57.026 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=tcp -45:57.026 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21275 network=tcp -45:57.026 [INFO] TestAgent: started state syncer -45:57.026 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21280 network=tcp -45:57.071 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.071 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21279 [Candidate]" term=2 -45:57.080 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.080 [DEBUG] TestAgent.server.raft: vote granted: from=c533e6f7-c443-1870-099b-da4dbf4a89f3 term=2 tally=1 -45:57.080 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.080 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21279 [Leader]" -45:57.080 [INFO] TestAgent.server: cluster leadership acquired -45:57.080 [INFO] TestAgent.server: New leader elected: payload=Node-c533e6f7-c443-1870-099b-da4dbf4a89f3 -45:57.085 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21279 -45:57.097 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.115 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.115 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.115 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.115 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.115 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.115 [INFO] TestAgent: Synced node info -45:57.116 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c533e6f7-c443-1870-099b-da4dbf4a89f3 -45:57.116 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c533e6f7-c443-1870-099b-da4dbf4a89f3 -45:57.121 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.125 [INFO] TestAgent: Synced service: service=two-sidecars -45:57.130 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy -45:57.134 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars -45:57.138 [INFO] TestAgent: Synced service: service=no-sidecar -45:57.141 [INFO] TestAgent: Synced service: service=one-sidecar -45:57.148 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy -45:57.148 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:57.148 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:57.148 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:57.148 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:57.414 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:51386 latency=309.943µs -45:57.415 [INFO] TestAgent: Requesting shutdown -45:57.415 [INFO] TestAgent.server: shutting down server -45:57.415 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:57.415 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:57.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:57.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:57.415 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:57.415 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:57.415 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:57.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:57.415 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:57.421 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.433Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.433Z [INFO] agent.router.manager: shutting down -45:57.433 [INFO] TestAgent: consul server down -45:57.433 [INFO] TestAgent: shutdown complete -45:57.433 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=tcp -45:57.433 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=udp -45:57.433 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21275 network=tcp -45:57.933 [INFO] TestAgent: Waiting for endpoints to shut down -45:57.933 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:37.580Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:37.618Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:37.666 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:07ee413d-1369-2e97-c565-3d706a34587b Address:127.0.0.1:21279}]" +32:37.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-07ee413d-1369-2e97-c565-3d706a34587b.dc1 127.0.0.1 +32:37.667 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-07ee413d-1369-2e97-c565-3d706a34587b 127.0.0.1 +2021-10-05T16:32:37.667Z [INFO] agent.router: Initializing LAN area manager +32:37.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=udp +32:37.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21279 [Follower]" leader= +32:37.668 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-07ee413d-1369-2e97-c565-3d706a34587b.dc1 area=wan +32:37.668 [INFO] TestAgent.server: Adding LAN server: server="Node-07ee413d-1369-2e97-c565-3d706a34587b (Addr: tcp/127.0.0.1:21279) (DC: dc1)" +32:37.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=tcp +32:37.669 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21275 network=tcp +32:37.669 [INFO] TestAgent: started state syncer +32:37.669 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21280 network=tcp +32:37.728 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:37.728 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21279 [Candidate]" term=2 +32:37.758 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:37.758 [DEBUG] TestAgent.server.raft: vote granted: from=07ee413d-1369-2e97-c565-3d706a34587b term=2 tally=1 +32:37.758 [INFO] TestAgent.server.raft: election won: tally=1 +32:37.758 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21279 [Leader]" +32:37.760 [INFO] TestAgent.server: cluster leadership acquired +32:37.760 [INFO] TestAgent.server: New leader elected: payload=Node-07ee413d-1369-2e97-c565-3d706a34587b +32:37.766 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21279 +32:37.783 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:37.832 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:45228 latency=433.856µs +32:37.836 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:37.836 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:37.836 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:37.836 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:37.836 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:37.836 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-07ee413d-1369-2e97-c565-3d706a34587b +32:37.836 [INFO] TestAgent.server: member joined, marking health alive: member=Node-07ee413d-1369-2e97-c565-3d706a34587b +32:37.836 [INFO] TestAgent: Requesting shutdown +32:37.836 [INFO] TestAgent.server: shutting down server +32:37.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:37.836 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:37.836 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:37.836 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:37.836 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:37.836 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:37.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:37.838 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:37.838 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:37.838 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:37.841 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:37.843Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:37.843Z [INFO] agent.router.manager: shutting down +32:37.856 [INFO] TestAgent: consul server down +32:37.856 [INFO] TestAgent: shutdown complete +32:37.856 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=tcp +32:37.856 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=udp +32:37.856 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21275 network=tcp +32:37.856 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-07ee413d-1369-2e97-c565-3d706a34587b 127.0.0.1 21277 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:07ee413d-1369-2e97-c565-3d706a34587b port:21279 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:21278] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +32:38.359 [INFO] TestAgent: Waiting for endpoints to shut down +32:38.359 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-sidecar-for,_multiple_sidecars -2022-11-07T22:45:57.944Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.954Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.975 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e06c15cc-a277-cb73-d585-1c884c63c087 Address:127.0.0.1:21286}]" -45:57.977 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21286 [Follower]" leader= -45:57.978 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e06c15cc-a277-cb73-d585-1c884c63c087.dc1 127.0.0.1 -45:57.979 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e06c15cc-a277-cb73-d585-1c884c63c087 127.0.0.1 -2022-11-07T22:45:57.979Z [INFO] agent.router: Initializing LAN area manager -45:57.979 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e06c15cc-a277-cb73-d585-1c884c63c087.dc1 area=wan -45:57.979 [INFO] TestAgent.server: Adding LAN server: server="Node-e06c15cc-a277-cb73-d585-1c884c63c087 (Addr: tcp/127.0.0.1:21286) (DC: dc1)" -45:57.979 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=tcp -45:57.979 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=udp -45:57.980 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21282 network=tcp -45:57.980 [INFO] TestAgent: started state syncer -45:57.980 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21287 network=tcp -45:58.023 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:58.023 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21286 [Candidate]" term=2 -45:58.031 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:58.031 [DEBUG] TestAgent.server.raft: vote granted: from=e06c15cc-a277-cb73-d585-1c884c63c087 term=2 tally=1 -45:58.031 [INFO] TestAgent.server.raft: election won: tally=1 -45:58.031 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21286 [Leader]" -45:58.032 [INFO] TestAgent.server: cluster leadership acquired -45:58.032 [INFO] TestAgent.server: New leader elected: payload=Node-e06c15cc-a277-cb73-d585-1c884c63c087 -45:58.037 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21286 -45:58.048 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:58.071 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:58.071 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:58.071 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:58.071 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:58.071 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:58.071 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e06c15cc-a277-cb73-d585-1c884c63c087 -45:58.071 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e06c15cc-a277-cb73-d585-1c884c63c087 -45:58.073 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:39036 latency=334.743µs -45:58.074 [INFO] TestAgent: Requesting shutdown -45:58.074 [INFO] TestAgent.server: shutting down server -45:58.074 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:58.074 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:58.074 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:58.074 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:58.074 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:58.074 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:58.074 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:58.074 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:58.074 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:58.080 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -45:58.089 [INFO] TestAgent.server: federation state anti-entropy synced -45:58.089 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:45:58.089Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.089Z [INFO] agent.router.manager: shutting down -45:58.089 [INFO] TestAgent: consul server down -45:58.089 [INFO] TestAgent: shutdown complete -45:58.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=tcp -45:58.089 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=udp -45:58.089 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21282 network=tcp -45:58.590 [INFO] TestAgent: Waiting for endpoints to shut down -45:58.590 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:38.389Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:38.425Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:38.484 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88e3f93a-5dd8-5406-e313-f0f878feafe4 Address:127.0.0.1:21286}]" +32:38.485 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88e3f93a-5dd8-5406-e313-f0f878feafe4.dc1 127.0.0.1 +32:38.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88e3f93a-5dd8-5406-e313-f0f878feafe4 127.0.0.1 +2021-10-05T16:32:38.486Z [INFO] agent.router: Initializing LAN area manager +32:38.486 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=udp +32:38.486 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21286 [Follower]" leader= +32:38.486 [INFO] TestAgent.server: Adding LAN server: server="Node-88e3f93a-5dd8-5406-e313-f0f878feafe4 (Addr: tcp/127.0.0.1:21286) (DC: dc1)" +32:38.486 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88e3f93a-5dd8-5406-e313-f0f878feafe4.dc1 area=wan +32:38.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=tcp +32:38.487 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21282 network=tcp +32:38.487 [INFO] TestAgent: started state syncer +32:38.488 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21287 network=tcp +32:38.552 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:38.552 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21286 [Candidate]" term=2 +32:38.577 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:38.577 [DEBUG] TestAgent.server.raft: vote granted: from=88e3f93a-5dd8-5406-e313-f0f878feafe4 term=2 tally=1 +32:38.578 [INFO] TestAgent.server.raft: election won: tally=1 +32:38.578 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21286 [Leader]" +32:38.578 [INFO] TestAgent.server: cluster leadership acquired +32:38.578 [INFO] TestAgent.server: New leader elected: payload=Node-88e3f93a-5dd8-5406-e313-f0f878feafe4 +32:38.584 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21286 +32:38.597 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:38.633 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:38.633 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:38.633 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:38.633 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:38.633 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:38.633 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88e3f93a-5dd8-5406-e313-f0f878feafe4 +32:38.633 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88e3f93a-5dd8-5406-e313-f0f878feafe4 +32:38.650 [INFO] TestAgent.server: federation state anti-entropy synced +32:38.694 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:38.698 [INFO] TestAgent: Synced node info +32:38.716 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy +32:38.729 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars +32:38.736 [INFO] TestAgent: Synced service: service=no-sidecar +32:38.745 [INFO] TestAgent: Synced service: service=one-sidecar +32:38.764 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy +32:38.769 [INFO] TestAgent: Synced service: service=two-sidecars +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 +32:38.769 [DEBUG] TestAgent: Node info in sync +32:38.769 [DEBUG] TestAgent: Service in sync: service=no-sidecar +32:38.769 [DEBUG] TestAgent: Service in sync: service=one-sidecar +32:38.769 [DEBUG] TestAgent: Service in sync: service=one-sidecar-sidecar-proxy +32:38.769 [DEBUG] TestAgent: Service in sync: service=two-sidecars +32:38.769 [DEBUG] TestAgent: Service in sync: service=two-sidecars-sidecar-proxy +32:38.769 [DEBUG] TestAgent: Service in sync: service=other-sidecar-for-two-sidecars +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 +32:38.769 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 +32:38.913 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:57996 latency=757.741µs +32:38.914 [INFO] TestAgent: Requesting shutdown +32:38.914 [INFO] TestAgent.server: shutting down server +32:38.914 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:38.914 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:38.914 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:38.914 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:38.914 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:38.914 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:38.914 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:38.914 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:38.914 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:38.914 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:38.937 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:38.938Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:38.938Z [INFO] agent.router.manager: shutting down +32:38.938 [INFO] TestAgent: consul server down +32:38.938 [INFO] TestAgent: shutdown complete +32:38.938 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=tcp +32:38.938 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=udp +32:38.938 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21282 network=tcp +32:39.445 [INFO] TestAgent: Waiting for endpoints to shut down +32:39.445 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-sidecar-for,_non-existent -2022-11-07T22:45:58.602Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:58.612Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:58.634 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3c2b0ca1-d8fa-c177-1bae-88d156c8d00d Address:127.0.0.1:21293}]" -45:58.634 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21293 [Follower]" leader= -45:58.634 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d.dc1 127.0.0.1 -45:58.635 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d 127.0.0.1 -2022-11-07T22:45:58.635Z [INFO] agent.router: Initializing LAN area manager -45:58.635 [INFO] TestAgent.server: Adding LAN server: server="Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d (Addr: tcp/127.0.0.1:21293) (DC: dc1)" -45:58.635 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d.dc1 area=wan -45:58.635 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21288 network=udp -45:58.635 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21288 network=tcp -45:58.635 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21289 network=tcp -45:58.635 [INFO] TestAgent: started state syncer -45:58.635 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21294 network=tcp -45:58.679 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:58.679 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21293 [Candidate]" term=2 -45:58.688 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:58.688 [DEBUG] TestAgent.server.raft: vote granted: from=3c2b0ca1-d8fa-c177-1bae-88d156c8d00d term=2 tally=1 -45:58.688 [INFO] TestAgent.server.raft: election won: tally=1 -45:58.688 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21293 [Leader]" -45:58.688 [INFO] TestAgent.server: cluster leadership acquired -45:58.688 [INFO] TestAgent.server: New leader elected: payload=Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d -45:58.694 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21293 -45:58.705 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:58.725 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:58.725 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:58.725 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:58.725 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:58.725 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:58.725 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d -45:58.725 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3c2b0ca1-d8fa-c177-1bae-88d156c8d00d -45:58.730 [INFO] TestAgent.server: federation state anti-entropy synced -45:58.770 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:43694 latency=311.192µs -45:58.771 [INFO] TestAgent: Requesting shutdown -45:58.771 [INFO] TestAgent.server: shutting down server -45:58.771 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:58.771 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:58.771 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:58.771 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:58.771 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:58.771 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:58.771 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:58.771 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -45:58.771 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:58.771 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:58.771 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:58.774 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:58.778Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.778Z [INFO] agent.router.manager: shutting down -45:58.778 [INFO] TestAgent: consul server down -45:58.778 [INFO] TestAgent: shutdown complete -45:58.778 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21288 network=tcp -45:58.778 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21288 network=udp -45:58.778 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21289 network=tcp -45:59.278 [INFO] TestAgent: Waiting for endpoints to shut down -45:59.278 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:39.475Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.491Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.553 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f6524e07-6891-5d0a-8d57-72c2597009c9 Address:127.0.0.1:21293}]" +32:39.554 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f6524e07-6891-5d0a-8d57-72c2597009c9.dc1 127.0.0.1 +32:39.555 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f6524e07-6891-5d0a-8d57-72c2597009c9 127.0.0.1 +2021-10-05T16:32:39.555Z [INFO] agent.router: Initializing LAN area manager +32:39.563 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21288 network=tcp +32:39.563 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21293 [Follower]" leader= +32:39.563 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f6524e07-6891-5d0a-8d57-72c2597009c9.dc1 area=wan +32:39.564 [INFO] TestAgent.server: Adding LAN server: server="Node-f6524e07-6891-5d0a-8d57-72c2597009c9 (Addr: tcp/127.0.0.1:21293) (DC: dc1)" +32:39.567 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21288 network=udp +32:39.568 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21289 network=tcp +32:39.568 [INFO] TestAgent: started state syncer +32:39.568 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21294 network=tcp +32:39.608 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:39.608 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21293 [Candidate]" term=2 +32:39.622 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:39.622 [DEBUG] TestAgent.server.raft: vote granted: from=f6524e07-6891-5d0a-8d57-72c2597009c9 term=2 tally=1 +32:39.622 [INFO] TestAgent.server.raft: election won: tally=1 +32:39.622 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21293 [Leader]" +32:39.624 [INFO] TestAgent.server: cluster leadership acquired +32:39.624 [INFO] TestAgent.server: New leader elected: payload=Node-f6524e07-6891-5d0a-8d57-72c2597009c9 +32:39.630 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21293 +32:39.646 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:39.697 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:39.697 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:39.697 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:39.697 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:39.697 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:39.697 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f6524e07-6891-5d0a-8d57-72c2597009c9 +32:39.697 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f6524e07-6891-5d0a-8d57-72c2597009c9 +32:39.719 [INFO] TestAgent.server: federation state anti-entropy synced +32:39.829 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:49346 latency=403.013µs +32:39.830 [INFO] TestAgent: Requesting shutdown +32:39.830 [INFO] TestAgent.server: shutting down server +32:39.830 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:39.830 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:39.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:39.830 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:39.830 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:39.830 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:39.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:39.830 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:39.830 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:39.830 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:39.830 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:39.840 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:39.848Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:39.848Z [INFO] agent.router.manager: shutting down +32:39.848 [INFO] TestAgent: consul server down +32:39.848 [INFO] TestAgent: shutdown complete +32:39.848 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21288 network=tcp +32:39.848 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21288 network=udp +32:39.848 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21289 network=tcp +32:40.352 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.352 [INFO] TestAgent: Endpoints down === RUN TestCommandConfigWatcher/-sidecar-for,_one_sidecar -2022-11-07T22:45:59.297Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:59.311Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:59.341 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:864b57fd-ffd2-5bad-2d47-f64a785a2bf4 Address:127.0.0.1:21300}]" -45:59.341 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21300 [Follower]" leader= -45:59.342 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4.dc1 127.0.0.1 -45:59.343 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4 127.0.0.1 -2022-11-07T22:45:59.343Z [INFO] agent.router: Initializing LAN area manager -45:59.347 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4.dc1 area=wan -45:59.347 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21295 network=tcp -45:59.347 [INFO] TestAgent.server: Adding LAN server: server="Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4 (Addr: tcp/127.0.0.1:21300) (DC: dc1)" -45:59.347 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21295 network=udp -45:59.348 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21296 network=tcp -45:59.348 [INFO] TestAgent: started state syncer -45:59.348 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21301 network=tcp -45:59.380 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:59.380 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21300 [Candidate]" term=2 -45:59.387 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:59.387 [DEBUG] TestAgent.server.raft: vote granted: from=864b57fd-ffd2-5bad-2d47-f64a785a2bf4 term=2 tally=1 -45:59.387 [INFO] TestAgent.server.raft: election won: tally=1 -45:59.387 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21300 [Leader]" -45:59.388 [INFO] TestAgent.server: cluster leadership acquired -45:59.389 [INFO] TestAgent.server: New leader elected: payload=Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4 -45:59.392 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21300 -45:59.403 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:59.425 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:59.425 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:59.425 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:59.425 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:59.425 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:59.425 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4 -45:59.425 [INFO] TestAgent.server: member joined, marking health alive: member=Node-864b57fd-ffd2-5bad-2d47-f64a785a2bf4 -45:59.443 [INFO] TestAgent.server: federation state anti-entropy synced -45:59.489 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:59.498 [INFO] TestAgent: Synced node info -45:59.502 [INFO] TestAgent: Synced service: service=no-sidecar -45:59.507 [INFO] TestAgent: Synced service: service=one-sidecar -45:59.516 [INFO] TestAgent: Synced service: service=one-sidecar-sidecar-proxy -45:59.520 [INFO] TestAgent: Synced service: service=two-sidecars -45:59.532 [INFO] TestAgent: Synced service: service=two-sidecars-sidecar-proxy -45:59.550 [INFO] TestAgent: Synced service: service=other-sidecar-for-two-sidecars -45:59.550 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:2 -45:59.550 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:1 -45:59.550 [DEBUG] TestAgent: Check in sync: check=service:one-sidecar-sidecar-proxy:2 -45:59.550 [DEBUG] TestAgent: Check in sync: check=service:two-sidecars-sidecar-proxy:1 -45:59.551 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:47156 latency=51.522745ms -45:59.552 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy from=127.0.0.1:47158 latency=179.431µs -45:59.553 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy from=127.0.0.1:47156 latency=440.606µs -45:59.553 [INFO] TestAgent: Requesting shutdown -45:59.553 [INFO] TestAgent.server: shutting down server -45:59.553 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:59.553 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:59.553 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:59.553 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:59.553 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:59.553 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:59.553 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:59.553 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:59.553 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:59.553 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:59.556 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:59.561Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:59.561Z [INFO] agent.router.manager: shutting down -45:59.562 [INFO] TestAgent: consul server down -45:59.562 [INFO] TestAgent: shutdown complete -45:59.562 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21295 network=tcp -45:59.562 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21295 network=udp -45:59.562 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21296 network=tcp -45:59.562 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy?hash=3bafbcac3734b91 from=127.0.0.1:47158 latency=9.464417ms -46:00.062 [INFO] TestAgent: Waiting for endpoints to shut down -46:00.062 [INFO] TestAgent: Endpoints down ---- PASS: TestCommandConfigWatcher (5.99s) - --- PASS: TestCommandConfigWatcher/-service_flag_only (0.81s) - --- PASS: TestCommandConfigWatcher/-service_flag_with_upstreams (0.64s) - --- PASS: TestCommandConfigWatcher/-service_flag_with_-service-addr (0.65s) - --- PASS: TestCommandConfigWatcher/-service,_-service-addr,_-listen (0.80s) - --- PASS: TestCommandConfigWatcher/-sidecar-for,_no_sidecar (0.95s) - --- PASS: TestCommandConfigWatcher/-sidecar-for,_multiple_sidecars (0.66s) - --- PASS: TestCommandConfigWatcher/-sidecar-for,_non-existent (0.69s) - --- PASS: TestCommandConfigWatcher/-sidecar-for,_one_sidecar (0.78s) +2021-10-05T16:32:40.382Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:40.400Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:40.457 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:96b873bb-be58-5f95-14c1-987f60ecc582 Address:127.0.0.1:21300}]" +32:40.458 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-96b873bb-be58-5f95-14c1-987f60ecc582.dc1 127.0.0.1 +32:40.459 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21300 [Follower]" leader= +32:40.460 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-96b873bb-be58-5f95-14c1-987f60ecc582 127.0.0.1 +2021-10-05T16:32:40.460Z [INFO] agent.router: Initializing LAN area manager +32:40.460 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21295 network=udp +32:40.460 [INFO] TestAgent.server: Adding LAN server: server="Node-96b873bb-be58-5f95-14c1-987f60ecc582 (Addr: tcp/127.0.0.1:21300) (DC: dc1)" +32:40.461 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-96b873bb-be58-5f95-14c1-987f60ecc582.dc1 area=wan +32:40.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21295 network=tcp +32:40.462 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21296 network=tcp +32:40.462 [INFO] TestAgent: started state syncer +32:40.462 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21301 network=tcp +32:40.496 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.496 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21300 [Candidate]" term=2 +32:40.513 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.513 [DEBUG] TestAgent.server.raft: vote granted: from=96b873bb-be58-5f95-14c1-987f60ecc582 term=2 tally=1 +32:40.513 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.513 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21300 [Leader]" +32:40.514 [INFO] TestAgent.server: cluster leadership acquired +32:40.514 [INFO] TestAgent.server: New leader elected: payload=Node-96b873bb-be58-5f95-14c1-987f60ecc582 +32:40.520 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:50046 latency=1.334304ms +32:40.527 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21300 +32:40.535 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy from=127.0.0.1:50048 latency=1.537971ms +32:40.536 [INFO] TestAgent: Requesting shutdown +32:40.536 [INFO] TestAgent.server: shutting down server +32:40.536 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:40.537 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:40.543 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +32:40.547 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy from=127.0.0.1:50046 latency=1.167826ms +2021-10-05T16:32:40.547Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:40.548Z [INFO] agent.router.manager: shutting down +32:40.552 [INFO] TestAgent: consul server down +32:40.552 [INFO] TestAgent: shutdown complete +32:40.552 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21295 network=tcp +32:40.552 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21295 network=udp +32:40.552 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21296 network=tcp +32:40.552 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +32:40.552 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:40.552 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/service/one-sidecar-sidecar-proxy?hash=3bafbcac3734b91 from=127.0.0.1:50048 latency=14.422618ms +32:41.056 [INFO] TestAgent: Waiting for endpoints to shut down +32:41.056 [INFO] TestAgent: Endpoints down +--- PASS: TestCommandConfigWatcher (7.03s) + --- PASS: TestCommandConfigWatcher/-service_flag_only (0.92s) + --- PASS: TestCommandConfigWatcher/-service_flag_with_upstreams (0.91s) + --- PASS: TestCommandConfigWatcher/-service_flag_with_-service-addr (0.73s) + --- PASS: TestCommandConfigWatcher/-service,_-service-addr,_-listen (0.97s) + --- PASS: TestCommandConfigWatcher/-sidecar-for,_no_sidecar (0.80s) + --- PASS: TestCommandConfigWatcher/-sidecar-for,_multiple_sidecars (1.09s) + --- PASS: TestCommandConfigWatcher/-sidecar-for,_non-existent (0.91s) + --- PASS: TestCommandConfigWatcher/-sidecar-for,_one_sidecar (0.70s) PASS -ok github.com/hashicorp/consul/command/connect/proxy 6.021s +ok github.com/hashicorp/consul/command/connect/proxy 7.117s === RUN TestEventCommand_noTabs === PAUSE TestEventCommand_noTabs === RUN TestEventCommand === PAUSE TestEventCommand === CONT TestEventCommand_noTabs +--- PASS: TestEventCommand_noTabs (0.00s) === CONT TestEventCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestEventCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:45:55.609Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:55.619Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:55.647 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7818bf5b-53a1-df7b-850f-955e6ca1e633 Address:127.0.0.1:10006}]" -45:55.647 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -45:55.647 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7818bf5b-53a1-df7b-850f-955e6ca1e633.dc1 127.0.0.1 -45:55.648 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7818bf5b-53a1-df7b-850f-955e6ca1e633 127.0.0.1 -2022-11-07T22:45:55.648Z [INFO] agent.router: Initializing LAN area manager -45:55.648 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7818bf5b-53a1-df7b-850f-955e6ca1e633.dc1 area=wan -45:55.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -45:55.648 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -45:55.648 [INFO] TestAgent.server: Adding LAN server: server="Node-7818bf5b-53a1-df7b-850f-955e6ca1e633 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -45:55.649 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -45:55.649 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -45:55.649 [INFO] TestAgent: started state syncer -45:55.688 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:55.688 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -45:55.696 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:55.696 [DEBUG] TestAgent.server.raft: vote granted: from=7818bf5b-53a1-df7b-850f-955e6ca1e633 term=2 tally=1 -45:55.696 [INFO] TestAgent.server.raft: election won: tally=1 -45:55.696 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -45:55.697 [INFO] TestAgent.server: cluster leadership acquired -45:55.697 [INFO] TestAgent.server: New leader elected: payload=Node-7818bf5b-53a1-df7b-850f-955e6ca1e633 -45:55.702 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10006 -45:55.715 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:55.737 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:55.737 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:55.737 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:55.737 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:55.737 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:55.737 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7818bf5b-53a1-df7b-850f-955e6ca1e633 -45:55.737 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7818bf5b-53a1-df7b-850f-955e6ca1e633 -45:55.748 [INFO] TestAgent.server: federation state anti-entropy synced -45:56.001 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:56.005 [INFO] TestAgent: Synced node info -45:56.066 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:37824 latency=1.101104ms -45:56.069 [DEBUG] TestAgent.server: User event: event=cmd -45:56.069 [DEBUG] TestAgent: new event: event_name=cmd event_id=882b524e-8796-6cd7-dad2-e80c3ef03169 -45:56.069 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/cmd from=127.0.0.1:37824 latency=333.989µs -45:56.071 [INFO] TestAgent: Requesting shutdown -45:56.071 [INFO] TestAgent.server: shutting down server -45:56.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:56.071 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:56.071 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:56.071 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:56.071 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:56.071 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:56.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:56.071 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:56.071 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:56.071 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:56.075 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:56.079Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:56.079Z [INFO] agent.router.manager: shutting down -45:56.079 [INFO] TestAgent: consul server down -45:56.079 [INFO] TestAgent: shutdown complete -45:56.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -45:56.079 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -45:56.079 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -45:56.579 [INFO] TestAgent: Waiting for endpoints to shut down -45:56.579 [INFO] TestAgent: Endpoints down ---- PASS: TestEventCommand (1.00s) +2021-10-05T16:32:37.117Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:37.152Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:37.177 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:edf8ca1a-814e-8181-f007-63501671d7ca Address:127.0.0.1:11027}]" +32:37.177 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:11027 [Follower]" leader= +32:37.178 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-edf8ca1a-814e-8181-f007-63501671d7ca.dc1 127.0.0.1 +32:37.179 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-edf8ca1a-814e-8181-f007-63501671d7ca 127.0.0.1 +2021-10-05T16:32:37.179Z [INFO] agent.router: Initializing LAN area manager +32:37.179 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=udp +32:37.180 [INFO] TestAgent.server: Adding LAN server: server="Node-edf8ca1a-814e-8181-f007-63501671d7ca (Addr: tcp/127.0.0.1:11027) (DC: dc1)" +32:37.180 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-edf8ca1a-814e-8181-f007-63501671d7ca.dc1 area=wan +32:37.180 [INFO] TestAgent: Started DNS server: address=127.0.0.1:11022 network=tcp +32:37.180 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:11023 network=tcp +32:37.180 [INFO] TestAgent: started state syncer +32:37.180 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:11028 network=tcp +32:37.237 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:37.237 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:11027 [Candidate]" term=2 +32:37.245 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:37.245 [DEBUG] TestAgent.server.raft: vote granted: from=edf8ca1a-814e-8181-f007-63501671d7ca term=2 tally=1 +32:37.245 [INFO] TestAgent.server.raft: election won: tally=1 +32:37.245 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:11027 [Leader]" +32:37.246 [INFO] TestAgent.server: cluster leadership acquired +32:37.246 [INFO] TestAgent.server: New leader elected: payload=Node-edf8ca1a-814e-8181-f007-63501671d7ca +32:37.251 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:11027 +32:37.261 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:37.287 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:37.287 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:37.287 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:37.287 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:37.287 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:37.287 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-edf8ca1a-814e-8181-f007-63501671d7ca +32:37.287 [INFO] TestAgent.server: member joined, marking health alive: member=Node-edf8ca1a-814e-8181-f007-63501671d7ca +32:37.316 [INFO] TestAgent.server: federation state anti-entropy synced +32:37.525 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:37.529 [INFO] TestAgent: Synced node info +32:37.529 [DEBUG] TestAgent: Node info in sync +32:37.542 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:33536 latency=2.52724ms +32:37.561 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/cmd from=127.0.0.1:33536 latency=1.470746ms +32:37.566 [DEBUG] TestAgent.server: User event: event=cmd +32:37.566 [DEBUG] TestAgent: new event: event_name=cmd event_id=f4c44b65-e2f9-7c5d-3e4f-38224525d497 +32:37.566 [INFO] TestAgent: Requesting shutdown +32:37.566 [INFO] TestAgent.server: shutting down server +32:37.566 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:37.566 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:37.566 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:37.566 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:37.567 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:37.567 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:37.567 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:37.567 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:37.567 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:37.567 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:37.571 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:37.575Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:37.575Z [INFO] agent.router.manager: shutting down +32:37.575 [INFO] TestAgent: consul server down +32:37.575 [INFO] TestAgent: shutdown complete +32:37.575 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=tcp +32:37.575 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:11022 network=udp +32:37.575 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:11023 network=tcp +32:38.076 [INFO] TestAgent: Waiting for endpoints to shut down +32:38.076 [INFO] TestAgent: Endpoints down +--- PASS: TestEventCommand (1.01s) PASS -ok github.com/hashicorp/consul/command/event 1.035s +ok github.com/hashicorp/consul/command/event 1.287s === RUN TestExecCommand_noTabs === PAUSE TestExecCommand_noTabs === RUN TestExecCommand @@ -83938,579 +84213,572 @@ === RUN TestExecCommand_StreamResults === PAUSE TestExecCommand_StreamResults === CONT TestExecCommand_noTabs -=== CONT TestExecCommand_Sessions -=== CONT TestExecCommand_StreamResults -=== CONT TestExecCommand_UploadDestroy -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestExecCommand_CrossDC -=== CONT TestExecCommand_Validate === CONT TestExecCommand_Sessions_Foreign ---- PASS: TestExecCommand_Validate (0.00s) +=== CONT TestExecCommand_CrossDC +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestExecCommand_StreamResults +=== CONT TestExecCommand_Sessions === CONT TestExecCommand_NoShell +=== CONT TestExecCommand_Validate +=== CONT TestExecCommand_UploadDestroy === CONT TestExecCommand +--- PASS: TestExecCommand_noTabs (0.01s) +--- PASS: TestExecCommand_Validate (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestExecCommand_noTabs (0.00s) -2022-11-07T22:45:57.385Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.405Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.486 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eb88119b-925a-edd3-df7f-707d60dc1979 Address:127.0.0.1:18188}]" -45:57.486 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18188 [Follower]" leader= -45:57.487 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eb88119b-925a-edd3-df7f-707d60dc1979.dc1 127.0.0.1 -45:57.487 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eb88119b-925a-edd3-df7f-707d60dc1979 127.0.0.1 -2022-11-07T22:45:57.488Z [INFO] agent.router: Initializing LAN area manager -45:57.488 [INFO] TestAgent.server: Adding LAN server: server="Node-eb88119b-925a-edd3-df7f-707d60dc1979 (Addr: tcp/127.0.0.1:18188) (DC: dc1)" -45:57.488 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eb88119b-925a-edd3-df7f-707d60dc1979.dc1 area=wan -45:57.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18183 network=tcp -45:57.488 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18183 network=udp -45:57.488 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18184 network=tcp -45:57.488 [INFO] TestAgent: started state syncer -45:57.488 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18189 network=tcp -45:57.533 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.533 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18188 [Candidate]" term=2 -45:57.552 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.552 [DEBUG] TestAgent.server.raft: vote granted: from=eb88119b-925a-edd3-df7f-707d60dc1979 term=2 tally=1 -45:57.552 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.552 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18188 [Leader]" -45:57.553 [INFO] TestAgent.server: cluster leadership acquired -45:57.553 [INFO] TestAgent.server: New leader elected: payload=Node-eb88119b-925a-edd3-df7f-707d60dc1979 -45:57.562 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18188 -45:57.582 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.618 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.618 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.618 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.618 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.618 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.618 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eb88119b-925a-edd3-df7f-707d60dc1979 -45:57.618 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eb88119b-925a-edd3-df7f-707d60dc1979 -45:57.624 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.639 [INFO] TestAgent: Synced node info -45:57.639 [DEBUG] TestAgent: Node info in sync -45:57.673 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:46048 latency=3.253879ms -45:57.677 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/5a9680a7-8d2d-2363-706e-cb7db1c3da70/job?acquire=5a9680a7-8d2d-2363-706e-cb7db1c3da70 from=127.0.0.1:46048 latency=3.191821ms -45:57.678 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/5a9680a7-8d2d-2363-706e-cb7db1c3da70/job from=127.0.0.1:46050 latency=258.108µs -45:57.681 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/5a9680a7-8d2d-2363-706e-cb7db1c3da70?recurse= from=127.0.0.1:46048 latency=2.999056ms -45:57.681 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/5a9680a7-8d2d-2363-706e-cb7db1c3da70/job from=127.0.0.1:46052 latency=43.858µs -45:57.682 [INFO] TestAgent: Requesting shutdown -45:57.682 [INFO] TestAgent.server: shutting down server -45:57.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:57.682 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:57.682 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:57.682 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:57.682 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.682 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:57.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:57.682 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:57.682 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:57.682 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:57.685 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.689Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.689Z [INFO] agent.router.manager: shutting down -45:57.689 [INFO] TestAgent: consul server down -45:57.689 [INFO] TestAgent: shutdown complete -45:57.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18183 network=tcp -45:57.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18183 network=udp -45:57.689 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18184 network=tcp -45:58.189 [INFO] TestAgent: Waiting for endpoints to shut down -45:58.189 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_UploadDestroy (0.85s) -2022-11-07T22:45:57.386Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.401Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.442 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:31e95af8-8f7e-700c-7f15-ea6e4b81622f Address:127.0.0.1:18174}]" -45:57.442 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -45:57.442 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f.dc1 127.0.0.1 -45:57.486 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f 127.0.0.1 -2022-11-07T22:45:57.487Z [INFO] agent.router: Initializing LAN area manager -45:57.487 [INFO] TestAgent.server: Adding LAN server: server="Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -45:57.487 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f.dc1 area=wan -45:57.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -45:57.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -45:57.488 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -45:57.488 [INFO] TestAgent: started state syncer -45:57.488 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -45:57.499 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.500 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -45:57.508 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.508 [DEBUG] TestAgent.server.raft: vote granted: from=31e95af8-8f7e-700c-7f15-ea6e4b81622f term=2 tally=1 -45:57.508 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.509 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -45:57.509 [INFO] TestAgent.server: cluster leadership acquired -45:57.509 [INFO] TestAgent.server: New leader elected: payload=Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f -45:57.516 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18174 -45:57.546 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.565 [INFO] TestAgent: Synced node info -45:57.580 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.580 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.580 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.580 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.580 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.580 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f -45:57.580 [INFO] TestAgent.server: member joined, marking health alive: member=Node-31e95af8-8f7e-700c-7f15-ea6e4b81622f -45:57.587 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.727 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:48258 latency=3.13487ms -45:57.728 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/13e32126-c835-31c6-ab33-cea38797b4d7 from=127.0.0.1:48260 latency=145.371µs -45:57.736 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/13e32126-c835-31c6-ab33-cea38797b4d7 from=127.0.0.1:48258 latency=4.783145ms -45:57.737 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/13e32126-c835-31c6-ab33-cea38797b4d7 from=127.0.0.1:48262 latency=109.852µs -45:57.737 [INFO] TestAgent: Requesting shutdown -45:57.737 [INFO] TestAgent.server: shutting down server -45:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:57.737 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:57.737 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:57.742 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.745Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.745Z [INFO] agent.router.manager: shutting down -45:57.745 [INFO] TestAgent: consul server down -45:57.745 [INFO] TestAgent: shutdown complete -45:57.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -45:57.745 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -45:57.745 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -45:58.245 [INFO] TestAgent: Waiting for endpoints to shut down -45:58.245 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_Sessions (0.91s) -2022-11-07T22:45:57.386Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.400Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.436 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:04aaafc2-df60-a159-359e-9af6baa84e44 Address:127.0.0.1:18181}]" -45:57.436 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18181 [Follower]" leader= -45:57.438 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-04aaafc2-df60-a159-359e-9af6baa84e44.dc1 127.0.0.1 -45:57.438 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-04aaafc2-df60-a159-359e-9af6baa84e44 127.0.0.1 -2022-11-07T22:45:57.439Z [INFO] agent.router: Initializing LAN area manager -45:57.439 [INFO] TestAgent.server: Adding LAN server: server="Node-04aaafc2-df60-a159-359e-9af6baa84e44 (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -45:57.439 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-04aaafc2-df60-a159-359e-9af6baa84e44.dc1 area=wan -45:57.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=udp -45:57.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=tcp -45:57.440 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18177 network=tcp -45:57.440 [INFO] TestAgent: started state syncer -45:57.440 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18182 network=tcp -45:57.485 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.485 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18181 [Candidate]" term=2 -45:57.497 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.497 [DEBUG] TestAgent.server.raft: vote granted: from=04aaafc2-df60-a159-359e-9af6baa84e44 term=2 tally=1 -45:57.497 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.497 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18181 [Leader]" -45:57.497 [INFO] TestAgent.server: cluster leadership acquired -45:57.497 [INFO] TestAgent.server: New leader elected: payload=Node-04aaafc2-df60-a159-359e-9af6baa84e44 -45:57.502 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18181 -45:57.518 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.557 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.557 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.557 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.557 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.557 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.557 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-04aaafc2-df60-a159-359e-9af6baa84e44 -45:57.557 [INFO] TestAgent.server: member joined, marking health alive: member=Node-04aaafc2-df60-a159-359e-9af6baa84e44 -45:57.565 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.592 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.598 [INFO] TestAgent: Synced node info -45:57.699 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.699 [DEBUG] TestAgent: Node info in sync -45:57.699 [DEBUG] TestAgent: Node info in sync -45:57.751 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:55270 latency=3.233887ms -45:57.751 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?keys=&wait=2000ms from=127.0.0.1:55270 latency=53.729µs -45:57.755 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/ack?acquire=b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc from=127.0.0.1:55272 latency=3.166685ms -45:57.755 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=1&keys=&wait=2000ms from=127.0.0.1:55270 latency=3.090662ms -45:57.758 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/exit?acquire=b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc from=127.0.0.1:55274 latency=2.946985ms -45:57.758 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=14&keys=&wait=2000ms from=127.0.0.1:55270 latency=3.147284ms -45:57.759 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/exit from=127.0.0.1:55270 latency=77.582µs -45:57.764 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/random?acquire=b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc from=127.0.0.1:55276 latency=5.150189ms -45:57.765 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=15&keys=&wait=2000ms from=127.0.0.1:55270 latency=5.601848ms -45:57.768 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/out/00000?acquire=b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc from=127.0.0.1:55278 latency=3.297603ms -45:57.769 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=16&keys=&wait=2000ms from=127.0.0.1:55270 latency=3.99755ms -45:57.770 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/out/00000 from=127.0.0.1:55270 latency=69.701µs -45:57.773 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/out/00001?acquire=b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc from=127.0.0.1:55280 latency=3.186548ms -45:57.774 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=17&keys=&wait=2000ms from=127.0.0.1:55270 latency=3.371175ms -45:57.774 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/foo/out/00001 from=127.0.0.1:55270 latency=79.318µs -45:57.774 [INFO] TestAgent: Requesting shutdown -45:57.774 [INFO] TestAgent.server: shutting down server -45:57.774 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:57.774 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:57.774 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:57.774 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:57.774 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:57.774 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.774 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:57.774 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:57.774 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:57.774 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:57.775 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b0c0bde3-7045-b8f6-fa12-dd41bb8c2fbc/?index=18&keys=&wait=2000ms from=127.0.0.1:55270 latency=187.84µs -45:57.778 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.781Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.781Z [INFO] agent.router.manager: shutting down -45:57.781 [INFO] TestAgent: consul server down -45:57.781 [INFO] TestAgent: shutdown complete -45:57.781 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=tcp -45:57.781 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=udp -45:57.781 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18177 network=tcp -45:58.281 [INFO] TestAgent: Waiting for endpoints to shut down -45:58.281 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_StreamResults (0.94s) -2022-11-07T22:45:57.385Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.401Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.436 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:31a39f26-20c1-eb86-4231-fa4cb815082c Address:127.0.0.1:18202}]" -45:57.436 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18202 [Follower]" leader= -45:57.436 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-31a39f26-20c1-eb86-4231-fa4cb815082c.dc1 127.0.0.1 -45:57.436 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-31a39f26-20c1-eb86-4231-fa4cb815082c 127.0.0.1 -2022-11-07T22:45:57.436Z [INFO] agent.router: Initializing LAN area manager -45:57.437 [INFO] TestAgent.server: Adding LAN server: server="Node-31a39f26-20c1-eb86-4231-fa4cb815082c (Addr: tcp/127.0.0.1:18202) (DC: dc1)" -45:57.437 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-31a39f26-20c1-eb86-4231-fa4cb815082c.dc1 area=wan -45:57.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18197 network=tcp -45:57.437 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18197 network=udp -45:57.438 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18198 network=tcp -45:57.438 [INFO] TestAgent: started state syncer -45:57.438 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18203 network=tcp -45:57.499 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.499 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18202 [Candidate]" term=2 -45:57.508 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.508 [DEBUG] TestAgent.server.raft: vote granted: from=31a39f26-20c1-eb86-4231-fa4cb815082c term=2 tally=1 -45:57.508 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.508 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18202 [Leader]" -45:57.509 [INFO] TestAgent.server: cluster leadership acquired -45:57.509 [INFO] TestAgent.server: New leader elected: payload=Node-31a39f26-20c1-eb86-4231-fa4cb815082c -45:57.519 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18202 -45:57.547 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.578 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.578 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.578 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.578 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.578 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.578 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-31a39f26-20c1-eb86-4231-fa4cb815082c -45:57.578 [INFO] TestAgent.server: member joined, marking health alive: member=Node-31a39f26-20c1-eb86-4231-fa4cb815082c -45:57.590 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.873 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.875 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/consul?passing=1 from=127.0.0.1:58420 latency=1.059569ms -45:57.877 [INFO] TestAgent: Synced node info -45:57.877 [DEBUG] TestAgent: Node info in sync -45:57.880 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:58420 latency=3.128349ms -45:57.880 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/e9bc5f38-fc00-cfb7-f5e9-fa2f29baa41f from=127.0.0.1:58422 latency=138.181µs -45:57.884 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/e9bc5f38-fc00-cfb7-f5e9-fa2f29baa41f from=127.0.0.1:58420 latency=3.260476ms -45:57.885 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/e9bc5f38-fc00-cfb7-f5e9-fa2f29baa41f from=127.0.0.1:58424 latency=199.489µs -45:57.885 [INFO] TestAgent: Requesting shutdown -45:57.885 [INFO] TestAgent.server: shutting down server -45:57.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:57.885 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:57.885 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:57.885 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:57.885 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:57.885 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:57.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:57.885 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:57.885 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:57.885 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:57.888 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:57.892Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:57.892Z [INFO] agent.router.manager: shutting down -45:57.892 [INFO] TestAgent: consul server down -45:57.892 [INFO] TestAgent: shutdown complete -45:57.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18197 network=tcp -45:57.892 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18197 network=udp -45:57.892 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18198 network=tcp -45:58.392 [INFO] TestAgent: Waiting for endpoints to shut down -45:58.392 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_Sessions_Foreign (1.05s) -2022-11-07T22:45:57.382Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.401Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.440 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b99d04a9-b1d6-d2c4-316f-3afe7281dceb Address:127.0.0.1:18209}]" -45:57.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18209 [Follower]" leader= -45:57.440 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb.dc1 127.0.0.1 -45:57.441 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb 127.0.0.1 -2022-11-07T22:45:57.441Z [INFO] agent.router: Initializing LAN area manager -45:57.441 [INFO] TestAgent.server: Adding LAN server: server="Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb (Addr: tcp/127.0.0.1:18209) (DC: dc1)" -45:57.441 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb.dc1 area=wan -45:57.441 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18204 network=udp -45:57.441 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18204 network=tcp -45:57.441 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18205 network=tcp -45:57.442 [INFO] TestAgent: started state syncer -45:57.442 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18210 network=tcp -45:57.509 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.509 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18209 [Candidate]" term=2 -45:57.526 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.526 [DEBUG] TestAgent.server.raft: vote granted: from=b99d04a9-b1d6-d2c4-316f-3afe7281dceb term=2 tally=1 -45:57.527 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.527 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18209 [Leader]" -45:57.527 [INFO] TestAgent.server: cluster leadership acquired -45:57.527 [INFO] TestAgent.server: New leader elected: payload=Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb -45:57.540 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18209 -45:57.560 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.587 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.587 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.587 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.587 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.587 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.587 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb -45:57.587 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb -45:57.598 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.714 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:34352 latency=1.871918ms -45:57.719 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:34352 latency=2.941327ms -45:57.723 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/job?acquire=c66e5535-129f-a988-c92b-ab21e88ac12b from=127.0.0.1:34352 latency=3.633253ms -45:57.773 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.775 [INFO] TestAgent: Synced node info -45:57.924 [DEBUG] TestAgent.server: User event: event=_rexec -45:57.924 [DEBUG] TestAgent: received remote exec event: id=666e2e09-6fb9-9f6e-9077-7a0834d8fdd5 -45:57.924 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec from=127.0.0.1:34352 latency=256.437µs -45:57.925 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/?keys=&wait=1000ms from=127.0.0.1:34352 latency=230.132µs -45:57.930 [INFO] TestAgent: remote exec script: script=uptime -45:57.930 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/?index=13&keys=&wait=1000ms from=127.0.0.1:34352 latency=5.446817ms -45:57.940 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/?index=15&keys=&wait=1000ms from=127.0.0.1:34352 latency=8.28671ms -45:57.941 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb/out/00000 from=127.0.0.1:34352 latency=136.944µs -45:57.943 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/?index=16&keys=&wait=1000ms from=127.0.0.1:34352 latency=1.816312ms -45:57.944 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/Node-b99d04a9-b1d6-d2c4-316f-3afe7281dceb/exit from=127.0.0.1:34352 latency=106.122µs -45:58.948 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b/?index=17&keys=&wait=1000ms from=127.0.0.1:34352 latency=1.003156505s -45:58.949 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/c66e5535-129f-a988-c92b-ab21e88ac12b from=127.0.0.1:34390 latency=3.809125ms -45:58.952 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/c66e5535-129f-a988-c92b-ab21e88ac12b?recurse= from=127.0.0.1:34390 latency=3.202695ms -45:58.956 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/c66e5535-129f-a988-c92b-ab21e88ac12b from=127.0.0.1:34390 latency=3.024569ms -45:58.956 [INFO] TestAgent: Requesting shutdown -45:58.956 [INFO] TestAgent.server: shutting down server -45:58.956 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:58.956 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:58.956 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:58.956 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:58.956 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:58.956 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:58.956 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:58.956 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:58.956 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:58.956 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:58.960 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:58.963Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.963Z [INFO] agent.router.manager: shutting down -45:58.963 [INFO] TestAgent: consul server down -45:58.963 [INFO] TestAgent: shutdown complete -45:58.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18204 network=tcp -45:58.963 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18204 network=udp -45:58.963 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18205 network=tcp -45:59.464 [INFO] TestAgent: Waiting for endpoints to shut down -45:59.464 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand (2.12s) -2022-11-07T22:45:57.383Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.403Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.486 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9f56cc86-7415-b330-f57e-d3b62ee0b492 Address:127.0.0.1:18216}]" -45:57.486 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18216 [Follower]" leader= -45:57.487 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9f56cc86-7415-b330-f57e-d3b62ee0b492.dc1 127.0.0.1 -45:57.487 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9f56cc86-7415-b330-f57e-d3b62ee0b492 127.0.0.1 -2022-11-07T22:45:57.487Z [INFO] agent.router: Initializing LAN area manager -45:57.487 [INFO] TestAgent.server: Adding LAN server: server="Node-9f56cc86-7415-b330-f57e-d3b62ee0b492 (Addr: tcp/127.0.0.1:18216) (DC: dc1)" -45:57.487 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9f56cc86-7415-b330-f57e-d3b62ee0b492.dc1 area=wan -45:57.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18211 network=udp -45:57.487 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18211 network=tcp -45:57.488 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18212 network=tcp -45:57.488 [INFO] TestAgent: started state syncer -45:57.488 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18217 network=tcp -45:57.529 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.529 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18216 [Candidate]" term=2 -45:57.548 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.548 [DEBUG] TestAgent.server.raft: vote granted: from=9f56cc86-7415-b330-f57e-d3b62ee0b492 term=2 tally=1 -45:57.548 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.548 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18216 [Leader]" -45:57.549 [INFO] TestAgent.server: cluster leadership acquired -45:57.549 [INFO] TestAgent.server: New leader elected: payload=Node-9f56cc86-7415-b330-f57e-d3b62ee0b492 -45:57.560 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18216 -45:57.579 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.618 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.618 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.618 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.618 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.619 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.619 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9f56cc86-7415-b330-f57e-d3b62ee0b492 -45:57.619 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9f56cc86-7415-b330-f57e-d3b62ee0b492 -45:57.637 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.694 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.697 [INFO] TestAgent: Synced node info -45:57.903 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.903 [DEBUG] TestAgent: Node info in sync -45:57.903 [DEBUG] TestAgent: Node info in sync -45:57.980 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:53664 latency=913.104µs -45:57.985 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:53664 latency=3.201005ms -45:57.988 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/job?acquire=b2805d6c-5783-b544-0184-3b1f60fedcc9 from=127.0.0.1:53664 latency=3.017696ms -45:58.189 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec from=127.0.0.1:53664 latency=261.041µs -45:58.189 [DEBUG] TestAgent.server: User event: event=_rexec -45:58.189 [DEBUG] TestAgent: received remote exec event: id=9a249abf-aef8-4568-1a16-6342ee9d4e90 -45:58.191 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/?keys=&wait=1000ms from=127.0.0.1:53664 latency=1.115826ms -45:58.195 [INFO] TestAgent: remote exec script: script= -45:58.196 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/?index=14&keys=&wait=1000ms from=127.0.0.1:53664 latency=3.670647ms -45:58.202 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/?index=15&keys=&wait=1000ms from=127.0.0.1:53664 latency=4.892807ms -45:58.203 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/Node-9f56cc86-7415-b330-f57e-d3b62ee0b492/out/00000 from=127.0.0.1:53664 latency=130.888µs -45:58.210 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/?index=16&keys=&wait=1000ms from=127.0.0.1:53664 latency=1.896944ms -45:58.211 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/Node-9f56cc86-7415-b330-f57e-d3b62ee0b492/exit from=127.0.0.1:53664 latency=468.847µs -45:59.222 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9/?index=17&keys=&wait=1000ms from=127.0.0.1:53664 latency=1.010898556s -45:59.292 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/b2805d6c-5783-b544-0184-3b1f60fedcc9 from=127.0.0.1:53678 latency=79.398946ms -45:59.316 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/b2805d6c-5783-b544-0184-3b1f60fedcc9?recurse= from=127.0.0.1:53678 latency=13.648956ms -45:59.322 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/b2805d6c-5783-b544-0184-3b1f60fedcc9 from=127.0.0.1:53678 latency=5.937314ms -45:59.322 [INFO] TestAgent: Requesting shutdown -45:59.322 [INFO] TestAgent.server: shutting down server -45:59.322 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:59.322 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:59.322 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:59.322 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:59.322 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:59.322 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:59.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:59.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:59.322 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:59.322 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:59.325 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:59.338Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:59.338Z [INFO] agent.router.manager: shutting down -45:59.338 [INFO] TestAgent: consul server down -45:59.338 [INFO] TestAgent: shutdown complete -45:59.338 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18211 network=tcp -45:59.338 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18211 network=udp -45:59.338 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18212 network=tcp -45:59.838 [INFO] TestAgent: Waiting for endpoints to shut down -45:59.838 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_NoShell (2.50s) -2022-11-07T22:45:57.594Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.603Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.642 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4d29de72-4075-5625-5aa6-f8dfd0a6366e Address:127.0.0.1:18223}]" -45:57.642 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18223 [Follower]" leader= -45:57.643 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e.dc2 127.0.0.1 -45:57.643 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e 127.0.0.1 -2022-11-07T22:45:57.643Z [INFO] agent.router: Initializing LAN area manager -45:57.644 [INFO] TestAgent.server: Adding LAN server: server="Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e (Addr: tcp/127.0.0.1:18223) (DC: dc2)" -45:57.644 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e.dc2 area=wan -45:57.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18218 network=udp -45:57.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18218 network=tcp -45:57.644 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18219 network=tcp -45:57.644 [INFO] TestAgent: started state syncer -45:57.644 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18224 network=tcp -45:57.687 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.687 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18223 [Candidate]" term=2 -45:57.695 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.695 [DEBUG] TestAgent.server.raft: vote granted: from=4d29de72-4075-5625-5aa6-f8dfd0a6366e term=2 tally=1 -45:57.695 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.695 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18223 [Leader]" -45:57.695 [INFO] TestAgent.server: cluster leadership acquired -45:57.695 [INFO] TestAgent.server: New leader elected: payload=Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e -45:57.700 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18223 -45:57.711 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.728 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.728 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.728 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.728 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.728 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.729 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e -45:57.729 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e -45:57.736 [INFO] TestAgent.server: federation state anti-entropy synced -45:58.080 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:58.084 [INFO] TestAgent: Synced node info -45:58.084 [DEBUG] TestAgent: Node info in sync -45:58.085 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:18194] -45:58.085 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:18194 -45:58.085 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60.dc1 127.0.0.1 -45:58.085 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -45:58.086 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60.dc1 area=wan -45:58.300 [DEBUG] TestAgent.server: User event: event=_rexec -45:58.300 [DEBUG] TestAgent: received remote exec event: id=362b526a-e70d-c7f5-81bf-6c3b281f9918 -45:58.307 [INFO] TestAgent: remote exec script: script=uptime -45:58.558 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:58.558 [DEBUG] TestAgent: Node info in sync -45:58.832 [INFO] TestAgent: Requesting shutdown -45:58.832 [INFO] TestAgent.server: shutting down server -45:58.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:58.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:58.832 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:58.832 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:58.832 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:58.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:58.832 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:58.832 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:58.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:58.832 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:58.836 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:58.839Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.839Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:58.839Z [INFO] agent.router.manager: shutting down -45:58.839 [INFO] TestAgent: consul server down -45:58.839 [INFO] TestAgent: shutdown complete -45:58.839 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18218 network=tcp -45:58.839 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18218 network=udp -45:58.839 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18219 network=tcp -45:59.339 [INFO] TestAgent: Waiting for endpoints to shut down -45:59.339 [INFO] TestAgent: Endpoints down -2022-11-07T22:45:57.382Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:45:57.397Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -45:57.433 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 Address:127.0.0.1:18195}]" -45:57.433 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18195 [Follower]" leader= -45:57.434 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60.dc1 127.0.0.1 -45:57.434 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 127.0.0.1 -2022-11-07T22:45:57.434Z [INFO] agent.router: Initializing LAN area manager -45:57.434 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60.dc1 area=wan -45:57.434 [INFO] TestAgent.server: Adding LAN server: server="Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 (Addr: tcp/127.0.0.1:18195) (DC: dc1)" -45:57.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18190 network=tcp -45:57.434 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18190 network=udp -45:57.435 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18191 network=tcp -45:57.435 [INFO] TestAgent: started state syncer -45:57.435 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18196 network=tcp -45:57.474 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -45:57.474 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18195 [Candidate]" term=2 -45:57.494 [DEBUG] TestAgent.server.raft: votes: needed=1 -45:57.494 [DEBUG] TestAgent.server.raft: vote granted: from=79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 term=2 tally=1 -45:57.494 [INFO] TestAgent.server.raft: election won: tally=1 -45:57.494 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18195 [Leader]" -45:57.495 [INFO] TestAgent.server: cluster leadership acquired -45:57.495 [INFO] TestAgent.server: New leader elected: payload=Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 -45:57.500 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18195 -45:57.514 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -45:57.556 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -45:57.556 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -45:57.556 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -45:57.556 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -45:57.556 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -45:57.556 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 -45:57.557 [INFO] TestAgent.server: member joined, marking health alive: member=Node-79b94ed3-0d99-e078-23bc-f7c6e8e5fd60 -45:57.573 [INFO] TestAgent.server: federation state anti-entropy synced -45:57.806 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -45:57.810 [INFO] TestAgent: Synced node info -45:57.810 [DEBUG] TestAgent: Node info in sync -45:58.085 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:42002 -45:58.085 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e.dc2 127.0.0.1 -45:58.085 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e.dc2 area=wan -45:58.087 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self?dc=dc2 from=127.0.0.1:49878 latency=935.108µs -45:58.090 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/consul?dc=dc2&passing=1 from=127.0.0.1:49878 latency=1.160777ms -45:58.095 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create?dc=dc2 from=127.0.0.1:49878 latency=4.39231ms -45:58.099 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/job?acquire=d06c4a9e-0132-1338-e0ed-967b95ecbc3a&dc=dc2 from=127.0.0.1:49878 latency=3.62917ms -45:58.301 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec?dc=dc2 from=127.0.0.1:49878 latency=1.186787ms -45:58.302 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/?dc=dc2&keys=&wait=500ms from=127.0.0.1:49878 latency=481.683µs -45:58.307 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/?dc=dc2&index=14&keys=&wait=500ms from=127.0.0.1:49878 latency=4.27264ms -45:58.315 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/?dc=dc2&index=15&keys=&wait=500ms from=127.0.0.1:49878 latency=6.371243ms -45:58.316 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e/out/00000?dc=dc2 from=127.0.0.1:49878 latency=735.226µs -45:58.317 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/?dc=dc2&index=16&keys=&wait=500ms from=127.0.0.1:49878 latency=407.118µs -45:58.318 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e/exit?dc=dc2 from=127.0.0.1:49878 latency=761.287µs -45:58.644 [DEBUG] TestAgent.server.serf.wan: serf: messageJoinType: Node-4d29de72-4075-5625-5aa6-f8dfd0a6366e.dc2 -45:58.823 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a/?dc=dc2&index=17&keys=&wait=500ms from=127.0.0.1:49878 latency=504.532736ms -45:58.823 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/d06c4a9e-0132-1338-e0ed-967b95ecbc3a?dc=dc2 from=127.0.0.1:49882 latency=3.943586ms -45:58.828 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/d06c4a9e-0132-1338-e0ed-967b95ecbc3a?dc=dc2&recurse= from=127.0.0.1:49882 latency=4.116493ms -45:58.832 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/d06c4a9e-0132-1338-e0ed-967b95ecbc3a?dc=dc2 from=127.0.0.1:49882 latency=3.318129ms -45:59.339 [INFO] TestAgent: Requesting shutdown -45:59.339 [INFO] TestAgent.server: shutting down server -45:59.339 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -45:59.339 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -45:59.339 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -45:59.339 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -45:59.339 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -45:59.339 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -45:59.339 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -45:59.340 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -45:59.340 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -45:59.340 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -45:59.343 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:45:59.346Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:59.346Z [INFO] agent.router.manager: shutting down -2022-11-07T22:45:59.346Z [INFO] agent.router.manager: shutting down -45:59.346 [INFO] TestAgent: consul server down -45:59.346 [INFO] TestAgent: shutdown complete -45:59.346 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18190 network=tcp -45:59.346 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18190 network=udp -45:59.346 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18191 network=tcp -45:59.847 [INFO] TestAgent: Waiting for endpoints to shut down -45:59.847 [INFO] TestAgent: Endpoints down ---- PASS: TestExecCommand_CrossDC (2.51s) +2021-10-05T16:32:39.897Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.927Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:40.002 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dc817e0b-f9e2-5974-d219-453264f6a229 Address:127.0.0.1:23314}]" +32:40.003 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dc817e0b-f9e2-5974-d219-453264f6a229.dc1 127.0.0.1 +32:40.003 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dc817e0b-f9e2-5974-d219-453264f6a229 127.0.0.1 +2021-10-05T16:32:40.003Z [INFO] agent.router: Initializing LAN area manager +32:40.003 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23309 network=udp +32:40.003 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23314 [Follower]" leader= +32:40.004 [INFO] TestAgent.server: Adding LAN server: server="Node-dc817e0b-f9e2-5974-d219-453264f6a229 (Addr: tcp/127.0.0.1:23314) (DC: dc1)" +32:40.004 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dc817e0b-f9e2-5974-d219-453264f6a229.dc1 area=wan +32:40.004 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23309 network=tcp +32:40.004 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23310 network=tcp +32:40.004 [INFO] TestAgent: started state syncer +32:40.005 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23315 network=tcp +32:40.056 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.060 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23314 [Candidate]" term=2 +32:40.089 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.089 [DEBUG] TestAgent.server.raft: vote granted: from=dc817e0b-f9e2-5974-d219-453264f6a229 term=2 tally=1 +32:40.089 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.089 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23314 [Leader]" +32:40.089 [INFO] TestAgent.server: cluster leadership acquired +32:40.089 [INFO] TestAgent.server: New leader elected: payload=Node-dc817e0b-f9e2-5974-d219-453264f6a229 +32:40.112 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23314 +32:40.141 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.221 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.221 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.221 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.221 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.221 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.221 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dc817e0b-f9e2-5974-d219-453264f6a229 +32:40.221 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dc817e0b-f9e2-5974-d219-453264f6a229 +32:40.236 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.245 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:48658 latency=5.919223ms +32:40.247 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/bdd9a06f-51a5-3313-6bec-b772096f4275 from=127.0.0.1:48660 latency=1.190318ms +32:40.251 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/bdd9a06f-51a5-3313-6bec-b772096f4275 from=127.0.0.1:48658 latency=2.883621ms +32:40.252 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/bdd9a06f-51a5-3313-6bec-b772096f4275 from=127.0.0.1:48662 latency=212.907µs +32:40.256 [INFO] TestAgent: Requesting shutdown +32:40.256 [INFO] TestAgent.server: shutting down server +32:40.256 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:40.256 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:40.256 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:40.256 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:40.256 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:40.256 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:40.256 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:40.256 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:40.256 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:40.256 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:40.256 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:40.263 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:40.266Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:40.266Z [INFO] agent.router.manager: shutting down +32:40.266 [INFO] TestAgent: consul server down +32:40.266 [INFO] TestAgent: shutdown complete +32:40.266 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23309 network=tcp +32:40.266 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23309 network=udp +32:40.266 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23310 network=tcp +32:40.773 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.773 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_Sessions (1.13s) +2021-10-05T16:32:39.743Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.835Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.905 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ee925226-c7c4-33d7-f21e-925f200e05ff Address:127.0.0.1:23321}]" +32:39.906 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ee925226-c7c4-33d7-f21e-925f200e05ff.dc1 127.0.0.1 +32:39.906 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ee925226-c7c4-33d7-f21e-925f200e05ff 127.0.0.1 +2021-10-05T16:32:39.906Z [INFO] agent.router: Initializing LAN area manager +32:39.907 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23316 network=udp +32:39.907 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23321 [Follower]" leader= +32:39.907 [INFO] TestAgent.server: Adding LAN server: server="Node-ee925226-c7c4-33d7-f21e-925f200e05ff (Addr: tcp/127.0.0.1:23321) (DC: dc1)" +32:39.907 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ee925226-c7c4-33d7-f21e-925f200e05ff.dc1 area=wan +32:39.907 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23316 network=tcp +32:39.915 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23317 network=tcp +32:39.915 [INFO] TestAgent: started state syncer +32:39.915 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23322 network=tcp +32:39.968 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:39.968 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23321 [Candidate]" term=2 +32:40.016 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.016 [DEBUG] TestAgent.server.raft: vote granted: from=ee925226-c7c4-33d7-f21e-925f200e05ff term=2 tally=1 +32:40.016 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.016 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23321 [Leader]" +32:40.016 [INFO] TestAgent.server: cluster leadership acquired +32:40.016 [INFO] TestAgent.server: New leader elected: payload=Node-ee925226-c7c4-33d7-f21e-925f200e05ff +32:40.032 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23321 +32:40.053 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.105 [INFO] TestAgent: Synced node info +32:40.105 [DEBUG] TestAgent: Node info in sync +32:40.117 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.117 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.117 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.117 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.117 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.117 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ee925226-c7c4-33d7-f21e-925f200e05ff +32:40.117 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ee925226-c7c4-33d7-f21e-925f200e05ff +32:40.137 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.310 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/consul?passing=1 from=127.0.0.1:42888 latency=1.057248ms +32:40.337 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:42888 latency=14.960777ms +32:40.342 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/1b742fe6-f795-b183-77a0-bdaec8163da9 from=127.0.0.1:42894 latency=513.166µs +32:40.358 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/1b742fe6-f795-b183-77a0-bdaec8163da9 from=127.0.0.1:42888 latency=14.74053ms +32:40.362 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/session/info/1b742fe6-f795-b183-77a0-bdaec8163da9 from=127.0.0.1:42896 latency=165.843µs +32:40.362 [INFO] TestAgent: Requesting shutdown +32:40.362 [INFO] TestAgent.server: shutting down server +32:40.362 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:40.362 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:40.362 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:40.362 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:40.362 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:40.363 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:40.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:40.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:40.363 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:40.363 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:40.376 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:40.384Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:40.384Z [INFO] agent.router.manager: shutting down +32:40.384 [INFO] TestAgent: consul server down +32:40.384 [INFO] TestAgent: shutdown complete +32:40.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23316 network=tcp +32:40.384 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23316 network=udp +32:40.384 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23317 network=tcp +32:40.884 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.884 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_Sessions_Foreign (1.25s) +2021-10-05T16:32:39.812Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.863Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.931 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9d3b55f2-9d91-e699-f57c-06777bff409f Address:127.0.0.1:23286}]" +32:39.932 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23286 [Follower]" leader= +32:39.932 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9d3b55f2-9d91-e699-f57c-06777bff409f.dc1 127.0.0.1 +32:39.974 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9d3b55f2-9d91-e699-f57c-06777bff409f 127.0.0.1 +2021-10-05T16:32:39.974Z [INFO] agent.router: Initializing LAN area manager +32:39.982 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9d3b55f2-9d91-e699-f57c-06777bff409f.dc1 area=wan +32:39.982 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=udp +32:39.982 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=tcp +32:39.983 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23282 network=tcp +32:39.983 [INFO] TestAgent: started state syncer +32:39.983 [INFO] TestAgent.server: Adding LAN server: server="Node-9d3b55f2-9d91-e699-f57c-06777bff409f (Addr: tcp/127.0.0.1:23286) (DC: dc1)" +32:39.983 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23287 network=tcp +32:40.000 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.000 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23286 [Candidate]" term=2 +32:40.029 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.029 [DEBUG] TestAgent.server.raft: vote granted: from=9d3b55f2-9d91-e699-f57c-06777bff409f term=2 tally=1 +32:40.029 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.029 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23286 [Leader]" +32:40.029 [INFO] TestAgent.server: cluster leadership acquired +32:40.029 [INFO] TestAgent.server: New leader elected: payload=Node-9d3b55f2-9d91-e699-f57c-06777bff409f +32:40.040 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23286 +32:40.064 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.110 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.110 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.110 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.110 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.110 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.110 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9d3b55f2-9d91-e699-f57c-06777bff409f +32:40.110 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9d3b55f2-9d91-e699-f57c-06777bff409f +32:40.136 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.184 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:40.213 [INFO] TestAgent: Synced node info +32:40.213 [DEBUG] TestAgent: Node info in sync +32:40.360 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44672 latency=35.278128ms +32:40.362 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?keys=&wait=2000ms from=127.0.0.1:44672 latency=64.56µs +32:40.367 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/ack?acquire=cf931db4-6f85-b8ae-187f-96337daf684d from=127.0.0.1:44680 latency=5.229745ms +32:40.367 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=1&keys=&wait=2000ms from=127.0.0.1:44672 latency=4.556464ms +32:40.375 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/exit?acquire=cf931db4-6f85-b8ae-187f-96337daf684d from=127.0.0.1:44682 latency=6.053716ms +32:40.375 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=14&keys=&wait=2000ms from=127.0.0.1:44672 latency=7.414787ms +32:40.378 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/exit from=127.0.0.1:44672 latency=1.538146ms +32:40.394 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/random?acquire=cf931db4-6f85-b8ae-187f-96337daf684d from=127.0.0.1:44684 latency=9.395302ms +32:40.394 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=15&keys=&wait=2000ms from=127.0.0.1:44672 latency=15.38349ms +32:40.402 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/out/00000?acquire=cf931db4-6f85-b8ae-187f-96337daf684d from=127.0.0.1:44688 latency=7.099895ms +32:40.403 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=16&keys=&wait=2000ms from=127.0.0.1:44672 latency=7.523601ms +32:40.408 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/out/00000 from=127.0.0.1:44672 latency=198.814µs +32:40.418 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/out/00001?acquire=cf931db4-6f85-b8ae-187f-96337daf684d from=127.0.0.1:44692 latency=5.420451ms +32:40.420 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=17&keys=&wait=2000ms from=127.0.0.1:44672 latency=8.037533ms +32:40.430 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/foo/out/00001 from=127.0.0.1:44672 latency=329.616µs +32:40.431 [INFO] TestAgent: Requesting shutdown +32:40.431 [INFO] TestAgent.server: shutting down server +32:40.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:40.431 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:40.431 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:40.431 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:40.431 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:40.431 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:40.431 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:40.431 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:40.431 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:40.431 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:40.432 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/cf931db4-6f85-b8ae-187f-96337daf684d/?index=18&keys=&wait=2000ms from=127.0.0.1:44672 latency=406.843µs +32:40.444 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:40.452Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:40.452Z [INFO] agent.router.manager: shutting down +32:40.452 [INFO] TestAgent: consul server down +32:40.452 [INFO] TestAgent: shutdown complete +32:40.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=tcp +32:40.452 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=udp +32:40.452 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23282 network=tcp +32:40.953 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.953 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_StreamResults (1.31s) +2021-10-05T16:32:39.915Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.974Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:40.014 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0b3b3277-4c56-52f3-5315-58a546d6e29c Address:127.0.0.1:23300}]" +32:40.014 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0b3b3277-4c56-52f3-5315-58a546d6e29c.dc1 127.0.0.1 +32:40.015 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0b3b3277-4c56-52f3-5315-58a546d6e29c 127.0.0.1 +2021-10-05T16:32:40.015Z [INFO] agent.router: Initializing LAN area manager +32:40.015 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=udp +32:40.015 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23300 [Follower]" leader= +32:40.016 [INFO] TestAgent.server: Adding LAN server: server="Node-0b3b3277-4c56-52f3-5315-58a546d6e29c (Addr: tcp/127.0.0.1:23300) (DC: dc1)" +32:40.016 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=tcp +32:40.016 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0b3b3277-4c56-52f3-5315-58a546d6e29c.dc1 area=wan +32:40.017 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23296 network=tcp +32:40.017 [INFO] TestAgent: started state syncer +32:40.017 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23301 network=tcp +32:40.064 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.064 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23300 [Candidate]" term=2 +32:40.094 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.094 [DEBUG] TestAgent.server.raft: vote granted: from=0b3b3277-4c56-52f3-5315-58a546d6e29c term=2 tally=1 +32:40.094 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.094 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23300 [Leader]" +32:40.094 [INFO] TestAgent.server: cluster leadership acquired +32:40.094 [INFO] TestAgent.server: New leader elected: payload=Node-0b3b3277-4c56-52f3-5315-58a546d6e29c +32:40.124 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23300 +32:40.149 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.176 [INFO] TestAgent: Synced node info +32:40.187 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.187 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.187 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.187 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.190 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.205 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0b3b3277-4c56-52f3-5315-58a546d6e29c +32:40.205 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0b3b3277-4c56-52f3-5315-58a546d6e29c +32:40.391 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:46442 latency=6.595686ms +32:40.402 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/2504c512-98ce-38c7-37e6-185bb1a048f9/job?acquire=2504c512-98ce-38c7-37e6-185bb1a048f9 from=127.0.0.1:46442 latency=9.861437ms +32:40.408 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/2504c512-98ce-38c7-37e6-185bb1a048f9/job from=127.0.0.1:46446 latency=397.245µs +32:40.432 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/2504c512-98ce-38c7-37e6-185bb1a048f9?recurse= from=127.0.0.1:46442 latency=17.221651ms +32:40.434 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/2504c512-98ce-38c7-37e6-185bb1a048f9/job from=127.0.0.1:46450 latency=82.566µs +32:40.436 [INFO] TestAgent: Requesting shutdown +32:40.436 [INFO] TestAgent.server: shutting down server +32:40.436 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:40.436 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:40.436 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:40.436 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:40.436 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:40.436 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" +32:40.436 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:40.436 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:40.436 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:40.436 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:40.449 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:40.459Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:40.459Z [INFO] agent.router.manager: shutting down +32:40.459 [INFO] TestAgent: consul server down +32:40.459 [INFO] TestAgent: shutdown complete +32:40.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=tcp +32:40.459 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=udp +32:40.459 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23296 network=tcp +32:40.960 [INFO] TestAgent: Waiting for endpoints to shut down +32:40.960 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_UploadDestroy (1.31s) +2021-10-05T16:32:39.833Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.862Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.984 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2c47baa2-dce5-be9d-5bf7-a1570e66da7b Address:127.0.0.1:23293}]" +32:39.984 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23293 [Follower]" leader= +32:39.985 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b.dc1 127.0.0.1 +32:39.997 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b 127.0.0.1 +2021-10-05T16:32:39.997Z [INFO] agent.router: Initializing LAN area manager +32:39.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=udp +32:39.997 [INFO] TestAgent.server: Adding LAN server: server="Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b (Addr: tcp/127.0.0.1:23293) (DC: dc1)" +32:39.997 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b.dc1 area=wan +32:39.997 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=tcp +32:39.998 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23289 network=tcp +32:39.998 [INFO] TestAgent: started state syncer +32:39.998 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23294 network=tcp +32:40.024 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.024 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23293 [Candidate]" term=2 +32:40.039 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.039 [DEBUG] TestAgent.server.raft: vote granted: from=2c47baa2-dce5-be9d-5bf7-a1570e66da7b term=2 tally=1 +32:40.039 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.039 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23293 [Leader]" +32:40.039 [INFO] TestAgent.server: cluster leadership acquired +32:40.039 [INFO] TestAgent.server: New leader elected: payload=Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b +32:40.050 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23293 +32:40.106 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.144 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.144 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.144 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.144 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.144 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.144 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b +32:40.144 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b +32:40.165 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.216 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:49028 latency=27.627508ms +32:40.241 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:49028 latency=12.819991ms +32:40.246 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/job?acquire=3551ea35-eb25-446e-8ad6-faec14dd4ae6 from=127.0.0.1:49028 latency=5.334986ms +32:40.352 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:40.360 [INFO] TestAgent: Synced node info +32:40.360 [DEBUG] TestAgent: Node info in sync +32:40.454 [DEBUG] TestAgent.server: User event: event=_rexec +32:40.454 [DEBUG] TestAgent: received remote exec event: id=5ebc5956-67db-ff21-8da3-c579b2ecdfc6 +32:40.454 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec from=127.0.0.1:49028 latency=330.318µs +32:40.456 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/?keys=&wait=1000ms from=127.0.0.1:49028 latency=237.96µs +32:40.463 [INFO] TestAgent: remote exec script: script= +32:40.466 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/?index=13&keys=&wait=1000ms from=127.0.0.1:49028 latency=10.077769ms +32:40.482 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/?index=15&keys=&wait=1000ms from=127.0.0.1:49028 latency=12.72644ms +32:40.483 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b/out/00000 from=127.0.0.1:49028 latency=230.69µs +32:40.487 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/?index=16&keys=&wait=1000ms from=127.0.0.1:49028 latency=3.063326ms +32:40.487 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/Node-2c47baa2-dce5-be9d-5bf7-a1570e66da7b/exit from=127.0.0.1:49028 latency=173.162µs +32:41.505 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6/?index=17&keys=&wait=1000ms from=127.0.0.1:49028 latency=1.016871041s +32:41.667 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/3551ea35-eb25-446e-8ad6-faec14dd4ae6 from=127.0.0.1:49080 latency=169.179983ms +32:41.675 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/3551ea35-eb25-446e-8ad6-faec14dd4ae6?recurse= from=127.0.0.1:49080 latency=5.689034ms +32:41.680 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/3551ea35-eb25-446e-8ad6-faec14dd4ae6 from=127.0.0.1:49080 latency=4.916424ms +32:41.680 [INFO] TestAgent: Requesting shutdown +32:41.681 [INFO] TestAgent.server: shutting down server +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:41.681 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:41.681 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:41.686 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:41.693Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:41.693Z [INFO] agent.router.manager: shutting down +32:41.693 [INFO] TestAgent: consul server down +32:41.693 [INFO] TestAgent: shutdown complete +32:41.693 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=tcp +32:41.693 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=udp +32:41.693 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23289 network=tcp +32:42.193 [INFO] TestAgent: Waiting for endpoints to shut down +32:42.193 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_NoShell (2.55s) +2021-10-05T16:32:39.858Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.906Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.988 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:eff9f8c3-b081-531c-0999-bef62eecc5d2 Address:127.0.0.1:23307}]" +32:39.992 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23307 [Follower]" leader= +32:40.000 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-eff9f8c3-b081-531c-0999-bef62eecc5d2.dc1 127.0.0.1 +32:40.001 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-eff9f8c3-b081-531c-0999-bef62eecc5d2 127.0.0.1 +2021-10-05T16:32:40.001Z [INFO] agent.router: Initializing LAN area manager +32:40.001 [INFO] TestAgent.server: Adding LAN server: server="Node-eff9f8c3-b081-531c-0999-bef62eecc5d2 (Addr: tcp/127.0.0.1:23307) (DC: dc1)" +32:40.001 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23302 network=udp +32:40.001 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-eff9f8c3-b081-531c-0999-bef62eecc5d2.dc1 area=wan +32:40.001 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23302 network=tcp +32:40.002 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23303 network=tcp +32:40.002 [INFO] TestAgent: started state syncer +32:40.002 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23308 network=tcp +32:40.041 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.041 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23307 [Candidate]" term=2 +32:40.067 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.067 [DEBUG] TestAgent.server.raft: vote granted: from=eff9f8c3-b081-531c-0999-bef62eecc5d2 term=2 tally=1 +32:40.067 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.067 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23307 [Leader]" +32:40.067 [INFO] TestAgent.server: cluster leadership acquired +32:40.068 [INFO] TestAgent.server: New leader elected: payload=Node-eff9f8c3-b081-531c-0999-bef62eecc5d2 +32:40.076 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23307 +32:40.103 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.145 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.145 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.145 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.145 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.145 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.145 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-eff9f8c3-b081-531c-0999-bef62eecc5d2 +32:40.145 [INFO] TestAgent.server: member joined, marking health alive: member=Node-eff9f8c3-b081-531c-0999-bef62eecc5d2 +32:40.167 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.178 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:40.223 [INFO] TestAgent: Synced node info +32:40.223 [DEBUG] TestAgent: Node info in sync +32:40.336 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:41146 latency=8.563372ms +32:40.361 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:41146 latency=19.007602ms +32:40.376 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/job?acquire=9f4e8daf-f448-e133-43ab-263d061b2cc9 from=127.0.0.1:41146 latency=13.907415ms +32:40.580 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec from=127.0.0.1:41146 latency=363.294µs +32:40.580 [DEBUG] TestAgent.server: User event: event=_rexec +32:40.580 [DEBUG] TestAgent: received remote exec event: id=9b558ba6-b2b6-b953-9467-4f5573e944dc +32:40.582 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/?keys=&wait=1000ms from=127.0.0.1:41146 latency=252.807µs +32:40.594 [INFO] TestAgent: remote exec script: script=uptime +32:40.598 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/?index=14&keys=&wait=1000ms from=127.0.0.1:41146 latency=14.39328ms +32:40.632 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/?index=15&keys=&wait=1000ms from=127.0.0.1:41146 latency=32.408672ms +32:40.640 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/Node-eff9f8c3-b081-531c-0999-bef62eecc5d2/out/00000 from=127.0.0.1:41146 latency=251.933µs +32:40.653 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/?index=16&keys=&wait=1000ms from=127.0.0.1:41146 latency=12.240679ms +32:40.654 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/Node-eff9f8c3-b081-531c-0999-bef62eecc5d2/exit from=127.0.0.1:41146 latency=177.398µs +32:41.666 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/9f4e8daf-f448-e133-43ab-263d061b2cc9 from=127.0.0.1:41190 latency=9.065854ms +32:41.666 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9/?index=17&keys=&wait=1000ms from=127.0.0.1:41146 latency=1.011606938s +32:41.684 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/9f4e8daf-f448-e133-43ab-263d061b2cc9?recurse= from=127.0.0.1:41190 latency=14.295609ms +32:41.702 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/9f4e8daf-f448-e133-43ab-263d061b2cc9 from=127.0.0.1:41190 latency=14.617292ms +32:41.703 [INFO] TestAgent: Requesting shutdown +32:41.703 [INFO] TestAgent.server: shutting down server +32:41.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:41.703 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:41.703 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:41.703 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:41.703 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:41.703 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:41.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:41.703 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:41.703 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:41.703 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:41.707 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:41.710Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:41.710Z [INFO] agent.router.manager: shutting down +32:41.710 [INFO] TestAgent: consul server down +32:41.710 [INFO] TestAgent: shutdown complete +32:41.710 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23302 network=tcp +32:41.711 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23302 network=udp +32:41.711 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23303 network=tcp +32:42.212 [INFO] TestAgent: Waiting for endpoints to shut down +32:42.212 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand (2.56s) +2021-10-05T16:32:40.185Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:40.231Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:40.271 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7a864460-268a-1a92-c2f3-ac27d97b8545 Address:127.0.0.1:23328}]" +32:40.272 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23328 [Follower]" leader= +32:40.272 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 127.0.0.1 +32:40.273 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7a864460-268a-1a92-c2f3-ac27d97b8545 127.0.0.1 +2021-10-05T16:32:40.273Z [INFO] agent.router: Initializing LAN area manager +32:40.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23323 network=udp +32:40.273 [INFO] TestAgent.server: Adding LAN server: server="Node-7a864460-268a-1a92-c2f3-ac27d97b8545 (Addr: tcp/127.0.0.1:23328) (DC: dc2)" +32:40.274 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 area=wan +32:40.274 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23323 network=tcp +32:40.274 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23324 network=tcp +32:40.274 [INFO] TestAgent: started state syncer +32:40.274 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23329 network=tcp +32:40.331 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.331 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23328 [Candidate]" term=2 +32:40.352 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.352 [DEBUG] TestAgent.server.raft: vote granted: from=7a864460-268a-1a92-c2f3-ac27d97b8545 term=2 tally=1 +32:40.352 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.352 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23328 [Leader]" +32:40.352 [INFO] TestAgent.server: cluster leadership acquired +32:40.352 [INFO] TestAgent.server: New leader elected: payload=Node-7a864460-268a-1a92-c2f3-ac27d97b8545 +32:40.362 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23328 +32:40.381 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.412 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.412 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.412 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.412 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.412 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.412 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7a864460-268a-1a92-c2f3-ac27d97b8545 +32:40.412 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7a864460-268a-1a92-c2f3-ac27d97b8545 +32:40.430 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.651 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:40.659 [INFO] TestAgent: Synced node info +32:40.659 [DEBUG] TestAgent: Node info in sync +32:40.712 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:23278] +32:40.713 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:23278 +32:40.715 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502.dc1 127.0.0.1 +32:40.715 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +32:40.715 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502.dc1 area=wan +32:40.945 [DEBUG] TestAgent.server: User event: event=_rexec +32:40.945 [DEBUG] TestAgent: received remote exec event: id=f29432d9-e789-6964-9962-43a5915acba7 +32:40.963 [INFO] TestAgent: remote exec script: script=uptime +32:41.004 [DEBUG] TestAgent.server.serf.wan: serf: messageJoinType: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 +32:41.501 [DEBUG] TestAgent.server.serf.wan: serf: messageJoinType: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 +32:41.681 [INFO] TestAgent: Requesting shutdown +32:41.681 [INFO] TestAgent.server: shutting down server +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:41.681 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:41.681 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:41.681 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:41.681 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:41.686 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:41.689Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:41.689Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:41.689Z [INFO] agent.router.manager: shutting down +32:41.689 [INFO] TestAgent: consul server down +32:41.689 [INFO] TestAgent: shutdown complete +32:41.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23323 network=tcp +32:41.689 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23323 network=udp +32:41.689 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23324 network=tcp +32:42.192 [INFO] TestAgent: Waiting for endpoints to shut down +32:42.192 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:39.802Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:39.925Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:39.990 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 Address:127.0.0.1:23279}]" +32:39.998 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502.dc1 127.0.0.1 +32:39.999 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 127.0.0.1 +2021-10-05T16:32:39.999Z [INFO] agent.router: Initializing LAN area manager +32:39.999 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp +32:39.999 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= +32:40.000 [INFO] TestAgent.server: Adding LAN server: server="Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 (Addr: tcp/127.0.0.1:23279) (DC: dc1)" +32:40.000 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502.dc1 area=wan +32:40.000 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp +32:40.000 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp +32:40.000 [INFO] TestAgent: started state syncer +32:40.001 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp +32:40.036 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:40.036 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 +32:40.050 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:40.050 [DEBUG] TestAgent.server.raft: vote granted: from=3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 term=2 tally=1 +32:40.050 [INFO] TestAgent.server.raft: election won: tally=1 +32:40.050 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" +32:40.050 [INFO] TestAgent.server: cluster leadership acquired +32:40.050 [INFO] TestAgent.server: New leader elected: payload=Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 +32:40.072 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 +32:40.093 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:40.123 [INFO] TestAgent: Synced node info +32:40.141 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:40.141 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:40.141 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:40.141 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:40.141 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:40.141 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 +32:40.141 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ae4ca34-099f-f8d4-66c7-f6c3a6c70502 +32:40.158 [INFO] TestAgent.server: federation state anti-entropy synced +32:40.713 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:42120 +32:40.714 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 127.0.0.1 +32:40.715 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 area=wan +32:40.718 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self?dc=dc2 from=127.0.0.1:36768 latency=2.1946ms +32:40.726 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/health/service/consul?dc=dc2&passing=1 from=127.0.0.1:36768 latency=2.555103ms +32:40.736 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create?dc=dc2 from=127.0.0.1:36768 latency=8.755921ms +32:40.740 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/job?acquire=0ea66e86-e192-53b2-1231-80778d011b83&dc=dc2 from=127.0.0.1:36768 latency=3.324799ms +32:40.774 [DEBUG] TestAgent.server.serf.wan: serf: messageJoinType: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 +32:40.948 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/event/fire/_rexec?dc=dc2 from=127.0.0.1:36768 latency=3.547323ms +32:40.953 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/?dc=dc2&keys=&wait=500ms from=127.0.0.1:36768 latency=484.643µs +32:40.970 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/?dc=dc2&index=14&keys=&wait=500ms from=127.0.0.1:36768 latency=16.389958ms +32:40.993 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/?dc=dc2&index=15&keys=&wait=500ms from=127.0.0.1:36768 latency=22.563252ms +32:41.005 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/Node-7a864460-268a-1a92-c2f3-ac27d97b8545/out/00000?dc=dc2 from=127.0.0.1:36768 latency=5.012547ms +32:41.011 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/?dc=dc2&index=16&keys=&wait=500ms from=127.0.0.1:36768 latency=505.354µs +32:41.011 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/Node-7a864460-268a-1a92-c2f3-ac27d97b8545/exit?dc=dc2 from=127.0.0.1:36768 latency=290.177µs +32:41.280 [DEBUG] TestAgent.server.serf.wan: serf: messageJoinType: Node-7a864460-268a-1a92-c2f3-ac27d97b8545.dc2 +32:41.521 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83/?dc=dc2&index=17&keys=&wait=500ms from=127.0.0.1:36768 latency=504.895149ms +32:41.669 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/0ea66e86-e192-53b2-1231-80778d011b83?dc=dc2 from=127.0.0.1:36780 latency=155.982109ms +32:41.674 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/_rexec/0ea66e86-e192-53b2-1231-80778d011b83?dc=dc2&recurse= from=127.0.0.1:36780 latency=4.35895ms +32:41.680 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/0ea66e86-e192-53b2-1231-80778d011b83?dc=dc2 from=127.0.0.1:36780 latency=5.614553ms +32:42.192 [INFO] TestAgent: Requesting shutdown +32:42.192 [INFO] TestAgent.server: shutting down server +32:42.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:42.192 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:42.192 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:42.192 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:42.192 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:42.192 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:42.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:42.192 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:42.192 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:42.192 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:42.204 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:42.212Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:42.212Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:42.212Z [INFO] agent.router.manager: shutting down +32:42.212 [INFO] TestAgent: consul server down +32:42.212 [INFO] TestAgent: shutdown complete +32:42.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp +32:42.212 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp +32:42.212 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp +32:42.716 [INFO] TestAgent: Waiting for endpoints to shut down +32:42.716 [INFO] TestAgent: Endpoints down +--- PASS: TestExecCommand_CrossDC (3.08s) PASS -ok github.com/hashicorp/consul/command/exec 2.536s +ok github.com/hashicorp/consul/command/exec 3.296s === RUN TestForceLeaveCommand_noTabs === PAUSE TestForceLeaveCommand_noTabs === RUN TestForceLeaveCommand @@ -84522,439 +84790,452 @@ === RUN TestForceLeaveCommand_noAddrs === PAUSE TestForceLeaveCommand_noAddrs === CONT TestForceLeaveCommand_noTabs +--- PASS: TestForceLeaveCommand_noTabs (0.00s) +=== CONT TestForceLeaveCommand_noAddrs +--- PASS: TestForceLeaveCommand_noAddrs (0.00s) === CONT TestForceLeaveCommand_prune +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestForceLeaveCommand_NoNodeWithName === CONT TestForceLeaveCommand -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestForceLeaveCommand_noAddrs ---- PASS: TestForceLeaveCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestForceLeaveCommand_noAddrs (0.00s) -2022-11-07T22:46:00.709Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.724Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.785 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e554e10d-d773-f925-e6fb-7e6f7bea7f2c Address:127.0.0.1:26349}]" -46:00.785 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= -46:00.786 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c.dc1 127.0.0.1 -46:00.786 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c 127.0.0.1 -2022-11-07T22:46:00.786Z [INFO] agent.router: Initializing LAN area manager -46:00.786 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp -46:00.786 [INFO] TestAgent.server: Adding LAN server: server="Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c (Addr: tcp/127.0.0.1:26349) (DC: dc1)" -46:00.786 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp -46:00.786 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c.dc1 area=wan -46:00.787 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp -46:00.787 [INFO] TestAgent: started state syncer -46:00.787 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp -46:00.838 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.838 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 -46:00.852 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.852 [DEBUG] TestAgent.server.raft: vote granted: from=e554e10d-d773-f925-e6fb-7e6f7bea7f2c term=2 tally=1 -46:00.852 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.852 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" -46:00.852 [INFO] TestAgent.server: cluster leadership acquired -46:00.852 [INFO] TestAgent.server: New leader elected: payload=Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c -46:00.861 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26349 -46:00.887 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.926 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.926 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.926 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.926 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.926 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.926 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c -46:00.926 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e554e10d-d773-f925-e6fb-7e6f7bea7f2c -46:00.938 [INFO] TestAgent.server: federation state anti-entropy synced -46:00.996 [INFO] TestAgent: Force leaving node: node=garbage-name -46:00.996 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/force-leave/garbage-name from=127.0.0.1:55908 error="agent: No node found with name 'garbage-name'" -46:00.998 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/garbage-name from=127.0.0.1:55908 latency=1.237485ms -46:00.998 [INFO] TestAgent: Requesting shutdown -46:00.998 [INFO] TestAgent.server: shutting down server -46:00.998 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:00.998 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:00.998 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:00.998 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:00.998 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:00.998 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:00.998 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:00.998 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:00.998 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:00.998 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:00.998 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:01.002 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.006Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.006Z [INFO] agent.router.manager: shutting down -46:01.006 [INFO] TestAgent: consul server down -46:01.006 [INFO] TestAgent: shutdown complete -46:01.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp -46:01.006 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp -46:01.006 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp -46:01.507 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.507 [INFO] TestAgent: Endpoints down ---- PASS: TestForceLeaveCommand_NoNodeWithName (0.84s) -2022-11-07T22:46:00.893Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.902Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.954 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bc7ba13-b56d-72d3-9a38-76400f512dd4 Address:127.0.0.1:26363}]" -46:00.954 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26363 [Follower]" leader= -46:00.955 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 127.0.0.1 -46:00.956 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 127.0.0.1 -2022-11-07T22:46:00.956Z [INFO] agent.router: Initializing LAN area manager -46:00.956 [INFO] TestAgent.server: Adding LAN server: server="Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (Addr: tcp/127.0.0.1:26363) (DC: dc1)" -46:00.956 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=tcp -46:00.956 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=udp -46:00.956 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 area=wan -46:00.957 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26359 network=tcp -46:00.957 [INFO] TestAgent: started state syncer -46:00.957 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26364 network=tcp -46:00.996 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.996 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26363 [Candidate]" term=2 -46:01.008 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:01.008 [DEBUG] TestAgent.server.raft: vote granted: from=9bc7ba13-b56d-72d3-9a38-76400f512dd4 term=2 tally=1 -46:01.008 [INFO] TestAgent.server.raft: election won: tally=1 -46:01.008 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26363 [Leader]" -46:01.008 [INFO] TestAgent.server: cluster leadership acquired -46:01.009 [INFO] TestAgent.server: New leader elected: payload=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 -46:01.016 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26363 -46:01.019 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:26354] -46:01.020 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:26354 -46:01.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 127.0.0.1 -46:01.021 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:01.021 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:01.021 [INFO] TestAgent: Requesting shutdown -46:01.021 [INFO] TestAgent.server: shutting down server -46:01.021 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:01.021 [INFO] TestAgent.server: Adding LAN server: server="Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 (Addr: tcp/127.0.0.1:26356) (DC: dc1)" -46:01.021 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:01.021 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:55568 -46:01.022 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 127.0.0.1:26355 -46:01.024 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 127.0.0.1 -46:01.024 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 area=wan -46:01.025 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 address=127.0.0.1:26355 -46:01.029 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:01.036 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:01.037Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.037Z [INFO] agent.router.manager: shutting down -46:01.045 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:01.045 [INFO] TestAgent: consul server down -46:01.045 [INFO] TestAgent: shutdown complete -46:01.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=tcp -46:01.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=udp -46:01.045 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26359 network=tcp -46:01.545 [INFO] TestAgent: Waiting for endpoints to shut down -46:01.545 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:00.714Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.729Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.782 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 Address:127.0.0.1:26356}]" -46:00.782 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26356 [Follower]" leader= -46:00.783 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 127.0.0.1 -46:00.783 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 127.0.0.1 -2022-11-07T22:46:00.783Z [INFO] agent.router: Initializing LAN area manager -46:00.783 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0.dc1 area=wan -46:00.783 [INFO] TestAgent.server: Adding LAN server: server="Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 (Addr: tcp/127.0.0.1:26356) (DC: dc1)" -46:00.783 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=udp -46:00.783 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=tcp -46:00.784 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26352 network=tcp -46:00.784 [INFO] TestAgent: started state syncer -46:00.784 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26357 network=tcp -46:00.829 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.829 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26356 [Candidate]" term=2 -46:00.847 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:00.847 [DEBUG] TestAgent.server.raft: vote granted: from=3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 term=2 tally=1 -46:00.847 [INFO] TestAgent.server.raft: election won: tally=1 -46:00.847 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26356 [Leader]" -46:00.847 [INFO] TestAgent.server: cluster leadership acquired -46:00.847 [INFO] TestAgent.server: New leader elected: payload=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 -46:00.857 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26356 -46:00.887 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.930 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.930 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:00.930 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:00.930 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:00.930 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:00.930 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 -46:00.930 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 -46:00.947 [INFO] TestAgent: Synced node info -46:00.947 [DEBUG] TestAgent: Node info in sync -46:00.957 [INFO] TestAgent.server: federation state anti-entropy synced -46:01.020 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:42336 -46:01.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 127.0.0.1 -46:01.021 [INFO] TestAgent.server: Adding LAN server: server="Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (Addr: tcp/127.0.0.1:26363) (DC: dc1)" -46:01.021 [INFO] TestAgent.server: New leader elected: payload=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 -46:01.021 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 other=Node-3e55aa55-c840-3f97-78ea-3d74ea8e2cd0 -46:01.021 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 -46:01.021 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 127.0.0.1:26362 -46:01.022 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:49232 -46:01.023 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 127.0.0.1 -46:01.023 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 area=wan -46:01.024 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4.dc1 address=127.0.0.1:26362 -46:01.546 [INFO] TestAgent: Force leaving node: node=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 -46:01.546 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 from=127.0.0.1:52900 latency=23.667µs -46:01.708 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:01.711 [INFO] TestAgent: Synced node info -46:02.283 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (timeout reached) -46:02.783 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 has failed, no acks received -46:02.870 [WARN] TestAgent: error getting server health from server: server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:26363: connect: connection refused" -46:02.921 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:02.921 [DEBUG] TestAgent: Node info in sync -46:03.870 [WARN] TestAgent: error getting server health from server: server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 error="context deadline exceeded" -46:04.283 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (timeout reached) -46:04.870 [WARN] TestAgent: error getting server health from server: server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:26363: connect: connection refused" -46:05.783 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 has failed, no acks received -46:05.870 [WARN] TestAgent: error getting server health from server: server=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 error="context deadline exceeded" -46:06.783 [INFO] TestAgent.server.memberlist.lan: memberlist: Marking Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 as failed, suspect timeout reached (0 peer confirmations) -46:06.783 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 127.0.0.1 -46:06.783 [INFO] TestAgent.server: Removing LAN server: server="Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (Addr: tcp/127.0.0.1:26363) (DC: dc1)" -46:06.783 [INFO] TestAgent.server: deregistering member: member=Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 reason=left -46:06.784 [INFO] TestAgent: Requesting shutdown -46:06.784 [INFO] TestAgent.server: shutting down server -46:06.784 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:06.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:06.784 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:06.784 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:06.784 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:06.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:06.784 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:06.784 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:06.784 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:06.784 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:06.789 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:06.799Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:06.799Z [INFO] agent.router.manager: shutting down -46:06.799 [INFO] TestAgent: consul server down -46:06.799 [INFO] TestAgent: shutdown complete -46:06.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=tcp -46:06.799 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=udp -46:06.799 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26352 network=tcp -46:07.284 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-9bc7ba13-b56d-72d3-9a38-76400f512dd4 (timeout reached) -46:07.299 [INFO] TestAgent: Waiting for endpoints to shut down -46:07.299 [INFO] TestAgent: Endpoints down ---- PASS: TestForceLeaveCommand (6.63s) -2022-11-07T22:46:01.070Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:01.079Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:01.104 [INFO] Agent2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:38b03adc-1239-366e-47d6-ff2992099317 Address:127.0.0.1:26370}]" -46:01.104 [INFO] Agent2.server.raft: entering follower state: follower="Node at 127.0.0.1:26370 [Follower]" leader= -46:01.105 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 127.0.0.1 -46:01.105 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-38b03adc-1239-366e-47d6-ff2992099317 127.0.0.1 -2022-11-07T22:46:01.105Z [INFO] agent.router: Initializing LAN area manager -46:01.106 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 area=wan -46:01.106 [INFO] Agent2.server: Adding LAN server: server="Node-38b03adc-1239-366e-47d6-ff2992099317 (Addr: tcp/127.0.0.1:26370) (DC: dc1)" -46:01.106 [INFO] Agent2: Started DNS server: address=127.0.0.1:26365 network=udp -46:01.106 [INFO] Agent2: Started DNS server: address=127.0.0.1:26365 network=tcp -46:01.106 [INFO] Agent2: Started HTTP server: address=127.0.0.1:26366 network=tcp -46:01.107 [INFO] Agent2: started state syncer -46:01.107 [INFO] Agent2: Started gRPC server: address=127.0.0.1:26371 network=tcp -46:01.153 [WARN] Agent2.server.raft: heartbeat timeout reached, starting election: last-leader= -46:01.153 [INFO] Agent2.server.raft: entering candidate state: node="Node at 127.0.0.1:26370 [Candidate]" term=2 -46:01.162 [DEBUG] Agent2.server.raft: votes: needed=1 -46:01.162 [DEBUG] Agent2.server.raft: vote granted: from=38b03adc-1239-366e-47d6-ff2992099317 term=2 tally=1 -46:01.162 [INFO] Agent2.server.raft: election won: tally=1 -46:01.162 [INFO] Agent2.server.raft: entering leader state: leader="Node at 127.0.0.1:26370 [Leader]" -46:01.162 [INFO] Agent2.server: cluster leadership acquired -46:01.162 [INFO] Agent2.server: New leader elected: payload=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:01.167 [DEBUG] Agent2.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26370 -46:01.177 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:01.191 [INFO] Agent2: Synced node info -46:01.198 [INFO] Agent2.server.connect: initialized primary datacenter CA with provider: provider=consul -46:01.198 [INFO] Agent2.leader: started routine: routine="federation state anti-entropy" -46:01.198 [INFO] Agent2.leader: started routine: routine="federation state pruning" -46:01.198 [INFO] Agent2.leader: started routine: routine="intermediate cert renew watch" -46:01.198 [INFO] Agent2.leader: started routine: routine="CA root pruning" -46:01.198 [DEBUG] Agent2.server: Skipping self join check for node since the cluster is too small: node=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:01.198 [INFO] Agent2.server: member joined, marking health alive: member=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:01.208 [INFO] Agent2.server: federation state anti-entropy synced -46:01.373 [INFO] Agent2: (LAN) joining: lan_addresses=[127.0.0.1:26340] -46:01.373 [DEBUG] Agent2.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:26340 -46:01.374 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 127.0.0.1 -46:01.374 [INFO] Agent2: (LAN) joined: number_of_nodes=1 -46:01.374 [DEBUG] Agent2: systemd notify failed: error="No socket" -46:01.374 [INFO] Agent2: Requesting shutdown -46:01.374 [INFO] Agent2.server: shutting down server -46:01.374 [DEBUG] Agent2.leader: stopping routine: routine="federation state anti-entropy" -46:01.374 [DEBUG] Agent2.leader: stopping routine: routine="federation state pruning" -46:01.374 [DEBUG] Agent2.leader: stopping routine: routine="intermediate cert renew watch" -46:01.374 [DEBUG] Agent2.leader: stopping routine: routine="CA root pruning" -46:01.374 [WARN] Agent2.server.serf.lan: serf: Shutdown without a Leave -46:01.374 [ERROR] Agent2.server: error performing anti-entropy sync of federation state: error="context canceled" -46:01.374 [DEBUG] Agent2.leader: stopped routine: routine="federation state anti-entropy" -46:01.374 [DEBUG] Agent2.leader: stopped routine: routine="federation state pruning" -46:01.374 [DEBUG] Agent2.leader: stopped routine: routine="intermediate cert renew watch" -46:01.374 [DEBUG] Agent2.leader: stopped routine: routine="CA root pruning" -46:01.374 [DEBUG] Agent2.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:52576 -46:01.375 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-cfde92e0-0790-1f02-162e-6c7fdde48de3.dc1 127.0.0.1 -46:01.375 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3.dc1 area=wan -46:01.379 [WARN] Agent2.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:01.385Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:01.385Z [INFO] agent.router.manager: shutting down -46:01.385 [INFO] Agent2: consul server down -46:01.385 [INFO] Agent2: shutdown complete -46:01.385 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:26365 network=tcp -46:01.385 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:26365 network=udp -46:01.385 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:26366 network=tcp -46:01.885 [INFO] Agent2: Waiting for endpoints to shut down -46:01.885 [INFO] Agent2: Endpoints down -2022-11-07T22:46:00.711Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:00.723Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:00.779 [INFO] Agent1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cfde92e0-0790-1f02-162e-6c7fdde48de3 Address:127.0.0.1:26342}]" -46:00.779 [INFO] Agent1.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= -46:00.779 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-cfde92e0-0790-1f02-162e-6c7fdde48de3.dc1 127.0.0.1 -46:00.780 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 127.0.0.1 -2022-11-07T22:46:00.780Z [INFO] agent.router: Initializing LAN area manager -46:00.780 [INFO] Agent1.server: Adding LAN server: server="Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" -46:00.780 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3.dc1 area=wan -46:00.780 [INFO] Agent1: Started DNS server: address=127.0.0.1:26337 network=tcp -46:00.781 [INFO] Agent1: Started DNS server: address=127.0.0.1:26337 network=udp -46:00.781 [INFO] Agent1: Started HTTP server: address=127.0.0.1:26338 network=tcp -46:00.781 [INFO] Agent1: started state syncer -46:00.781 [INFO] Agent1: Started gRPC server: address=127.0.0.1:26343 network=tcp -46:00.832 [WARN] Agent1.server.raft: heartbeat timeout reached, starting election: last-leader= -46:00.832 [INFO] Agent1.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 -46:00.854 [DEBUG] Agent1.server.raft: votes: needed=1 -46:00.854 [DEBUG] Agent1.server.raft: vote granted: from=cfde92e0-0790-1f02-162e-6c7fdde48de3 term=2 tally=1 -46:00.854 [INFO] Agent1.server.raft: election won: tally=1 -46:00.854 [INFO] Agent1.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" -46:00.854 [INFO] Agent1.server: cluster leadership acquired -46:00.854 [INFO] Agent1.server: New leader elected: payload=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 -46:00.872 [DEBUG] Agent1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26342 -46:00.893 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:00.927 [INFO] Agent1: Synced node info -46:00.947 [INFO] Agent1.server.connect: initialized primary datacenter CA with provider: provider=consul -46:00.947 [INFO] Agent1.leader: started routine: routine="federation state anti-entropy" -46:00.947 [INFO] Agent1.leader: started routine: routine="federation state pruning" -46:00.947 [INFO] Agent1.leader: started routine: routine="intermediate cert renew watch" -46:00.947 [INFO] Agent1.leader: started routine: routine="CA root pruning" -46:00.947 [DEBUG] Agent1.server: Skipping self join check for node since the cluster is too small: node=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 -46:00.947 [INFO] Agent1.server: member joined, marking health alive: member=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 -46:00.957 [INFO] Agent1.server: federation state anti-entropy synced -46:01.373 [DEBUG] Agent1.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:49508 -46:01.374 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-38b03adc-1239-366e-47d6-ff2992099317 127.0.0.1 -46:01.374 [INFO] Agent1.server: Adding LAN server: server="Node-38b03adc-1239-366e-47d6-ff2992099317 (Addr: tcp/127.0.0.1:26370) (DC: dc1)" -46:01.374 [INFO] Agent1.server: New leader elected: payload=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:01.374 [ERROR] Agent1.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-38b03adc-1239-366e-47d6-ff2992099317 other=Node-cfde92e0-0790-1f02-162e-6c7fdde48de3 -46:01.374 [INFO] Agent1.server: member joined, marking health alive: member=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:01.374 [DEBUG] Agent1.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 127.0.0.1:26369 -46:01.375 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 127.0.0.1 -46:01.375 [DEBUG] Agent1.server: Successfully performed flood-join for server at address: server=Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 address=127.0.0.1:26369 -46:01.375 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 area=wan -46:01.886 [INFO] Agent1: Force leaving node: node=Node-38b03adc-1239-366e-47d6-ff2992099317 -46:02.280 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-38b03adc-1239-366e-47d6-ff2992099317 (timeout reached) -46:02.780 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-38b03adc-1239-366e-47d6-ff2992099317 has failed, no acks received -46:04.280 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-38b03adc-1239-366e-47d6-ff2992099317 (timeout reached) -46:05.780 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-38b03adc-1239-366e-47d6-ff2992099317 has failed, no acks received -46:06.780 [INFO] Agent1.server.memberlist.lan: memberlist: Marking Node-38b03adc-1239-366e-47d6-ff2992099317 as failed, suspect timeout reached (0 peer confirmations) -46:07.281 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-38b03adc-1239-366e-47d6-ff2992099317 (timeout reached) -46:08.780 [DEBUG] Agent1.server.memberlist.wan: memberlist: Failed ping: Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 (timeout reached) -46:09.886 [INFO] Agent1.server.serf.lan: serf: EventMemberReap (forced): Node-38b03adc-1239-366e-47d6-ff2992099317 127.0.0.1 -46:09.887 [INFO] Agent1.server: Removing LAN server: server="Node-38b03adc-1239-366e-47d6-ff2992099317 (Addr: tcp/127.0.0.1:26370) (DC: dc1)" -46:09.887 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-38b03adc-1239-366e-47d6-ff2992099317 has failed, no acks received -46:09.887 [INFO] Agent1.server: deregistering member: member=Node-38b03adc-1239-366e-47d6-ff2992099317 reason=reaped -46:09.887 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth -46:09.887 [DEBUG] Agent1: Node info in sync -46:09.887 [DEBUG] Agent1: Node info in sync -46:10.197 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth -46:10.197 [DEBUG] Agent1: Node info in sync -46:10.780 [INFO] Agent1.server.memberlist.wan: memberlist: Suspect Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 has failed, no acks received -46:17.887 [INFO] Agent1.server.serf.wan: serf: EventMemberReap (forced): Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 127.0.0.1 -46:17.887 [DEBUG] Agent1.http: Request finished: method=PUT url=/v1/agent/force-leave/Node-38b03adc-1239-366e-47d6-ff2992099317?prune=1 from=127.0.0.1:57888 latency=16.000494065s -46:17.887 [INFO] Agent1.server: Handled event for server in area: event=member-reap server=Node-38b03adc-1239-366e-47d6-ff2992099317.dc1 area=wan -46:17.887 [INFO] Agent1: Requesting shutdown -46:17.887 [INFO] Agent1.server: shutting down server -46:17.887 [DEBUG] Agent1.leader: stopping routine: routine="federation state anti-entropy" -46:17.887 [DEBUG] Agent1.leader: stopping routine: routine="federation state pruning" -46:17.887 [DEBUG] Agent1.leader: stopping routine: routine="intermediate cert renew watch" -46:17.887 [DEBUG] Agent1.leader: stopping routine: routine="CA root pruning" -46:17.887 [WARN] Agent1.server.serf.lan: serf: Shutdown without a Leave -46:17.887 [ERROR] Agent1.server: error performing anti-entropy sync of federation state: error="context canceled" -46:17.887 [DEBUG] Agent1.leader: stopped routine: routine="CA root pruning" -46:17.887 [DEBUG] Agent1.leader: stopped routine: routine="federation state anti-entropy" -46:17.887 [DEBUG] Agent1.leader: stopped routine: routine="federation state pruning" -46:17.887 [DEBUG] Agent1.leader: stopped routine: routine="intermediate cert renew watch" -46:17.896 [WARN] Agent1.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:17.906Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:17.906Z [INFO] agent.router.manager: shutting down -46:17.906 [INFO] Agent1: consul server down -46:17.906 [INFO] Agent1: shutdown complete -46:17.906 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp -46:17.906 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp -46:17.906 [INFO] Agent1: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp -46:18.406 [INFO] Agent1: Waiting for endpoints to shut down -46:18.406 [INFO] Agent1: Endpoints down ---- PASS: TestForceLeaveCommand_prune (17.74s) +2021-10-05T16:32:50.336Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:50.420Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:50.542 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0bff7d34-5243-cbd4-31d5-6f2a232f886e Address:127.0.0.1:19202}]" +32:50.542 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19202 [Follower]" leader= +32:50.542 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e.dc1 127.0.0.1 +32:50.543 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e 127.0.0.1 +2021-10-05T16:32:50.543Z [INFO] agent.router: Initializing LAN area manager +32:50.543 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=udp +32:50.543 [INFO] TestAgent.server: Adding LAN server: server="Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e (Addr: tcp/127.0.0.1:19202) (DC: dc1)" +32:50.543 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e.dc1 area=wan +32:50.544 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=tcp +32:50.552 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19198 network=tcp +32:50.552 [INFO] TestAgent: started state syncer +32:50.552 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19203 network=tcp +32:50.582 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:50.582 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19202 [Candidate]" term=2 +32:50.624 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:50.624 [DEBUG] TestAgent.server.raft: vote granted: from=0bff7d34-5243-cbd4-31d5-6f2a232f886e term=2 tally=1 +32:50.624 [INFO] TestAgent.server.raft: election won: tally=1 +32:50.624 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19202 [Leader]" +32:50.624 [INFO] TestAgent.server: cluster leadership acquired +32:50.624 [INFO] TestAgent.server: New leader elected: payload=Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e +32:50.638 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19202 +32:50.654 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:50.683 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:50.683 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:50.683 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:50.683 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:50.683 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:50.683 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e +32:50.683 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0bff7d34-5243-cbd4-31d5-6f2a232f886e +32:50.697 [INFO] TestAgent.server: federation state anti-entropy synced +32:50.892 [INFO] TestAgent: Force leaving node: node=garbage-name +32:50.892 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/agent/force-leave/garbage-name from=127.0.0.1:53500 error="agent: No node found with name 'garbage-name'" +32:50.893 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/garbage-name from=127.0.0.1:53500 latency=962.275µs +32:50.896 [INFO] TestAgent: Requesting shutdown +32:50.896 [INFO] TestAgent.server: shutting down server +32:50.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:50.896 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:50.896 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:50.896 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:50.896 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:50.896 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:50.896 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:50.897 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:50.897 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:50.897 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:50.897 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:50.912 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:50.920Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:50.920Z [INFO] agent.router.manager: shutting down +32:50.920 [INFO] TestAgent: consul server down +32:50.920 [INFO] TestAgent: shutdown complete +32:50.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=tcp +32:50.920 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=udp +32:50.920 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19198 network=tcp +32:51.421 [INFO] TestAgent: Waiting for endpoints to shut down +32:51.421 [INFO] TestAgent: Endpoints down +--- PASS: TestForceLeaveCommand_NoNodeWithName (1.17s) +2021-10-05T16:32:50.929Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:50.993Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:51.081 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1a4bff87-3311-01d9-8f26-927cd155c523 Address:127.0.0.1:19223}]" +32:51.082 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 127.0.0.1 +32:51.082 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a4bff87-3311-01d9-8f26-927cd155c523 127.0.0.1 +2021-10-05T16:32:51.082Z [INFO] agent.router: Initializing LAN area manager +32:51.082 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19218 network=udp +32:51.082 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19223 [Follower]" leader= +32:51.083 [INFO] TestAgent.server: Adding LAN server: server="Node-1a4bff87-3311-01d9-8f26-927cd155c523 (Addr: tcp/127.0.0.1:19223) (DC: dc1)" +32:51.083 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 area=wan +32:51.083 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19218 network=tcp +32:51.083 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19219 network=tcp +32:51.084 [INFO] TestAgent: started state syncer +32:51.084 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19224 network=tcp +32:51.135 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:51.135 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19223 [Candidate]" term=2 +32:51.149 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:51.149 [DEBUG] TestAgent.server.raft: vote granted: from=1a4bff87-3311-01d9-8f26-927cd155c523 term=2 tally=1 +32:51.150 [INFO] TestAgent.server.raft: election won: tally=1 +32:51.150 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19223 [Leader]" +32:51.151 [INFO] TestAgent.server: cluster leadership acquired +32:51.151 [INFO] TestAgent.server: New leader elected: payload=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.177 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19223 +32:51.207 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:51.218 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:19207] +32:51.224 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:19207 +32:51.229 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:51.229 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:51.229 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:51.229 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:51.229 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:51.229 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.229 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.232 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:50416 +32:51.236 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4add2533-d712-c1c1-59e0-3dcfff85cc99.dc1 127.0.0.1 +32:51.236 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99.dc1 area=wan +32:51.241 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 127.0.0.1 +32:51.241 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +32:51.241 [DEBUG] TestAgent: systemd notify failed: error="No socket" +32:51.241 [INFO] TestAgent: Requesting shutdown +32:51.241 [INFO] TestAgent.server: shutting down server +32:51.241 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:51.241 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:51.241 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:51.241 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:51.241 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:51.241 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:51.241 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:51.241 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:51.241 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:51.248 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:51.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:51.252 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:51.260Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:51.260Z [INFO] agent.router.manager: shutting down +32:51.260 [INFO] TestAgent: consul server down +32:51.260 [INFO] TestAgent: shutdown complete +32:51.260 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19218 network=tcp +32:51.260 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19218 network=udp +32:51.260 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19219 network=tcp +32:51.764 [INFO] TestAgent: Waiting for endpoints to shut down +32:51.764 [INFO] TestAgent: Endpoints down +2021-10-05T16:32:50.333Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:50.358Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:50.468 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4add2533-d712-c1c1-59e0-3dcfff85cc99 Address:127.0.0.1:19209}]" +32:50.469 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4add2533-d712-c1c1-59e0-3dcfff85cc99.dc1 127.0.0.1 +32:50.469 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19209 [Follower]" leader= +32:50.475 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 127.0.0.1 +2021-10-05T16:32:50.475Z [INFO] agent.router: Initializing LAN area manager +32:50.475 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19204 network=udp +32:50.475 [INFO] TestAgent.server: Adding LAN server: server="Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 (Addr: tcp/127.0.0.1:19209) (DC: dc1)" +32:50.475 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99.dc1 area=wan +32:50.476 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19204 network=tcp +32:50.476 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19205 network=tcp +32:50.476 [INFO] TestAgent: started state syncer +32:50.476 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19210 network=tcp +32:50.536 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:50.536 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19209 [Candidate]" term=2 +32:50.584 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:50.584 [DEBUG] TestAgent.server.raft: vote granted: from=4add2533-d712-c1c1-59e0-3dcfff85cc99 term=2 tally=1 +32:50.584 [INFO] TestAgent.server.raft: election won: tally=1 +32:50.584 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19209 [Leader]" +32:50.588 [INFO] TestAgent.server: cluster leadership acquired +32:50.588 [INFO] TestAgent.server: New leader elected: payload=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 +32:50.612 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19209 +32:50.663 [INFO] TestAgent: Synced node info +32:50.663 [DEBUG] TestAgent: Node info in sync +32:50.663 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:50.686 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:50.686 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:50.686 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:50.686 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:50.686 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:50.686 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 +32:50.686 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 +32:50.701 [INFO] TestAgent.server: federation state anti-entropy synced +32:51.224 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:50072 +32:51.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1a4bff87-3311-01d9-8f26-927cd155c523 127.0.0.1 +32:51.229 [INFO] TestAgent.server: Adding LAN server: server="Node-1a4bff87-3311-01d9-8f26-927cd155c523 (Addr: tcp/127.0.0.1:19223) (DC: dc1)" +32:51.229 [INFO] TestAgent.server: New leader elected: payload=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.229 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-1a4bff87-3311-01d9-8f26-927cd155c523 other=Node-4add2533-d712-c1c1-59e0-3dcfff85cc99 +32:51.229 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.232 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 127.0.0.1:19222 +32:51.240 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 127.0.0.1 +32:51.240 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 address=127.0.0.1:19222 +32:51.240 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1a4bff87-3311-01d9-8f26-927cd155c523.dc1 area=wan +32:51.768 [INFO] TestAgent: Force leaving node: node=Node-1a4bff87-3311-01d9-8f26-927cd155c523 +32:51.768 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/force-leave/Node-1a4bff87-3311-01d9-8f26-927cd155c523 from=127.0.0.1:39116 latency=30.143µs +32:51.981 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-1a4bff87-3311-01d9-8f26-927cd155c523 (timeout reached) +32:52.484 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-1a4bff87-3311-01d9-8f26-927cd155c523 has failed, no acks received +32:52.628 [WARN] TestAgent: error getting server health from server: server=Node-1a4bff87-3311-01d9-8f26-927cd155c523 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:19223: connect: connection refused" +32:53.138 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:53.138 [DEBUG] TestAgent: Node info in sync +32:53.628 [WARN] TestAgent: error getting server health from server: server=Node-1a4bff87-3311-01d9-8f26-927cd155c523 error="context deadline exceeded" +32:53.980 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-1a4bff87-3311-01d9-8f26-927cd155c523 (timeout reached) +32:54.088 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:54.088 [DEBUG] TestAgent: Node info in sync +32:54.628 [WARN] TestAgent: error getting server health from server: server=Node-1a4bff87-3311-01d9-8f26-927cd155c523 error="rpc error getting client: failed to get conn: dial tcp 127.0.0.1:0->127.0.0.1:19223: connect: connection refused" +32:55.480 [INFO] TestAgent.server.memberlist.lan: memberlist: Suspect Node-1a4bff87-3311-01d9-8f26-927cd155c523 has failed, no acks received +32:55.629 [WARN] TestAgent: error getting server health from server: server=Node-1a4bff87-3311-01d9-8f26-927cd155c523 error="context deadline exceeded" +32:56.488 [INFO] TestAgent.server.memberlist.lan: memberlist: Marking Node-1a4bff87-3311-01d9-8f26-927cd155c523 as failed, suspect timeout reached (0 peer confirmations) +32:56.488 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-1a4bff87-3311-01d9-8f26-927cd155c523 127.0.0.1 +32:56.488 [INFO] TestAgent.server: Removing LAN server: server="Node-1a4bff87-3311-01d9-8f26-927cd155c523 (Addr: tcp/127.0.0.1:19223) (DC: dc1)" +32:56.488 [INFO] TestAgent.server: deregistering member: member=Node-1a4bff87-3311-01d9-8f26-927cd155c523 reason=left +32:56.508 [INFO] TestAgent: Requesting shutdown +32:56.508 [INFO] TestAgent.server: shutting down server +32:56.508 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:56.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:56.508 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:56.508 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:56.508 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:56.508 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:56.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:56.508 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:56.508 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:56.508 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:56.520 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:56.520Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:56.520Z [INFO] agent.router.manager: shutting down +32:56.520 [INFO] TestAgent: consul server down +32:56.520 [INFO] TestAgent: shutdown complete +32:56.520 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19204 network=tcp +32:56.520 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19204 network=udp +32:56.520 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19205 network=tcp +32:56.977 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-1a4bff87-3311-01d9-8f26-927cd155c523 (timeout reached) +32:57.021 [INFO] TestAgent: Waiting for endpoints to shut down +32:57.021 [INFO] TestAgent: Endpoints down +--- PASS: TestForceLeaveCommand (6.77s) +2021-10-05T16:32:50.801Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:50.831Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:50.920 [INFO] Agent2.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d9f3d23b-bfde-515d-8446-6415a9580675 Address:127.0.0.1:19216}]" +32:50.921 [INFO] Agent2.server.raft: entering follower state: follower="Node at 127.0.0.1:19216 [Follower]" leader= +32:50.921 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 127.0.0.1 +32:50.921 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-d9f3d23b-bfde-515d-8446-6415a9580675 127.0.0.1 +2021-10-05T16:32:50.921Z [INFO] agent.router: Initializing LAN area manager +32:50.922 [INFO] Agent2: Started DNS server: address=127.0.0.1:19211 network=udp +32:50.922 [INFO] Agent2.server: Handled event for server in area: event=member-join server=Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 area=wan +32:50.922 [INFO] Agent2.server: Adding LAN server: server="Node-d9f3d23b-bfde-515d-8446-6415a9580675 (Addr: tcp/127.0.0.1:19216) (DC: dc1)" +32:50.922 [INFO] Agent2: Started DNS server: address=127.0.0.1:19211 network=tcp +32:50.922 [INFO] Agent2: Started HTTP server: address=127.0.0.1:19212 network=tcp +32:50.922 [INFO] Agent2: started state syncer +32:50.922 [INFO] Agent2: Started gRPC server: address=127.0.0.1:19217 network=tcp +32:50.960 [WARN] Agent2.server.raft: heartbeat timeout reached, starting election: last-leader= +32:50.960 [INFO] Agent2.server.raft: entering candidate state: node="Node at 127.0.0.1:19216 [Candidate]" term=2 +32:51.028 [DEBUG] Agent2.server.raft: votes: needed=1 +32:51.028 [DEBUG] Agent2.server.raft: vote granted: from=d9f3d23b-bfde-515d-8446-6415a9580675 term=2 tally=1 +32:51.028 [INFO] Agent2.server.raft: election won: tally=1 +32:51.028 [INFO] Agent2.server.raft: entering leader state: leader="Node at 127.0.0.1:19216 [Leader]" +32:51.029 [INFO] Agent2.server: cluster leadership acquired +32:51.029 [INFO] Agent2.server: New leader elected: payload=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:51.057 [DEBUG] Agent2.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19216 +32:51.108 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:51.150 [INFO] Agent2.server.connect: initialized primary datacenter CA with provider: provider=consul +32:51.150 [INFO] Agent2.leader: started routine: routine="federation state anti-entropy" +32:51.150 [INFO] Agent2.leader: started routine: routine="federation state pruning" +32:51.150 [INFO] Agent2.leader: started routine: routine="intermediate cert renew watch" +32:51.150 [INFO] Agent2.leader: started routine: routine="CA root pruning" +32:51.150 [DEBUG] Agent2.server: Skipping self join check for node since the cluster is too small: node=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:51.150 [INFO] Agent2.server: member joined, marking health alive: member=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:51.185 [INFO] Agent2.server: federation state anti-entropy synced +32:51.192 [INFO] Agent2: (LAN) joining: lan_addresses=[127.0.0.1:19193] +32:51.196 [DEBUG] Agent2.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:19193 +32:51.198 [INFO] Agent2.server.serf.lan: serf: EventMemberJoin: Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 127.0.0.1 +32:51.198 [INFO] Agent2: (LAN) joined: number_of_nodes=1 +32:51.198 [DEBUG] Agent2: systemd notify failed: error="No socket" +32:51.198 [INFO] Agent2: Requesting shutdown +32:51.198 [INFO] Agent2.server: shutting down server +32:51.198 [DEBUG] Agent2.leader: stopping routine: routine="federation state pruning" +32:51.198 [DEBUG] Agent2.leader: stopping routine: routine="intermediate cert renew watch" +32:51.198 [DEBUG] Agent2.leader: stopping routine: routine="CA root pruning" +32:51.198 [DEBUG] Agent2.leader: stopping routine: routine="federation state anti-entropy" +32:51.198 [WARN] Agent2.server.serf.lan: serf: Shutdown without a Leave +32:51.198 [ERROR] Agent2.server: error performing anti-entropy sync of federation state: error="context canceled" +32:51.198 [DEBUG] Agent2.leader: stopped routine: routine="federation state anti-entropy" +32:51.198 [ERROR] Agent2.anti_entropy: failed to sync remote state: error="No cluster leader" +32:51.198 [DEBUG] Agent2.leader: stopped routine: routine="federation state pruning" +32:51.198 [DEBUG] Agent2.leader: stopped routine: routine="intermediate cert renew watch" +32:51.198 [DEBUG] Agent2.leader: stopped routine: routine="CA root pruning" +32:51.199 [DEBUG] Agent2.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:50494 +32:51.204 [WARN] Agent2.server.serf.wan: serf: Shutdown without a Leave +32:51.205 [INFO] Agent2.server.serf.wan: serf: EventMemberJoin: Node-daa82dd6-56d4-0bbd-9f53-badbdf966324.dc1 127.0.0.1 +2021-10-05T16:32:51.212Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:51.212Z [INFO] agent.router.manager: shutting down +32:51.212 [INFO] Agent2: consul server down +32:51.212 [INFO] Agent2: shutdown complete +32:51.212 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:19211 network=tcp +32:51.212 [INFO] Agent2: Stopping server: protocol=DNS address=127.0.0.1:19211 network=udp +32:51.212 [INFO] Agent2: Stopping server: protocol=HTTP address=127.0.0.1:19212 network=tcp +32:51.712 [INFO] Agent2: Waiting for endpoints to shut down +32:51.712 [INFO] Agent2: Endpoints down +2021-10-05T16:32:50.328Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:50.412Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:50.495 [INFO] Agent1.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:daa82dd6-56d4-0bbd-9f53-badbdf966324 Address:127.0.0.1:19195}]" +32:50.496 [INFO] Agent1.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= +32:50.496 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-daa82dd6-56d4-0bbd-9f53-badbdf966324.dc1 127.0.0.1 +32:50.497 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 127.0.0.1 +2021-10-05T16:32:50.498Z [INFO] agent.router: Initializing LAN area manager +32:50.498 [INFO] Agent1: Started DNS server: address=127.0.0.1:19190 network=udp +32:50.498 [INFO] Agent1.server: Adding LAN server: server="Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 (Addr: tcp/127.0.0.1:19195) (DC: dc1)" +32:50.498 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-daa82dd6-56d4-0bbd-9f53-badbdf966324.dc1 area=wan +32:50.498 [INFO] Agent1: Started DNS server: address=127.0.0.1:19190 network=tcp +32:50.499 [INFO] Agent1: Started HTTP server: address=127.0.0.1:19191 network=tcp +32:50.536 [WARN] Agent1.server.raft: heartbeat timeout reached, starting election: last-leader= +32:50.536 [INFO] Agent1.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 +32:50.542 [INFO] Agent1: started state syncer +32:50.542 [INFO] Agent1: Started gRPC server: address=127.0.0.1:19196 network=tcp +32:50.557 [DEBUG] Agent1.server.raft: votes: needed=1 +32:50.557 [DEBUG] Agent1.server.raft: vote granted: from=daa82dd6-56d4-0bbd-9f53-badbdf966324 term=2 tally=1 +32:50.557 [INFO] Agent1.server.raft: election won: tally=1 +32:50.557 [INFO] Agent1.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" +32:50.557 [INFO] Agent1.server: cluster leadership acquired +32:50.558 [INFO] Agent1.server: New leader elected: payload=Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 +32:50.571 [DEBUG] Agent1.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19195 +32:50.583 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:50.640 [INFO] Agent1.server.connect: initialized primary datacenter CA with provider: provider=consul +32:50.640 [INFO] Agent1.leader: started routine: routine="federation state anti-entropy" +32:50.640 [INFO] Agent1.leader: started routine: routine="federation state pruning" +32:50.640 [INFO] Agent1.leader: started routine: routine="intermediate cert renew watch" +32:50.640 [INFO] Agent1.leader: started routine: routine="CA root pruning" +32:50.640 [DEBUG] Agent1.server: Skipping self join check for node since the cluster is too small: node=Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 +32:50.640 [INFO] Agent1.server: member joined, marking health alive: member=Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 +32:50.680 [INFO] Agent1.server: federation state anti-entropy synced +32:50.988 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth +32:51.012 [INFO] Agent1: Synced node info +32:51.012 [DEBUG] Agent1: Node info in sync +32:51.196 [DEBUG] Agent1.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:34870 +32:51.197 [INFO] Agent1.server.serf.lan: serf: EventMemberJoin: Node-d9f3d23b-bfde-515d-8446-6415a9580675 127.0.0.1 +32:51.197 [INFO] Agent1.server: Adding LAN server: server="Node-d9f3d23b-bfde-515d-8446-6415a9580675 (Addr: tcp/127.0.0.1:19216) (DC: dc1)" +32:51.197 [INFO] Agent1.server: New leader elected: payload=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:51.197 [ERROR] Agent1.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-d9f3d23b-bfde-515d-8446-6415a9580675 other=Node-daa82dd6-56d4-0bbd-9f53-badbdf966324 +32:51.197 [INFO] Agent1.server: member joined, marking health alive: member=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:51.199 [DEBUG] Agent1.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 127.0.0.1:19215 +32:51.206 [INFO] Agent1.server.serf.wan: serf: EventMemberJoin: Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 127.0.0.1 +32:51.206 [DEBUG] Agent1.server: Successfully performed flood-join for server at address: server=Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 address=127.0.0.1:19215 +32:51.206 [INFO] Agent1.server: Handled event for server in area: event=member-join server=Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 area=wan +32:51.388 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth +32:51.388 [DEBUG] Agent1: Node info in sync +32:51.716 [INFO] Agent1: Force leaving node: node=Node-d9f3d23b-bfde-515d-8446-6415a9580675 +32:52.008 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-d9f3d23b-bfde-515d-8446-6415a9580675 (timeout reached) +32:52.500 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-d9f3d23b-bfde-515d-8446-6415a9580675 has failed, no acks received +32:54.002 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-d9f3d23b-bfde-515d-8446-6415a9580675 (timeout reached) +32:55.500 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-d9f3d23b-bfde-515d-8446-6415a9580675 has failed, no acks received +32:56.504 [INFO] Agent1.server.memberlist.lan: memberlist: Marking Node-d9f3d23b-bfde-515d-8446-6415a9580675 as failed, suspect timeout reached (0 peer confirmations) +32:57.000 [DEBUG] Agent1.server.memberlist.lan: memberlist: Failed ping: Node-d9f3d23b-bfde-515d-8446-6415a9580675 (timeout reached) +32:58.504 [DEBUG] Agent1.server.memberlist.wan: memberlist: Failed ping: Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 (timeout reached) +32:59.720 [INFO] Agent1.server.serf.lan: serf: EventMemberReap (forced): Node-d9f3d23b-bfde-515d-8446-6415a9580675 127.0.0.1 +32:59.720 [INFO] Agent1.server: Removing LAN server: server="Node-d9f3d23b-bfde-515d-8446-6415a9580675 (Addr: tcp/127.0.0.1:19216) (DC: dc1)" +32:59.720 [INFO] Agent1.server: deregistering member: member=Node-d9f3d23b-bfde-515d-8446-6415a9580675 reason=reaped +32:59.724 [DEBUG] Agent1: Skipping remote check since it is managed automatically: check=serfHealth +32:59.724 [DEBUG] Agent1: Node info in sync +32:59.725 [INFO] Agent1.server.memberlist.lan: memberlist: Suspect Node-d9f3d23b-bfde-515d-8446-6415a9580675 has failed, no acks received +33:00.501 [INFO] Agent1.server.memberlist.wan: memberlist: Suspect Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 has failed, no acks received +33:07.724 [INFO] Agent1.server.serf.wan: serf: EventMemberReap (forced): Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 127.0.0.1 +33:07.724 [DEBUG] Agent1.http: Request finished: method=PUT url=/v1/agent/force-leave/Node-d9f3d23b-bfde-515d-8446-6415a9580675?prune=1 from=127.0.0.1:50632 latency=16.007566534s +33:07.724 [INFO] Agent1.server: Handled event for server in area: event=member-reap server=Node-d9f3d23b-bfde-515d-8446-6415a9580675.dc1 area=wan +33:07.724 [INFO] Agent1: Requesting shutdown +33:07.724 [INFO] Agent1.server: shutting down server +33:07.724 [DEBUG] Agent1.leader: stopping routine: routine="federation state anti-entropy" +33:07.724 [DEBUG] Agent1.leader: stopping routine: routine="federation state pruning" +33:07.724 [DEBUG] Agent1.leader: stopping routine: routine="intermediate cert renew watch" +33:07.724 [DEBUG] Agent1.leader: stopping routine: routine="CA root pruning" +33:07.724 [WARN] Agent1.server.serf.lan: serf: Shutdown without a Leave +33:07.724 [ERROR] Agent1.server: error performing anti-entropy sync of federation state: error="context canceled" +33:07.724 [DEBUG] Agent1.leader: stopped routine: routine="federation state anti-entropy" +33:07.724 [DEBUG] Agent1.leader: stopped routine: routine="federation state pruning" +33:07.724 [DEBUG] Agent1.leader: stopped routine: routine="intermediate cert renew watch" +33:07.724 [DEBUG] Agent1.leader: stopped routine: routine="CA root pruning" +33:07.726 [WARN] Agent1.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:07.727Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:07.727Z [INFO] agent.router.manager: shutting down +33:07.728 [INFO] Agent1: consul server down +33:07.728 [INFO] Agent1: shutdown complete +33:07.728 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp +33:07.728 [INFO] Agent1: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp +33:07.728 [INFO] Agent1: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp +33:08.232 [INFO] Agent1: Waiting for endpoints to shut down +33:08.232 [INFO] Agent1: Endpoints down +--- PASS: TestForceLeaveCommand_prune (17.99s) PASS -ok github.com/hashicorp/consul/command/forceleave 17.768s +ok github.com/hashicorp/consul/command/forceleave 18.132s ? github.com/hashicorp/consul/command/helpers [no test files] === RUN TestInfoCommand_noTabs === PAUSE TestInfoCommand_noTabs === RUN TestInfoCommand === PAUSE TestInfoCommand === CONT TestInfoCommand_noTabs -=== CONT TestInfoCommand --- PASS: TestInfoCommand_noTabs (0.00s) +=== CONT TestInfoCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:03.956Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:03.968Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:03.994 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:129e195f-cdda-6944-bda5-aa67e1c4e715 Address:127.0.0.1:16132}]" -46:03.994 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:16132 [Follower]" leader= -46:03.994 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-129e195f-cdda-6944-bda5-aa67e1c4e715.dc1 127.0.0.1 -46:03.995 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-129e195f-cdda-6944-bda5-aa67e1c4e715 127.0.0.1 -2022-11-07T22:46:03.995Z [INFO] agent.router: Initializing LAN area manager -46:03.995 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-129e195f-cdda-6944-bda5-aa67e1c4e715.dc1 area=wan -46:03.995 [INFO] TestAgent.server: Adding LAN server: server="Node-129e195f-cdda-6944-bda5-aa67e1c4e715 (Addr: tcp/127.0.0.1:16132) (DC: dc1)" -46:03.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=udp -46:03.995 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=tcp -46:03.996 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:16128 network=tcp -46:03.996 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:16133 network=tcp -46:03.996 [INFO] TestAgent: started state syncer -46:04.042 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.042 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:16132 [Candidate]" term=2 -46:04.054 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.054 [DEBUG] TestAgent.server.raft: vote granted: from=129e195f-cdda-6944-bda5-aa67e1c4e715 term=2 tally=1 -46:04.054 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.054 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:16132 [Leader]" -46:04.054 [INFO] TestAgent.server: cluster leadership acquired -46:04.054 [INFO] TestAgent.server: New leader elected: payload=Node-129e195f-cdda-6944-bda5-aa67e1c4e715 -46:04.061 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:16132 -46:04.088 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.125 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.125 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.125 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.125 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.125 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.125 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-129e195f-cdda-6944-bda5-aa67e1c4e715 -46:04.125 [INFO] TestAgent.server: member joined, marking health alive: member=Node-129e195f-cdda-6944-bda5-aa67e1c4e715 -46:04.146 [INFO] TestAgent.server: federation state anti-entropy synced -46:04.152 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:34454 latency=1.023831ms -46:04.154 [INFO] TestAgent: Requesting shutdown -46:04.154 [INFO] TestAgent.server: shutting down server -46:04.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:04.154 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:04.154 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:04.154 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:04.154 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:04.154 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:04.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:04.154 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:04.154 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:04.154 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:04.154 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:04.160 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:04.167Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:04.167Z [INFO] agent.router.manager: shutting down -46:04.167 [INFO] TestAgent: consul server down -46:04.167 [INFO] TestAgent: shutdown complete -46:04.167 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=tcp -46:04.167 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=udp -46:04.167 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:16128 network=tcp -46:04.667 [INFO] TestAgent: Waiting for endpoints to shut down -46:04.667 [INFO] TestAgent: Endpoints down ---- PASS: TestInfoCommand (0.75s) +2021-10-05T16:32:51.615Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:51.648Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:51.719 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:00d6b490-3eb2-a74e-5634-ee8ab24192f0 Address:127.0.0.1:16132}]" +32:51.720 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:16132 [Follower]" leader= +32:51.721 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0.dc1 127.0.0.1 +32:51.721 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0 127.0.0.1 +2021-10-05T16:32:51.721Z [INFO] agent.router: Initializing LAN area manager +32:51.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=udp +32:51.722 [INFO] TestAgent.server: Adding LAN server: server="Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0 (Addr: tcp/127.0.0.1:16132) (DC: dc1)" +32:51.722 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0.dc1 area=wan +32:51.722 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=tcp +32:51.723 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:16128 network=tcp +32:51.723 [INFO] TestAgent: started state syncer +32:51.723 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:16133 network=tcp +32:51.777 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:51.777 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:16132 [Candidate]" term=2 +32:51.791 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:51.792 [DEBUG] TestAgent.server.raft: vote granted: from=00d6b490-3eb2-a74e-5634-ee8ab24192f0 term=2 tally=1 +32:51.792 [INFO] TestAgent.server.raft: election won: tally=1 +32:51.792 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:16132 [Leader]" +32:51.792 [INFO] TestAgent.server: cluster leadership acquired +32:51.792 [INFO] TestAgent.server: New leader elected: payload=Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0 +32:51.798 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:16132 +32:51.858 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:51.891 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:51.891 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:51.891 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:51.891 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:51.891 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:51.891 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0 +32:51.891 [INFO] TestAgent.server: member joined, marking health alive: member=Node-00d6b490-3eb2-a74e-5634-ee8ab24192f0 +32:51.920 [INFO] TestAgent.server: federation state anti-entropy synced +32:52.072 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:52.088 [INFO] TestAgent: Synced node info +32:52.088 [DEBUG] TestAgent: Node info in sync +32:52.102 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:48768 latency=1.603993ms +32:52.108 [INFO] TestAgent: Requesting shutdown +32:52.108 [INFO] TestAgent.server: shutting down server +32:52.108 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:52.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:52.108 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:52.108 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:52.108 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:52.108 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:52.108 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:52.109 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:52.109 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:52.109 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:52.122 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:52.122Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:52.122Z [INFO] agent.router.manager: shutting down +32:52.122 [INFO] TestAgent: consul server down +32:52.122 [INFO] TestAgent: shutdown complete +32:52.122 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=tcp +32:52.123 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=udp +32:52.123 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:16128 network=tcp +32:52.624 [INFO] TestAgent: Waiting for endpoints to shut down +32:52.624 [INFO] TestAgent: Endpoints down +--- PASS: TestInfoCommand (1.07s) PASS -ok github.com/hashicorp/consul/command/info 0.777s +ok github.com/hashicorp/consul/command/info 1.306s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === CONT TestCommand_noTabs --- PASS: TestCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/intention 0.005s +ok github.com/hashicorp/consul/command/intention 0.060s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -84969,75 +85250,75 @@ === RUN TestCommand_Validation/0_args === RUN TestCommand_Validation/1_args === RUN TestCommand_Validation/3_args -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCommand_Validation (0.01s) +--- PASS: TestCommand_Validation (0.00s) --- PASS: TestCommand_Validation/0_args (0.00s) --- PASS: TestCommand_Validation/1_args (0.00s) --- PASS: TestCommand_Validation/3_args (0.00s) -2022-11-07T22:46:04.644Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:04.654Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:04.687 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:92c430ee-13a3-8567-ccae-0150d6614167 Address:127.0.0.1:14090}]" -46:04.688 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= -46:04.688 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-92c430ee-13a3-8567-ccae-0150d6614167.dc1 127.0.0.1 -46:04.689 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-92c430ee-13a3-8567-ccae-0150d6614167 127.0.0.1 -2022-11-07T22:46:04.689Z [INFO] agent.router: Initializing LAN area manager -46:04.689 [INFO] TestAgent.server: Adding LAN server: server="Node-92c430ee-13a3-8567-ccae-0150d6614167 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" -46:04.689 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp -46:04.689 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp -46:04.689 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-92c430ee-13a3-8567-ccae-0150d6614167.dc1 area=wan -46:04.690 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp -46:04.690 [INFO] TestAgent: started state syncer -46:04.691 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp -46:04.746 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:04.746 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 -46:04.755 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:04.755 [DEBUG] TestAgent.server.raft: vote granted: from=92c430ee-13a3-8567-ccae-0150d6614167 term=2 tally=1 -46:04.755 [INFO] TestAgent.server.raft: election won: tally=1 -46:04.755 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" -46:04.755 [INFO] TestAgent.server: cluster leadership acquired -46:04.755 [INFO] TestAgent.server: New leader elected: payload=Node-92c430ee-13a3-8567-ccae-0150d6614167 -46:04.761 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 -46:04.779 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:04.811 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:04.811 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:04.811 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:04.811 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:04.811 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:04.811 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-92c430ee-13a3-8567-ccae-0150d6614167 -46:04.811 [INFO] TestAgent.server: member joined, marking health alive: member=Node-92c430ee-13a3-8567-ccae-0150d6614167 -46:04.826 [INFO] TestAgent.server: federation state anti-entropy synced -46:05.079 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:05.084 [INFO] TestAgent: Synced node info -46:05.084 [DEBUG] TestAgent: Node info in sync -46:05.089 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57504 latency=6.312101ms -46:05.091 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check?destination=db&source=foo&source-type=consul from=127.0.0.1:57506 latency=928.133µs -46:05.094 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check?destination=db&source=web&source-type=consul from=127.0.0.1:57508 latency=936.234µs -46:05.094 [INFO] TestAgent: Requesting shutdown -46:05.094 [INFO] TestAgent.server: shutting down server -46:05.094 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:05.094 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:05.094 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:05.094 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:05.094 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:05.094 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:05.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:05.094 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:05.094 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:05.094 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:05.101 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:05.110Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:05.110Z [INFO] agent.router.manager: shutting down -46:05.110 [INFO] TestAgent: consul server down -46:05.110 [INFO] TestAgent: shutdown complete -46:05.110 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp -46:05.110 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp -46:05.110 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp -46:05.610 [INFO] TestAgent: Waiting for endpoints to shut down -46:05.610 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand (1.01s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:32:53.288Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:53.314Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:53.346 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:70efa02f-8f28-fbc7-31e9-5eef978d7f65 Address:127.0.0.1:10006}]" +32:53.347 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65.dc1 127.0.0.1 +32:53.347 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65 127.0.0.1 +2021-10-05T16:32:53.348Z [INFO] agent.router: Initializing LAN area manager +32:53.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= +32:53.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp +32:53.348 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65.dc1 area=wan +32:53.348 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp +32:53.348 [INFO] TestAgent.server: Adding LAN server: server="Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" +32:53.349 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp +32:53.349 [INFO] TestAgent: started state syncer +32:53.349 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp +32:53.402 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:53.402 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 +32:53.411 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:53.411 [DEBUG] TestAgent.server.raft: vote granted: from=70efa02f-8f28-fbc7-31e9-5eef978d7f65 term=2 tally=1 +32:53.411 [INFO] TestAgent.server.raft: election won: tally=1 +32:53.411 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" +32:53.412 [INFO] TestAgent.server: cluster leadership acquired +32:53.412 [INFO] TestAgent.server: New leader elected: payload=Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65 +32:53.419 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10006 +32:53.442 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:53.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:53.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:53.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:53.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:53.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:53.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65 +32:53.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-70efa02f-8f28-fbc7-31e9-5eef978d7f65 +32:53.513 [INFO] TestAgent.server: federation state anti-entropy synced +32:53.747 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:53.764 [INFO] TestAgent: Synced node info +32:53.764 [DEBUG] TestAgent: Node info in sync +32:53.797 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:49690 latency=21.057406ms +32:53.829 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check?destination=db&source=foo&source-type=consul from=127.0.0.1:49692 latency=512.045µs +32:53.830 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/check?destination=db&source=web&source-type=consul from=127.0.0.1:49694 latency=431.632µs +32:53.832 [INFO] TestAgent: Requesting shutdown +32:53.832 [INFO] TestAgent.server: shutting down server +32:53.832 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:53.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:53.832 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:53.832 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:53.832 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:53.832 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:53.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:53.832 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:53.832 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:53.832 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:53.839 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:53.844Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:53.844Z [INFO] agent.router.manager: shutting down +32:53.844 [INFO] TestAgent: consul server down +32:53.844 [INFO] TestAgent: shutdown complete +32:53.844 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp +32:53.844 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp +32:53.844 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp +32:54.348 [INFO] TestAgent: Waiting for endpoints to shut down +32:54.348 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand (1.13s) PASS -ok github.com/hashicorp/consul/command/intention/check 1.036s +ok github.com/hashicorp/consul/command/intention/check 1.300s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -85055,348 +85336,349 @@ === RUN TestCommand_replace === PAUSE TestCommand_replace === CONT TestCommand_noTabs -=== CONT TestCommand_File === CONT TestCommand_deny === CONT TestCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestCommand_meta -=== CONT TestCommand_Validation -=== CONT TestCommand_FileNoExist -=== CONT TestCommand_replace --- PASS: TestCommand_noTabs (0.00s) +=== CONT TestCommand_FileNoExist +=== CONT TestCommand_File +=== CONT TestCommand_Validation === RUN TestCommand_Validation/-allow_and_-deny -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== CONT TestCommand_replace +=== CONT TestCommand_meta --- PASS: TestCommand_Validation (0.00s) --- PASS: TestCommand_Validation/-allow_and_-deny (0.00s) -2022-11-07T22:46:08.977Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.992Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.035 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:66597580-aa10-2b8b-ff15-70e05b438557 Address:127.0.0.1:31454}]" -46:09.035 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31454 [Follower]" leader= -46:09.037 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-66597580-aa10-2b8b-ff15-70e05b438557.dc1 127.0.0.1 -46:09.038 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-66597580-aa10-2b8b-ff15-70e05b438557 127.0.0.1 -2022-11-07T22:46:09.038Z [INFO] agent.router: Initializing LAN area manager -46:09.039 [INFO] TestAgent.server: Adding LAN server: server="Node-66597580-aa10-2b8b-ff15-70e05b438557 (Addr: tcp/127.0.0.1:31454) (DC: dc1)" -46:09.039 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=udp -46:09.039 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-66597580-aa10-2b8b-ff15-70e05b438557.dc1 area=wan -46:09.039 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=tcp -46:09.040 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31450 network=tcp -46:09.040 [INFO] TestAgent: started state syncer -46:09.040 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31455 network=tcp -46:09.092 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.092 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31454 [Candidate]" term=2 -46:09.114 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.114 [DEBUG] TestAgent.server.raft: vote granted: from=66597580-aa10-2b8b-ff15-70e05b438557 term=2 tally=1 -46:09.114 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.114 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31454 [Leader]" -46:09.114 [INFO] TestAgent.server: cluster leadership acquired -46:09.114 [INFO] TestAgent.server: New leader elected: payload=Node-66597580-aa10-2b8b-ff15-70e05b438557 -46:09.134 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31454 -46:09.146 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57814 latency=13.661855ms -46:09.148 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:57816 latency=867.337µs -46:09.149 [INFO] TestAgent: Requesting shutdown -46:09.149 [INFO] TestAgent.server: shutting down server -46:09.149 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.149 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.161 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.180Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.180Z [INFO] agent.router.manager: shutting down -46:09.192 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" -46:09.192 [INFO] TestAgent: consul server down -46:09.192 [INFO] TestAgent: shutdown complete -46:09.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=tcp -46:09.192 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=udp -46:09.192 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31450 network=tcp -46:09.692 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.692 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_deny (0.76s) -2022-11-07T22:46:08.977Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.998Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.044 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8631e1ef-825e-5928-a259-b8956411a797 Address:127.0.0.1:31447}]" -46:09.044 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= -46:09.044 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8631e1ef-825e-5928-a259-b8956411a797.dc1 127.0.0.1 -46:09.066 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8631e1ef-825e-5928-a259-b8956411a797 127.0.0.1 -2022-11-07T22:46:09.066Z [INFO] agent.router: Initializing LAN area manager -46:09.066 [INFO] TestAgent.server: Adding LAN server: server="Node-8631e1ef-825e-5928-a259-b8956411a797 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" -46:09.066 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8631e1ef-825e-5928-a259-b8956411a797.dc1 area=wan -46:09.066 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp -46:09.066 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp -46:09.067 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp -46:09.067 [INFO] TestAgent: started state syncer -46:09.067 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp -46:09.089 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.089 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 -46:09.113 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.113 [DEBUG] TestAgent.server.raft: vote granted: from=8631e1ef-825e-5928-a259-b8956411a797 term=2 tally=1 -46:09.113 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.113 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" -46:09.113 [INFO] TestAgent.server: cluster leadership acquired -46:09.113 [INFO] TestAgent.server: New leader elected: payload=Node-8631e1ef-825e-5928-a259-b8956411a797 -46:09.134 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 -46:09.178 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57334 latency=25.501497ms -46:09.181 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:57336 latency=1.896582ms -46:09.184 [INFO] TestAgent: Requesting shutdown -46:09.184 [INFO] TestAgent.server: shutting down server -46:09.184 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.184 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:09.189 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:09.197 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:09.199Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.199Z [INFO] agent.router.manager: shutting down -46:09.202 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" -46:09.202 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:09.202 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" -46:09.202 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" -46:09.202 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" -46:09.202 [INFO] TestAgent: consul server down -46:09.202 [INFO] TestAgent: shutdown complete -46:09.202 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp -46:09.202 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp -46:09.202 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp -46:09.702 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.702 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_File (0.77s) -2022-11-07T22:46:08.979Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.995Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.039 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f13392b6-10a0-fbaa-dffc-7944dd23d6da Address:127.0.0.1:31475}]" -46:09.039 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31475 [Follower]" leader= -46:09.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da.dc1 127.0.0.1 -46:09.040 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da 127.0.0.1 -2022-11-07T22:46:09.040Z [INFO] agent.router: Initializing LAN area manager -46:09.040 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da.dc1 area=wan -46:09.040 [INFO] TestAgent.server: Adding LAN server: server="Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da (Addr: tcp/127.0.0.1:31475) (DC: dc1)" -46:09.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31470 network=udp -46:09.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31470 network=tcp -46:09.040 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31471 network=tcp -46:09.040 [INFO] TestAgent: started state syncer -46:09.040 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31476 network=tcp -46:09.085 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.085 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31475 [Candidate]" term=2 -46:09.107 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.107 [DEBUG] TestAgent.server.raft: vote granted: from=f13392b6-10a0-fbaa-dffc-7944dd23d6da term=2 tally=1 -46:09.107 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.107 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31475 [Leader]" -46:09.107 [INFO] TestAgent.server: cluster leadership acquired -46:09.107 [INFO] TestAgent.server: New leader elected: payload=Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da -46:09.119 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31475 -46:09.158 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.205 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.205 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.205 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.205 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.205 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.205 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da -46:09.205 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f13392b6-10a0-fbaa-dffc-7944dd23d6da -46:09.228 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.232 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.237 [INFO] TestAgent: Synced node info -46:09.237 [DEBUG] TestAgent: Node info in sync -46:09.246 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:38758 latency=5.452813ms -46:09.248 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:38760 latency=766.143µs -46:09.248 [INFO] TestAgent: Requesting shutdown -46:09.248 [INFO] TestAgent.server: shutting down server -46:09.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.248 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.248 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.248 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.248 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.248 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.248 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.248 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.248 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.255 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.265Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.265Z [INFO] agent.router.manager: shutting down -46:09.265 [INFO] TestAgent: consul server down -46:09.265 [INFO] TestAgent: shutdown complete -46:09.265 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31470 network=tcp -46:09.265 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31470 network=udp -46:09.265 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31471 network=tcp -46:09.765 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.765 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_meta (0.83s) -2022-11-07T22:46:08.977Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.999Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.066 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7d9c276e-6461-b73f-533e-5e6fdd610b68 Address:127.0.0.1:31482}]" -46:09.066 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31482 [Follower]" leader= -46:09.066 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7d9c276e-6461-b73f-533e-5e6fdd610b68.dc1 127.0.0.1 -46:09.067 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7d9c276e-6461-b73f-533e-5e6fdd610b68 127.0.0.1 -2022-11-07T22:46:09.067Z [INFO] agent.router: Initializing LAN area manager -46:09.067 [INFO] TestAgent.server: Adding LAN server: server="Node-7d9c276e-6461-b73f-533e-5e6fdd610b68 (Addr: tcp/127.0.0.1:31482) (DC: dc1)" -46:09.067 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7d9c276e-6461-b73f-533e-5e6fdd610b68.dc1 area=wan -46:09.067 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31477 network=tcp -46:09.067 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31477 network=udp -46:09.068 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31478 network=tcp -46:09.068 [INFO] TestAgent: started state syncer -46:09.068 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31483 network=tcp -46:09.109 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.109 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31482 [Candidate]" term=2 -46:09.135 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.135 [DEBUG] TestAgent.server.raft: vote granted: from=7d9c276e-6461-b73f-533e-5e6fdd610b68 term=2 tally=1 -46:09.135 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.135 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31482 [Leader]" -46:09.135 [INFO] TestAgent.server: cluster leadership acquired -46:09.135 [INFO] TestAgent.server: New leader elected: payload=Node-7d9c276e-6461-b73f-533e-5e6fdd610b68 -46:09.148 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31482 -46:09.175 [INFO] TestAgent: Synced node info -46:09.207 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.228 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.228 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.228 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.228 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.228 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.228 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7d9c276e-6461-b73f-533e-5e6fdd610b68 -46:09.228 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7d9c276e-6461-b73f-533e-5e6fdd610b68 -46:09.247 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.282 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:45820 latency=8.201563ms -46:09.284 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:45822 latency=332.402µs -46:09.290 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions from=127.0.0.1:45824 error="duplicate intention found: ALLOW default/foo => default/bar (ID: eca2363b-e939-d91d-b347-8d2e9c86e5f4, Precedence: 9)" -46:09.291 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:45824 latency=5.590607ms -46:09.292 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:45826 latency=148.49µs -46:09.298 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/eca2363b-e939-d91d-b347-8d2e9c86e5f4 from=127.0.0.1:45826 latency=6.284308ms -46:09.299 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:45822 latency=111.995µs -46:09.299 [INFO] TestAgent: Requesting shutdown -46:09.299 [INFO] TestAgent.server: shutting down server -46:09.299 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.299 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.299 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.299 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.299 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.299 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.299 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.299 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.299 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.299 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.304 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.316Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.316Z [INFO] agent.router.manager: shutting down -46:09.316 [INFO] TestAgent: consul server down -46:09.316 [INFO] TestAgent: shutdown complete -46:09.316 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31477 network=tcp -46:09.316 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31477 network=udp -46:09.316 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31478 network=tcp -46:09.816 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.816 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_replace (0.88s) -2022-11-07T22:46:08.977Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.994Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.038 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9440e9c6-2915-d13e-e3bc-f9c65d2e362c Address:127.0.0.1:31468}]" -46:09.038 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31468 [Follower]" leader= -46:09.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c.dc1 127.0.0.1 -46:09.040 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c 127.0.0.1 -2022-11-07T22:46:09.040Z [INFO] agent.router: Initializing LAN area manager -46:09.040 [INFO] TestAgent.server: Adding LAN server: server="Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c (Addr: tcp/127.0.0.1:31468) (DC: dc1)" -46:09.040 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c.dc1 area=wan -46:09.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31463 network=udp -46:09.040 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31463 network=tcp -46:09.040 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31464 network=tcp -46:09.040 [INFO] TestAgent: started state syncer -46:09.041 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31469 network=tcp -46:09.107 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.107 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31468 [Candidate]" term=2 -46:09.134 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.134 [DEBUG] TestAgent.server.raft: vote granted: from=9440e9c6-2915-d13e-e3bc-f9c65d2e362c term=2 tally=1 -46:09.134 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.134 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31468 [Leader]" -46:09.134 [INFO] TestAgent.server: cluster leadership acquired -46:09.134 [INFO] TestAgent.server: New leader elected: payload=Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c -46:09.149 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31468 -46:09.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.227 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.227 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.227 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.227 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.227 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.227 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c -46:09.227 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9440e9c6-2915-d13e-e3bc-f9c65d2e362c -46:09.244 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.246 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.248 [INFO] TestAgent: Synced node info -46:09.351 [INFO] TestAgent: Requesting shutdown -46:09.351 [INFO] TestAgent.server: shutting down server -46:09.351 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.351 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.351 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.351 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.351 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.351 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.351 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.351 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.351 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.351 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.358 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.370Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.370Z [INFO] agent.router.manager: shutting down -46:09.371 [INFO] TestAgent: consul server down -46:09.371 [INFO] TestAgent: shutdown complete -46:09.371 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31463 network=tcp -46:09.371 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31463 network=udp -46:09.371 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31464 network=tcp -46:09.871 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.871 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_FileNoExist (0.94s) -2022-11-07T22:46:08.973Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:08.989Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.025 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:326daf0d-301e-9411-14fc-9db37bc3fba3 Address:127.0.0.1:31461}]" -46:09.025 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31461 [Follower]" leader= -46:09.026 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-326daf0d-301e-9411-14fc-9db37bc3fba3.dc1 127.0.0.1 -46:09.026 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-326daf0d-301e-9411-14fc-9db37bc3fba3 127.0.0.1 -2022-11-07T22:46:09.026Z [INFO] agent.router: Initializing LAN area manager -46:09.026 [INFO] TestAgent.server: Adding LAN server: server="Node-326daf0d-301e-9411-14fc-9db37bc3fba3 (Addr: tcp/127.0.0.1:31461) (DC: dc1)" -46:09.027 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-326daf0d-301e-9411-14fc-9db37bc3fba3.dc1 area=wan -46:09.027 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31456 network=udp -46:09.027 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31456 network=tcp -46:09.027 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31457 network=tcp -46:09.027 [INFO] TestAgent: started state syncer -46:09.027 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31462 network=tcp -46:09.066 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.066 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31461 [Candidate]" term=2 -46:09.077 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.077 [DEBUG] TestAgent.server.raft: vote granted: from=326daf0d-301e-9411-14fc-9db37bc3fba3 term=2 tally=1 -46:09.077 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.077 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31461 [Leader]" -46:09.077 [INFO] TestAgent.server: cluster leadership acquired -46:09.077 [INFO] TestAgent.server: New leader elected: payload=Node-326daf0d-301e-9411-14fc-9db37bc3fba3 -46:09.086 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31461 -46:09.109 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:09.176 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:09.176 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:09.176 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:09.176 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:09.176 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:09.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-326daf0d-301e-9411-14fc-9db37bc3fba3 -46:09.176 [INFO] TestAgent.server: member joined, marking health alive: member=Node-326daf0d-301e-9411-14fc-9db37bc3fba3 -46:09.199 [INFO] TestAgent.server: federation state anti-entropy synced -46:09.343 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:09.350 [INFO] TestAgent: Synced node info -46:09.396 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:37586 latency=4.425454ms -46:09.397 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:37588 latency=259.198µs -46:09.397 [INFO] TestAgent: Requesting shutdown -46:09.397 [INFO] TestAgent.server: shutting down server -46:09.397 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:09.397 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:09.397 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:09.397 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:09.397 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:09.397 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:09.397 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:09.397 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:09.397 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:09.397 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:09.402 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:09.407Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:09.407Z [INFO] agent.router.manager: shutting down -46:09.407 [INFO] TestAgent: consul server down -46:09.407 [INFO] TestAgent: shutdown complete -46:09.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31456 network=tcp -46:09.407 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31456 network=udp -46:09.407 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31457 network=tcp -46:09.907 [INFO] TestAgent: Waiting for endpoints to shut down -46:09.907 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand (0.98s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:32:54.724Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.887 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f41510b-775e-1a5c-8eb5-8e5554c12c8b Address:127.0.0.1:17160}]" +32:54.887 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17160 [Follower]" leader= +32:54.888 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f41510b-775e-1a5c-8eb5-8e5554c12c8b.dc1 127.0.0.1 +32:54.888 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f41510b-775e-1a5c-8eb5-8e5554c12c8b 127.0.0.1 +2021-10-05T16:32:54.889Z [INFO] agent.router: Initializing LAN area manager +32:54.889 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f41510b-775e-1a5c-8eb5-8e5554c12c8b.dc1 area=wan +32:54.889 [INFO] TestAgent.server: Adding LAN server: server="Node-8f41510b-775e-1a5c-8eb5-8e5554c12c8b (Addr: tcp/127.0.0.1:17160) (DC: dc1)" +32:54.889 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=udp +32:54.889 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=tcp +32:54.890 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17156 network=tcp +32:54.890 [INFO] TestAgent: started state syncer +32:54.890 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17161 network=tcp +32:54.958 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:54.958 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17160 [Candidate]" term=2 +32:54.972 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:54.972 [DEBUG] TestAgent.server.raft: vote granted: from=8f41510b-775e-1a5c-8eb5-8e5554c12c8b term=2 tally=1 +32:54.972 [INFO] TestAgent.server.raft: election won: tally=1 +32:54.972 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17160 [Leader]" +32:54.972 [INFO] TestAgent.server: cluster leadership acquired +32:54.972 [INFO] TestAgent.server: New leader elected: payload=Node-8f41510b-775e-1a5c-8eb5-8e5554c12c8b +32:54.982 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17160 +32:55.002 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.007 [INFO] TestAgent: Requesting shutdown +32:55.008 [INFO] TestAgent.server: shutting down server +32:55.008 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.008 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:55.014 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.015Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.015Z [INFO] agent.router.manager: shutting down +32:55.019 [INFO] TestAgent: consul server down +32:55.019 [INFO] TestAgent: shutdown complete +32:55.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=tcp +32:55.019 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=udp +32:55.019 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17156 network=tcp +32:55.019 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +32:55.019 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +32:55.520 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.520 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_FileNoExist (1.05s) +2021-10-05T16:32:54.758Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.815Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.936 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fba2629e-4620-0f13-0285-e2ff569f5b58 Address:127.0.0.1:17188}]" +32:54.937 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fba2629e-4620-0f13-0285-e2ff569f5b58.dc1 127.0.0.1 +32:54.938 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fba2629e-4620-0f13-0285-e2ff569f5b58 127.0.0.1 +2021-10-05T16:32:54.938Z [INFO] agent.router: Initializing LAN area manager +32:54.944 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17188 [Follower]" leader= +32:54.948 [INFO] TestAgent.server: Adding LAN server: server="Node-fba2629e-4620-0f13-0285-e2ff569f5b58 (Addr: tcp/127.0.0.1:17188) (DC: dc1)" +32:54.957 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fba2629e-4620-0f13-0285-e2ff569f5b58.dc1 area=wan +32:54.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17183 network=tcp +32:54.958 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17183 network=udp +32:54.959 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17184 network=tcp +32:54.959 [INFO] TestAgent: started state syncer +32:54.959 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17189 network=tcp +32:54.981 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:54.981 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17188 [Candidate]" term=2 +32:54.993 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:54.993 [DEBUG] TestAgent.server.raft: vote granted: from=fba2629e-4620-0f13-0285-e2ff569f5b58 term=2 tally=1 +32:54.993 [INFO] TestAgent.server.raft: election won: tally=1 +32:54.993 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17188 [Leader]" +32:54.995 [INFO] TestAgent.server: cluster leadership acquired +32:54.995 [INFO] TestAgent.server: New leader elected: payload=Node-fba2629e-4620-0f13-0285-e2ff569f5b58 +32:55.007 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17188 +32:55.034 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.065 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:59484 latency=8.124128ms +32:55.068 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:59486 latency=1.194714ms +32:55.069 [INFO] TestAgent: Requesting shutdown +32:55.069 [INFO] TestAgent.server: shutting down server +32:55.069 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.069 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +32:55.076 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.084Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.084Z [INFO] agent.router.manager: shutting down +32:55.092 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +32:55.096 [INFO] TestAgent: consul server down +32:55.096 [INFO] TestAgent: shutdown complete +32:55.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17183 network=tcp +32:55.096 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17183 network=udp +32:55.096 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17184 network=tcp +32:55.604 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.604 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_deny (1.13s) +2021-10-05T16:32:54.701Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.804Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.928 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:156f9021-0d7c-0027-4d08-4296c4dcd2bf Address:127.0.0.1:17174}]" +32:54.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf.dc1 127.0.0.1 +32:54.930 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17174 [Follower]" leader= +32:54.943 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf 127.0.0.1 +2021-10-05T16:32:54.943Z [INFO] agent.router: Initializing LAN area manager +32:54.956 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf.dc1 area=wan +32:54.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=tcp +32:54.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=udp +32:54.958 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17170 network=tcp +32:54.958 [INFO] TestAgent: started state syncer +32:54.958 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17175 network=tcp +32:54.958 [INFO] TestAgent.server: Adding LAN server: server="Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf (Addr: tcp/127.0.0.1:17174) (DC: dc1)" +32:54.999 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:54.999 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17174 [Candidate]" term=2 +32:55.030 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:55.030 [DEBUG] TestAgent.server.raft: vote granted: from=156f9021-0d7c-0027-4d08-4296c4dcd2bf term=2 tally=1 +32:55.030 [INFO] TestAgent.server.raft: election won: tally=1 +32:55.030 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17174 [Leader]" +32:55.031 [INFO] TestAgent.server: cluster leadership acquired +32:55.031 [INFO] TestAgent.server: New leader elected: payload=Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf +32:55.075 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:49576 latency=20.305743ms +32:55.075 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17174 +32:55.085 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:49582 latency=6.917373ms +32:55.144 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.167 [ERROR] TestAgent.http: Request error: method=POST url=/v1/connect/intentions from=127.0.0.1:49584 error="duplicate intention found: ALLOW default/foo => default/bar (ID: 99dfa392-a0d2-1f0e-3f1a-05e204a5c3b3, Precedence: 9)" +32:55.170 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:49584 latency=38.533827ms +32:55.177 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:49586 latency=492.259µs +32:55.200 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:55.200 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:55.200 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:55.200 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:55.200 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:55.200 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf +32:55.200 [INFO] TestAgent.server: member joined, marking health alive: member=Node-156f9021-0d7c-0027-4d08-4296c4dcd2bf +32:55.208 [INFO] TestAgent: Synced node info +32:55.209 [DEBUG] TestAgent: Node info in sync +32:55.209 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/connect/intentions/99dfa392-a0d2-1f0e-3f1a-05e204a5c3b3 from=127.0.0.1:49586 latency=24.931767ms +32:55.210 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:49582 latency=502.256µs +32:55.210 [INFO] TestAgent: Requesting shutdown +32:55.210 [INFO] TestAgent.server: shutting down server +32:55.210 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:55.210 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:55.210 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:55.210 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:55.210 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.210 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:55.210 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:55.210 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:55.211 [INFO] TestAgent.server: federation state anti-entropy synced +32:55.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:55.214 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.218Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.218Z [INFO] agent.router.manager: shutting down +32:55.218 [INFO] TestAgent: consul server down +32:55.218 [INFO] TestAgent: shutdown complete +32:55.218 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=tcp +32:55.218 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=udp +32:55.218 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17170 network=tcp +32:55.719 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.719 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_replace (1.24s) +2021-10-05T16:32:54.669Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.728Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.891 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:903300e4-3f5f-9cf7-eea1-2664f699005e Address:127.0.0.1:17167}]" +32:54.891 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17167 [Follower]" leader= +32:54.892 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-903300e4-3f5f-9cf7-eea1-2664f699005e.dc1 127.0.0.1 +32:54.928 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-903300e4-3f5f-9cf7-eea1-2664f699005e 127.0.0.1 +2021-10-05T16:32:54.929Z [INFO] agent.router: Initializing LAN area manager +32:54.929 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=udp +32:54.929 [INFO] TestAgent.server: Adding LAN server: server="Node-903300e4-3f5f-9cf7-eea1-2664f699005e (Addr: tcp/127.0.0.1:17167) (DC: dc1)" +32:54.929 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-903300e4-3f5f-9cf7-eea1-2664f699005e.dc1 area=wan +32:54.943 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=tcp +32:54.944 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17163 network=tcp +32:54.944 [INFO] TestAgent: started state syncer +32:54.944 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17168 network=tcp +32:54.956 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:54.956 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17167 [Candidate]" term=2 +32:54.970 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:54.970 [DEBUG] TestAgent.server.raft: vote granted: from=903300e4-3f5f-9cf7-eea1-2664f699005e term=2 tally=1 +32:54.970 [INFO] TestAgent.server.raft: election won: tally=1 +32:54.970 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17167 [Leader]" +32:54.972 [INFO] TestAgent.server: cluster leadership acquired +32:54.972 [INFO] TestAgent.server: New leader elected: payload=Node-903300e4-3f5f-9cf7-eea1-2664f699005e +32:54.980 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17167 +32:55.016 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.070 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:55.070 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:55.070 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:55.070 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:55.070 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:55.071 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-903300e4-3f5f-9cf7-eea1-2664f699005e +32:55.071 [INFO] TestAgent.server: member joined, marking health alive: member=Node-903300e4-3f5f-9cf7-eea1-2664f699005e +32:55.097 [INFO] TestAgent.server: federation state anti-entropy synced +32:55.296 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:55.311 [INFO] TestAgent: Synced node info +32:55.311 [DEBUG] TestAgent: Node info in sync +32:55.322 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:40684 latency=20.81185ms +32:55.323 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:40690 latency=260.943µs +32:55.324 [INFO] TestAgent: Requesting shutdown +32:55.324 [INFO] TestAgent.server: shutting down server +32:55.324 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:55.324 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:55.324 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:55.324 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:55.324 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.324 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:55.324 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:55.324 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:55.324 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:55.324 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:55.329 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.330Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.330Z [INFO] agent.router.manager: shutting down +32:55.330 [INFO] TestAgent: consul server down +32:55.330 [INFO] TestAgent: shutdown complete +32:55.330 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=tcp +32:55.330 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=udp +32:55.330 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17163 network=tcp +32:55.830 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.830 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_File (1.35s) +2021-10-05T16:32:54.741Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.806Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.942 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ad29110c-7010-bf18-5d3a-041c4123be18 Address:127.0.0.1:17181}]" +32:54.943 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ad29110c-7010-bf18-5d3a-041c4123be18.dc1 127.0.0.1 +32:54.944 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ad29110c-7010-bf18-5d3a-041c4123be18 127.0.0.1 +2021-10-05T16:32:54.944Z [INFO] agent.router: Initializing LAN area manager +32:54.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17176 network=udp +32:54.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17176 network=tcp +32:54.950 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17181 [Follower]" leader= +32:54.957 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ad29110c-7010-bf18-5d3a-041c4123be18.dc1 area=wan +32:54.957 [INFO] TestAgent.server: Adding LAN server: server="Node-ad29110c-7010-bf18-5d3a-041c4123be18 (Addr: tcp/127.0.0.1:17181) (DC: dc1)" +32:54.958 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17177 network=tcp +32:54.958 [INFO] TestAgent: started state syncer +32:54.958 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17182 network=tcp +32:55.017 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:55.017 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17181 [Candidate]" term=2 +32:55.065 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:55.065 [DEBUG] TestAgent.server.raft: vote granted: from=ad29110c-7010-bf18-5d3a-041c4123be18 term=2 tally=1 +32:55.065 [INFO] TestAgent.server.raft: election won: tally=1 +32:55.065 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17181 [Leader]" +32:55.065 [INFO] TestAgent.server: cluster leadership acquired +32:55.065 [INFO] TestAgent.server: New leader elected: payload=Node-ad29110c-7010-bf18-5d3a-041c4123be18 +32:55.090 [INFO] TestAgent: Synced node info +32:55.090 [DEBUG] TestAgent: Node info in sync +32:55.090 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17181 +32:55.160 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.185 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:55.185 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:55.185 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:55.185 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:55.185 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:55.185 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ad29110c-7010-bf18-5d3a-041c4123be18 +32:55.185 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ad29110c-7010-bf18-5d3a-041c4123be18 +32:55.200 [INFO] TestAgent.server: federation state anti-entropy synced +32:55.319 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:45790 latency=9.791232ms +32:55.320 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:45792 latency=298.205µs +32:55.320 [INFO] TestAgent: Requesting shutdown +32:55.320 [INFO] TestAgent.server: shutting down server +32:55.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:55.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:55.320 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:55.320 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:55.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.320 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:55.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:55.321 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:55.321 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:55.321 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:55.328 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.332Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.332Z [INFO] agent.router.manager: shutting down +32:55.332 [INFO] TestAgent: consul server down +32:55.332 [INFO] TestAgent: shutdown complete +32:55.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17176 network=tcp +32:55.332 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17176 network=udp +32:55.332 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17177 network=tcp +32:55.833 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.833 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_meta (1.35s) +2021-10-05T16:32:54.613Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:32:54.707Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +32:54.928 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cd3df830-ee9e-15fc-0c7f-619fb238e16f Address:127.0.0.1:17153}]" +32:54.929 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f.dc1 127.0.0.1 +32:54.929 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f 127.0.0.1 +2021-10-05T16:32:54.929Z [INFO] agent.router: Initializing LAN area manager +32:54.929 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= +32:54.930 [INFO] TestAgent.server: Adding LAN server: server="Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f (Addr: tcp/127.0.0.1:17153) (DC: dc1)" +32:54.930 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f.dc1 area=wan +32:54.940 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=udp +32:54.940 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=tcp +32:54.941 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17149 network=tcp +32:54.942 [INFO] TestAgent: started state syncer +32:54.942 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17154 network=tcp +32:54.985 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +32:54.985 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 +32:55.000 [DEBUG] TestAgent.server.raft: votes: needed=1 +32:55.000 [DEBUG] TestAgent.server.raft: vote granted: from=cd3df830-ee9e-15fc-0c7f-619fb238e16f term=2 tally=1 +32:55.000 [INFO] TestAgent.server.raft: election won: tally=1 +32:55.000 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" +32:55.004 [INFO] TestAgent.server: cluster leadership acquired +32:55.004 [INFO] TestAgent.server: New leader elected: payload=Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f +32:55.014 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17153 +32:55.044 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +32:55.095 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +32:55.096 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +32:55.096 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +32:55.096 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +32:55.096 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +32:55.096 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f +32:55.096 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cd3df830-ee9e-15fc-0c7f-619fb238e16f +32:55.126 [INFO] TestAgent.server: federation state anti-entropy synced +32:55.328 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:55.344 [INFO] TestAgent: Synced node info +32:55.380 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +32:55.380 [DEBUG] TestAgent: Node info in sync +32:55.380 [DEBUG] TestAgent: Node info in sync +32:55.396 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:47400 latency=23.418708ms +32:55.404 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:47402 latency=316.423µs +32:55.405 [INFO] TestAgent: Requesting shutdown +32:55.405 [INFO] TestAgent.server: shutting down server +32:55.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +32:55.405 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +32:55.405 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +32:55.405 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +32:55.405 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +32:55.405 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +32:55.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +32:55.405 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +32:55.405 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +32:55.405 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +32:55.416 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:32:55.424Z [INFO] agent.router.manager: shutting down +2021-10-05T16:32:55.424Z [INFO] agent.router.manager: shutting down +32:55.424 [INFO] TestAgent: consul server down +32:55.424 [INFO] TestAgent: shutdown complete +32:55.424 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp +32:55.424 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp +32:55.424 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp +32:55.928 [INFO] TestAgent: Waiting for endpoints to shut down +32:55.928 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand (1.45s) PASS -ok github.com/hashicorp/consul/command/intention/create 1.006s +ok github.com/hashicorp/consul/command/intention/create 1.562s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -85404,147 +85686,147 @@ === RUN TestCommand === PAUSE TestCommand === CONT TestCommand_noTabs +--- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand -=== CONT TestCommand_Validation [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestCommand_noTabs (0.00s) +=== CONT TestCommand_Validation +=== RUN TestCommand_Validation/3_args === RUN TestCommand_Validation/0_args [INFO] freeport: detected ephemeral port range of [32768, 60999] -=== RUN TestCommand_Validation/3_args [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCommand_Validation (0.00s) - --- PASS: TestCommand_Validation/0_args (0.00s) +--- PASS: TestCommand_Validation (0.01s) --- PASS: TestCommand_Validation/3_args (0.00s) -2022-11-07T22:46:09.878Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:09.892Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:09.919 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bfc14aa4-881d-2998-80ed-3c9945f5af66 Address:127.0.0.1:22258}]" -46:09.919 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= -46:09.920 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bfc14aa4-881d-2998-80ed-3c9945f5af66.dc1 127.0.0.1 -46:09.921 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bfc14aa4-881d-2998-80ed-3c9945f5af66 127.0.0.1 -2022-11-07T22:46:09.921Z [INFO] agent.router: Initializing LAN area manager -46:09.921 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bfc14aa4-881d-2998-80ed-3c9945f5af66.dc1 area=wan -46:09.921 [INFO] TestAgent.server: Adding LAN server: server="Node-bfc14aa4-881d-2998-80ed-3c9945f5af66 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" -46:09.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp -46:09.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp -46:09.922 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp -46:09.923 [INFO] TestAgent: started state syncer -46:09.923 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp -46:09.982 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:09.982 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 -46:09.991 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:09.991 [DEBUG] TestAgent.server.raft: vote granted: from=bfc14aa4-881d-2998-80ed-3c9945f5af66 term=2 tally=1 -46:09.991 [INFO] TestAgent.server.raft: election won: tally=1 -46:09.991 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" -46:09.991 [INFO] TestAgent.server: cluster leadership acquired -46:09.991 [INFO] TestAgent.server: New leader elected: payload=Node-bfc14aa4-881d-2998-80ed-3c9945f5af66 -46:09.997 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 -46:10.009 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:10.032 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:10.032 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:10.032 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:10.032 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:10.032 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:10.033 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bfc14aa4-881d-2998-80ed-3c9945f5af66 -46:10.033 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bfc14aa4-881d-2998-80ed-3c9945f5af66 -46:10.046 [INFO] TestAgent.server: federation state anti-entropy synced -46:10.183 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:10.188 [INFO] TestAgent: Synced node info -46:10.188 [DEBUG] TestAgent: Node info in sync -46:10.272 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:39330 latency=5.353855ms -46:10.276 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:39332 latency=2.973846ms -46:10.282 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/630f59e1-762f-cf66-e58e-45c4d3d3c2c7 from=127.0.0.1:39332 latency=4.470158ms -46:10.282 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:39330 latency=145.876µs -46:10.282 [INFO] TestAgent: Requesting shutdown -46:10.282 [INFO] TestAgent.server: shutting down server -46:10.282 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:10.282 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:10.282 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:10.282 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:10.282 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:10.283 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:10.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:10.283 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:10.283 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:10.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:10.287 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:10.295Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:10.295Z [INFO] agent.router.manager: shutting down -46:10.295 [INFO] TestAgent: consul server down -46:10.295 [INFO] TestAgent: shutdown complete -46:10.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp -46:10.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp -46:10.295 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp -46:10.795 [INFO] TestAgent: Waiting for endpoints to shut down -46:10.795 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand (0.95s) + --- PASS: TestCommand_Validation/0_args (0.00s) +2021-10-05T16:33:03.727Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:03.750Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:03.811 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:707552f4-5307-2b5a-0dad-abaf06ee3684 Address:127.0.0.1:25321}]" +33:03.812 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-707552f4-5307-2b5a-0dad-abaf06ee3684.dc1 127.0.0.1 +33:03.813 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-707552f4-5307-2b5a-0dad-abaf06ee3684 127.0.0.1 +2021-10-05T16:33:03.813Z [INFO] agent.router: Initializing LAN area manager +33:03.813 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp +33:03.814 [INFO] TestAgent.server: Adding LAN server: server="Node-707552f4-5307-2b5a-0dad-abaf06ee3684 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" +33:03.814 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-707552f4-5307-2b5a-0dad-abaf06ee3684.dc1 area=wan +33:03.814 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp +33:03.814 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp +33:03.815 [INFO] TestAgent: started state syncer +33:03.815 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp +33:03.815 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= +33:03.860 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:03.860 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 +33:03.872 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:03.872 [DEBUG] TestAgent.server.raft: vote granted: from=707552f4-5307-2b5a-0dad-abaf06ee3684 term=2 tally=1 +33:03.872 [INFO] TestAgent.server.raft: election won: tally=1 +33:03.872 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" +33:03.873 [INFO] TestAgent.server: cluster leadership acquired +33:03.874 [INFO] TestAgent.server: New leader elected: payload=Node-707552f4-5307-2b5a-0dad-abaf06ee3684 +33:03.881 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25321 +33:03.896 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:03.916 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:03.916 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:03.916 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:03.916 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:03.916 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:03.916 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-707552f4-5307-2b5a-0dad-abaf06ee3684 +33:03.916 [INFO] TestAgent.server: member joined, marking health alive: member=Node-707552f4-5307-2b5a-0dad-abaf06ee3684 +33:03.931 [INFO] TestAgent.server: federation state anti-entropy synced +33:04.068 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:55134 latency=11.215548ms +33:04.075 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:55136 latency=686.707µs +33:04.081 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/connect/intentions/63bfeb5d-73b1-8da1-3655-2b23eaae7b62 from=127.0.0.1:55136 latency=5.362846ms +33:04.082 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:55134 latency=187.776µs +33:04.082 [INFO] TestAgent: Requesting shutdown +33:04.082 [INFO] TestAgent.server: shutting down server +33:04.082 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:04.082 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:04.082 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:04.082 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:04.082 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:04.083 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:04.083 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:04.083 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:04.083 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:04.083 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:04.083 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:04.088 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:04.091Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:04.091Z [INFO] agent.router.manager: shutting down +33:04.091 [INFO] TestAgent: consul server down +33:04.091 [INFO] TestAgent: shutdown complete +33:04.091 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp +33:04.091 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp +33:04.092 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp +33:04.592 [INFO] TestAgent: Waiting for endpoints to shut down +33:04.592 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand (0.94s) PASS -ok github.com/hashicorp/consul/command/intention/delete 0.979s +ok github.com/hashicorp/consul/command/intention/delete 1.135s === RUN TestFinder === PAUSE TestFinder === CONT TestFinder [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:14.138Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.149Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:14.181 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a3299cec-fdf5-e588-fccb-167deeec147f Address:127.0.0.1:18174}]" -46:14.181 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -46:14.181 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a3299cec-fdf5-e588-fccb-167deeec147f.dc1 127.0.0.1 -46:14.182 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a3299cec-fdf5-e588-fccb-167deeec147f 127.0.0.1 -2022-11-07T22:46:14.182Z [INFO] agent.router: Initializing LAN area manager -46:14.182 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a3299cec-fdf5-e588-fccb-167deeec147f.dc1 area=wan -46:14.183 [INFO] TestAgent.server: Adding LAN server: server="Node-a3299cec-fdf5-e588-fccb-167deeec147f (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -46:14.183 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -46:14.183 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -46:14.183 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -46:14.183 [INFO] TestAgent: started state syncer -46:14.183 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -46:14.217 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:14.217 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -46:14.226 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:14.226 [DEBUG] TestAgent.server.raft: vote granted: from=a3299cec-fdf5-e588-fccb-167deeec147f term=2 tally=1 -46:14.226 [INFO] TestAgent.server.raft: election won: tally=1 -46:14.226 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -46:14.226 [INFO] TestAgent.server: cluster leadership acquired -46:14.227 [INFO] TestAgent.server: New leader elected: payload=Node-a3299cec-fdf5-e588-fccb-167deeec147f -46:14.234 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:18174 -46:14.258 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:14.301 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:14.301 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:14.301 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:14.301 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:14.301 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:14.301 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a3299cec-fdf5-e588-fccb-167deeec147f -46:14.301 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a3299cec-fdf5-e588-fccb-167deeec147f -46:14.326 [INFO] TestAgent.server: federation state anti-entropy synced -46:14.349 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:48468 latency=7.75244ms -46:14.351 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:48468 latency=350.487µs -46:14.351 [INFO] TestAgent: Requesting shutdown -46:14.351 [INFO] TestAgent.server: shutting down server -46:14.351 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:14.351 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:14.351 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:14.351 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:14.351 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:14.352 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:14.352 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:14.352 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:14.352 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:14.352 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:14.352 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:14.357 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:14.364Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:14.364Z [INFO] agent.router.manager: shutting down -46:14.364 [INFO] TestAgent: consul server down -46:14.364 [INFO] TestAgent: shutdown complete -46:14.364 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -46:14.364 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -46:14.364 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -46:14.864 [INFO] TestAgent: Waiting for endpoints to shut down -46:14.864 [INFO] TestAgent: Endpoints down ---- PASS: TestFinder (0.76s) +2021-10-05T16:33:04.154Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:04.168Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:04.199 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5ce5b782-64d8-fb9c-2fbc-ec01d05b150c Address:127.0.0.1:23279}]" +33:04.200 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c.dc1 127.0.0.1 +33:04.200 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c 127.0.0.1 +2021-10-05T16:33:04.200Z [INFO] agent.router: Initializing LAN area manager +33:04.201 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp +33:04.201 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= +33:04.201 [INFO] TestAgent.server: Adding LAN server: server="Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c (Addr: tcp/127.0.0.1:23279) (DC: dc1)" +33:04.201 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c.dc1 area=wan +33:04.202 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp +33:04.211 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp +33:04.211 [INFO] TestAgent: started state syncer +33:04.211 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp +33:04.245 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:04.245 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 +33:04.255 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:04.255 [DEBUG] TestAgent.server.raft: vote granted: from=5ce5b782-64d8-fb9c-2fbc-ec01d05b150c term=2 tally=1 +33:04.255 [INFO] TestAgent.server.raft: election won: tally=1 +33:04.255 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" +33:04.255 [INFO] TestAgent.server: cluster leadership acquired +33:04.255 [INFO] TestAgent.server: New leader elected: payload=Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c +33:04.265 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 +33:04.283 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:04.304 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:04.304 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:04.304 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:04.304 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:04.304 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:04.304 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c +33:04.304 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5ce5b782-64d8-fb9c-2fbc-ec01d05b150c +33:04.314 [INFO] TestAgent.server: federation state anti-entropy synced +33:04.596 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:04.605 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:36914 latency=20.431369ms +33:04.606 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:36914 latency=337.544µs +33:04.606 [INFO] TestAgent: Requesting shutdown +33:04.606 [INFO] TestAgent.server: shutting down server +33:04.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:04.606 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:04.606 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:04.606 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:04.606 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:04.606 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:04.607 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:04.607 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:04.607 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:04.607 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:04.610 [INFO] TestAgent: Synced node info +33:04.610 [DEBUG] TestAgent: Node info in sync +33:04.612 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:04.618Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:04.618Z [INFO] agent.router.manager: shutting down +33:04.618 [INFO] TestAgent: consul server down +33:04.618 [INFO] TestAgent: shutdown complete +33:04.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp +33:04.618 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp +33:04.618 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp +33:05.120 [INFO] TestAgent: Waiting for endpoints to shut down +33:05.120 [INFO] TestAgent: Endpoints down +--- PASS: TestFinder (1.07s) PASS -ok github.com/hashicorp/consul/command/intention/finder 0.787s +ok github.com/hashicorp/consul/command/intention/finder 1.182s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -85556,197 +85838,198 @@ === RUN TestCommand_verticalBar === PAUSE TestCommand_verticalBar === CONT TestCommand_noTabs +--- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand_verticalBar -=== CONT TestCommand_id [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestCommand_srcDst +=== CONT TestCommand_id === CONT TestCommand_Validation ---- PASS: TestCommand_noTabs (0.00s) -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== CONT TestCommand_srcDst === RUN TestCommand_Validation/0_args === RUN TestCommand_Validation/3_args ---- PASS: TestCommand_Validation (0.01s) +--- PASS: TestCommand_Validation (0.00s) --- PASS: TestCommand_Validation/0_args (0.00s) --- PASS: TestCommand_Validation/3_args (0.00s) -2022-11-07T22:46:14.980Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.992Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.029 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae52ccfa-4983-6234-f82c-19ee8896dbb6 Address:127.0.0.1:10006}]" -46:15.029 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -46:15.029 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6.dc1 127.0.0.1 -46:15.030 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6 127.0.0.1 -2022-11-07T22:46:15.030Z [INFO] agent.router: Initializing LAN area manager -46:15.030 [INFO] TestAgent.server: Adding LAN server: server="Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6 (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -46:15.030 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6.dc1 area=wan -46:15.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -46:15.030 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -46:15.031 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -46:15.031 [INFO] TestAgent: started state syncer -46:15.031 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -46:15.093 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.093 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -46:15.109 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.109 [DEBUG] TestAgent.server.raft: vote granted: from=ae52ccfa-4983-6234-f82c-19ee8896dbb6 term=2 tally=1 -46:15.109 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -46:15.109 [INFO] TestAgent.server: cluster leadership acquired -46:15.109 [INFO] TestAgent.server: New leader elected: payload=Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6 -46:15.114 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10006 -46:15.135 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.166 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.166 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.166 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.166 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.166 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.166 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6 -46:15.166 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ae52ccfa-4983-6234-f82c-19ee8896dbb6 -46:15.172 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.217 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:38048 latency=5.809519ms -46:15.218 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/ad0e5871-f950-0672-9d10-e916d5177780 from=127.0.0.1:38050 latency=585.773µs -46:15.219 [INFO] TestAgent: Requesting shutdown -46:15.219 [INFO] TestAgent.server: shutting down server -46:15.219 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.219 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.219 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.219 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.219 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.219 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.219 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.219 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.219 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.219 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.219 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.226 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.231Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.231Z [INFO] agent.router.manager: shutting down -46:15.231 [INFO] TestAgent: consul server down -46:15.231 [INFO] TestAgent: shutdown complete -46:15.231 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -46:15.231 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -46:15.231 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -46:15.731 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.731 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_verticalBar (0.79s) -2022-11-07T22:46:14.981Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.996Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.030 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a89c9371-ceeb-0687-283d-2e9a601f2392 Address:127.0.0.1:10020}]" -46:15.030 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10020 [Follower]" leader= -46:15.031 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a89c9371-ceeb-0687-283d-2e9a601f2392.dc1 127.0.0.1 -46:15.031 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a89c9371-ceeb-0687-283d-2e9a601f2392 127.0.0.1 -2022-11-07T22:46:15.031Z [INFO] agent.router: Initializing LAN area manager -46:15.031 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a89c9371-ceeb-0687-283d-2e9a601f2392.dc1 area=wan -46:15.031 [INFO] TestAgent.server: Adding LAN server: server="Node-a89c9371-ceeb-0687-283d-2e9a601f2392 (Addr: tcp/127.0.0.1:10020) (DC: dc1)" -46:15.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10015 network=tcp -46:15.031 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10015 network=udp -46:15.033 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10016 network=tcp -46:15.033 [INFO] TestAgent: started state syncer -46:15.033 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10021 network=tcp -46:15.075 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.075 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10020 [Candidate]" term=2 -46:15.087 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.087 [DEBUG] TestAgent.server.raft: vote granted: from=a89c9371-ceeb-0687-283d-2e9a601f2392 term=2 tally=1 -46:15.087 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.087 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10020 [Leader]" -46:15.087 [INFO] TestAgent.server: cluster leadership acquired -46:15.087 [INFO] TestAgent.server: New leader elected: payload=Node-a89c9371-ceeb-0687-283d-2e9a601f2392 -46:15.099 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10020 -46:15.116 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.146 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.146 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.146 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.146 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.146 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.147 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a89c9371-ceeb-0687-283d-2e9a601f2392 -46:15.147 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a89c9371-ceeb-0687-283d-2e9a601f2392 -46:15.157 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.264 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:42412 latency=3.970011ms -46:15.268 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:42414 latency=369.616µs -46:15.268 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/b825b9a6-229d-b321-c8c5-3cc674f43f16 from=127.0.0.1:42414 latency=121.104µs -46:15.269 [INFO] TestAgent: Requesting shutdown -46:15.269 [INFO] TestAgent.server: shutting down server -46:15.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.269 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.269 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.269 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.269 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.269 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.269 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:15.269 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.269 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.269 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.275 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.281Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.281Z [INFO] agent.router.manager: shutting down -46:15.281 [INFO] TestAgent: consul server down -46:15.281 [INFO] TestAgent: shutdown complete -46:15.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10015 network=tcp -46:15.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10015 network=udp -46:15.281 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10016 network=tcp -46:15.781 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.781 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_srcDst (0.84s) -2022-11-07T22:46:14.980Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:14.994Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:15.026 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 Address:127.0.0.1:10013}]" -46:15.026 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10013 [Follower]" leader= -46:15.026 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8.dc1 127.0.0.1 -46:15.027 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 127.0.0.1 -2022-11-07T22:46:15.027Z [INFO] agent.router: Initializing LAN area manager -46:15.027 [INFO] TestAgent.server: Adding LAN server: server="Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 (Addr: tcp/127.0.0.1:10013) (DC: dc1)" -46:15.027 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8.dc1 area=wan -46:15.027 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=tcp -46:15.027 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10008 network=udp -46:15.027 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10009 network=tcp -46:15.027 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10014 network=tcp -46:15.027 [INFO] TestAgent: started state syncer -46:15.093 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:15.094 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10013 [Candidate]" term=2 -46:15.110 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:15.110 [DEBUG] TestAgent.server.raft: vote granted: from=d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 term=2 tally=1 -46:15.110 [INFO] TestAgent.server.raft: election won: tally=1 -46:15.110 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10013 [Leader]" -46:15.110 [INFO] TestAgent.server: cluster leadership acquired -46:15.110 [INFO] TestAgent.server: New leader elected: payload=Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 -46:15.120 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10013 -46:15.139 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:15.169 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:15.169 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:15.169 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:15.169 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:15.169 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:15.169 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 -46:15.169 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d36b0c63-05a6-152c-2aa8-99ea3f6ceef8 -46:15.186 [INFO] TestAgent.server: federation state anti-entropy synced -46:15.398 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:15.402 [INFO] TestAgent: Synced node info -46:15.421 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46274 latency=7.074808ms -46:15.422 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/ad04e542-8c0f-bed8-6ab8-8e9147156011 from=127.0.0.1:46276 latency=230.798µs -46:15.423 [INFO] TestAgent: Requesting shutdown -46:15.423 [INFO] TestAgent.server: shutting down server -46:15.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:15.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:15.423 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:15.423 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:15.423 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:15.423 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:15.423 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:15.423 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:15.423 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:15.423 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:15.429 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:15.441Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:15.441Z [INFO] agent.router.manager: shutting down -46:15.441 [INFO] TestAgent: consul server down -46:15.441 [INFO] TestAgent: shutdown complete -46:15.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=tcp -46:15.441 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10008 network=udp -46:15.441 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10009 network=tcp -46:15.942 [INFO] TestAgent: Waiting for endpoints to shut down -46:15.942 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_id (1.00s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:33:06.297Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:06.354Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:06.398 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2a43f8d5-6964-8d52-56a1-13feeb076fb4 Address:127.0.0.1:31447}]" +33:06.398 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4.dc1 127.0.0.1 +33:06.399 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4 127.0.0.1 +2021-10-05T16:33:06.399Z [INFO] agent.router: Initializing LAN area manager +33:06.399 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp +33:06.399 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= +33:06.400 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4.dc1 area=wan +33:06.400 [INFO] TestAgent.server: Adding LAN server: server="Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" +33:06.400 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp +33:06.400 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp +33:06.400 [INFO] TestAgent: started state syncer +33:06.401 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp +33:06.448 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:06.448 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 +33:06.490 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:06.490 [DEBUG] TestAgent.server.raft: vote granted: from=2a43f8d5-6964-8d52-56a1-13feeb076fb4 term=2 tally=1 +33:06.490 [INFO] TestAgent.server.raft: election won: tally=1 +33:06.490 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" +33:06.490 [INFO] TestAgent.server: cluster leadership acquired +33:06.490 [INFO] TestAgent.server: New leader elected: payload=Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4 +33:06.502 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 +33:06.521 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:06.560 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:06.560 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:06.560 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:06.560 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:06.560 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:06.560 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4 +33:06.560 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2a43f8d5-6964-8d52-56a1-13feeb076fb4 +33:06.580 [INFO] TestAgent.server: federation state anti-entropy synced +33:06.585 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:06.591 [INFO] TestAgent: Synced node info +33:06.591 [DEBUG] TestAgent: Node info in sync +33:06.603 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46324 latency=4.053101ms +33:06.606 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/aa9a21ca-fd36-5f2f-facf-1b4109c01b51 from=127.0.0.1:46326 latency=1.409328ms +33:06.613 [INFO] TestAgent: Requesting shutdown +33:06.613 [INFO] TestAgent.server: shutting down server +33:06.613 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:06.613 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:06.613 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:06.613 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:06.613 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:06.613 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:06.613 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:06.613 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:06.613 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:06.613 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:06.619 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:06.623Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:06.623Z [INFO] agent.router.manager: shutting down +33:06.624 [INFO] TestAgent: consul server down +33:06.624 [INFO] TestAgent: shutdown complete +33:06.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp +33:06.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp +33:06.624 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp +33:07.128 [INFO] TestAgent: Waiting for endpoints to shut down +33:07.128 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_verticalBar (0.93s) +2021-10-05T16:33:06.306Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:06.371Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:06.416 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:08ce6db4-ed40-f653-566b-915a1df1414e Address:127.0.0.1:31461}]" +33:06.416 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-08ce6db4-ed40-f653-566b-915a1df1414e.dc1 127.0.0.1 +33:06.417 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-08ce6db4-ed40-f653-566b-915a1df1414e 127.0.0.1 +2021-10-05T16:33:06.417Z [INFO] agent.router: Initializing LAN area manager +33:06.417 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31456 network=udp +33:06.417 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31461 [Follower]" leader= +33:06.428 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-08ce6db4-ed40-f653-566b-915a1df1414e.dc1 area=wan +33:06.428 [INFO] TestAgent.server: Adding LAN server: server="Node-08ce6db4-ed40-f653-566b-915a1df1414e (Addr: tcp/127.0.0.1:31461) (DC: dc1)" +33:06.452 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31456 network=tcp +33:06.460 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31457 network=tcp +33:06.460 [INFO] TestAgent: started state syncer +33:06.460 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31462 network=tcp +33:06.461 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:06.461 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31461 [Candidate]" term=2 +33:06.482 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:06.482 [DEBUG] TestAgent.server.raft: vote granted: from=08ce6db4-ed40-f653-566b-915a1df1414e term=2 tally=1 +33:06.482 [INFO] TestAgent.server.raft: election won: tally=1 +33:06.482 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31461 [Leader]" +33:06.483 [INFO] TestAgent.server: cluster leadership acquired +33:06.483 [INFO] TestAgent.server: New leader elected: payload=Node-08ce6db4-ed40-f653-566b-915a1df1414e +33:06.497 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31461 +33:06.516 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:06.559 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:06.559 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:06.559 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:06.559 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:06.559 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:06.559 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-08ce6db4-ed40-f653-566b-915a1df1414e +33:06.560 [INFO] TestAgent.server: member joined, marking health alive: member=Node-08ce6db4-ed40-f653-566b-915a1df1414e +33:06.575 [INFO] TestAgent.server: federation state anti-entropy synced +33:06.680 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:49788 latency=5.141703ms +33:06.685 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions from=127.0.0.1:49790 latency=340.788µs +33:06.692 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/3ead9351-cc0d-8413-ccb7-85c03607ff2a from=127.0.0.1:49790 latency=233.945µs +33:06.696 [INFO] TestAgent: Requesting shutdown +33:06.696 [INFO] TestAgent.server: shutting down server +33:06.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:06.696 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:06.696 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:06.696 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:06.696 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:06.696 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:06.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:06.696 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:06.696 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:06.696 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:06.696 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:06.707 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:06.716Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:06.716Z [INFO] agent.router.manager: shutting down +33:06.716 [INFO] TestAgent: consul server down +33:06.716 [INFO] TestAgent: shutdown complete +33:06.716 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31456 network=tcp +33:06.716 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31456 network=udp +33:06.716 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31457 network=tcp +33:07.220 [INFO] TestAgent: Waiting for endpoints to shut down +33:07.220 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_srcDst (1.02s) +2021-10-05T16:33:06.309Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:06.379Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:06.424 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5c6a5595-b156-6778-3929-a2e9b2fc090d Address:127.0.0.1:31454}]" +33:06.428 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31454 [Follower]" leader= +33:06.452 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5c6a5595-b156-6778-3929-a2e9b2fc090d.dc1 127.0.0.1 +33:06.460 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5c6a5595-b156-6778-3929-a2e9b2fc090d 127.0.0.1 +2021-10-05T16:33:06.460Z [INFO] agent.router: Initializing LAN area manager +33:06.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=udp +33:06.461 [INFO] TestAgent.server: Adding LAN server: server="Node-5c6a5595-b156-6778-3929-a2e9b2fc090d (Addr: tcp/127.0.0.1:31454) (DC: dc1)" +33:06.461 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5c6a5595-b156-6778-3929-a2e9b2fc090d.dc1 area=wan +33:06.461 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=tcp +33:06.461 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31450 network=tcp +33:06.461 [INFO] TestAgent: started state syncer +33:06.461 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31455 network=tcp +33:06.472 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:06.472 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31454 [Candidate]" term=2 +33:06.502 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:06.502 [DEBUG] TestAgent.server.raft: vote granted: from=5c6a5595-b156-6778-3929-a2e9b2fc090d term=2 tally=1 +33:06.502 [INFO] TestAgent.server.raft: election won: tally=1 +33:06.502 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31454 [Leader]" +33:06.504 [INFO] TestAgent.server: cluster leadership acquired +33:06.504 [INFO] TestAgent.server: New leader elected: payload=Node-5c6a5595-b156-6778-3929-a2e9b2fc090d +33:06.515 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31454 +33:06.538 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:06.572 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:06.572 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:06.572 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:06.572 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:06.572 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:06.573 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5c6a5595-b156-6778-3929-a2e9b2fc090d +33:06.573 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5c6a5595-b156-6778-3929-a2e9b2fc090d +33:06.590 [INFO] TestAgent.server: federation state anti-entropy synced +33:06.808 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:46498 latency=8.777302ms +33:06.810 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/9f62fd09-64b6-cc40-af40-119d8c482378 from=127.0.0.1:46500 latency=264.345µs +33:06.810 [INFO] TestAgent: Requesting shutdown +33:06.810 [INFO] TestAgent.server: shutting down server +33:06.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:06.810 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:06.810 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:06.810 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:06.810 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:06.810 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:06.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:06.810 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:06.810 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:06.810 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:06.810 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:06.816 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:06.824Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:06.824Z [INFO] agent.router.manager: shutting down +33:06.824 [INFO] TestAgent: consul server down +33:06.824 [INFO] TestAgent: shutdown complete +33:06.824 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=tcp +33:06.824 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=udp +33:06.824 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31450 network=tcp +33:07.328 [INFO] TestAgent: Waiting for endpoints to shut down +33:07.328 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_id (1.13s) PASS -ok github.com/hashicorp/consul/command/intention/get 1.033s +ok github.com/hashicorp/consul/command/intention/get 1.235s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -85756,142 +86039,143 @@ === RUN TestCommand_matchSource === PAUSE TestCommand_matchSource === CONT TestCommand_noTabs +--- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand_matchSource +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestCommand_matchDst ---- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand_Validation === RUN TestCommand_Validation/0_args -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestCommand_Validation/3_args === RUN TestCommand_Validation/both_source_and_dest -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCommand_Validation (0.00s) +--- PASS: TestCommand_Validation (0.01s) --- PASS: TestCommand_Validation/0_args (0.00s) --- PASS: TestCommand_Validation/3_args (0.00s) --- PASS: TestCommand_Validation/both_source_and_dest (0.00s) -2022-11-07T22:46:19.017Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.070 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:65caafa5-2e8c-65b7-3837-aa05bd290a45 Address:127.0.0.1:31454}]" -46:19.070 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31454 [Follower]" leader= -46:19.071 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-65caafa5-2e8c-65b7-3837-aa05bd290a45.dc1 127.0.0.1 -46:19.071 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-65caafa5-2e8c-65b7-3837-aa05bd290a45 127.0.0.1 -2022-11-07T22:46:19.071Z [INFO] agent.router: Initializing LAN area manager -46:19.071 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-65caafa5-2e8c-65b7-3837-aa05bd290a45.dc1 area=wan -46:19.071 [INFO] TestAgent.server: Adding LAN server: server="Node-65caafa5-2e8c-65b7-3837-aa05bd290a45 (Addr: tcp/127.0.0.1:31454) (DC: dc1)" -46:19.071 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=tcp -46:19.071 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31449 network=udp -46:19.072 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31450 network=tcp -46:19.072 [INFO] TestAgent: started state syncer -46:19.072 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31455 network=tcp -46:19.112 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.112 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31454 [Candidate]" term=2 -46:19.125 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.125 [DEBUG] TestAgent.server.raft: vote granted: from=65caafa5-2e8c-65b7-3837-aa05bd290a45 term=2 tally=1 -46:19.125 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.125 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31454 [Leader]" -46:19.125 [INFO] TestAgent.server: cluster leadership acquired -46:19.126 [INFO] TestAgent.server: New leader elected: payload=Node-65caafa5-2e8c-65b7-3837-aa05bd290a45 -46:19.134 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31454 -46:19.151 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.182 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.182 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.182 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.182 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.182 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.182 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-65caafa5-2e8c-65b7-3837-aa05bd290a45 -46:19.182 [INFO] TestAgent.server: member joined, marking health alive: member=Node-65caafa5-2e8c-65b7-3837-aa05bd290a45 -46:19.192 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.402 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57898 latency=4.501938ms -46:19.409 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57898 latency=4.60462ms -46:19.414 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57898 latency=4.299275ms -46:19.415 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match?by=destination&name=db from=127.0.0.1:57900 latency=233.512µs -46:19.415 [INFO] TestAgent: Requesting shutdown -46:19.415 [INFO] TestAgent.server: shutting down server -46:19.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.415 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.415 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.415 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.415 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.415 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.415 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.415 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.415 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.415 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.424 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.431Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.431Z [INFO] agent.router.manager: shutting down -46:19.431 [INFO] TestAgent: consul server down -46:19.431 [INFO] TestAgent: shutdown complete -46:19.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=tcp -46:19.431 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31449 network=udp -46:19.431 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31450 network=tcp -46:19.931 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.931 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_matchDst (0.95s) -2022-11-07T22:46:19.016Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:19.029Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:19.062 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c Address:127.0.0.1:31447}]" -46:19.062 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= -46:19.063 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c.dc1 127.0.0.1 -46:19.063 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c 127.0.0.1 -2022-11-07T22:46:19.063Z [INFO] agent.router: Initializing LAN area manager -46:19.063 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c.dc1 area=wan -46:19.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp -46:19.064 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp -46:19.065 [INFO] TestAgent.server: Adding LAN server: server="Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c (Addr: tcp/127.0.0.1:31447) (DC: dc1)" -46:19.065 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp -46:19.065 [INFO] TestAgent: started state syncer -46:19.066 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp -46:19.114 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:19.114 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 -46:19.127 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:19.127 [DEBUG] TestAgent.server.raft: vote granted: from=02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c term=2 tally=1 -46:19.127 [INFO] TestAgent.server.raft: election won: tally=1 -46:19.127 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" -46:19.127 [INFO] TestAgent.server: cluster leadership acquired -46:19.127 [INFO] TestAgent.server: New leader elected: payload=Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c -46:19.133 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 -46:19.149 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:19.177 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:19.177 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:19.177 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:19.177 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:19.177 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:19.177 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c -46:19.177 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02cd7cc3-58ce-c6c7-dd4e-cbba6d5c8e6c -46:19.196 [INFO] TestAgent.server: federation state anti-entropy synced -46:19.462 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57418 latency=4.061374ms -46:19.467 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57418 latency=4.407521ms -46:19.471 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:57418 latency=4.353844ms -46:19.472 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match?by=source&name=foo from=127.0.0.1:57420 latency=195.16µs -46:19.473 [INFO] TestAgent: Requesting shutdown -46:19.473 [INFO] TestAgent.server: shutting down server -46:19.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:19.473 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:19.473 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:19.473 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:19.473 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:19.473 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:19.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:19.473 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:19.473 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:19.473 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:19.473 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:19.478 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:19.481Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:19.481Z [INFO] agent.router.manager: shutting down -46:19.482 [INFO] TestAgent: consul server down -46:19.482 [INFO] TestAgent: shutdown complete -46:19.482 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp -46:19.482 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp -46:19.482 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp -46:19.982 [INFO] TestAgent: Waiting for endpoints to shut down -46:19.982 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_matchSource (1.00s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:33:16.186Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.243Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.299 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:902e3671-8b7a-4e65-696f-9cf95e2ac89d Address:127.0.0.1:15118}]" +33:16.299 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= +33:16.299 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d.dc1 127.0.0.1 +33:16.300 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d 127.0.0.1 +2021-10-05T16:33:16.300Z [INFO] agent.router: Initializing LAN area manager +33:16.301 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp +33:16.301 [INFO] TestAgent.server: Adding LAN server: server="Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +33:16.301 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp +33:16.301 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d.dc1 area=wan +33:16.301 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp +33:16.301 [INFO] TestAgent: started state syncer +33:16.302 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp +33:16.360 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.360 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 +33:16.374 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:16.374 [DEBUG] TestAgent.server.raft: vote granted: from=902e3671-8b7a-4e65-696f-9cf95e2ac89d term=2 tally=1 +33:16.374 [INFO] TestAgent.server.raft: election won: tally=1 +33:16.374 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" +33:16.375 [INFO] TestAgent.server: cluster leadership acquired +33:16.375 [INFO] TestAgent.server: New leader elected: payload=Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d +33:16.388 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15118 +33:16.405 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:16.417 [INFO] TestAgent: Synced node info +33:16.420 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:38712 latency=8.230728ms +33:16.437 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:38712 latency=13.417019ms +33:16.437 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:16.437 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:16.437 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:16.437 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:16.437 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:16.437 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d +33:16.438 [INFO] TestAgent.server: member joined, marking health alive: member=Node-902e3671-8b7a-4e65-696f-9cf95e2ac89d +33:16.445 [INFO] TestAgent.server: federation state anti-entropy synced +33:16.460 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:38712 latency=12.31934ms +33:16.463 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match?by=destination&name=db from=127.0.0.1:38716 latency=989.285µs +33:16.463 [INFO] TestAgent: Requesting shutdown +33:16.463 [INFO] TestAgent.server: shutting down server +33:16.463 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:16.463 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:16.463 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:16.463 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:16.463 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:16.463 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:16.463 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:16.463 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:16.463 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:16.463 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:16.467 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:16.470Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:16.470Z [INFO] agent.router.manager: shutting down +33:16.470 [INFO] TestAgent: consul server down +33:16.470 [INFO] TestAgent: shutdown complete +33:16.470 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp +33:16.470 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp +33:16.470 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp +33:16.971 [INFO] TestAgent: Waiting for endpoints to shut down +33:16.971 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_matchDst (0.85s) +2021-10-05T16:33:16.215Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.231Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.263 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6075e30e-3897-d70e-5ed0-dc5b60219842 Address:127.0.0.1:15111}]" +33:16.264 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +33:16.264 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-6075e30e-3897-d70e-5ed0-dc5b60219842.dc1 127.0.0.1 +33:16.265 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-6075e30e-3897-d70e-5ed0-dc5b60219842 127.0.0.1 +2021-10-05T16:33:16.265Z [INFO] agent.router: Initializing LAN area manager +33:16.265 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +33:16.265 [INFO] TestAgent.server: Adding LAN server: server="Node-6075e30e-3897-d70e-5ed0-dc5b60219842 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +33:16.265 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-6075e30e-3897-d70e-5ed0-dc5b60219842.dc1 area=wan +33:16.265 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +33:16.266 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +33:16.266 [INFO] TestAgent: started state syncer +33:16.266 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +33:16.318 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.318 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +33:16.331 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:16.331 [DEBUG] TestAgent.server.raft: vote granted: from=6075e30e-3897-d70e-5ed0-dc5b60219842 term=2 tally=1 +33:16.331 [INFO] TestAgent.server.raft: election won: tally=1 +33:16.331 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +33:16.332 [INFO] TestAgent.server: cluster leadership acquired +33:16.332 [INFO] TestAgent.server: New leader elected: payload=Node-6075e30e-3897-d70e-5ed0-dc5b60219842 +33:16.337 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 +33:16.351 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:16.360 [INFO] TestAgent: Synced node info +33:16.360 [DEBUG] TestAgent: Node info in sync +33:16.385 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:16.385 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:16.385 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:16.385 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:16.385 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:16.385 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-6075e30e-3897-d70e-5ed0-dc5b60219842 +33:16.385 [INFO] TestAgent.server: member joined, marking health alive: member=Node-6075e30e-3897-d70e-5ed0-dc5b60219842 +33:16.402 [INFO] TestAgent.server: federation state anti-entropy synced +33:16.642 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:56710 latency=4.309488ms +33:16.649 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:56710 latency=6.528321ms +33:16.672 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:56710 latency=7.90288ms +33:16.683 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/connect/intentions/match?by=source&name=foo from=127.0.0.1:56712 latency=2.223201ms +33:16.684 [INFO] TestAgent: Requesting shutdown +33:16.684 [INFO] TestAgent.server: shutting down server +33:16.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:16.684 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:16.684 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:16.684 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:16.684 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:16.684 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:16.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:16.684 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:16.684 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:16.684 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:16.687 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:16.694Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:16.694Z [INFO] agent.router.manager: shutting down +33:16.694 [INFO] TestAgent: consul server down +33:16.694 [INFO] TestAgent: shutdown complete +33:16.694 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +33:16.694 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +33:16.694 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +33:17.194 [INFO] TestAgent: Waiting for endpoints to shut down +33:17.194 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_matchSource (1.08s) PASS -ok github.com/hashicorp/consul/command/intention/match 1.031s +ok github.com/hashicorp/consul/command/intention/match 1.160s === RUN TestJoinCommand_noTabs === PAUSE TestJoinCommand_noTabs === RUN TestJoinCommandJoin_lan @@ -85901,275 +86185,275 @@ === RUN TestJoinCommand_noAddrs === PAUSE TestJoinCommand_noAddrs === CONT TestJoinCommand_noTabs +--- PASS: TestJoinCommand_noTabs (0.00s) === CONT TestJoinCommand_noAddrs -=== CONT TestJoinCommandJoin_lan +--- PASS: TestJoinCommand_noAddrs (0.00s) === CONT TestJoinCommand_wan [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestJoinCommand_noTabs (0.00s) ---- PASS: TestJoinCommand_noAddrs (0.00s) +=== CONT TestJoinCommandJoin_lan [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:20.278Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.292Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.332 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c9f60756-7f84-4cb0-5842-5c2816295ca2 Address:127.0.0.1:23293}]" -46:20.332 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23293 [Follower]" leader= -46:20.332 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9f60756-7f84-4cb0-5842-5c2816295ca2.dc1 127.0.0.1 -46:20.333 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c9f60756-7f84-4cb0-5842-5c2816295ca2 127.0.0.1 -2022-11-07T22:46:20.333Z [INFO] agent.router: Initializing LAN area manager -46:20.333 [INFO] TestAgent.server: Adding LAN server: server="Node-c9f60756-7f84-4cb0-5842-5c2816295ca2 (Addr: tcp/127.0.0.1:23293) (DC: dc1)" -46:20.333 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9f60756-7f84-4cb0-5842-5c2816295ca2.dc1 area=wan -46:20.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=tcp -46:20.333 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=udp -46:20.333 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23289 network=tcp -46:20.334 [INFO] TestAgent: started state syncer -46:20.334 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23294 network=tcp -46:20.386 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.386 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23293 [Candidate]" term=2 -46:20.400 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.400 [DEBUG] TestAgent.server.raft: vote granted: from=c9f60756-7f84-4cb0-5842-5c2816295ca2 term=2 tally=1 -46:20.400 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.400 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23293 [Leader]" -46:20.400 [INFO] TestAgent.server: cluster leadership acquired -46:20.400 [INFO] TestAgent.server: New leader elected: payload=Node-c9f60756-7f84-4cb0-5842-5c2816295ca2 -46:20.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23293 -46:20.428 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.458 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.459 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.459 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.459 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.459 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.459 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c9f60756-7f84-4cb0-5842-5c2816295ca2 -46:20.459 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c9f60756-7f84-4cb0-5842-5c2816295ca2 -46:20.472 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.531 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.533 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:57288 -46:20.534 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a12b2f77-1924-e6d1-fee5-d63857341439.dc1 127.0.0.1 -46:20.534 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a12b2f77-1924-e6d1-fee5-d63857341439.dc1 area=wan -46:20.534 [INFO] TestAgent: Requesting shutdown -46:20.534 [INFO] TestAgent.server: shutting down server -46:20.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.534 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.534 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.534 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.534 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.534 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.534 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.534 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.534 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.535 [INFO] TestAgent: Synced node info -46:20.539 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.545Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.545Z [INFO] agent.router.manager: shutting down -46:20.545 [INFO] TestAgent: consul server down -46:20.545 [INFO] TestAgent: shutdown complete -46:20.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=tcp -46:20.545 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=udp -46:20.545 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23289 network=tcp -46:21.045 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.045 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.149Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.160Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.192 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a12b2f77-1924-e6d1-fee5-d63857341439 Address:127.0.0.1:23286}]" -46:20.192 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23286 [Follower]" leader= -46:20.192 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a12b2f77-1924-e6d1-fee5-d63857341439.dc1 127.0.0.1 -46:20.195 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a12b2f77-1924-e6d1-fee5-d63857341439 127.0.0.1 -2022-11-07T22:46:20.195Z [INFO] agent.router: Initializing LAN area manager -46:20.195 [INFO] TestAgent.server: Adding LAN server: server="Node-a12b2f77-1924-e6d1-fee5-d63857341439 (Addr: tcp/127.0.0.1:23286) (DC: dc1)" -46:20.195 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a12b2f77-1924-e6d1-fee5-d63857341439.dc1 area=wan -46:20.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=udp -46:20.195 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=tcp -46:20.196 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23282 network=tcp -46:20.196 [INFO] TestAgent: started state syncer -46:20.196 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23287 network=tcp -46:20.238 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.238 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23286 [Candidate]" term=2 -46:20.248 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.248 [DEBUG] TestAgent.server.raft: vote granted: from=a12b2f77-1924-e6d1-fee5-d63857341439 term=2 tally=1 -46:20.248 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.248 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23286 [Leader]" -46:20.248 [INFO] TestAgent.server: cluster leadership acquired -46:20.248 [INFO] TestAgent.server: New leader elected: payload=Node-a12b2f77-1924-e6d1-fee5-d63857341439 -46:20.260 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23286 -46:20.290 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.326 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.326 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.326 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.326 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.326 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.326 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a12b2f77-1924-e6d1-fee5-d63857341439 -46:20.326 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a12b2f77-1924-e6d1-fee5-d63857341439 -46:20.345 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.495 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.498 [INFO] TestAgent: Synced node info -46:20.498 [DEBUG] TestAgent: Node info in sync -46:20.533 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:23292] -46:20.533 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:23292 -46:20.534 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c9f60756-7f84-4cb0-5842-5c2816295ca2.dc1 127.0.0.1 -46:20.534 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 -46:20.534 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/127.0.0.1:23292?wan=1 from=127.0.0.1:57850 latency=1.221372ms -46:20.534 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c9f60756-7f84-4cb0-5842-5c2816295ca2.dc1 area=wan -46:21.045 [INFO] TestAgent: Requesting shutdown -46:21.045 [INFO] TestAgent.server: shutting down server -46:21.045 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.045 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.045 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.045 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.045 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.045 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.045 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.045 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.045 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.045 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.049 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.055Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.055Z [INFO] agent.router.manager: shutting down -46:21.055 [INFO] TestAgent: consul server down -46:21.055 [INFO] TestAgent: shutdown complete -46:21.055 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=tcp -46:21.055 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=udp -46:21.055 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23282 network=tcp -46:21.556 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.556 [INFO] TestAgent: Endpoints down ---- PASS: TestJoinCommand_wan (1.44s) -2022-11-07T22:46:20.292Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.308Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.347 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:af204203-ebfd-55a3-10bd-395c1252555b Address:127.0.0.1:23300}]" -46:20.348 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23300 [Follower]" leader= -46:20.348 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-af204203-ebfd-55a3-10bd-395c1252555b.dc1 127.0.0.1 -46:20.349 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-af204203-ebfd-55a3-10bd-395c1252555b 127.0.0.1 -2022-11-07T22:46:20.349Z [INFO] agent.router: Initializing LAN area manager -46:20.349 [INFO] TestAgent.server: Adding LAN server: server="Node-af204203-ebfd-55a3-10bd-395c1252555b (Addr: tcp/127.0.0.1:23300) (DC: dc1)" -46:20.349 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-af204203-ebfd-55a3-10bd-395c1252555b.dc1 area=wan -46:20.349 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=udp -46:20.349 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=tcp -46:20.350 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23296 network=tcp -46:20.350 [INFO] TestAgent: started state syncer -46:20.350 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23301 network=tcp -46:20.410 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.410 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23300 [Candidate]" term=2 -46:20.424 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.424 [DEBUG] TestAgent.server.raft: vote granted: from=af204203-ebfd-55a3-10bd-395c1252555b term=2 tally=1 -46:20.424 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.424 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23300 [Leader]" -46:20.424 [INFO] TestAgent.server: cluster leadership acquired -46:20.424 [INFO] TestAgent.server: New leader elected: payload=Node-af204203-ebfd-55a3-10bd-395c1252555b -46:20.432 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23300 -46:20.449 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.476 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.476 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.476 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.476 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.476 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.476 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-af204203-ebfd-55a3-10bd-395c1252555b -46:20.476 [INFO] TestAgent.server: member joined, marking health alive: member=Node-af204203-ebfd-55a3-10bd-395c1252555b -46:20.490 [INFO] TestAgent: Synced node info -46:20.490 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.723 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:37678 -46:20.724 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f 127.0.0.1 -46:20.724 [INFO] TestAgent.server: Adding LAN server: server="Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f (Addr: tcp/127.0.0.1:23279) (DC: dc1)" -46:20.724 [INFO] TestAgent.server: New leader elected: payload=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.724 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f other=Node-af204203-ebfd-55a3-10bd-395c1252555b -46:20.724 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.724 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 127.0.0.1:23278 -46:20.724 [INFO] TestAgent: Requesting shutdown -46:20.724 [INFO] TestAgent.server: shutting down server -46:20.724 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:20.724 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:20.724 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:20.724 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:20.724 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:20.724 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:20.724 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:20.724 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:20.724 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:20.724 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:20.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 127.0.0.1 -46:20.725 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 address=127.0.0.1:23278 -46:20.725 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 area=wan -46:20.732 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:20.741Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:20.741Z [INFO] agent.router.manager: shutting down -46:20.741 [INFO] TestAgent: consul server down -46:20.741 [INFO] TestAgent: shutdown complete -46:20.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=tcp -46:20.741 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=udp -46:20.741 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23296 network=tcp -46:21.241 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.241 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:20.149Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:20.164Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:20.197 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1221bd6c-16ac-a67e-dd8a-3c905d44463f Address:127.0.0.1:23279}]" -46:20.197 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= -46:20.198 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 127.0.0.1 -46:20.198 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f 127.0.0.1 -2022-11-07T22:46:20.199Z [INFO] agent.router: Initializing LAN area manager -46:20.199 [INFO] TestAgent.server: Adding LAN server: server="Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f (Addr: tcp/127.0.0.1:23279) (DC: dc1)" -46:20.199 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f.dc1 area=wan -46:20.199 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp -46:20.199 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp -46:20.199 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp -46:20.199 [INFO] TestAgent: started state syncer -46:20.199 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp -46:20.236 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:20.236 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 -46:20.244 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:20.244 [DEBUG] TestAgent.server.raft: vote granted: from=1221bd6c-16ac-a67e-dd8a-3c905d44463f term=2 tally=1 -46:20.244 [INFO] TestAgent.server.raft: election won: tally=1 -46:20.244 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" -46:20.244 [INFO] TestAgent.server: cluster leadership acquired -46:20.244 [INFO] TestAgent.server: New leader elected: payload=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.252 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 -46:20.271 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:20.317 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:20.317 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:20.317 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:20.317 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:20.317 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:20.317 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.317 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.333 [INFO] TestAgent.server: federation state anti-entropy synced -46:20.367 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:20.370 [INFO] TestAgent: Synced node info -46:20.370 [DEBUG] TestAgent: Node info in sync -46:20.723 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:23298] -46:20.723 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:23298 -46:20.724 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-af204203-ebfd-55a3-10bd-395c1252555b 127.0.0.1 -46:20.724 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:43868 -46:20.724 [INFO] TestAgent.server: Adding LAN server: server="Node-af204203-ebfd-55a3-10bd-395c1252555b (Addr: tcp/127.0.0.1:23300) (DC: dc1)" -46:20.724 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 -46:20.724 [DEBUG] TestAgent: systemd notify failed: error="No socket" -46:20.724 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/127.0.0.1:23298 from=127.0.0.1:40366 latency=1.305243ms -46:20.724 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-af204203-ebfd-55a3-10bd-395c1252555b other=Node-1221bd6c-16ac-a67e-dd8a-3c905d44463f -46:20.724 [INFO] TestAgent.server: member joined, marking health alive: member=Node-af204203-ebfd-55a3-10bd-395c1252555b -46:20.725 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-af204203-ebfd-55a3-10bd-395c1252555b.dc1 127.0.0.1 -46:20.725 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-af204203-ebfd-55a3-10bd-395c1252555b.dc1 area=wan -46:21.241 [INFO] TestAgent: Requesting shutdown -46:21.241 [INFO] TestAgent.server: shutting down server -46:21.241 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:21.241 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:21.241 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:21.241 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:21.241 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:21.242 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:21.242 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:21.242 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:21.242 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:21.242 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:21.249 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:21.258Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:21.258Z [INFO] agent.router.manager: shutting down -46:21.258 [INFO] TestAgent: consul server down -46:21.258 [INFO] TestAgent: shutdown complete -46:21.258 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp -46:21.258 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp -46:21.259 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp -46:21.699 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-af204203-ebfd-55a3-10bd-395c1252555b (timeout reached) -46:21.759 [INFO] TestAgent: Waiting for endpoints to shut down -46:21.759 [INFO] TestAgent: Endpoints down ---- PASS: TestJoinCommandJoin_lan (1.65s) +2021-10-05T16:33:16.870Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.883Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.919 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:84fa357a-2943-b6a3-f616-563e117d4fad Address:127.0.0.1:17174}]" +33:16.920 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17174 [Follower]" leader= +33:16.920 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-84fa357a-2943-b6a3-f616-563e117d4fad.dc1 127.0.0.1 +33:16.921 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-84fa357a-2943-b6a3-f616-563e117d4fad 127.0.0.1 +2021-10-05T16:33:16.921Z [INFO] agent.router: Initializing LAN area manager +33:16.921 [INFO] TestAgent.server: Adding LAN server: server="Node-84fa357a-2943-b6a3-f616-563e117d4fad (Addr: tcp/127.0.0.1:17174) (DC: dc1)" +33:16.921 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-84fa357a-2943-b6a3-f616-563e117d4fad.dc1 area=wan +33:16.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=tcp +33:16.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=udp +33:16.922 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17170 network=tcp +33:16.922 [INFO] TestAgent: started state syncer +33:16.922 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17175 network=tcp +33:16.989 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.989 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17174 [Candidate]" term=2 +33:17.002 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:17.002 [DEBUG] TestAgent.server.raft: vote granted: from=84fa357a-2943-b6a3-f616-563e117d4fad term=2 tally=1 +33:17.002 [INFO] TestAgent.server.raft: election won: tally=1 +33:17.002 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17174 [Leader]" +33:17.004 [INFO] TestAgent.server: cluster leadership acquired +33:17.004 [INFO] TestAgent.server: New leader elected: payload=Node-84fa357a-2943-b6a3-f616-563e117d4fad +33:17.013 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17174 +33:17.048 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:17.080 [INFO] TestAgent: Synced node info +33:17.080 [DEBUG] TestAgent: Node info in sync +33:17.112 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:17.112 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:17.112 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:17.112 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:17.112 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:17.112 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-84fa357a-2943-b6a3-f616-563e117d4fad +33:17.112 [INFO] TestAgent.server: member joined, marking health alive: member=Node-84fa357a-2943-b6a3-f616-563e117d4fad +33:17.125 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Stream connection from=127.0.0.1:45176 +33:17.126 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 127.0.0.1 +33:17.126 [INFO] TestAgent.server: Adding LAN server: server="Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 (Addr: tcp/127.0.0.1:17160) (DC: dc1)" +33:17.126 [INFO] TestAgent.server: New leader elected: payload=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:17.127 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 127.0.0.1:17159 +33:17.128 [INFO] TestAgent: Requesting shutdown +33:17.128 [INFO] TestAgent.server: shutting down server +33:17.128 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:17.128 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:17.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:17.128 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:17.128 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:17.141 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 other=Node-84fa357a-2943-b6a3-f616-563e117d4fad +33:17.141 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:17.142 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 127.0.0.1 +33:17.142 [DEBUG] TestAgent.server: Successfully performed flood-join for server at address: server=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 address=127.0.0.1:17159 +33:17.143 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 area=wan +33:17.144 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +33:17.144 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:17.144 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:33:17.172Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:17.172Z [INFO] agent.router.manager: shutting down +33:17.188 [INFO] TestAgent: consul server down +33:17.188 [INFO] TestAgent: shutdown complete +33:17.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=tcp +33:17.188 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=udp +33:17.188 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17170 network=tcp +33:17.188 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 127.0.0.1 17158 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:5162f780-b159-6ff9-01d8-7b7a30a3cc12 port:17160 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:17159] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +33:17.692 [INFO] TestAgent: Waiting for endpoints to shut down +33:17.692 [INFO] TestAgent: Endpoints down +2021-10-05T16:33:16.470Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.488Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.528 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5162f780-b159-6ff9-01d8-7b7a30a3cc12 Address:127.0.0.1:17160}]" +33:16.529 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 127.0.0.1 +33:16.530 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 127.0.0.1 +2021-10-05T16:33:16.530Z [INFO] agent.router: Initializing LAN area manager +33:16.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=udp +33:16.530 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17160 [Follower]" leader= +33:16.530 [INFO] TestAgent.server: Adding LAN server: server="Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 (Addr: tcp/127.0.0.1:17160) (DC: dc1)" +33:16.530 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12.dc1 area=wan +33:16.530 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=tcp +33:16.531 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17156 network=tcp +33:16.531 [INFO] TestAgent: started state syncer +33:16.531 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17161 network=tcp +33:16.580 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.580 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17160 [Candidate]" term=2 +33:16.587 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:16.587 [DEBUG] TestAgent.server.raft: vote granted: from=5162f780-b159-6ff9-01d8-7b7a30a3cc12 term=2 tally=1 +33:16.587 [INFO] TestAgent.server.raft: election won: tally=1 +33:16.587 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17160 [Leader]" +33:16.588 [INFO] TestAgent.server: cluster leadership acquired +33:16.588 [INFO] TestAgent.server: New leader elected: payload=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:16.592 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17160 +33:16.606 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:16.637 [INFO] TestAgent: Synced node info +33:16.665 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:16.665 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:16.665 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:16.665 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:16.665 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:16.665 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:16.665 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:16.684 [INFO] TestAgent.server: federation state anti-entropy synced +33:17.125 [INFO] TestAgent: (LAN) joining: lan_addresses=[127.0.0.1:17172] +33:17.125 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.1:17172 +33:17.127 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:38308 +33:17.127 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-84fa357a-2943-b6a3-f616-563e117d4fad 127.0.0.1 +33:17.127 [INFO] TestAgent: (LAN) joined: number_of_nodes=1 +33:17.127 [DEBUG] TestAgent: systemd notify failed: error="No socket" +33:17.127 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/127.0.0.1:17172 from=127.0.0.1:57334 latency=2.09233ms +33:17.127 [INFO] TestAgent.server: Adding LAN server: server="Node-84fa357a-2943-b6a3-f616-563e117d4fad (Addr: tcp/127.0.0.1:17174) (DC: dc1)" +33:17.127 [ERROR] TestAgent.server: Two nodes are in bootstrap mode. Only one node should be in bootstrap mode, not adding Raft peer.: node_to_add=Node-84fa357a-2943-b6a3-f616-563e117d4fad other=Node-5162f780-b159-6ff9-01d8-7b7a30a3cc12 +33:17.127 [INFO] TestAgent.server: member joined, marking health alive: member=Node-84fa357a-2943-b6a3-f616-563e117d4fad +33:17.128 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-84fa357a-2943-b6a3-f616-563e117d4fad.dc1 127.0.0.1 +33:17.128 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-84fa357a-2943-b6a3-f616-563e117d4fad.dc1 area=wan +33:17.471 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:17.471 [DEBUG] TestAgent: Node info in sync +33:17.471 [DEBUG] TestAgent: Node info in sync +33:17.692 [INFO] TestAgent: Requesting shutdown +33:17.692 [INFO] TestAgent.server: shutting down server +33:17.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:17.692 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:17.692 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:17.692 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:17.692 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:17.692 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:17.692 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:17.692 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:17.692 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:17.692 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:17.768 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:17.772Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:17.772Z [INFO] agent.router.manager: shutting down +33:17.772 [INFO] TestAgent: consul server down +33:17.772 [INFO] TestAgent: shutdown complete +33:17.772 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=tcp +33:17.772 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=udp +33:17.772 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17156 network=tcp +33:18.031 [DEBUG] TestAgent.server.memberlist.lan: memberlist: Failed ping: Node-84fa357a-2943-b6a3-f616-563e117d4fad (timeout reached) +33:18.273 [INFO] TestAgent: Waiting for endpoints to shut down +33:18.273 [INFO] TestAgent: Endpoints down +--- PASS: TestJoinCommandJoin_lan (1.90s) +2021-10-05T16:33:16.757Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.815Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.898 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:49effe1b-69ef-dbdb-32b1-c3b237d1496b Address:127.0.0.1:17167}]" +33:16.899 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17167 [Follower]" leader= +33:16.899 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b.dc1 127.0.0.1 +33:16.900 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b 127.0.0.1 +2021-10-05T16:33:16.900Z [INFO] agent.router: Initializing LAN area manager +33:16.900 [INFO] TestAgent.server: Adding LAN server: server="Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b (Addr: tcp/127.0.0.1:17167) (DC: dc1)" +33:16.900 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b.dc1 area=wan +33:16.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=udp +33:16.911 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=tcp +33:16.912 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17163 network=tcp +33:16.912 [INFO] TestAgent: started state syncer +33:16.912 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17168 network=tcp +33:16.956 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.956 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17167 [Candidate]" term=2 +33:16.968 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:16.969 [DEBUG] TestAgent.server.raft: vote granted: from=49effe1b-69ef-dbdb-32b1-c3b237d1496b term=2 tally=1 +33:16.969 [INFO] TestAgent.server.raft: election won: tally=1 +33:16.969 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17167 [Leader]" +33:16.969 [INFO] TestAgent.server: cluster leadership acquired +33:16.969 [INFO] TestAgent.server: New leader elected: payload=Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b +33:16.975 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17167 +33:16.998 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:17.036 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:17.036 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:17.036 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:17.036 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:17.036 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:17.036 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b +33:17.036 [INFO] TestAgent.server: member joined, marking health alive: member=Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b +33:17.096 [INFO] TestAgent.server: federation state anti-entropy synced +33:17.096 [INFO] TestAgent: Synced node info +33:17.096 [DEBUG] TestAgent: Node info in sync +33:17.126 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Stream connection from=127.0.0.1:37892 +33:17.126 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5.dc1 127.0.0.1 +33:17.126 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5.dc1 area=wan +33:17.127 [INFO] TestAgent: Requesting shutdown +33:17.127 [INFO] TestAgent.server: shutting down server +33:17.127 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:17.127 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:17.127 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:17.127 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:17.127 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:17.128 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:17.128 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:17.143 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:17.172Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:17.172Z [INFO] agent.router.manager: shutting down +33:17.172 [INFO] TestAgent: consul server down +33:17.172 [INFO] TestAgent: shutdown complete +33:17.172 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=tcp +33:17.172 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=udp +33:17.172 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17163 network=tcp +33:17.676 [INFO] TestAgent: Waiting for endpoints to shut down +33:17.676 [INFO] TestAgent: Endpoints down +2021-10-05T16:33:16.490Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:16.525Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:16.566 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8f1936c0-e338-d3f4-b23c-f1e08387cef5 Address:127.0.0.1:17153}]" +33:16.567 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5.dc1 127.0.0.1 +33:16.567 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5 127.0.0.1 +2021-10-05T16:33:16.567Z [INFO] agent.router: Initializing LAN area manager +33:16.567 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= +33:16.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=udp +33:16.568 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=tcp +33:16.568 [INFO] TestAgent.server: Adding LAN server: server="Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5 (Addr: tcp/127.0.0.1:17153) (DC: dc1)" +33:16.568 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5.dc1 area=wan +33:16.568 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17149 network=tcp +33:16.568 [INFO] TestAgent: started state syncer +33:16.568 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17154 network=tcp +33:16.612 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:16.612 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 +33:16.644 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:16.644 [DEBUG] TestAgent.server.raft: vote granted: from=8f1936c0-e338-d3f4-b23c-f1e08387cef5 term=2 tally=1 +33:16.644 [INFO] TestAgent.server.raft: election won: tally=1 +33:16.644 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" +33:16.644 [INFO] TestAgent.server: cluster leadership acquired +33:16.644 [INFO] TestAgent.server: New leader elected: payload=Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5 +33:16.663 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17153 +33:16.693 [INFO] TestAgent: Synced node info +33:16.693 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:16.708 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:16.708 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:16.708 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:16.708 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:16.708 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:16.708 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5 +33:16.708 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8f1936c0-e338-d3f4-b23c-f1e08387cef5 +33:16.752 [INFO] TestAgent.server: federation state anti-entropy synced +33:17.125 [INFO] TestAgent: (WAN) joining: wan_addresses=[127.0.0.1:17166] +33:17.126 [DEBUG] TestAgent.server.memberlist.wan: memberlist: Initiating push/pull sync with: 127.0.0.1:17166 +33:17.127 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b.dc1 127.0.0.1 +33:17.127 [INFO] TestAgent: (WAN) joined: number_of_nodes=1 +33:17.127 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/join/127.0.0.1:17166?wan=1 from=127.0.0.1:47506 latency=1.657666ms +33:17.127 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-49effe1b-69ef-dbdb-32b1-c3b237d1496b.dc1 area=wan +33:17.676 [INFO] TestAgent: Requesting shutdown +33:17.676 [INFO] TestAgent.server: shutting down server +33:17.676 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:17.676 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:17.676 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:17.676 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:17.676 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:17.676 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:17.676 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:17.676 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:17.676 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:17.676 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:17.768 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:17.782Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:17.782Z [INFO] agent.router.manager: shutting down +33:17.782 [INFO] TestAgent: consul server down +33:17.782 [INFO] TestAgent: shutdown complete +33:17.782 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp +33:17.782 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp +33:17.782 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp +33:18.283 [INFO] TestAgent: Waiting for endpoints to shut down +33:18.283 [INFO] TestAgent: Endpoints down +--- PASS: TestJoinCommand_wan (1.91s) PASS -ok github.com/hashicorp/consul/command/join 1.676s +ok github.com/hashicorp/consul/command/join 1.999s === RUN TestKeygenCommand_noTabs === PAUSE TestKeygenCommand_noTabs === RUN TestKeygenCommand @@ -86179,7 +86463,7 @@ === CONT TestKeygenCommand --- PASS: TestKeygenCommand (0.00s) PASS -ok github.com/hashicorp/consul/command/keygen 0.005s +ok github.com/hashicorp/consul/command/keygen 0.066s === RUN TestKeyringCommand_noTabs === PAUSE TestKeyringCommand_noTabs === RUN TestKeyringCommand @@ -86204,105 +86488,117 @@ --- PASS: TestKeyringCommand_numActions (0.00s) === CONT TestKeyringCommand_noTabs === CONT TestKeyringCommand_invalidLocalOnly +--- PASS: TestKeyringCommand_noTabs (0.00s) +=== CONT TestKeyringCommand_failedConnection === CONT TestKeyringCommand_help -=== CONT TestKeyringCommand +--- PASS: TestKeyringCommand_invalidLocalOnly (0.00s) === CONT TestKeyringCommand_invalidRelayFactor -=== CONT TestKeyringCommand_failedConnection +--- PASS: TestKeyringCommand_help (0.00s) +=== CONT TestKeyringCommand +--- PASS: TestKeyringCommand_invalidRelayFactor (0.00s) +--- PASS: TestKeyringCommand_failedConnection (0.00s) [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestKeyringCommand_noTabs (0.00s) ---- PASS: TestKeyringCommand_invalidRelayFactor (0.01s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestKeyringCommand_invalidLocalOnly (0.01s) ---- PASS: TestKeyringCommand_failedConnection (0.01s) ---- PASS: TestKeyringCommand_help (0.01s) -2022-11-07T22:46:22.907Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:22.918Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:22.942 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f114b9ff-dc17-d59c-1111-806e64d725f3 Address:127.0.0.1:15111}]" -46:22.942 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= -46:22.943 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 127.0.0.1 -46:22.944 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 127.0.0.1 -2022-11-07T22:46:22.944Z [INFO] agent.router: Initializing LAN area manager -46:22.944 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 area=wan -46:22.944 [INFO] TestAgent.server: Adding LAN server: server="Node-f114b9ff-dc17-d59c-1111-806e64d725f3 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" -46:22.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp -46:22.944 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp -46:22.944 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp -46:22.944 [INFO] TestAgent: started state syncer -46:22.944 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp -46:22.992 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:22.992 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 -46:23.001 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:23.001 [DEBUG] TestAgent.server.raft: vote granted: from=f114b9ff-dc17-d59c-1111-806e64d725f3 term=2 tally=1 -46:23.001 [INFO] TestAgent.server.raft: election won: tally=1 -46:23.001 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" -46:23.002 [INFO] TestAgent.server: cluster leadership acquired -46:23.002 [INFO] TestAgent.server: New leader elected: payload=Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.010 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 -46:23.020 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:23.020 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.020 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:23.021 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.021 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:48244 latency=1.094821ms -46:23.022 [INFO] TestAgent.server.serf.wan: serf: Received install-key query -46:23.022 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.022 [INFO] TestAgent.server.serf.lan: serf: Received install-key query -46:23.023 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.023 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/keyring from=127.0.0.1:48246 latency=889.987µs -46:23.024 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:23.024 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.024 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:23.024 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.025 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:48248 latency=1.371967ms -46:23.027 [INFO] TestAgent.server.serf.wan: serf: Received use-key query -46:23.027 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.027 [INFO] TestAgent.server.serf.lan: serf: Received use-key query -46:23.027 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.027 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/keyring from=127.0.0.1:48250 latency=632.872µs -46:23.027 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:23.028 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:23.028 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.028 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:23.028 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.028 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:48252 latency=611.714µs -46:23.029 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query -46:23.030 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.030 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query -46:23.030 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.030 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/keyring from=127.0.0.1:48254 latency=781.641µs -46:23.031 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query -46:23.031 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3.dc1 -46:23.031 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query -46:23.031 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-f114b9ff-dc17-d59c-1111-806e64d725f3 -46:23.031 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:48256 latency=629.338µs -46:23.032 [INFO] TestAgent: Requesting shutdown -46:23.032 [INFO] TestAgent.server: shutting down server -46:23.032 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:23.032 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:23.038 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:23.044Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:23.044Z [INFO] agent.router.manager: shutting down -46:23.044 [INFO] TestAgent: consul server down -46:23.044 [INFO] TestAgent: shutdown complete -46:23.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp -46:23.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp -46:23.044 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp -46:23.045 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" -46:23.045 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:23.045 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" -46:23.045 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" -46:23.045 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" -46:23.545 [INFO] TestAgent: Waiting for endpoints to shut down -46:23.545 [INFO] TestAgent: Endpoints down ---- PASS: TestKeyringCommand (0.67s) +2021-10-05T16:33:20.892Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:20.916Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:20.954 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4b37bedc-0d46-96bc-bea9-c637603e1c63 Address:127.0.0.1:30426}]" +33:20.955 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 127.0.0.1 +33:20.956 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= +33:20.956 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 127.0.0.1 +2021-10-05T16:33:20.956Z [INFO] agent.router: Initializing LAN area manager +33:20.956 [INFO] TestAgent.server: Adding LAN server: server="Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" +33:20.956 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 area=wan +33:20.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp +33:20.957 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp +33:20.957 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp +33:20.957 [INFO] TestAgent: started state syncer +33:20.958 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp +33:21.003 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:21.004 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 +33:21.020 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:21.020 [DEBUG] TestAgent.server.raft: vote granted: from=4b37bedc-0d46-96bc-bea9-c637603e1c63 term=2 tally=1 +33:21.020 [INFO] TestAgent.server.raft: election won: tally=1 +33:21.020 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" +33:21.024 [INFO] TestAgent.server: cluster leadership acquired +33:21.024 [INFO] TestAgent.server: New leader elected: payload=Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.030 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30426 +33:21.051 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:21.085 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +33:21.085 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.086 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +33:21.086 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.096 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:55526 latency=11.695191ms +33:21.098 [INFO] TestAgent.server.serf.wan: serf: Received install-key query +33:21.099 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.099 [INFO] TestAgent.server.serf.lan: serf: Received install-key query +33:21.099 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.099 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/operator/keyring from=127.0.0.1:55528 latency=1.076412ms +33:21.104 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +33:21.105 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.105 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +33:21.105 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.106 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:55530 latency=1.651048ms +33:21.113 [INFO] TestAgent.server.serf.wan: serf: Received use-key query +33:21.114 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.114 [INFO] TestAgent.server.serf.lan: serf: Received use-key query +33:21.114 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.114 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/keyring from=127.0.0.1:55532 latency=1.10662ms +33:21.115 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +33:21.115 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.116 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +33:21.125 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.126 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:55534 latency=11.403694ms +33:21.128 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:21.128 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:21.128 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:21.128 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:21.128 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:21.128 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.128 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.157 [INFO] TestAgent.server.serf.wan: serf: Received remove-key query +33:21.159 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.159 [INFO] TestAgent.server.serf.lan: serf: Received remove-key query +33:21.159 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.159 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/keyring from=127.0.0.1:55536 latency=1.988636ms +33:21.162 [INFO] TestAgent.server: federation state anti-entropy synced +33:21.197 [INFO] TestAgent.server.serf.wan: serf: Received list-keys query +33:21.200 [DEBUG] TestAgent.server.serf.wan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63.dc1 +33:21.200 [INFO] TestAgent.server.serf.lan: serf: Received list-keys query +33:21.200 [DEBUG] TestAgent.server.serf.lan: serf: messageQueryResponseType: Node-4b37bedc-0d46-96bc-bea9-c637603e1c63 +33:21.200 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/keyring from=127.0.0.1:55538 latency=3.922783ms +33:21.201 [INFO] TestAgent: Requesting shutdown +33:21.201 [INFO] TestAgent.server: shutting down server +33:21.201 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:21.201 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:21.201 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:21.201 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:21.201 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:21.201 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:21.201 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:21.201 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:21.201 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:21.201 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:21.201 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:21.212 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:21.220Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:21.220Z [INFO] agent.router.manager: shutting down +33:21.220 [INFO] TestAgent: consul server down +33:21.220 [INFO] TestAgent: shutdown complete +33:21.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp +33:21.220 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp +33:21.220 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp +33:21.724 [INFO] TestAgent: Waiting for endpoints to shut down +33:21.724 [INFO] TestAgent: Endpoints down +--- PASS: TestKeyringCommand (0.93s) PASS -ok github.com/hashicorp/consul/command/keyring 0.709s +ok github.com/hashicorp/consul/command/keyring 1.139s === RUN TestKVCommand_noTabs === PAUSE TestKVCommand_noTabs === CONT TestKVCommand_noTabs --- PASS: TestKVCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/kv 0.005s +ok github.com/hashicorp/consul/command/kv 0.060s === RUN TestKVDeleteCommand_noTabs === PAUSE TestKVDeleteCommand_noTabs === RUN TestKVDeleteCommand_Validation @@ -86314,273 +86610,276 @@ === RUN TestKVDeleteCommand_CAS === PAUSE TestKVDeleteCommand_CAS === CONT TestKVDeleteCommand_noTabs -=== CONT TestKVDeleteCommand_Recurse -=== CONT TestKVDeleteCommand === CONT TestKVDeleteCommand_CAS -=== CONT TestKVDeleteCommand_Validation +=== CONT TestKVDeleteCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== CONT TestKVDeleteCommand_Recurse +=== CONT TestKVDeleteCommand_Validation --- PASS: TestKVDeleteCommand_noTabs (0.01s) --- PASS: TestKVDeleteCommand_Validation (0.01s) -2022-11-07T22:46:24.481Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.499Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.548 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0321a07a-92ef-d155-ca06-b97658cc5c94 Address:127.0.0.1:25335}]" -46:24.548 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25335 [Follower]" leader= -46:24.549 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0321a07a-92ef-d155-ca06-b97658cc5c94.dc1 127.0.0.1 -46:24.549 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0321a07a-92ef-d155-ca06-b97658cc5c94 127.0.0.1 -2022-11-07T22:46:24.549Z [INFO] agent.router: Initializing LAN area manager -46:24.549 [INFO] TestAgent.server: Adding LAN server: server="Node-0321a07a-92ef-d155-ca06-b97658cc5c94 (Addr: tcp/127.0.0.1:25335) (DC: dc1)" -46:24.549 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0321a07a-92ef-d155-ca06-b97658cc5c94.dc1 area=wan -46:24.549 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25330 network=udp -46:24.549 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25330 network=tcp -46:24.550 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25331 network=tcp -46:24.550 [INFO] TestAgent: started state syncer -46:24.550 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25336 network=tcp -46:24.584 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.584 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25335 [Candidate]" term=2 -46:24.593 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.593 [DEBUG] TestAgent.server.raft: vote granted: from=0321a07a-92ef-d155-ca06-b97658cc5c94 term=2 tally=1 -46:24.593 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.593 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25335 [Leader]" -46:24.593 [INFO] TestAgent.server: cluster leadership acquired -46:24.593 [INFO] TestAgent.server: New leader elected: payload=Node-0321a07a-92ef-d155-ca06-b97658cc5c94 -46:24.600 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25335 -46:24.615 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.656 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:51044 latency=26.73155ms -46:24.666 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.666 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.666 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.666 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.666 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.666 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0321a07a-92ef-d155-ca06-b97658cc5c94 -46:24.666 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0321a07a-92ef-d155-ca06-b97658cc5c94 -46:24.675 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?cas=1 from=127.0.0.1:51046 latency=17.803978ms -46:24.676 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:51044 latency=228.05µs -46:24.688 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?cas=8 from=127.0.0.1:51048 latency=11.871478ms -46:24.688 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:51044 latency=32.355µs -46:24.689 [INFO] TestAgent: Requesting shutdown -46:24.689 [INFO] TestAgent.server: shutting down server -46:24.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.689 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.689 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.689 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.689 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.689 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.689 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.689 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.689 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:24.692 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.692 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.696 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.702Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.702Z [INFO] agent.router.manager: shutting down -46:24.702 [INFO] TestAgent: consul server down -46:24.702 [INFO] TestAgent: shutdown complete -46:24.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25330 network=tcp -46:24.702 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25330 network=udp -46:24.703 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25331 network=tcp -46:25.203 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.203 [INFO] TestAgent: Endpoints down ---- PASS: TestKVDeleteCommand_CAS (0.77s) -2022-11-07T22:46:24.483Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.501Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.544 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:339778d4-0f83-02e6-0793-4c88344791e3 Address:127.0.0.1:25328}]" -46:24.546 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25328 [Follower]" leader= -46:24.547 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-339778d4-0f83-02e6-0793-4c88344791e3.dc1 127.0.0.1 -46:24.547 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-339778d4-0f83-02e6-0793-4c88344791e3 127.0.0.1 -2022-11-07T22:46:24.548Z [INFO] agent.router: Initializing LAN area manager -46:24.548 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-339778d4-0f83-02e6-0793-4c88344791e3.dc1 area=wan -46:24.548 [INFO] TestAgent.server: Adding LAN server: server="Node-339778d4-0f83-02e6-0793-4c88344791e3 (Addr: tcp/127.0.0.1:25328) (DC: dc1)" -46:24.548 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=udp -46:24.548 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=tcp -46:24.548 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25324 network=tcp -46:24.548 [INFO] TestAgent: started state syncer -46:24.548 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25329 network=tcp -46:24.610 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.610 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25328 [Candidate]" term=2 -46:24.629 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.629 [DEBUG] TestAgent.server.raft: vote granted: from=339778d4-0f83-02e6-0793-4c88344791e3 term=2 tally=1 -46:24.629 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.629 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25328 [Leader]" -46:24.629 [INFO] TestAgent.server: cluster leadership acquired -46:24.629 [INFO] TestAgent.server: New leader elected: payload=Node-339778d4-0f83-02e6-0793-4c88344791e3 -46:24.638 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25328 -46:24.654 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.680 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.680 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.680 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.680 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.680 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.680 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-339778d4-0f83-02e6-0793-4c88344791e3 -46:24.680 [INFO] TestAgent.server: member joined, marking health alive: member=Node-339778d4-0f83-02e6-0793-4c88344791e3 -46:24.695 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.710 [INFO] TestAgent: Synced node info -46:24.718 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:35296 latency=5.049158ms -46:24.724 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo from=127.0.0.1:35298 latency=4.565726ms -46:24.725 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:35296 latency=78.763µs -46:24.725 [INFO] TestAgent: Requesting shutdown -46:24.725 [INFO] TestAgent.server: shutting down server -46:24.725 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.725 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.725 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.725 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.725 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.725 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.725 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.725 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.725 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.725 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.731 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.737Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.737Z [INFO] agent.router.manager: shutting down -46:24.737 [INFO] TestAgent: consul server down -46:24.737 [INFO] TestAgent: shutdown complete -46:24.737 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=tcp -46:24.737 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=udp -46:24.737 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25324 network=tcp -46:25.238 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.238 [INFO] TestAgent: Endpoints down ---- PASS: TestKVDeleteCommand (0.80s) -2022-11-07T22:46:24.477Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:24.494Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:24.532 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:23131cbe-04bc-b0e0-1f08-b7a4644e7aab Address:127.0.0.1:25321}]" -46:24.532 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= -46:24.532 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab.dc1 127.0.0.1 -46:24.533 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab 127.0.0.1 -2022-11-07T22:46:24.533Z [INFO] agent.router: Initializing LAN area manager -46:24.533 [INFO] TestAgent.server: Adding LAN server: server="Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab (Addr: tcp/127.0.0.1:25321) (DC: dc1)" -46:24.533 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab.dc1 area=wan -46:24.533 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp -46:24.533 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp -46:24.534 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp -46:24.534 [INFO] TestAgent: started state syncer -46:24.534 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp -46:24.593 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:24.593 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 -46:24.605 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:24.605 [DEBUG] TestAgent.server.raft: vote granted: from=23131cbe-04bc-b0e0-1f08-b7a4644e7aab term=2 tally=1 -46:24.605 [INFO] TestAgent.server.raft: election won: tally=1 -46:24.605 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" -46:24.605 [INFO] TestAgent.server: cluster leadership acquired -46:24.605 [INFO] TestAgent.server: New leader elected: payload=Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab -46:24.613 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25321 -46:24.634 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:24.673 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:24.673 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:24.673 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:24.673 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:24.673 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:24.673 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab -46:24.673 [INFO] TestAgent.server: member joined, marking health alive: member=Node-23131cbe-04bc-b0e0-1f08-b7a4644e7aab -46:24.682 [INFO] TestAgent.server: federation state anti-entropy synced -46:24.811 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:24.820 [INFO] TestAgent: Synced node info -46:24.918 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:53102 latency=6.719658ms -46:24.924 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:53102 latency=5.227257ms -46:24.929 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/food from=127.0.0.1:53102 latency=5.522086ms -46:24.935 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?recurse= from=127.0.0.1:53106 latency=5.296642ms -46:24.935 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/a from=127.0.0.1:53102 latency=33.169µs -46:24.936 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/b from=127.0.0.1:53102 latency=27.454µs -46:24.936 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/food from=127.0.0.1:53102 latency=18.378µs -46:24.936 [INFO] TestAgent: Requesting shutdown -46:24.936 [INFO] TestAgent.server: shutting down server -46:24.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:24.936 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:24.936 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:24.936 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:24.936 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:24.936 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:24.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:24.936 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:24.936 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:24.936 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:24.943 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:24.951Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:24.951Z [INFO] agent.router.manager: shutting down -46:24.951 [INFO] TestAgent: consul server down -46:24.951 [INFO] TestAgent: shutdown complete -46:24.951 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp -46:24.951 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp -46:24.951 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp -46:25.451 [INFO] TestAgent: Waiting for endpoints to shut down -46:25.451 [INFO] TestAgent: Endpoints down ---- PASS: TestKVDeleteCommand_Recurse (1.01s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:33:20.785Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:20.810Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:20.889 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3782e10d-22f1-6b78-85b7-90b874b15f33 Address:127.0.0.1:29405}]" +33:20.889 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3782e10d-22f1-6b78-85b7-90b874b15f33.dc1 127.0.0.1 +33:20.890 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3782e10d-22f1-6b78-85b7-90b874b15f33 127.0.0.1 +2021-10-05T16:33:20.890Z [INFO] agent.router: Initializing LAN area manager +33:20.890 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= +33:20.891 [INFO] TestAgent.server: Adding LAN server: server="Node-3782e10d-22f1-6b78-85b7-90b874b15f33 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" +33:20.891 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3782e10d-22f1-6b78-85b7-90b874b15f33.dc1 area=wan +33:20.891 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp +33:20.891 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp +33:20.892 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp +33:20.895 [INFO] TestAgent: started state syncer +33:20.895 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp +33:20.932 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:20.932 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 +33:20.947 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:20.947 [DEBUG] TestAgent.server.raft: vote granted: from=3782e10d-22f1-6b78-85b7-90b874b15f33 term=2 tally=1 +33:20.947 [INFO] TestAgent.server.raft: election won: tally=1 +33:20.947 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" +33:20.947 [INFO] TestAgent.server: cluster leadership acquired +33:20.947 [INFO] TestAgent.server: New leader elected: payload=Node-3782e10d-22f1-6b78-85b7-90b874b15f33 +33:20.956 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 +33:20.980 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:20.999 [INFO] TestAgent: Synced node info +33:21.015 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:21.016 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:21.016 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:21.016 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:21.016 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:21.016 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3782e10d-22f1-6b78-85b7-90b874b15f33 +33:21.016 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3782e10d-22f1-6b78-85b7-90b874b15f33 +33:21.037 [INFO] TestAgent.server: federation state anti-entropy synced +33:21.199 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:34954 latency=6.907639ms +33:21.208 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?cas=1 from=127.0.0.1:34956 latency=4.160213ms +33:21.212 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:34954 latency=375.495µs +33:21.217 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?cas=13 from=127.0.0.1:34958 latency=3.976113ms +33:21.235 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:34954 latency=81.998µs +33:21.235 [INFO] TestAgent: Requesting shutdown +33:21.235 [INFO] TestAgent.server: shutting down server +33:21.235 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:21.235 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:21.235 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:21.235 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:21.235 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:21.235 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:21.235 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:21.236 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:21.236 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:21.236 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:21.240 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:21.244Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:21.244Z [INFO] agent.router.manager: shutting down +33:21.244 [INFO] TestAgent: consul server down +33:21.244 [INFO] TestAgent: shutdown complete +33:21.244 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp +33:21.244 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp +33:21.244 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp +33:21.748 [INFO] TestAgent: Waiting for endpoints to shut down +33:21.748 [INFO] TestAgent: Endpoints down +--- PASS: TestKVDeleteCommand_CAS (1.15s) +2021-10-05T16:33:20.737Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:20.788Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:20.962 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:73e113ac-1f6a-42de-4c19-aae54bb9cd6d Address:127.0.0.1:29419}]" +33:20.963 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d.dc1 127.0.0.1 +33:20.963 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d 127.0.0.1 +2021-10-05T16:33:20.963Z [INFO] agent.router: Initializing LAN area manager +33:20.964 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29419 [Follower]" leader= +33:20.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=udp +33:20.964 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d.dc1 area=wan +33:20.964 [INFO] TestAgent.server: Adding LAN server: server="Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d (Addr: tcp/127.0.0.1:29419) (DC: dc1)" +33:20.964 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=tcp +33:20.965 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29415 network=tcp +33:20.965 [INFO] TestAgent: started state syncer +33:20.965 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29420 network=tcp +33:21.030 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:21.030 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29419 [Candidate]" term=2 +33:21.046 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:21.046 [DEBUG] TestAgent.server.raft: vote granted: from=73e113ac-1f6a-42de-4c19-aae54bb9cd6d term=2 tally=1 +33:21.046 [INFO] TestAgent.server.raft: election won: tally=1 +33:21.046 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29419 [Leader]" +33:21.048 [INFO] TestAgent.server: cluster leadership acquired +33:21.048 [INFO] TestAgent.server: New leader elected: payload=Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d +33:21.055 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29419 +33:21.085 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:21.129 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:21.130 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:21.130 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:21.130 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:21.130 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:21.130 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d +33:21.130 [INFO] TestAgent.server: member joined, marking health alive: member=Node-73e113ac-1f6a-42de-4c19-aae54bb9cd6d +33:21.141 [INFO] TestAgent.server: federation state anti-entropy synced +33:21.267 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:45364 latency=6.246951ms +33:21.272 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:45364 latency=4.582326ms +33:21.277 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/food from=127.0.0.1:45364 latency=4.214781ms +33:21.281 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo?recurse= from=127.0.0.1:45370 latency=3.624968ms +33:21.284 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/a from=127.0.0.1:45364 latency=50.134µs +33:21.284 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/b from=127.0.0.1:45364 latency=31.026µs +33:21.284 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/food from=127.0.0.1:45364 latency=28.131µs +33:21.285 [INFO] TestAgent: Requesting shutdown +33:21.285 [INFO] TestAgent.server: shutting down server +33:21.285 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:21.285 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:21.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:21.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:21.285 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:21.285 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:21.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:21.285 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:21.285 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:21.285 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:21.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:21.291 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:21.295Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:21.295Z [INFO] agent.router.manager: shutting down +33:21.295 [INFO] TestAgent: consul server down +33:21.295 [INFO] TestAgent: shutdown complete +33:21.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=tcp +33:21.295 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=udp +33:21.295 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29415 network=tcp +33:21.796 [INFO] TestAgent: Waiting for endpoints to shut down +33:21.796 [INFO] TestAgent: Endpoints down +--- PASS: TestKVDeleteCommand_Recurse (1.20s) +2021-10-05T16:33:20.761Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:20.805Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:20.892 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:895ffced-97d5-78bd-389b-23bb0d896495 Address:127.0.0.1:29412}]" +33:20.893 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-895ffced-97d5-78bd-389b-23bb0d896495.dc1 127.0.0.1 +33:20.893 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-895ffced-97d5-78bd-389b-23bb0d896495 127.0.0.1 +2021-10-05T16:33:20.893Z [INFO] agent.router: Initializing LAN area manager +33:20.894 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp +33:20.894 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= +33:20.894 [INFO] TestAgent.server: Adding LAN server: server="Node-895ffced-97d5-78bd-389b-23bb0d896495 (Addr: tcp/127.0.0.1:29412) (DC: dc1)" +33:20.894 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-895ffced-97d5-78bd-389b-23bb0d896495.dc1 area=wan +33:20.894 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp +33:20.895 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp +33:20.936 [INFO] TestAgent: started state syncer +33:20.936 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp +33:20.964 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:20.964 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 +33:20.982 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:20.982 [DEBUG] TestAgent.server.raft: vote granted: from=895ffced-97d5-78bd-389b-23bb0d896495 term=2 tally=1 +33:20.982 [INFO] TestAgent.server.raft: election won: tally=1 +33:20.982 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" +33:20.982 [INFO] TestAgent.server: cluster leadership acquired +33:20.982 [INFO] TestAgent.server: New leader elected: payload=Node-895ffced-97d5-78bd-389b-23bb0d896495 +33:21.002 [INFO] TestAgent: Synced node info +33:21.002 [DEBUG] TestAgent: Node info in sync +33:21.002 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 +33:21.031 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:21.057 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:21.057 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:21.057 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:21.057 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:21.057 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:21.057 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-895ffced-97d5-78bd-389b-23bb0d896495 +33:21.057 [INFO] TestAgent.server: member joined, marking health alive: member=Node-895ffced-97d5-78bd-389b-23bb0d896495 +33:21.085 [INFO] TestAgent.server: federation state anti-entropy synced +33:21.276 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:44460 latency=4.86681ms +33:21.291 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/foo from=127.0.0.1:44462 latency=14.011579ms +33:21.292 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:44460 latency=42.122µs +33:21.296 [INFO] TestAgent: Requesting shutdown +33:21.296 [INFO] TestAgent.server: shutting down server +33:21.296 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:21.296 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:21.296 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:21.296 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:21.296 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:21.296 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:21.296 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:21.296 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:21.296 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:21.296 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:21.301 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:21.320Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:21.320Z [INFO] agent.router.manager: shutting down +33:21.320 [INFO] TestAgent: consul server down +33:21.320 [INFO] TestAgent: shutdown complete +33:21.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp +33:21.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp +33:21.320 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp +33:21.824 [INFO] TestAgent: Waiting for endpoints to shut down +33:21.824 [INFO] TestAgent: Endpoints down +--- PASS: TestKVDeleteCommand (1.22s) PASS -ok github.com/hashicorp/consul/command/kv/del 1.043s +ok github.com/hashicorp/consul/command/kv/del 1.648s === RUN TestKVExportCommand_noTabs === PAUSE TestKVExportCommand_noTabs === RUN TestKVExportCommand === PAUSE TestKVExportCommand === CONT TestKVExportCommand_noTabs -=== CONT TestKVExportCommand --- PASS: TestKVExportCommand_noTabs (0.00s) +=== CONT TestKVExportCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:25.972Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:25.983Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:26.017 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 Address:127.0.0.1:29405}]" -46:26.017 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= -46:26.018 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5.dc1 127.0.0.1 -46:26.018 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 127.0.0.1 -2022-11-07T22:46:26.018Z [INFO] agent.router: Initializing LAN area manager -46:26.018 [INFO] TestAgent.server: Adding LAN server: server="Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" -46:26.018 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5.dc1 area=wan -46:26.018 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp -46:26.018 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp -46:26.019 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp -46:26.019 [INFO] TestAgent: started state syncer -46:26.019 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp -46:26.063 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:26.063 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 -46:26.072 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:26.072 [DEBUG] TestAgent.server.raft: vote granted: from=9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 term=2 tally=1 -46:26.072 [INFO] TestAgent.server.raft: election won: tally=1 -46:26.072 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" -46:26.072 [INFO] TestAgent.server: cluster leadership acquired -46:26.072 [INFO] TestAgent.server: New leader elected: payload=Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 -46:26.079 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 -46:26.094 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:26.113 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:26.113 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:26.113 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:26.113 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:26.113 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:26.113 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 -46:26.113 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9bfbcd28-1d4e-8a4e-4944-6bfd2c035ff5 -46:26.119 [INFO] TestAgent.server: federation state anti-entropy synced -46:26.177 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:55274 latency=4.101506ms -46:26.181 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:55274 latency=3.843536ms -46:26.186 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:55274 latency=4.533589ms -46:26.190 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/bar from=127.0.0.1:55274 latency=3.98119ms -46:26.193 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:55276 latency=996.666µs -46:26.194 [INFO] TestAgent: Requesting shutdown -46:26.194 [INFO] TestAgent.server: shutting down server -46:26.194 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:26.194 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:26.194 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:26.194 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:26.194 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:26.194 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:26.194 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:26.194 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:26.194 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:26.194 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:26.194 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:26.200 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:26.203Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:26.203Z [INFO] agent.router.manager: shutting down -46:26.203 [INFO] TestAgent: consul server down -46:26.203 [INFO] TestAgent: shutdown complete -46:26.203 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp -46:26.203 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp -46:26.203 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp -46:26.703 [INFO] TestAgent: Waiting for endpoints to shut down -46:26.703 [INFO] TestAgent: Endpoints down ---- PASS: TestKVExportCommand (0.77s) +2021-10-05T16:33:27.070Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:27.109Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:27.218 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a2d12ab6-b524-acf6-d1f9-bf75a687da39 Address:127.0.0.1:28384}]" +33:27.218 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39.dc1 127.0.0.1 +33:27.219 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39 127.0.0.1 +2021-10-05T16:33:27.219Z [INFO] agent.router: Initializing LAN area manager +33:27.219 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=udp +33:27.219 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28384 [Follower]" leader= +33:27.220 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39.dc1 area=wan +33:27.220 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=tcp +33:27.224 [INFO] TestAgent.server: Adding LAN server: server="Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" +33:27.224 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28380 network=tcp +33:27.224 [INFO] TestAgent: started state syncer +33:27.224 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28385 network=tcp +33:27.280 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:27.280 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28384 [Candidate]" term=2 +33:27.328 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:27.328 [DEBUG] TestAgent.server.raft: vote granted: from=a2d12ab6-b524-acf6-d1f9-bf75a687da39 term=2 tally=1 +33:27.328 [INFO] TestAgent.server.raft: election won: tally=1 +33:27.328 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28384 [Leader]" +33:27.332 [INFO] TestAgent.server: cluster leadership acquired +33:27.332 [INFO] TestAgent.server: New leader elected: payload=Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39 +33:27.363 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28384 +33:27.420 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:33420 latency=35.255351ms +33:27.456 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:27.456 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:33420 latency=34.528529ms +33:27.512 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:33420 latency=54.686995ms +33:27.514 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:27.514 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:27.514 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:27.514 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:27.514 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:27.514 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39 +33:27.514 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a2d12ab6-b524-acf6-d1f9-bf75a687da39 +33:27.552 [INFO] TestAgent.server: federation state anti-entropy synced +33:27.555 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/bar from=127.0.0.1:33420 latency=34.890891ms +33:27.572 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:27.581 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:33422 latency=21.520625ms +33:27.584 [INFO] TestAgent: Requesting shutdown +33:27.584 [INFO] TestAgent.server: shutting down server +33:27.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:27.584 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:27.584 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:27.584 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:27.584 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:27.584 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:27.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:27.584 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:27.584 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:27.584 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:27.596 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +33:27.596 [INFO] TestAgent: Synced node info +33:27.596 [DEBUG] TestAgent: Node info in sync +2021-10-05T16:33:27.624Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:27.624Z [INFO] agent.router.manager: shutting down +33:27.624 [INFO] TestAgent: consul server down +33:27.624 [INFO] TestAgent: shutdown complete +33:27.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=tcp +33:27.624 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=udp +33:27.624 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28380 network=tcp +33:28.128 [INFO] TestAgent: Waiting for endpoints to shut down +33:28.128 [INFO] TestAgent: Endpoints down +--- PASS: TestKVExportCommand (1.11s) PASS -ok github.com/hashicorp/consul/command/kv/exp 0.799s +ok github.com/hashicorp/consul/command/kv/exp 1.225s === RUN TestKVGetCommand_noTabs === PAUSE TestKVGetCommand_noTabs === RUN TestKVGetCommand_Validation @@ -86604,627 +86903,496 @@ === RUN TestKVGetCommand_DetailedBase64 === PAUSE TestKVGetCommand_DetailedBase64 === CONT TestKVGetCommand_noTabs -=== CONT TestKVGetCommand_Detailed -=== CONT TestKVGetCommand_RecurseBase64 -=== CONT TestKVGetCommand_Empty -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestKVGetCommand_Recurse -=== CONT TestKVGetCommand_Missing -=== CONT TestKVGetCommand -=== CONT TestKVGetCommand_Keys --- PASS: TestKVGetCommand_noTabs (0.00s) === CONT TestKVGetCommand_DetailedBase64 +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestKVGetCommand_Empty === CONT TestKVGetCommand_Base64 -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +=== CONT TestKVGetCommand_Missing +=== CONT TestKVGetCommand_Keys +=== CONT TestKVGetCommand_Recurse +=== CONT TestKVGetCommand === CONT TestKVGetCommand_Validation --- PASS: TestKVGetCommand_Validation (0.00s) -2022-11-07T22:46:27.781Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.809Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bcca0f4b-6ce0-9962-325f-8f53ff03f70d Address:127.0.0.1:23307}]" -46:27.860 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23307 [Follower]" leader= -46:27.860 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d.dc1 127.0.0.1 -46:27.862 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d 127.0.0.1 -2022-11-07T22:46:27.862Z [INFO] agent.router: Initializing LAN area manager -46:27.862 [INFO] TestAgent.server: Adding LAN server: server="Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d (Addr: tcp/127.0.0.1:23307) (DC: dc1)" -46:27.862 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23302 network=udp -46:27.862 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d.dc1 area=wan -46:27.862 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23302 network=tcp -46:27.863 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23303 network=tcp -46:27.863 [INFO] TestAgent: started state syncer -46:27.863 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23308 network=tcp -46:27.900 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.900 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23307 [Candidate]" term=2 -46:27.921 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.921 [DEBUG] TestAgent.server.raft: vote granted: from=bcca0f4b-6ce0-9962-325f-8f53ff03f70d term=2 tally=1 -46:27.921 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.921 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23307 [Leader]" -46:27.921 [INFO] TestAgent.server: cluster leadership acquired -46:27.921 [INFO] TestAgent.server: New leader elected: payload=Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d -46:27.936 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23307 -46:27.971 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:27.998 [INFO] TestAgent: Synced node info -46:27.999 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:58530 latency=21.82862ms -46:28.001 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:58532 latency=502.744µs -46:28.003 [INFO] TestAgent: Requesting shutdown -46:28.003 [INFO] TestAgent.server: shutting down server -46:28.003 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.012 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:28.015 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.015 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.015 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.015 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.015 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.015 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d -46:28.015 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d -2022-11-07T22:46:28.021Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.021Z [INFO] agent.router.manager: shutting down -46:28.021 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-bcca0f4b-6ce0-9962-325f-8f53ff03f70d 127.0.0.1 23305 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:bcca0f4b-6ce0-9962-325f-8f53ff03f70d port:23307 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:23306] alive 1 5 2 2 5 4}" error="raft is already shutdown" -46:28.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.021 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.021 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.021 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.021 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.021 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.021 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.024 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:28.024 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.024 [INFO] TestAgent: consul server down -46:28.024 [INFO] TestAgent: shutdown complete -46:28.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23302 network=tcp -46:28.024 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23302 network=udp -46:28.024 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23303 network=tcp -46:28.524 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.524 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Base64 (0.79s) -2022-11-07T22:46:27.786Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.811Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.902 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:088479ef-3dfb-48b7-af96-a6df5ed289b6 Address:127.0.0.1:23335}]" -46:27.902 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23335 [Follower]" leader= -46:27.906 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-088479ef-3dfb-48b7-af96-a6df5ed289b6.dc1 127.0.0.1 -46:27.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-088479ef-3dfb-48b7-af96-a6df5ed289b6 127.0.0.1 -2022-11-07T22:46:27.908Z [INFO] agent.router: Initializing LAN area manager -46:27.908 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-088479ef-3dfb-48b7-af96-a6df5ed289b6.dc1 area=wan -46:27.908 [INFO] TestAgent.server: Adding LAN server: server="Node-088479ef-3dfb-48b7-af96-a6df5ed289b6 (Addr: tcp/127.0.0.1:23335) (DC: dc1)" -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23330 network=tcp -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23330 network=udp -46:27.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23331 network=tcp -46:27.909 [INFO] TestAgent: started state syncer -46:27.909 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23336 network=tcp -46:27.954 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.954 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23335 [Candidate]" term=2 -46:27.977 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.977 [DEBUG] TestAgent.server.raft: vote granted: from=088479ef-3dfb-48b7-af96-a6df5ed289b6 term=2 tally=1 -46:27.977 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.977 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23335 [Leader]" -46:27.977 [INFO] TestAgent.server: cluster leadership acquired -46:27.977 [INFO] TestAgent.server: New leader elected: payload=Node-088479ef-3dfb-48b7-af96-a6df5ed289b6 -46:27.989 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23335 -46:28.011 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.045 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.045 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.045 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.045 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.045 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.045 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-088479ef-3dfb-48b7-af96-a6df5ed289b6 -46:28.045 [INFO] TestAgent.server: member joined, marking health alive: member=Node-088479ef-3dfb-48b7-af96-a6df5ed289b6 -46:28.073 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.074 [INFO] TestAgent: Synced node info -46:28.074 [DEBUG] TestAgent: Node info in sync -46:28.131 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/bar from=127.0.0.1:58470 latency=5.344565ms -46:28.136 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/baz from=127.0.0.1:58470 latency=4.902459ms -46:28.145 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/zip from=127.0.0.1:58470 latency=8.088753ms -46:28.147 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/?keys=&separator=%2F from=127.0.0.1:58474 latency=2.051517ms -46:28.148 [INFO] TestAgent: Requesting shutdown -46:28.148 [INFO] TestAgent.server: shutting down server -46:28.148 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.148 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.148 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.148 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.148 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.148 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.148 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.152 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.161Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.161Z [INFO] agent.router.manager: shutting down -46:28.161 [INFO] TestAgent: consul server down -46:28.161 [INFO] TestAgent: shutdown complete -46:28.161 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23330 network=tcp -46:28.161 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23330 network=udp -46:28.161 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23331 network=tcp -46:28.661 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.661 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Keys (0.93s) -2022-11-07T22:46:27.788Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.813Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.902 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dfe01f7f-0fce-33f0-d624-977ac65ca821 Address:127.0.0.1:23300}]" -46:27.902 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23300 [Follower]" leader= -46:27.903 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dfe01f7f-0fce-33f0-d624-977ac65ca821.dc1 127.0.0.1 -46:27.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dfe01f7f-0fce-33f0-d624-977ac65ca821 127.0.0.1 -2022-11-07T22:46:27.908Z [INFO] agent.router: Initializing LAN area manager -46:27.908 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dfe01f7f-0fce-33f0-d624-977ac65ca821.dc1 area=wan -46:27.908 [INFO] TestAgent.server: Adding LAN server: server="Node-dfe01f7f-0fce-33f0-d624-977ac65ca821 (Addr: tcp/127.0.0.1:23300) (DC: dc1)" -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=udp -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=tcp -46:27.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23296 network=tcp -46:27.909 [INFO] TestAgent: started state syncer -46:27.909 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23301 network=tcp -46:27.949 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.949 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23300 [Candidate]" term=2 -46:27.966 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.966 [DEBUG] TestAgent.server.raft: vote granted: from=dfe01f7f-0fce-33f0-d624-977ac65ca821 term=2 tally=1 -46:27.966 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.966 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23300 [Leader]" -46:27.966 [INFO] TestAgent.server: cluster leadership acquired -46:27.966 [INFO] TestAgent.server: New leader elected: payload=Node-dfe01f7f-0fce-33f0-d624-977ac65ca821 -46:27.979 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23300 -46:28.001 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.048 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.048 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.048 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.048 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.048 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.048 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dfe01f7f-0fce-33f0-d624-977ac65ca821 -46:28.048 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dfe01f7f-0fce-33f0-d624-977ac65ca821 -46:28.067 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.074 [INFO] TestAgent: Synced node info -46:28.145 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:48144 latency=7.74682ms -46:28.150 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:48144 latency=4.92583ms -46:28.154 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:48144 latency=3.686173ms -46:28.156 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:48148 latency=857.49µs -46:28.156 [INFO] TestAgent: Requesting shutdown -46:28.156 [INFO] TestAgent.server: shutting down server -46:28.156 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.156 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.156 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.156 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.156 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.156 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.156 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.156 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.156 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.156 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.164 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.171Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.171Z [INFO] agent.router.manager: shutting down -46:28.171 [INFO] TestAgent: consul server down -46:28.171 [INFO] TestAgent: shutdown complete -46:28.171 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=tcp -46:28.171 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=udp -46:28.171 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23296 network=tcp -46:28.671 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.671 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Recurse (0.94s) -2022-11-07T22:46:27.782Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.807Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.862 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4c5c36d8-376b-0894-4e62-a2f4a6089fee Address:127.0.0.1:23279}]" -46:27.862 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= -46:27.863 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee.dc1 127.0.0.1 -46:27.906 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee 127.0.0.1 -2022-11-07T22:46:27.906Z [INFO] agent.router: Initializing LAN area manager -46:27.906 [INFO] TestAgent.server: Adding LAN server: server="Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee (Addr: tcp/127.0.0.1:23279) (DC: dc1)" -46:27.907 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp -46:27.907 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee.dc1 area=wan -46:27.907 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp -46:27.908 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp -46:27.908 [INFO] TestAgent: started state syncer -46:27.908 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp -46:27.923 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.923 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 -46:27.944 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.944 [DEBUG] TestAgent.server.raft: vote granted: from=4c5c36d8-376b-0894-4e62-a2f4a6089fee term=2 tally=1 -46:27.944 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.944 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" -46:27.944 [INFO] TestAgent.server: cluster leadership acquired -46:27.944 [INFO] TestAgent.server: New leader elected: payload=Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee -46:27.960 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 -46:27.983 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.021 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.021 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.021 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.021 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee -46:28.021 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4c5c36d8-376b-0894-4e62-a2f4a6089fee -46:28.034 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.210 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:40486 latency=5.179328ms -46:28.210 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:40490 latency=178.697µs -46:28.210 [INFO] TestAgent: Requesting shutdown -46:28.211 [INFO] TestAgent.server: shutting down server -46:28.211 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.211 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.211 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.211 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.211 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.211 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.211 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.211 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:28.211 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.219 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.229Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.229Z [INFO] agent.router.manager: shutting down -46:28.229 [INFO] TestAgent: consul server down -46:28.229 [INFO] TestAgent: shutdown complete -46:28.229 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp -46:28.229 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp -46:28.229 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp -46:28.729 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.729 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Detailed (1.00s) -2022-11-07T22:46:27.785Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.858 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a1c46eb4-2e57-f2b6-f982-8f176a07fed8 Address:127.0.0.1:23293}]" -46:27.858 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23293 [Follower]" leader= -46:27.859 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8.dc1 127.0.0.1 -46:27.860 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8 127.0.0.1 -2022-11-07T22:46:27.860Z [INFO] agent.router: Initializing LAN area manager -46:27.861 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8.dc1 area=wan -46:27.861 [INFO] TestAgent.server: Adding LAN server: server="Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8 (Addr: tcp/127.0.0.1:23293) (DC: dc1)" -46:27.861 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=udp -46:27.861 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=tcp -46:27.862 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23289 network=tcp -46:27.862 [INFO] TestAgent: started state syncer -46:27.862 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23294 network=tcp -46:27.901 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.901 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23293 [Candidate]" term=2 -46:27.925 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.925 [DEBUG] TestAgent.server.raft: vote granted: from=a1c46eb4-2e57-f2b6-f982-8f176a07fed8 term=2 tally=1 -46:27.925 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.925 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23293 [Leader]" -46:27.926 [INFO] TestAgent.server: cluster leadership acquired -46:27.926 [INFO] TestAgent.server: New leader elected: payload=Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8 -46:27.942 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23293 -46:27.978 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.021 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.021 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.021 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.021 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8 -46:28.021 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a1c46eb4-2e57-f2b6-f982-8f176a07fed8 -46:28.028 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.201 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:59290 latency=4.892646ms -46:28.210 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:59290 latency=8.233238ms -46:28.217 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:59290 latency=6.646484ms -46:28.217 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:59300 latency=163.862µs -46:28.217 [INFO] TestAgent: Requesting shutdown -46:28.217 [INFO] TestAgent.server: shutting down server -46:28.217 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.217 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.217 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.217 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.217 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.218 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.218 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:28.218 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.218 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.222 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.233Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.233Z [INFO] agent.router.manager: shutting down -46:28.234 [INFO] TestAgent: consul server down -46:28.234 [INFO] TestAgent: shutdown complete -46:28.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=tcp -46:28.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=udp -46:28.234 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23289 network=tcp -46:28.734 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.734 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:27.788Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.814Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.902 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c88a94c7-958c-632b-5d6c-59eead9320ba Address:127.0.0.1:23328}]" -46:27.902 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23328 [Follower]" leader= -46:27.906 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c88a94c7-958c-632b-5d6c-59eead9320ba.dc1 127.0.0.1 -46:27.908 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c88a94c7-958c-632b-5d6c-59eead9320ba 127.0.0.1 -2022-11-07T22:46:27.908Z [INFO] agent.router: Initializing LAN area manager -46:27.908 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c88a94c7-958c-632b-5d6c-59eead9320ba.dc1 area=wan -46:27.908 [INFO] TestAgent.server: Adding LAN server: server="Node-c88a94c7-958c-632b-5d6c-59eead9320ba (Addr: tcp/127.0.0.1:23328) (DC: dc1)" -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23323 network=udp -46:27.908 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23323 network=tcp -46:27.909 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23324 network=tcp -46:27.909 [INFO] TestAgent: started state syncer -46:27.909 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23329 network=tcp -46:27.942 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.942 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23328 [Candidate]" term=2 -46:27.963 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.963 [DEBUG] TestAgent.server.raft: vote granted: from=c88a94c7-958c-632b-5d6c-59eead9320ba term=2 tally=1 -46:27.963 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.963 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23328 [Leader]" -46:27.963 [INFO] TestAgent.server: cluster leadership acquired -46:27.963 [INFO] TestAgent.server: New leader elected: payload=Node-c88a94c7-958c-632b-5d6c-59eead9320ba -46:27.978 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23328 -46:27.988 [INFO] TestAgent: Synced node info -46:27.988 [DEBUG] TestAgent: Node info in sync -46:28.021 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.044 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.044 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.044 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.044 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.044 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.044 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c88a94c7-958c-632b-5d6c-59eead9320ba -46:28.044 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c88a94c7-958c-632b-5d6c-59eead9320ba -46:28.074 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.213 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:52974 latency=7.167139ms -46:28.214 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:52978 latency=247.543µs -46:28.214 [INFO] TestAgent: Requesting shutdown -46:28.214 [INFO] TestAgent.server: shutting down server -46:28.214 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.214 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.214 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.214 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.214 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.214 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.214 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.214 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.214 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.222 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.233Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.233Z [INFO] agent.router.manager: shutting down -46:28.234 [INFO] TestAgent: consul server down -46:28.234 [INFO] TestAgent: shutdown complete -46:28.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23323 network=tcp -46:28.234 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23323 network=udp -46:28.234 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23324 network=tcp -46:28.734 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.734 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand (1.00s) ---- PASS: TestKVGetCommand_RecurseBase64 (1.00s) -2022-11-07T22:46:27.781Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.861 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b5f583d5-a0b7-8903-7be4-fb515638f21e Address:127.0.0.1:23314}]" -46:27.861 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23314 [Follower]" leader= -46:27.863 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b5f583d5-a0b7-8903-7be4-fb515638f21e.dc1 127.0.0.1 -46:27.906 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b5f583d5-a0b7-8903-7be4-fb515638f21e 127.0.0.1 -2022-11-07T22:46:27.906Z [INFO] agent.router: Initializing LAN area manager -46:27.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23309 network=udp -46:27.906 [INFO] TestAgent.server: Adding LAN server: server="Node-b5f583d5-a0b7-8903-7be4-fb515638f21e (Addr: tcp/127.0.0.1:23314) (DC: dc1)" -46:27.906 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23309 network=tcp -46:27.907 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b5f583d5-a0b7-8903-7be4-fb515638f21e.dc1 area=wan -46:27.908 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23310 network=tcp -46:27.908 [INFO] TestAgent: started state syncer -46:27.908 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23315 network=tcp -46:27.925 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.926 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23314 [Candidate]" term=2 -46:27.952 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.952 [DEBUG] TestAgent.server.raft: vote granted: from=b5f583d5-a0b7-8903-7be4-fb515638f21e term=2 tally=1 -46:27.952 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.952 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23314 [Leader]" -46:27.952 [INFO] TestAgent.server: cluster leadership acquired -46:27.952 [INFO] TestAgent.server: New leader elected: payload=Node-b5f583d5-a0b7-8903-7be4-fb515638f21e -46:27.963 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23314 -46:27.989 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.032 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.032 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.032 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.032 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.032 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.033 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b5f583d5-a0b7-8903-7be4-fb515638f21e -46:28.033 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b5f583d5-a0b7-8903-7be4-fb515638f21e -46:28.044 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.156 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.160 [INFO] TestAgent: Synced node info -46:28.160 [DEBUG] TestAgent: Node info in sync -46:28.265 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/not-a-real-key from=127.0.0.1:37024 latency=67.421µs -46:28.265 [INFO] TestAgent: Requesting shutdown -46:28.265 [INFO] TestAgent.server: shutting down server -46:28.265 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.265 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.265 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.265 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.265 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.265 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.265 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.265 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.265 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.265 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.271 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.282Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.282Z [INFO] agent.router.manager: shutting down -46:28.282 [INFO] TestAgent: consul server down -46:28.282 [INFO] TestAgent: shutdown complete -46:28.282 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23309 network=tcp -46:28.282 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23309 network=udp -46:28.282 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23310 network=tcp -46:28.783 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.783 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Missing (1.05s) -2022-11-07T22:46:27.780Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:aef754c6-582d-2900-3199-a366417265b2 Address:127.0.0.1:23321}]" -46:27.860 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23321 [Follower]" leader= -46:27.862 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-aef754c6-582d-2900-3199-a366417265b2.dc1 127.0.0.1 -46:27.863 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-aef754c6-582d-2900-3199-a366417265b2 127.0.0.1 -2022-11-07T22:46:27.863Z [INFO] agent.router: Initializing LAN area manager -46:27.863 [INFO] TestAgent.server: Adding LAN server: server="Node-aef754c6-582d-2900-3199-a366417265b2 (Addr: tcp/127.0.0.1:23321) (DC: dc1)" -46:27.863 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-aef754c6-582d-2900-3199-a366417265b2.dc1 area=wan -46:27.863 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23316 network=udp -46:27.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23316 network=tcp -46:27.903 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23317 network=tcp -46:27.903 [INFO] TestAgent: started state syncer -46:27.903 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23322 network=tcp -46:27.910 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.910 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23321 [Candidate]" term=2 -46:27.927 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.927 [DEBUG] TestAgent.server.raft: vote granted: from=aef754c6-582d-2900-3199-a366417265b2 term=2 tally=1 -46:27.927 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.927 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23321 [Leader]" -46:27.927 [INFO] TestAgent.server: cluster leadership acquired -46:27.927 [INFO] TestAgent.server: New leader elected: payload=Node-aef754c6-582d-2900-3199-a366417265b2 -46:27.945 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23321 -46:27.979 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.024 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.024 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.024 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.024 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.024 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.024 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-aef754c6-582d-2900-3199-a366417265b2 -46:28.024 [INFO] TestAgent.server: member joined, marking health alive: member=Node-aef754c6-582d-2900-3199-a366417265b2 -46:28.044 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.270 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.275 [INFO] TestAgent: Synced node info -46:28.282 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:42380 latency=6.253167ms -46:28.282 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:42382 latency=204.342µs -46:28.283 [INFO] TestAgent: Requesting shutdown -46:28.283 [INFO] TestAgent.server: shutting down server -46:28.283 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.283 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.283 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.283 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.283 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.283 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.283 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.283 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.283 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.294 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.304Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.304Z [INFO] agent.router.manager: shutting down -46:28.304 [INFO] TestAgent: consul server down -46:28.304 [INFO] TestAgent: shutdown complete -46:28.304 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23316 network=tcp -46:28.304 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23316 network=udp -46:28.304 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23317 network=tcp -46:28.804 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.804 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_DetailedBase64 (1.07s) -2022-11-07T22:46:27.781Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:27.812Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:27.862 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e474798-3a23-12bb-524d-4b326d8e5ae1 Address:127.0.0.1:23286}]" -46:27.862 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23286 [Follower]" leader= -46:27.863 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e474798-3a23-12bb-524d-4b326d8e5ae1.dc1 127.0.0.1 -46:27.863 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e474798-3a23-12bb-524d-4b326d8e5ae1 127.0.0.1 -2022-11-07T22:46:27.863Z [INFO] agent.router: Initializing LAN area manager -46:27.863 [INFO] TestAgent.server: Adding LAN server: server="Node-1e474798-3a23-12bb-524d-4b326d8e5ae1 (Addr: tcp/127.0.0.1:23286) (DC: dc1)" -46:27.863 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e474798-3a23-12bb-524d-4b326d8e5ae1.dc1 area=wan -46:27.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=udp -46:27.902 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=tcp -46:27.903 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23282 network=tcp -46:27.903 [INFO] TestAgent: started state syncer -46:27.903 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23287 network=tcp -46:27.909 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:27.909 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23286 [Candidate]" term=2 -46:27.936 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:27.936 [DEBUG] TestAgent.server.raft: vote granted: from=1e474798-3a23-12bb-524d-4b326d8e5ae1 term=2 tally=1 -46:27.936 [INFO] TestAgent.server.raft: election won: tally=1 -46:27.936 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23286 [Leader]" -46:27.936 [INFO] TestAgent.server: cluster leadership acquired -46:27.936 [INFO] TestAgent.server: New leader elected: payload=Node-1e474798-3a23-12bb-524d-4b326d8e5ae1 -46:27.951 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23286 -46:27.979 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:28.021 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:28.021 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:28.021 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:28.021 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:28.021 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e474798-3a23-12bb-524d-4b326d8e5ae1 -46:28.021 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e474798-3a23-12bb-524d-4b326d8e5ae1 -46:28.037 [INFO] TestAgent.server: federation state anti-entropy synced -46:28.325 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:28.333 [INFO] TestAgent: Synced node info -46:28.344 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/empty from=127.0.0.1:57992 latency=4.875855ms -46:28.345 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/empty from=127.0.0.1:57994 latency=177.713µs -46:28.345 [INFO] TestAgent: Requesting shutdown -46:28.345 [INFO] TestAgent.server: shutting down server -46:28.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:28.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:28.345 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:28.345 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:28.345 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:28.345 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:28.345 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:28.345 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:28.345 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:28.345 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:28.349 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:28.359Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:28.359Z [INFO] agent.router.manager: shutting down -46:28.359 [INFO] TestAgent: consul server down -46:28.359 [INFO] TestAgent: shutdown complete -46:28.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=tcp -46:28.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=udp -46:28.359 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23282 network=tcp -46:28.859 [INFO] TestAgent: Waiting for endpoints to shut down -46:28.859 [INFO] TestAgent: Endpoints down ---- PASS: TestKVGetCommand_Empty (1.13s) +=== CONT TestKVGetCommand_RecurseBase64 +=== CONT TestKVGetCommand_Detailed +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:33:29.670Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.720Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.276 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:47ec7649-ccf1-f14f-fce6-789015870f9b Address:127.0.0.1:19244}]" +33:30.276 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19244 [Follower]" leader= +33:30.277 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-47ec7649-ccf1-f14f-fce6-789015870f9b.dc1 127.0.0.1 +33:30.277 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-47ec7649-ccf1-f14f-fce6-789015870f9b 127.0.0.1 +2021-10-05T16:33:30.278Z [INFO] agent.router: Initializing LAN area manager +33:30.279 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19239 network=udp +33:30.279 [INFO] TestAgent.server: Adding LAN server: server="Node-47ec7649-ccf1-f14f-fce6-789015870f9b (Addr: tcp/127.0.0.1:19244) (DC: dc1)" +33:30.279 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-47ec7649-ccf1-f14f-fce6-789015870f9b.dc1 area=wan +33:30.279 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19239 network=tcp +33:30.280 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19240 network=tcp +33:30.284 [INFO] TestAgent: started state syncer +33:30.284 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19245 network=tcp +33:30.336 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.336 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19244 [Candidate]" term=2 +33:30.392 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.392 [DEBUG] TestAgent.server.raft: vote granted: from=47ec7649-ccf1-f14f-fce6-789015870f9b term=2 tally=1 +33:30.392 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.392 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19244 [Leader]" +33:30.393 [INFO] TestAgent.server: cluster leadership acquired +33:30.393 [INFO] TestAgent.server: New leader elected: payload=Node-47ec7649-ccf1-f14f-fce6-789015870f9b +33:30.572 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19244 +33:30.589 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:59938 latency=49.276624ms +33:30.591 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:59942 latency=328.043µs +33:30.591 [INFO] TestAgent: Requesting shutdown +33:30.591 [INFO] TestAgent.server: shutting down server +33:30.591 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.591 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:30.672 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.678Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.678Z [INFO] agent.router.manager: shutting down +33:30.678 [WARN] TestAgent: Syncing node info failed.: error="raft is already shutdown" +33:30.678 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="raft is already shutdown" +33:30.716 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +33:30.726 [INFO] TestAgent: consul server down +33:30.726 [INFO] TestAgent: shutdown complete +33:30.726 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19239 network=tcp +33:30.733 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19239 network=udp +33:30.733 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19240 network=tcp +33:31.233 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.234 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Detailed (1.72s) +2021-10-05T16:33:29.804Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.849Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.243 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:929dcfbb-ff98-d9ba-a316-8197430fe588 Address:127.0.0.1:19209}]" +33:30.251 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19209 [Follower]" leader= +33:30.275 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-929dcfbb-ff98-d9ba-a316-8197430fe588.dc1 127.0.0.1 +33:30.275 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-929dcfbb-ff98-d9ba-a316-8197430fe588 127.0.0.1 +2021-10-05T16:33:30.275Z [INFO] agent.router: Initializing LAN area manager +33:30.276 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-929dcfbb-ff98-d9ba-a316-8197430fe588.dc1 area=wan +33:30.276 [INFO] TestAgent.server: Adding LAN server: server="Node-929dcfbb-ff98-d9ba-a316-8197430fe588 (Addr: tcp/127.0.0.1:19209) (DC: dc1)" +33:30.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19204 network=udp +33:30.277 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19204 network=tcp +33:30.277 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19205 network=tcp +33:30.277 [INFO] TestAgent: started state syncer +33:30.277 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19210 network=tcp +33:30.288 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.288 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19209 [Candidate]" term=2 +33:30.396 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.396 [DEBUG] TestAgent.server.raft: vote granted: from=929dcfbb-ff98-d9ba-a316-8197430fe588 term=2 tally=1 +33:30.396 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.396 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19209 [Leader]" +33:30.396 [INFO] TestAgent.server: cluster leadership acquired +33:30.396 [INFO] TestAgent.server: New leader elected: payload=Node-929dcfbb-ff98-d9ba-a316-8197430fe588 +33:30.440 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19209 +33:30.569 [INFO] TestAgent: Synced node info +33:30.590 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/not-a-real-key from=127.0.0.1:39356 latency=143.906µs +33:30.590 [INFO] TestAgent: Requesting shutdown +33:30.590 [INFO] TestAgent.server: shutting down server +33:30.590 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.672 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.673Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.673Z [INFO] agent.router.manager: shutting down +33:30.701 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +33:30.726 [INFO] TestAgent: consul server down +33:30.726 [INFO] TestAgent: shutdown complete +33:30.726 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19204 network=tcp +33:30.734 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19204 network=udp +33:30.734 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19205 network=tcp +33:31.234 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.234 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Missing (1.72s) +2021-10-05T16:33:29.751Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.892Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.241 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1d03f50c-9afe-1f97-3728-c3302c76763f Address:127.0.0.1:19251}]" +33:30.242 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1d03f50c-9afe-1f97-3728-c3302c76763f.dc1 127.0.0.1 +33:30.243 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1d03f50c-9afe-1f97-3728-c3302c76763f 127.0.0.1 +2021-10-05T16:33:30.243Z [INFO] agent.router: Initializing LAN area manager +33:30.243 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19246 network=udp +33:30.243 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19251 [Follower]" leader= +33:30.250 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19246 network=tcp +33:30.251 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19247 network=tcp +33:30.251 [INFO] TestAgent: started state syncer +33:30.251 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1d03f50c-9afe-1f97-3728-c3302c76763f.dc1 area=wan +33:30.272 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19252 network=tcp +33:30.276 [INFO] TestAgent.server: Adding LAN server: server="Node-1d03f50c-9afe-1f97-3728-c3302c76763f (Addr: tcp/127.0.0.1:19251) (DC: dc1)" +33:30.280 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.280 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19251 [Candidate]" term=2 +33:30.388 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.388 [DEBUG] TestAgent.server.raft: vote granted: from=1d03f50c-9afe-1f97-3728-c3302c76763f term=2 tally=1 +33:30.388 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.388 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19251 [Leader]" +33:30.391 [INFO] TestAgent.server: cluster leadership acquired +33:30.391 [INFO] TestAgent.server: New leader elected: payload=Node-1d03f50c-9afe-1f97-3728-c3302c76763f +33:30.439 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19251 +33:30.673 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/empty from=127.0.0.1:46802 latency=141.091468ms +33:30.720 [INFO] TestAgent: Synced node info +33:30.733 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/empty from=127.0.0.1:46812 latency=52.389008ms +33:30.734 [INFO] TestAgent: Requesting shutdown +33:30.734 [INFO] TestAgent.server: shutting down server +33:30.734 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.780 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:30.801 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.802Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.802Z [INFO] agent.router.manager: shutting down +33:30.819 [INFO] TestAgent: consul server down +33:30.819 [INFO] TestAgent: shutdown complete +33:30.819 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19246 network=tcp +33:30.819 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19246 network=udp +33:30.819 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19247 network=tcp +33:30.819 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +33:30.819 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:31.323 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.323 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Empty (1.81s) +2021-10-05T16:33:29.800Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.922Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.372 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c30aa3d5-55ed-8464-991a-d1d64e069fea Address:127.0.0.1:19230}]" +33:30.373 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c30aa3d5-55ed-8464-991a-d1d64e069fea.dc1 127.0.0.1 +33:30.373 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c30aa3d5-55ed-8464-991a-d1d64e069fea 127.0.0.1 +2021-10-05T16:33:30.373Z [INFO] agent.router: Initializing LAN area manager +33:30.373 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19225 network=udp +33:30.373 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19230 [Follower]" leader= +33:30.374 [INFO] TestAgent.server: Adding LAN server: server="Node-c30aa3d5-55ed-8464-991a-d1d64e069fea (Addr: tcp/127.0.0.1:19230) (DC: dc1)" +33:30.374 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c30aa3d5-55ed-8464-991a-d1d64e069fea.dc1 area=wan +33:30.374 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19225 network=tcp +33:30.374 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19226 network=tcp +33:30.374 [INFO] TestAgent: started state syncer +33:30.375 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19231 network=tcp +33:30.435 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.435 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19230 [Candidate]" term=2 +33:30.695 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.695 [DEBUG] TestAgent.server.raft: vote granted: from=c30aa3d5-55ed-8464-991a-d1d64e069fea term=2 tally=1 +33:30.695 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.695 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19230 [Leader]" +33:30.705 [INFO] TestAgent.server: cluster leadership acquired +33:30.733 [INFO] TestAgent.server: New leader elected: payload=Node-c30aa3d5-55ed-8464-991a-d1d64e069fea +33:30.783 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19230 +33:30.821 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:54994 latency=75.542256ms +33:30.822 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:54996 latency=246.148µs +33:30.822 [INFO] TestAgent: Requesting shutdown +33:30.822 [INFO] TestAgent.server: shutting down server +33:30.822 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.829 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.831Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.831Z [INFO] agent.router.manager: shutting down +33:30.837 [INFO] TestAgent: consul server down +33:30.837 [INFO] TestAgent: shutdown complete +33:30.837 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19225 network=tcp +33:30.837 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19225 network=udp +33:30.837 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19226 network=tcp +33:30.838 [ERROR] TestAgent.server.autopilot: failed to initialize config: error="leadership lost while committing log" +33:30.838 [ERROR] TestAgent.server: failed to establish leadership: error="raft is already shutdown" +33:30.838 [WARN] TestAgent: Syncing node info failed.: error="leadership lost while committing log" +33:30.838 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="leadership lost while committing log" +33:31.340 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.340 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand (1.83s) +2021-10-05T16:33:29.723Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.850Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.164 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:77364fb1-f114-54d1-d290-07c63f1be7b4 Address:127.0.0.1:19202}]" +33:30.165 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19202 [Follower]" leader= +33:30.165 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-77364fb1-f114-54d1-d290-07c63f1be7b4.dc1 127.0.0.1 +33:30.166 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-77364fb1-f114-54d1-d290-07c63f1be7b4 127.0.0.1 +2021-10-05T16:33:30.166Z [INFO] agent.router: Initializing LAN area manager +33:30.167 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=udp +33:30.167 [INFO] TestAgent.server: Adding LAN server: server="Node-77364fb1-f114-54d1-d290-07c63f1be7b4 (Addr: tcp/127.0.0.1:19202) (DC: dc1)" +33:30.167 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-77364fb1-f114-54d1-d290-07c63f1be7b4.dc1 area=wan +33:30.239 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.239 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19202 [Candidate]" term=2 +33:30.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19197 network=tcp +33:30.245 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19198 network=tcp +33:30.245 [INFO] TestAgent: started state syncer +33:30.246 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19203 network=tcp +33:30.372 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.372 [DEBUG] TestAgent.server.raft: vote granted: from=77364fb1-f114-54d1-d290-07c63f1be7b4 term=2 tally=1 +33:30.372 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.372 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19202 [Leader]" +33:30.375 [INFO] TestAgent.server: cluster leadership acquired +33:30.375 [INFO] TestAgent.server: New leader elected: payload=Node-77364fb1-f114-54d1-d290-07c63f1be7b4 +33:30.412 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19202 +33:30.828 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:30.829 [INFO] TestAgent: Synced node info +33:30.839 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:53762 latency=106.270349ms +33:30.845 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:53772 latency=257.373µs +33:30.845 [INFO] TestAgent: Requesting shutdown +33:30.845 [INFO] TestAgent.server: shutting down server +33:30.845 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.852 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.853Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.853Z [INFO] agent.router.manager: shutting down +33:30.853 [INFO] TestAgent: consul server down +33:30.853 [INFO] TestAgent: shutdown complete +33:30.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=tcp +33:30.853 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19197 network=udp +33:30.853 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19198 network=tcp +33:30.853 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" +33:31.356 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.356 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Base64 (1.84s) +2021-10-05T16:33:29.705Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.870Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22 Address:127.0.0.1:19195}]" +33:30.247 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22.dc1 127.0.0.1 +33:30.274 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22 127.0.0.1 +2021-10-05T16:33:30.274Z [INFO] agent.router: Initializing LAN area manager +33:30.275 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=udp +33:30.275 [INFO] TestAgent.server: Adding LAN server: server="Node-c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22 (Addr: tcp/127.0.0.1:19195) (DC: dc1)" +33:30.275 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22.dc1 area=wan +33:30.276 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= +33:30.276 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=tcp +33:30.277 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19191 network=tcp +33:30.277 [INFO] TestAgent: started state syncer +33:30.277 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19196 network=tcp +33:30.316 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.316 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 +33:30.428 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.428 [DEBUG] TestAgent.server.raft: vote granted: from=c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22 term=2 tally=1 +33:30.428 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.428 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" +33:30.428 [INFO] TestAgent.server: cluster leadership acquired +33:30.428 [INFO] TestAgent.server: New leader elected: payload=Node-c6a5d279-0e76-cd4c-f224-9c1e0f7a1c22 +33:30.619 [INFO] TestAgent: Synced node info +33:30.619 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19195 +33:30.842 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:50884 latency=97.241647ms +33:30.845 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:50892 latency=348.225µs +33:30.845 [INFO] TestAgent: Requesting shutdown +33:30.845 [INFO] TestAgent.server: shutting down server +33:30.845 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.852 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.852Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.852Z [INFO] agent.router.manager: shutting down +33:30.858 [INFO] TestAgent: consul server down +33:30.858 [INFO] TestAgent: shutdown complete +33:30.858 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp +33:30.858 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp +33:30.858 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp +33:30.859 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +33:31.359 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.359 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_DetailedBase64 (1.85s) +2021-10-05T16:33:29.709Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.845Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.244 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:70970fc3-0839-3d19-a499-29f33b011dc8 Address:127.0.0.1:19237}]" +33:30.245 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-70970fc3-0839-3d19-a499-29f33b011dc8.dc1 127.0.0.1 +33:30.246 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-70970fc3-0839-3d19-a499-29f33b011dc8 127.0.0.1 +2021-10-05T16:33:30.246Z [INFO] agent.router: Initializing LAN area manager +33:30.246 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19232 network=udp +33:30.246 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19237 [Follower]" leader= +33:30.273 [INFO] TestAgent.server: Adding LAN server: server="Node-70970fc3-0839-3d19-a499-29f33b011dc8 (Addr: tcp/127.0.0.1:19237) (DC: dc1)" +33:30.273 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-70970fc3-0839-3d19-a499-29f33b011dc8.dc1 area=wan +33:30.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19232 network=tcp +33:30.274 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19233 network=tcp +33:30.274 [INFO] TestAgent: started state syncer +33:30.274 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19238 network=tcp +33:30.300 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.300 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19237 [Candidate]" term=2 +33:30.396 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.396 [DEBUG] TestAgent.server.raft: vote granted: from=70970fc3-0839-3d19-a499-29f33b011dc8 term=2 tally=1 +33:30.396 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.396 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19237 [Leader]" +33:30.396 [INFO] TestAgent.server: cluster leadership acquired +33:30.397 [INFO] TestAgent.server: New leader elected: payload=Node-70970fc3-0839-3d19-a499-29f33b011dc8 +33:30.444 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19237 +33:30.448 [INFO] TestAgent: Synced node info +33:30.618 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:59778 latency=112.992302ms +33:30.763 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:30.764 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:59778 latency=143.88202ms +33:30.830 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:59778 latency=64.568281ms +33:30.837 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:59800 latency=306.784µs +33:30.837 [INFO] TestAgent: Requesting shutdown +33:30.837 [INFO] TestAgent.server: shutting down server +33:30.837 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.848 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.851Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.851Z [INFO] agent.router.manager: shutting down +33:30.859 [INFO] TestAgent: consul server down +33:30.859 [INFO] TestAgent: shutdown complete +33:30.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19232 network=tcp +33:30.859 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19232 network=udp +33:30.859 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19233 network=tcp +33:30.859 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" +33:31.360 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.360 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_RecurseBase64 (1.85s) +2021-10-05T16:33:29.603Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.759Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.212 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8 Address:127.0.0.1:19223}]" +33:30.213 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19223 [Follower]" leader= +33:30.255 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8.dc1 127.0.0.1 +33:30.273 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8 127.0.0.1 +2021-10-05T16:33:30.273Z [INFO] agent.router: Initializing LAN area manager +33:30.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19218 network=udp +33:30.273 [INFO] TestAgent.server: Adding LAN server: server="Node-bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8 (Addr: tcp/127.0.0.1:19223) (DC: dc1)" +33:30.273 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8.dc1 area=wan +33:30.273 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19218 network=tcp +33:30.274 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19219 network=tcp +33:30.274 [INFO] TestAgent: started state syncer +33:30.274 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19224 network=tcp +33:30.324 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.324 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19223 [Candidate]" term=2 +33:30.392 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.392 [DEBUG] TestAgent.server.raft: vote granted: from=bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8 term=2 tally=1 +33:30.392 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.392 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19223 [Leader]" +33:30.392 [INFO] TestAgent.server: cluster leadership acquired +33:30.392 [INFO] TestAgent.server: New leader elected: payload=Node-bb74c76a-e1d5-1a5b-d0ec-e0c15a7da9b8 +33:30.564 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:55718 latency=131.982561ms +33:30.564 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19223 +33:30.680 [INFO] TestAgent: Synced node info +33:30.680 [DEBUG] TestAgent: Node info in sync +33:30.726 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/b from=127.0.0.1:55718 latency=157.220556ms +33:30.846 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/c from=127.0.0.1:55718 latency=111.881827ms +33:30.849 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo?recurse= from=127.0.0.1:55746 latency=310.565µs +33:30.850 [INFO] TestAgent: Requesting shutdown +33:30.850 [INFO] TestAgent.server: shutting down server +33:30.850 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.856 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:30.859 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.860Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.860Z [INFO] agent.router.manager: shutting down +33:30.863 [INFO] TestAgent: consul server down +33:30.863 [INFO] TestAgent: shutdown complete +33:30.863 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19218 network=tcp +33:30.863 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19218 network=udp +33:30.863 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19219 network=tcp +33:30.863 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +33:31.364 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.364 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Recurse (1.85s) +2021-10-05T16:33:29.725Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:29.781Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:30.208 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf53eb54-1c80-954e-cefd-bca98a5b9a77 Address:127.0.0.1:19216}]" +33:30.210 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77.dc1 127.0.0.1 +33:30.210 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77 127.0.0.1 +2021-10-05T16:33:30.210Z [INFO] agent.router: Initializing LAN area manager +33:30.211 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19211 network=udp +33:30.211 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19216 [Follower]" leader= +33:30.211 [INFO] TestAgent.server: Adding LAN server: server="Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77 (Addr: tcp/127.0.0.1:19216) (DC: dc1)" +33:30.211 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77.dc1 area=wan +33:30.211 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19211 network=tcp +33:30.212 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19212 network=tcp +33:30.213 [INFO] TestAgent: started state syncer +33:30.272 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19217 network=tcp +33:30.276 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:30.276 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19216 [Candidate]" term=2 +33:30.384 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:30.384 [DEBUG] TestAgent.server.raft: vote granted: from=bf53eb54-1c80-954e-cefd-bca98a5b9a77 term=2 tally=1 +33:30.384 [INFO] TestAgent.server.raft: election won: tally=1 +33:30.384 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19216 [Leader]" +33:30.384 [INFO] TestAgent.server: cluster leadership acquired +33:30.384 [INFO] TestAgent.server: New leader elected: payload=Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77 +33:30.408 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19216 +33:30.673 [INFO] TestAgent: Synced node info +33:30.673 [DEBUG] TestAgent: Node info in sync +33:30.779 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:30.783 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/bar from=127.0.0.1:52748 latency=162.403512ms +33:30.832 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/baz from=127.0.0.1:52748 latency=49.444824ms +33:30.868 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:30.868 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:30.868 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:30.868 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:30.868 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:30.869 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77 +33:30.869 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bf53eb54-1c80-954e-cefd-bca98a5b9a77 +33:30.872 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/zip from=127.0.0.1:52748 latency=35.573856ms +33:30.876 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/?keys=&separator=%2F from=127.0.0.1:52766 latency=286.338µs +33:30.880 [INFO] TestAgent: Requesting shutdown +33:30.880 [INFO] TestAgent.server: shutting down server +33:30.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:30.880 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:30.880 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:30.880 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:30.880 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:30.880 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:30.880 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:30.880 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:30.886 [INFO] TestAgent.server: federation state anti-entropy synced +33:30.886 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:30.892 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:30.896Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:30.896Z [INFO] agent.router.manager: shutting down +33:30.896 [INFO] TestAgent: consul server down +33:30.896 [INFO] TestAgent: shutdown complete +33:30.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19211 network=tcp +33:30.896 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19211 network=udp +33:30.896 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19212 network=tcp +33:31.396 [INFO] TestAgent: Waiting for endpoints to shut down +33:31.396 [INFO] TestAgent: Endpoints down +--- PASS: TestKVGetCommand_Keys (1.88s) PASS -ok github.com/hashicorp/consul/command/kv/get 1.159s +ok github.com/hashicorp/consul/command/kv/get 2.398s === RUN TestKVImportCommand_noTabs === PAUSE TestKVImportCommand_noTabs === RUN TestKVImportCommand === PAUSE TestKVImportCommand === CONT TestKVImportCommand_noTabs +--- PASS: TestKVImportCommand_noTabs (0.00s) === CONT TestKVImportCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestKVImportCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:29.715Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:29.725Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:29.766 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 Address:127.0.0.1:23279}]" -46:29.766 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= -46:29.767 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1.dc1 127.0.0.1 -46:29.767 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 127.0.0.1 -2022-11-07T22:46:29.768Z [INFO] agent.router: Initializing LAN area manager -46:29.768 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1.dc1 area=wan -46:29.768 [INFO] TestAgent.server: Adding LAN server: server="Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 (Addr: tcp/127.0.0.1:23279) (DC: dc1)" -46:29.768 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp -46:29.768 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp -46:29.768 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp -46:29.768 [INFO] TestAgent: started state syncer -46:29.769 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp -46:29.806 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:29.806 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 -46:29.819 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:29.819 [DEBUG] TestAgent.server.raft: vote granted: from=a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 term=2 tally=1 -46:29.819 [INFO] TestAgent.server.raft: election won: tally=1 -46:29.819 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" -46:29.819 [INFO] TestAgent.server: cluster leadership acquired -46:29.820 [INFO] TestAgent.server: New leader elected: payload=Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 -46:29.825 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 -46:29.840 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:29.862 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:29.862 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:29.862 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:29.862 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:29.862 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:29.862 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 -46:29.862 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a9bc1df8-f9c4-db9b-6d25-b6282a4f2ae1 -46:29.873 [INFO] TestAgent.server: federation state anti-entropy synced -46:29.890 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:40512 latency=4.093419ms -46:29.895 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:40512 latency=3.911927ms -46:29.896 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:40514 latency=910.253µs -46:29.896 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/a from=127.0.0.1:40514 latency=79.346µs -46:29.897 [INFO] TestAgent: Requesting shutdown -46:29.897 [INFO] TestAgent.server: shutting down server -46:29.897 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:29.897 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:29.897 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:29.897 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:29.897 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:29.897 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:29.897 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:29.897 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:29.897 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:29.897 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:29.897 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:29.902 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:29.907Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:29.907Z [INFO] agent.router.manager: shutting down -46:29.907 [INFO] TestAgent: consul server down -46:29.907 [INFO] TestAgent: shutdown complete -46:29.907 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp -46:29.907 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp -46:29.907 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp -46:30.408 [INFO] TestAgent: Waiting for endpoints to shut down -46:30.408 [INFO] TestAgent: Endpoints down ---- PASS: TestKVImportCommand (0.73s) +2021-10-05T16:33:33.666Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.709Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:33.802 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:457e0594-24a8-d041-6134-fef29c9b44d5 Address:127.0.0.1:23279}]" +33:33.802 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= +33:33.803 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-457e0594-24a8-d041-6134-fef29c9b44d5.dc1 127.0.0.1 +33:33.805 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-457e0594-24a8-d041-6134-fef29c9b44d5 127.0.0.1 +2021-10-05T16:33:33.805Z [INFO] agent.router: Initializing LAN area manager +33:33.805 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp +33:33.805 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-457e0594-24a8-d041-6134-fef29c9b44d5.dc1 area=wan +33:33.806 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp +33:33.806 [INFO] TestAgent.server: Adding LAN server: server="Node-457e0594-24a8-d041-6134-fef29c9b44d5 (Addr: tcp/127.0.0.1:23279) (DC: dc1)" +33:33.806 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp +33:33.806 [INFO] TestAgent: started state syncer +33:33.807 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp +33:33.850 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:33.851 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 +33:33.885 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:33.885 [DEBUG] TestAgent.server.raft: vote granted: from=457e0594-24a8-d041-6134-fef29c9b44d5 term=2 tally=1 +33:33.885 [INFO] TestAgent.server.raft: election won: tally=1 +33:33.885 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" +33:33.886 [INFO] TestAgent.server: cluster leadership acquired +33:33.886 [INFO] TestAgent.server: New leader elected: payload=Node-457e0594-24a8-d041-6134-fef29c9b44d5 +33:33.897 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 +33:33.943 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:33.983 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:33.983 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:33.983 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:33.983 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:33.983 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:33.983 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-457e0594-24a8-d041-6134-fef29c9b44d5 +33:33.983 [INFO] TestAgent.server: member joined, marking health alive: member=Node-457e0594-24a8-d041-6134-fef29c9b44d5 +33:33.989 [INFO] TestAgent: Synced node info +33:33.997 [INFO] TestAgent.server: federation state anti-entropy synced +33:34.042 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:37104 latency=24.34322ms +33:34.077 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo/a from=127.0.0.1:37104 latency=29.000479ms +33:34.082 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:37108 latency=759.186µs +33:34.084 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo/a from=127.0.0.1:37108 latency=454.121µs +33:34.084 [INFO] TestAgent: Requesting shutdown +33:34.085 [INFO] TestAgent.server: shutting down server +33:34.085 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.085 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.085 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.085 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.085 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.085 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:34.085 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.085 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.085 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.085 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.096 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.108Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.108Z [INFO] agent.router.manager: shutting down +33:34.108 [INFO] TestAgent: consul server down +33:34.108 [INFO] TestAgent: shutdown complete +33:34.108 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp +33:34.108 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp +33:34.108 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp +33:34.612 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.612 [INFO] TestAgent: Endpoints down +--- PASS: TestKVImportCommand (1.07s) PASS -ok github.com/hashicorp/consul/command/kv/imp 0.760s +ok github.com/hashicorp/consul/command/kv/imp 1.222s ? github.com/hashicorp/consul/command/kv/impexp [no test files] === RUN TestKVPutCommand_noTabs === PAUSE TestKVPutCommand_noTabs @@ -87249,465 +87417,453 @@ === RUN TestKVPutCommand_CAS === PAUSE TestKVPutCommand_CAS === CONT TestKVPutCommand_noTabs -=== CONT TestKVPutCommand_Flags -=== CONT TestKVPutCommand_NegativeVal +--- PASS: TestKVPutCommand_noTabs (0.00s) +=== CONT TestKVPutCommand_CAS === CONT TestKVPutCommand_File [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestKVPutCommand_EmptyDataQuoted -=== CONT TestKVPutCommand_Base64 -=== CONT TestKVPutCommand -[INFO] freeport: detected ephemeral port range of [32768, 60999] +=== CONT TestKVPutCommand_Flags +=== CONT TestKVPutCommand_NegativeVal === CONT TestKVPutCommand_Stdin -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== CONT TestKVPutCommand_Validation === CONT TestKVPutCommand_FileNoExist ---- PASS: TestKVPutCommand_noTabs (0.00s) -=== CONT TestKVPutCommand_CAS +=== CONT TestKVPutCommand_EmptyDataQuoted --- PASS: TestKVPutCommand_FileNoExist (0.00s) +=== CONT TestKVPutCommand +=== CONT TestKVPutCommand_Base64 +=== CONT TestKVPutCommand_Validation --- PASS: TestKVPutCommand_Validation (0.01s) -2022-11-07T22:46:31.055Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.076Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.132 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8c3c2912-f9f1-b369-a297-16f305fd5b79 Address:127.0.0.1:29419}]" -46:31.132 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29419 [Follower]" leader= -46:31.132 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8c3c2912-f9f1-b369-a297-16f305fd5b79.dc1 127.0.0.1 -46:31.133 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8c3c2912-f9f1-b369-a297-16f305fd5b79 127.0.0.1 -2022-11-07T22:46:31.133Z [INFO] agent.router: Initializing LAN area manager -46:31.133 [INFO] TestAgent.server: Adding LAN server: server="Node-8c3c2912-f9f1-b369-a297-16f305fd5b79 (Addr: tcp/127.0.0.1:29419) (DC: dc1)" -46:31.133 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8c3c2912-f9f1-b369-a297-16f305fd5b79.dc1 area=wan -46:31.133 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=udp -46:31.133 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=tcp -46:31.134 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29415 network=tcp -46:31.134 [INFO] TestAgent: started state syncer -46:31.134 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29420 network=tcp -46:31.179 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.179 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29419 [Candidate]" term=2 -46:31.195 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.195 [DEBUG] TestAgent.server.raft: vote granted: from=8c3c2912-f9f1-b369-a297-16f305fd5b79 term=2 tally=1 -46:31.195 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.195 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29419 [Leader]" -46:31.195 [INFO] TestAgent.server: cluster leadership acquired -46:31.195 [INFO] TestAgent.server: New leader elected: payload=Node-8c3c2912-f9f1-b369-a297-16f305fd5b79 -46:31.207 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29419 -46:31.248 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:59060 latency=32.896446ms -46:31.249 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:59064 latency=276.36µs -46:31.249 [INFO] TestAgent: Requesting shutdown -46:31.250 [INFO] TestAgent.server: shutting down server -46:31.250 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.250 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:31.259 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:31.270 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -2022-11-07T22:46:31.272Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.273Z [INFO] agent.router.manager: shutting down -46:31.273 [INFO] TestAgent: consul server down -46:31.273 [INFO] TestAgent: shutdown complete -46:31.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=tcp -46:31.273 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=udp -46:31.273 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29415 network=tcp -46:31.273 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: raft is already shutdown" -46:31.773 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.773 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_Base64 (0.77s) -2022-11-07T22:46:31.056Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.079Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.140 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:405cd39d-286f-e8f0-8f67-a1e2a1665d96 Address:127.0.0.1:29447}]" -46:31.140 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29447 [Follower]" leader= -46:31.140 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-405cd39d-286f-e8f0-8f67-a1e2a1665d96.dc1 127.0.0.1 -46:31.142 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-405cd39d-286f-e8f0-8f67-a1e2a1665d96 127.0.0.1 -2022-11-07T22:46:31.142Z [INFO] agent.router: Initializing LAN area manager -46:31.142 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-405cd39d-286f-e8f0-8f67-a1e2a1665d96.dc1 area=wan -46:31.142 [INFO] TestAgent.server: Adding LAN server: server="Node-405cd39d-286f-e8f0-8f67-a1e2a1665d96 (Addr: tcp/127.0.0.1:29447) (DC: dc1)" -46:31.142 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=udp -46:31.142 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29442 network=tcp -46:31.143 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29443 network=tcp -46:31.143 [INFO] TestAgent: started state syncer -46:31.143 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29448 network=tcp -46:31.209 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.209 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29447 [Candidate]" term=2 -46:31.229 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.229 [DEBUG] TestAgent.server.raft: vote granted: from=405cd39d-286f-e8f0-8f67-a1e2a1665d96 term=2 tally=1 -46:31.229 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.229 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29447 [Leader]" -46:31.229 [INFO] TestAgent.server: cluster leadership acquired -46:31.229 [INFO] TestAgent.server: New leader elected: payload=Node-405cd39d-286f-e8f0-8f67-a1e2a1665d96 -46:31.244 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29447 -46:31.271 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:53080 latency=19.536446ms -46:31.275 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:53082 latency=2.006311ms -46:31.277 [INFO] TestAgent: Requesting shutdown -46:31.277 [INFO] TestAgent.server: shutting down server -46:31.277 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.277 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:31.286 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.287 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.298Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.298Z [INFO] agent.router.manager: shutting down -46:31.305 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" -46:31.305 [INFO] TestAgent: consul server down -46:31.305 [INFO] TestAgent: shutdown complete -46:31.305 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=tcp -46:31.305 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29442 network=udp -46:31.305 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29443 network=tcp -46:31.805 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.805 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_Stdin (0.80s) -2022-11-07T22:46:31.055Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.080Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.144 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fd98073c-51e4-c4eb-7501-2b096d7b2b85 Address:127.0.0.1:29454}]" -46:31.144 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29454 [Follower]" leader= -46:31.182 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85.dc1 127.0.0.1 -46:31.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 127.0.0.1 -2022-11-07T22:46:31.184Z [INFO] agent.router: Initializing LAN area manager -46:31.184 [INFO] TestAgent.server: Adding LAN server: server="Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 (Addr: tcp/127.0.0.1:29454) (DC: dc1)" -46:31.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85.dc1 area=wan -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=udp -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29449 network=tcp -46:31.185 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29450 network=tcp -46:31.185 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29455 network=tcp -46:31.185 [INFO] TestAgent: started state syncer -46:31.191 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.191 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29454 [Candidate]" term=2 -46:31.206 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.206 [DEBUG] TestAgent.server.raft: vote granted: from=fd98073c-51e4-c4eb-7501-2b096d7b2b85 term=2 tally=1 -46:31.206 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.206 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29454 [Leader]" -46:31.206 [INFO] TestAgent.server: cluster leadership acquired -46:31.206 [INFO] TestAgent.server: New leader elected: payload=Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 -46:31.214 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29454 -46:31.239 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.273 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:56092 latency=27.270602ms -46:31.282 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?cas=123 from=127.0.0.1:56100 latency=4.843854ms -46:31.282 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:56092 latency=166.364µs -46:31.294 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?cas=8 from=127.0.0.1:56102 latency=11.047972ms -46:31.296 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:56092 latency=1.25953ms -46:31.296 [INFO] TestAgent: Requesting shutdown -46:31.296 [INFO] TestAgent.server: shutting down server -46:31.296 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.296 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:31.305 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:31.309 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.309 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.309 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.309 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.309 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.309 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 -46:31.309 [INFO] TestAgent.server: member joined, marking health alive: member=Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 -2022-11-07T22:46:31.313Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.313Z [INFO] agent.router.manager: shutting down -46:31.313 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-fd98073c-51e4-c4eb-7501-2b096d7b2b85 127.0.0.1 29452 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:fd98073c-51e4-c4eb-7501-2b096d7b2b85 port:29454 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:29453] alive 1 5 2 2 5 4}" error="raft is already shutdown" -46:31.313 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.313 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.313 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.313 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.313 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.313 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.313 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.322 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:31.322 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.322 [INFO] TestAgent: consul server down -46:31.322 [INFO] TestAgent: shutdown complete -46:31.322 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=tcp -46:31.322 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29449 network=udp -46:31.322 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29450 network=tcp -46:31.822 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.822 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_CAS (0.82s) -2022-11-07T22:46:31.056Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.080Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.182 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:83e2bc99-9753-ae57-7feb-957506214c46 Address:127.0.0.1:29440}]" -46:31.182 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29440 [Follower]" leader= -46:31.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-83e2bc99-9753-ae57-7feb-957506214c46.dc1 127.0.0.1 -46:31.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-83e2bc99-9753-ae57-7feb-957506214c46 127.0.0.1 -2022-11-07T22:46:31.184Z [INFO] agent.router: Initializing LAN area manager -46:31.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-83e2bc99-9753-ae57-7feb-957506214c46.dc1 area=wan -46:31.184 [INFO] TestAgent.server: Adding LAN server: server="Node-83e2bc99-9753-ae57-7feb-957506214c46 (Addr: tcp/127.0.0.1:29440) (DC: dc1)" -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=udp -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29435 network=tcp -46:31.186 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29436 network=tcp -46:31.186 [INFO] TestAgent: started state syncer -46:31.186 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29441 network=tcp -46:31.217 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.217 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29440 [Candidate]" term=2 -46:31.246 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.246 [DEBUG] TestAgent.server.raft: vote granted: from=83e2bc99-9753-ae57-7feb-957506214c46 term=2 tally=1 -46:31.246 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.246 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29440 [Leader]" -46:31.246 [INFO] TestAgent.server: cluster leadership acquired -46:31.246 [INFO] TestAgent.server: New leader elected: payload=Node-83e2bc99-9753-ae57-7feb-957506214c46 -46:31.256 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29440 -46:31.279 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.326 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.326 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.326 [INFO] TestAgent: Synced node info -46:31.326 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.326 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.326 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.326 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-83e2bc99-9753-ae57-7feb-957506214c46 -46:31.326 [INFO] TestAgent.server: member joined, marking health alive: member=Node-83e2bc99-9753-ae57-7feb-957506214c46 -46:31.335 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.404 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:37036 latency=2.522635ms -46:31.406 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:37038 latency=795.129µs -46:31.406 [INFO] TestAgent: Requesting shutdown -46:31.406 [INFO] TestAgent.server: shutting down server -46:31.406 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.406 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.406 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.406 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.406 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.406 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.406 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.406 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.406 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.406 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.410 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.413Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.413Z [INFO] agent.router.manager: shutting down -46:31.414 [INFO] TestAgent: consul server down -46:31.414 [INFO] TestAgent: shutdown complete -46:31.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=tcp -46:31.414 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29435 network=udp -46:31.414 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29436 network=tcp -46:31.914 [INFO] TestAgent: Waiting for endpoints to shut down -46:31.914 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand (0.91s) -2022-11-07T22:46:31.054Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.085Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.183 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:13bd14eb-df2d-b753-f17a-47da60fcdfc5 Address:127.0.0.1:29405}]" -46:31.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5.dc1 127.0.0.1 -46:31.184 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= -46:31.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5 127.0.0.1 -2022-11-07T22:46:31.184Z [INFO] agent.router: Initializing LAN area manager -46:31.184 [INFO] TestAgent.server: Adding LAN server: server="Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" -46:31.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5.dc1 area=wan -46:31.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp -46:31.185 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp -46:31.186 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp -46:31.186 [INFO] TestAgent: started state syncer -46:31.186 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp -46:31.245 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.245 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 -46:31.265 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.265 [DEBUG] TestAgent.server.raft: vote granted: from=13bd14eb-df2d-b753-f17a-47da60fcdfc5 term=2 tally=1 -46:31.265 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.265 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" -46:31.265 [INFO] TestAgent.server: cluster leadership acquired -46:31.265 [INFO] TestAgent.server: New leader elected: payload=Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5 -46:31.275 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 -46:31.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.333 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.333 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.333 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.333 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.333 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.333 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5 -46:31.333 [INFO] TestAgent.server: member joined, marking health alive: member=Node-13bd14eb-df2d-b753-f17a-47da60fcdfc5 -46:31.338 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.517 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?flags=12345 from=127.0.0.1:55340 latency=3.384478ms -46:31.520 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:55342 latency=1.696862ms -46:31.522 [INFO] TestAgent: Requesting shutdown -46:31.522 [INFO] TestAgent.server: shutting down server -46:31.522 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.522 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.522 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.522 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.522 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.522 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.522 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.522 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.522 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.522 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.522 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:31.526 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.531Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.531Z [INFO] agent.router.manager: shutting down -46:31.531 [INFO] TestAgent: consul server down -46:31.531 [INFO] TestAgent: shutdown complete -46:31.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp -46:31.531 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp -46:31.531 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp -46:32.031 [INFO] TestAgent: Waiting for endpoints to shut down -46:32.031 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_Flags (1.03s) -2022-11-07T22:46:31.060Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.083Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.182 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:defcbd58-5675-7c4d-0b19-42dc88921022 Address:127.0.0.1:29426}]" -46:31.182 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29426 [Follower]" leader= -46:31.184 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-defcbd58-5675-7c4d-0b19-42dc88921022.dc1 127.0.0.1 -46:31.184 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-defcbd58-5675-7c4d-0b19-42dc88921022 127.0.0.1 -2022-11-07T22:46:31.184Z [INFO] agent.router: Initializing LAN area manager -46:31.184 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-defcbd58-5675-7c4d-0b19-42dc88921022.dc1 area=wan -46:31.184 [INFO] TestAgent.server: Adding LAN server: server="Node-defcbd58-5675-7c4d-0b19-42dc88921022 (Addr: tcp/127.0.0.1:29426) (DC: dc1)" -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=udp -46:31.184 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29421 network=tcp -46:31.186 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29422 network=tcp -46:31.186 [INFO] TestAgent: started state syncer -46:31.186 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29427 network=tcp -46:31.235 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.235 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29426 [Candidate]" term=2 -46:31.258 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.258 [DEBUG] TestAgent.server.raft: vote granted: from=defcbd58-5675-7c4d-0b19-42dc88921022 term=2 tally=1 -46:31.258 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.258 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29426 [Leader]" -46:31.258 [INFO] TestAgent.server: cluster leadership acquired -46:31.258 [INFO] TestAgent.server: New leader elected: payload=Node-defcbd58-5675-7c4d-0b19-42dc88921022 -46:31.274 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29426 -46:31.303 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.335 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.335 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.335 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.335 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.335 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.335 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-defcbd58-5675-7c4d-0b19-42dc88921022 -46:31.335 [INFO] TestAgent.server: member joined, marking health alive: member=Node-defcbd58-5675-7c4d-0b19-42dc88921022 -46:31.350 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.497 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:31.500 [INFO] TestAgent: Synced node info -46:31.525 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:35138 latency=4.536555ms -46:31.526 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:35140 latency=284.618µs -46:31.526 [INFO] TestAgent: Requesting shutdown -46:31.527 [INFO] TestAgent.server: shutting down server -46:31.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.527 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.527 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.531 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.537Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.537Z [INFO] agent.router.manager: shutting down -46:31.537 [INFO] TestAgent: consul server down -46:31.537 [INFO] TestAgent: shutdown complete -46:31.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=tcp -46:31.537 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29421 network=udp -46:31.537 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29422 network=tcp -46:32.037 [INFO] TestAgent: Waiting for endpoints to shut down -46:32.037 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_EmptyDataQuoted (1.04s) -2022-11-07T22:46:31.055Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.077Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.131 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:36a230b9-bfc9-51d9-a5cf-c10324db1ee9 Address:127.0.0.1:29412}]" -46:31.131 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= -46:31.132 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9.dc1 127.0.0.1 -46:31.132 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9 127.0.0.1 -2022-11-07T22:46:31.132Z [INFO] agent.router: Initializing LAN area manager -46:31.133 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9.dc1 area=wan -46:31.133 [INFO] TestAgent.server: Adding LAN server: server="Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9 (Addr: tcp/127.0.0.1:29412) (DC: dc1)" -46:31.133 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp -46:31.133 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp -46:31.133 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp -46:31.134 [INFO] TestAgent: started state syncer -46:31.134 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp -46:31.191 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.191 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 -46:31.207 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.207 [DEBUG] TestAgent.server.raft: vote granted: from=36a230b9-bfc9-51d9-a5cf-c10324db1ee9 term=2 tally=1 -46:31.207 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.207 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" -46:31.207 [INFO] TestAgent.server: cluster leadership acquired -46:31.208 [INFO] TestAgent.server: New leader elected: payload=Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9 -46:31.224 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 -46:31.247 [INFO] TestAgent: Synced node info -46:31.247 [DEBUG] TestAgent: Node info in sync -46:31.279 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.308 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.308 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.308 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.308 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.308 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.308 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9 -46:31.308 [INFO] TestAgent.server: member joined, marking health alive: member=Node-36a230b9-bfc9-51d9-a5cf-c10324db1ee9 -46:31.319 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.536 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:34160 latency=4.463101ms -46:31.537 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:34162 latency=144.505µs -46:31.537 [INFO] TestAgent: Requesting shutdown -46:31.537 [INFO] TestAgent.server: shutting down server -46:31.537 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.537 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.537 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.537 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.537 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.537 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.537 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.537 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.540 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.543Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.543Z [INFO] agent.router.manager: shutting down -46:31.543 [INFO] TestAgent: consul server down -46:31.543 [INFO] TestAgent: shutdown complete -46:31.543 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp -46:31.543 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp -46:31.543 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp -46:32.044 [INFO] TestAgent: Waiting for endpoints to shut down -46:32.044 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_NegativeVal (1.04s) -2022-11-07T22:46:31.055Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:31.074Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:31.142 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a6dfb6ea-f154-0814-0645-007dc7705109 Address:127.0.0.1:29433}]" -46:31.142 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29433 [Follower]" leader= -46:31.142 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a6dfb6ea-f154-0814-0645-007dc7705109.dc1 127.0.0.1 -46:31.143 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a6dfb6ea-f154-0814-0645-007dc7705109 127.0.0.1 -2022-11-07T22:46:31.143Z [INFO] agent.router: Initializing LAN area manager -46:31.143 [INFO] TestAgent.server: Adding LAN server: server="Node-a6dfb6ea-f154-0814-0645-007dc7705109 (Addr: tcp/127.0.0.1:29433) (DC: dc1)" -46:31.143 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a6dfb6ea-f154-0814-0645-007dc7705109.dc1 area=wan -46:31.143 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=udp -46:31.143 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29428 network=tcp -46:31.143 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29429 network=tcp -46:31.144 [INFO] TestAgent: started state syncer -46:31.144 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29434 network=tcp -46:31.208 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:31.208 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29433 [Candidate]" term=2 -46:31.232 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:31.232 [DEBUG] TestAgent.server.raft: vote granted: from=a6dfb6ea-f154-0814-0645-007dc7705109 term=2 tally=1 -46:31.232 [INFO] TestAgent.server.raft: election won: tally=1 -46:31.232 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29433 [Leader]" -46:31.232 [INFO] TestAgent.server: cluster leadership acquired -46:31.232 [INFO] TestAgent.server: New leader elected: payload=Node-a6dfb6ea-f154-0814-0645-007dc7705109 -46:31.251 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29433 -46:31.279 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:31.318 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:31.319 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:31.319 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:31.319 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:31.319 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:31.319 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a6dfb6ea-f154-0814-0645-007dc7705109 -46:31.319 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a6dfb6ea-f154-0814-0645-007dc7705109 -46:31.327 [INFO] TestAgent.server: federation state anti-entropy synced -46:31.360 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:31.363 [INFO] TestAgent: Synced node info -46:31.565 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:31.565 [DEBUG] TestAgent: Node info in sync -46:31.565 [DEBUG] TestAgent: Node info in sync -46:31.576 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:42602 latency=2.833963ms -46:31.577 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:42604 latency=121.007µs -46:31.577 [INFO] TestAgent: Requesting shutdown -46:31.577 [INFO] TestAgent.server: shutting down server -46:31.577 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:31.577 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:31.577 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:31.577 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:31.577 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:31.577 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:31.577 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:31.577 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:31.577 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:31.577 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:31.581 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:31.585Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:31.585Z [INFO] agent.router.manager: shutting down -46:31.585 [INFO] TestAgent: consul server down -46:31.585 [INFO] TestAgent: shutdown complete -46:31.585 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=tcp -46:31.585 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29428 network=udp -46:31.585 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29429 network=tcp -46:32.086 [INFO] TestAgent: Waiting for endpoints to shut down -46:32.086 [INFO] TestAgent: Endpoints down ---- PASS: TestKVPutCommand_File (1.09s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:33:33.631Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.826Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:33.961 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:651cd9f3-5c9d-994f-7ee8-3c45c2db31a8 Address:127.0.0.1:21237}]" +33:33.962 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-651cd9f3-5c9d-994f-7ee8-3c45c2db31a8.dc1 127.0.0.1 +33:33.962 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-651cd9f3-5c9d-994f-7ee8-3c45c2db31a8 127.0.0.1 +2021-10-05T16:33:33.962Z [INFO] agent.router: Initializing LAN area manager +33:33.963 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp +33:33.963 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= +33:33.972 [INFO] TestAgent.server: Adding LAN server: server="Node-651cd9f3-5c9d-994f-7ee8-3c45c2db31a8 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" +33:33.972 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp +33:33.976 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-651cd9f3-5c9d-994f-7ee8-3c45c2db31a8.dc1 area=wan +33:33.976 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp +33:34.016 [INFO] TestAgent: started state syncer +33:34.030 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.030 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 +33:34.033 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp +33:34.055 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.055 [DEBUG] TestAgent.server.raft: vote granted: from=651cd9f3-5c9d-994f-7ee8-3c45c2db31a8 term=2 tally=1 +33:34.056 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.056 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" +33:34.056 [INFO] TestAgent.server: cluster leadership acquired +33:34.056 [INFO] TestAgent.server: New leader elected: payload=Node-651cd9f3-5c9d-994f-7ee8-3c45c2db31a8 +33:34.089 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:35876 latency=29.210223ms +33:34.089 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21237 +33:34.107 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?cas=123 from=127.0.0.1:35880 latency=12.77499ms +33:34.109 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:35876 latency=1.198701ms +33:34.159 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?cas=4 from=127.0.0.1:35882 latency=45.44335ms +33:34.161 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:35876 latency=538.759µs +33:34.162 [INFO] TestAgent: Requesting shutdown +33:34.162 [INFO] TestAgent.server: shutting down server +33:34.162 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.162 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:34.174 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.176 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.204Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.204Z [INFO] agent.router.manager: shutting down +33:34.214 [INFO] TestAgent: consul server down +33:34.214 [INFO] TestAgent: shutdown complete +33:34.214 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp +33:34.214 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp +33:34.214 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp +33:34.214 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +33:34.214 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:34.716 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.716 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_CAS (1.21s) +2021-10-05T16:33:33.778Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.932Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.075 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d67e30ba-d320-dd1c-9906-8b4bb1937be5 Address:127.0.0.1:21272}]" +33:34.076 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d67e30ba-d320-dd1c-9906-8b4bb1937be5.dc1 127.0.0.1 +33:34.076 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21272 [Follower]" leader= +33:34.077 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d67e30ba-d320-dd1c-9906-8b4bb1937be5 127.0.0.1 +2021-10-05T16:33:34.077Z [INFO] agent.router: Initializing LAN area manager +33:34.078 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=udp +33:34.078 [INFO] TestAgent.server: Adding LAN server: server="Node-d67e30ba-d320-dd1c-9906-8b4bb1937be5 (Addr: tcp/127.0.0.1:21272) (DC: dc1)" +33:34.078 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d67e30ba-d320-dd1c-9906-8b4bb1937be5.dc1 area=wan +33:34.078 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21267 network=tcp +33:34.079 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21268 network=tcp +33:34.079 [INFO] TestAgent: started state syncer +33:34.080 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21273 network=tcp +33:34.129 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.129 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21272 [Candidate]" term=2 +33:34.169 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.169 [DEBUG] TestAgent.server.raft: vote granted: from=d67e30ba-d320-dd1c-9906-8b4bb1937be5 term=2 tally=1 +33:34.169 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.169 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21272 [Leader]" +33:34.172 [INFO] TestAgent.server: cluster leadership acquired +33:34.174 [INFO] TestAgent.server: New leader elected: payload=Node-d67e30ba-d320-dd1c-9906-8b4bb1937be5 +33:34.214 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21272 +33:34.246 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:37708 latency=12.493831ms +33:34.247 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:37710 latency=500.848µs +33:34.248 [INFO] TestAgent: Requesting shutdown +33:34.248 [INFO] TestAgent.server: shutting down server +33:34.248 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.248 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:34.255 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.267Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.267Z [INFO] agent.router.manager: shutting down +33:34.282 [ERROR] TestAgent.server: failed to establish leadership: error="error configuring provider: leadership lost while committing log" +33:34.292 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:34.292 [INFO] TestAgent: consul server down +33:34.292 [INFO] TestAgent: shutdown complete +33:34.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=tcp +33:34.292 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21267 network=udp +33:34.292 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21268 network=tcp +33:34.796 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.796 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_EmptyDataQuoted (1.28s) +2021-10-05T16:33:33.858Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.931Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.059 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f7e52f82-b835-fbd3-6286-e9aafca0e354 Address:127.0.0.1:21279}]" +33:34.060 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21279 [Follower]" leader= +33:34.060 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f7e52f82-b835-fbd3-6286-e9aafca0e354.dc1 127.0.0.1 +33:34.060 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f7e52f82-b835-fbd3-6286-e9aafca0e354 127.0.0.1 +2021-10-05T16:33:34.060Z [INFO] agent.router: Initializing LAN area manager +33:34.061 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=udp +33:34.061 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f7e52f82-b835-fbd3-6286-e9aafca0e354.dc1 area=wan +33:34.061 [INFO] TestAgent.server: Adding LAN server: server="Node-f7e52f82-b835-fbd3-6286-e9aafca0e354 (Addr: tcp/127.0.0.1:21279) (DC: dc1)" +33:34.061 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21274 network=tcp +33:34.073 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21275 network=tcp +33:34.074 [INFO] TestAgent: started state syncer +33:34.074 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21280 network=tcp +33:34.113 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.113 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21279 [Candidate]" term=2 +33:34.160 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.160 [DEBUG] TestAgent.server.raft: vote granted: from=f7e52f82-b835-fbd3-6286-e9aafca0e354 term=2 tally=1 +33:34.160 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.160 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21279 [Leader]" +33:34.161 [INFO] TestAgent.server: cluster leadership acquired +33:34.161 [INFO] TestAgent.server: New leader elected: payload=Node-f7e52f82-b835-fbd3-6286-e9aafca0e354 +33:34.193 [INFO] TestAgent: Synced node info +33:34.193 [DEBUG] TestAgent: Node info in sync +33:34.193 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21279 +33:34.272 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.272 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:45644 latency=11.880871ms +33:34.278 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:45648 latency=464.172µs +33:34.296 [INFO] TestAgent: Requesting shutdown +33:34.296 [INFO] TestAgent.server: shutting down server +33:34.296 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.308 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.324Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.324Z [INFO] agent.router.manager: shutting down +33:34.324 [INFO] TestAgent: consul server down +33:34.324 [INFO] TestAgent: shutdown complete +33:34.324 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=tcp +33:34.324 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21274 network=udp +33:34.324 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21275 network=tcp +33:34.324 [ERROR] TestAgent.server.connect: Raft apply failed: error="raft is already shutdown" +33:34.324 [ERROR] TestAgent.server: failed to establish leadership: error="raft is already shutdown" +33:34.324 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:34.826 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.826 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_Base64 (1.31s) +2021-10-05T16:33:33.678Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.898Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.012 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 Address:127.0.0.1:21265}]" +33:34.032 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1.dc1 127.0.0.1 +33:34.032 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21265 [Follower]" leader= +33:34.032 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 127.0.0.1 +2021-10-05T16:33:34.032Z [INFO] agent.router: Initializing LAN area manager +33:34.033 [INFO] TestAgent.server: Adding LAN server: server="Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 (Addr: tcp/127.0.0.1:21265) (DC: dc1)" +33:34.033 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1.dc1 area=wan +33:34.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=tcp +33:34.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21260 network=udp +33:34.034 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21261 network=tcp +33:34.034 [INFO] TestAgent: started state syncer +33:34.034 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21266 network=tcp +33:34.078 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.078 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21265 [Candidate]" term=2 +33:34.111 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.111 [DEBUG] TestAgent.server.raft: vote granted: from=9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 term=2 tally=1 +33:34.111 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.111 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21265 [Leader]" +33:34.111 [INFO] TestAgent.server: cluster leadership acquired +33:34.111 [INFO] TestAgent.server: New leader elected: payload=Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 +33:34.151 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21265 +33:34.276 [INFO] TestAgent: Synced node info +33:34.276 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.313 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:34370 latency=84.373122ms +33:34.317 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:34.317 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:34.317 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:34.317 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:34.317 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:34.320 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 +33:34.320 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 +33:34.321 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:34382 latency=4.367406ms +33:34.321 [INFO] TestAgent: Requesting shutdown +33:34.321 [INFO] TestAgent.server: shutting down server +33:34.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.321 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.321 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.321 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.321 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.321 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.321 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.324 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.327 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.336Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.336Z [INFO] agent.router.manager: shutting down +33:34.368 [INFO] TestAgent: consul server down +33:34.368 [INFO] TestAgent: shutdown complete +33:34.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=tcp +33:34.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21260 network=udp +33:34.368 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21261 network=tcp +33:34.368 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 127.0.0.1 21263 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:9880dcf0-c7b5-d875-a34b-5b618d0f6bb1 port:21265 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:21264] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +33:34.368 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" +33:34.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.870 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.870 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_Stdin (1.36s) +2021-10-05T16:33:33.676Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.768Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.013 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0116787e-d8d3-5c6a-ab87-93c60836507d Address:127.0.0.1:21244}]" +33:34.032 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0116787e-d8d3-5c6a-ab87-93c60836507d.dc1 127.0.0.1 +33:34.033 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0116787e-d8d3-5c6a-ab87-93c60836507d 127.0.0.1 +2021-10-05T16:33:34.033Z [INFO] agent.router: Initializing LAN area manager +33:34.033 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp +33:34.033 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= +33:34.034 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp +33:34.034 [INFO] TestAgent.server: Adding LAN server: server="Node-0116787e-d8d3-5c6a-ab87-93c60836507d (Addr: tcp/127.0.0.1:21244) (DC: dc1)" +33:34.034 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0116787e-d8d3-5c6a-ab87-93c60836507d.dc1 area=wan +33:34.034 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp +33:34.035 [INFO] TestAgent: started state syncer +33:34.035 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp +33:34.077 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.077 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 +33:34.109 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.109 [DEBUG] TestAgent.server.raft: vote granted: from=0116787e-d8d3-5c6a-ab87-93c60836507d term=2 tally=1 +33:34.109 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.109 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" +33:34.109 [INFO] TestAgent.server: cluster leadership acquired +33:34.109 [INFO] TestAgent.server: New leader elected: payload=Node-0116787e-d8d3-5c6a-ab87-93c60836507d +33:34.132 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21244 +33:34.172 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.268 [INFO] TestAgent: Synced node info +33:34.268 [DEBUG] TestAgent: Node info in sync +33:34.284 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:34.284 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:34.284 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:34.284 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:34.284 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:34.284 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0116787e-d8d3-5c6a-ab87-93c60836507d +33:34.286 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0116787e-d8d3-5c6a-ab87-93c60836507d +33:34.286 [DEBUG] TestAgent: Node info in sync +33:34.341 [INFO] TestAgent.server: federation state anti-entropy synced +33:34.341 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:37292 latency=65.105797ms +33:34.363 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:37298 latency=239.183µs +33:34.363 [INFO] TestAgent: Requesting shutdown +33:34.363 [INFO] TestAgent.server: shutting down server +33:34.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.363 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.363 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.363 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.363 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.363 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:34.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.363 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.363 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.363 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.371 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.379Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.379Z [INFO] agent.router.manager: shutting down +33:34.379 [INFO] TestAgent: consul server down +33:34.379 [INFO] TestAgent: shutdown complete +33:34.379 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp +33:34.379 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp +33:34.379 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp +33:34.884 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.884 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_File (1.38s) +2021-10-05T16:33:33.769Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.879Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:33.972 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9f212213-c039-dd31-cfc2-16b770421ba0 Address:127.0.0.1:21258}]" +33:33.972 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21258 [Follower]" leader= +33:33.973 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9f212213-c039-dd31-cfc2-16b770421ba0.dc1 127.0.0.1 +33:34.030 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.031 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21258 [Candidate]" term=2 +33:34.034 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9f212213-c039-dd31-cfc2-16b770421ba0 127.0.0.1 +2021-10-05T16:33:34.034Z [INFO] agent.router: Initializing LAN area manager +33:34.034 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=udp +33:34.034 [INFO] TestAgent.server: Adding LAN server: server="Node-9f212213-c039-dd31-cfc2-16b770421ba0 (Addr: tcp/127.0.0.1:21258) (DC: dc1)" +33:34.034 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9f212213-c039-dd31-cfc2-16b770421ba0.dc1 area=wan +33:34.034 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21253 network=tcp +33:34.035 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21254 network=tcp +33:34.035 [INFO] TestAgent: started state syncer +33:34.035 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21259 network=tcp +33:34.055 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.055 [DEBUG] TestAgent.server.raft: vote granted: from=9f212213-c039-dd31-cfc2-16b770421ba0 term=2 tally=1 +33:34.055 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.055 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21258 [Leader]" +33:34.055 [INFO] TestAgent.server: cluster leadership acquired +33:34.056 [INFO] TestAgent.server: New leader elected: payload=Node-9f212213-c039-dd31-cfc2-16b770421ba0 +33:34.076 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21258 +33:34.132 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.180 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:34.180 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:34.180 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:34.180 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:34.180 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:34.180 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9f212213-c039-dd31-cfc2-16b770421ba0 +33:34.180 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9f212213-c039-dd31-cfc2-16b770421ba0 +33:34.244 [INFO] TestAgent: Synced node info +33:34.244 [DEBUG] TestAgent: Node info in sync +33:34.244 [INFO] TestAgent.server: federation state anti-entropy synced +33:34.418 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:59636 latency=6.326173ms +33:34.420 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:59638 latency=219.98µs +33:34.421 [INFO] TestAgent: Requesting shutdown +33:34.421 [INFO] TestAgent.server: shutting down server +33:34.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.421 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.421 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.421 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.421 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.421 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:34.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.421 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.421 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.421 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.428 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.435Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.435Z [INFO] agent.router.manager: shutting down +33:34.435 [INFO] TestAgent: consul server down +33:34.435 [INFO] TestAgent: shutdown complete +33:34.435 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=tcp +33:34.435 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21253 network=udp +33:34.435 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21254 network=tcp +33:34.936 [INFO] TestAgent: Waiting for endpoints to shut down +33:34.936 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_NegativeVal (1.43s) +2021-10-05T16:33:33.930Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.962Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.089 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:34227373-eb46-397d-7067-7ae4611ab7a3 Address:127.0.0.1:21286}]" +33:34.090 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-34227373-eb46-397d-7067-7ae4611ab7a3.dc1 127.0.0.1 +33:34.090 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-34227373-eb46-397d-7067-7ae4611ab7a3 127.0.0.1 +2021-10-05T16:33:34.090Z [INFO] agent.router: Initializing LAN area manager +33:34.091 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=udp +33:34.091 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21286 [Follower]" leader= +33:34.091 [INFO] TestAgent.server: Adding LAN server: server="Node-34227373-eb46-397d-7067-7ae4611ab7a3 (Addr: tcp/127.0.0.1:21286) (DC: dc1)" +33:34.091 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-34227373-eb46-397d-7067-7ae4611ab7a3.dc1 area=wan +33:34.091 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21281 network=tcp +33:34.092 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21282 network=tcp +33:34.092 [INFO] TestAgent: started state syncer +33:34.092 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21287 network=tcp +33:34.161 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.161 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21286 [Candidate]" term=2 +33:34.215 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.215 [DEBUG] TestAgent.server.raft: vote granted: from=34227373-eb46-397d-7067-7ae4611ab7a3 term=2 tally=1 +33:34.215 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.215 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21286 [Leader]" +33:34.216 [INFO] TestAgent.server: cluster leadership acquired +33:34.216 [INFO] TestAgent.server: New leader elected: payload=Node-34227373-eb46-397d-7067-7ae4611ab7a3 +33:34.245 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21286 +33:34.275 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.333 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:34.333 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:34.333 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:34.333 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:34.333 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:34.333 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-34227373-eb46-397d-7067-7ae4611ab7a3 +33:34.333 [INFO] TestAgent.server: member joined, marking health alive: member=Node-34227373-eb46-397d-7067-7ae4611ab7a3 +33:34.405 [INFO] TestAgent.server: federation state anti-entropy synced +33:34.406 [INFO] TestAgent: Synced node info +33:34.460 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:34.460 [DEBUG] TestAgent: Node info in sync +33:34.460 [DEBUG] TestAgent: Node info in sync +33:34.484 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo from=127.0.0.1:58416 latency=4.340599ms +33:34.489 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:58418 latency=290.847µs +33:34.490 [INFO] TestAgent: Requesting shutdown +33:34.490 [INFO] TestAgent.server: shutting down server +33:34.490 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.490 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.490 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.490 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.490 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.490 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:34.490 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.490 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.490 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.494 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.498Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.498Z [INFO] agent.router.manager: shutting down +33:34.498 [INFO] TestAgent: consul server down +33:34.498 [INFO] TestAgent: shutdown complete +33:34.498 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=tcp +33:34.498 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21281 network=udp +33:34.498 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21282 network=tcp +33:35.000 [INFO] TestAgent: Waiting for endpoints to shut down +33:35.000 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand (1.49s) +2021-10-05T16:33:33.875Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:33.960Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:34.114 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 Address:127.0.0.1:21251}]" +33:34.115 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8.dc1 127.0.0.1 +33:34.116 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 127.0.0.1 +2021-10-05T16:33:34.116Z [INFO] agent.router: Initializing LAN area manager +33:34.116 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=udp +33:34.116 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21251 [Follower]" leader= +33:34.116 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8.dc1 area=wan +33:34.116 [INFO] TestAgent.server: Adding LAN server: server="Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 (Addr: tcp/127.0.0.1:21251) (DC: dc1)" +33:34.117 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21246 network=tcp +33:34.117 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21247 network=tcp +33:34.117 [INFO] TestAgent: started state syncer +33:34.117 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21252 network=tcp +33:34.156 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:34.157 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21251 [Candidate]" term=2 +33:34.234 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:34.234 [DEBUG] TestAgent.server.raft: vote granted: from=e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 term=2 tally=1 +33:34.234 [INFO] TestAgent.server.raft: election won: tally=1 +33:34.234 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21251 [Leader]" +33:34.235 [INFO] TestAgent.server: cluster leadership acquired +33:34.235 [INFO] TestAgent.server: New leader elected: payload=Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 +33:34.248 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:21251 +33:34.281 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:34.410 [INFO] TestAgent: Synced node info +33:34.419 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:34.419 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:34.419 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:34.419 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:34.419 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:34.419 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 +33:34.419 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e0789ff8-a8fa-dfe5-116d-60f4d7edd1b8 +33:34.434 [INFO] TestAgent.server: federation state anti-entropy synced +33:34.545 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/foo?flags=12345 from=127.0.0.1:33676 latency=24.101241ms +33:34.552 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/foo from=127.0.0.1:33678 latency=257.005µs +33:34.556 [INFO] TestAgent: Requesting shutdown +33:34.556 [INFO] TestAgent.server: shutting down server +33:34.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:34.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:34.556 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:34.556 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:34.556 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:34.556 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:34.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:34.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:34.556 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:34.556 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:34.564 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:34.572Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:34.572Z [INFO] agent.router.manager: shutting down +33:34.572 [INFO] TestAgent: consul server down +33:34.572 [INFO] TestAgent: shutdown complete +33:34.572 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=tcp +33:34.572 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21246 network=udp +33:34.572 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21247 network=tcp +33:35.076 [INFO] TestAgent: Waiting for endpoints to shut down +33:35.076 [INFO] TestAgent: Endpoints down +--- PASS: TestKVPutCommand_Flags (1.57s) PASS -ok github.com/hashicorp/consul/command/kv/put 1.116s +ok github.com/hashicorp/consul/command/kv/put 1.775s === RUN TestLeaveCommand_noTabs === PAUSE TestLeaveCommand_noTabs === RUN TestLeaveCommand @@ -87715,142 +87871,141 @@ === RUN TestLeaveCommand_FailOnNonFlagArgs === PAUSE TestLeaveCommand_FailOnNonFlagArgs === CONT TestLeaveCommand_noTabs +--- PASS: TestLeaveCommand_noTabs (0.00s) === CONT TestLeaveCommand_FailOnNonFlagArgs -=== CONT TestLeaveCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestLeaveCommand_noTabs (0.00s) +=== CONT TestLeaveCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:33.190Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:33.200Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:33.237 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4537e7ad-d029-8ca7-4790-87d22c79b416 Address:127.0.0.1:28391}]" -46:33.237 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28391 [Follower]" leader= -46:33.237 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4537e7ad-d029-8ca7-4790-87d22c79b416.dc1 127.0.0.1 -46:33.238 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4537e7ad-d029-8ca7-4790-87d22c79b416 127.0.0.1 -2022-11-07T22:46:33.238Z [INFO] agent.router: Initializing LAN area manager -46:33.238 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=udp -46:33.238 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4537e7ad-d029-8ca7-4790-87d22c79b416.dc1 area=wan -46:33.238 [INFO] TestAgent.server: Adding LAN server: server="Node-4537e7ad-d029-8ca7-4790-87d22c79b416 (Addr: tcp/127.0.0.1:28391) (DC: dc1)" -46:33.238 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=tcp -46:33.239 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28387 network=tcp -46:33.239 [INFO] TestAgent: started state syncer -46:33.239 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28392 network=tcp -46:33.295 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:33.295 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28391 [Candidate]" term=2 -46:33.307 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:33.307 [DEBUG] TestAgent.server.raft: vote granted: from=4537e7ad-d029-8ca7-4790-87d22c79b416 term=2 tally=1 -46:33.307 [INFO] TestAgent.server.raft: election won: tally=1 -46:33.307 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28391 [Leader]" -46:33.307 [INFO] TestAgent.server: cluster leadership acquired -46:33.307 [INFO] TestAgent.server: New leader elected: payload=Node-4537e7ad-d029-8ca7-4790-87d22c79b416 -46:33.312 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28391 -46:33.330 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:33.362 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:33.362 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:33.362 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:33.362 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:33.362 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:33.362 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4537e7ad-d029-8ca7-4790-87d22c79b416 -46:33.362 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4537e7ad-d029-8ca7-4790-87d22c79b416 -46:33.371 [INFO] TestAgent.server: federation state anti-entropy synced -46:33.562 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:33.569 [INFO] TestAgent: Synced node info -46:33.582 [INFO] TestAgent: Requesting shutdown -46:33.582 [INFO] TestAgent.server: shutting down server -46:33.582 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:33.582 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:33.582 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:33.582 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:33.582 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:33.582 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:33.582 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:33.582 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:33.582 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:33.582 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:33.590 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:33.594Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:33.594Z [INFO] agent.router.manager: shutting down -46:33.594 [INFO] TestAgent: consul server down -46:33.594 [INFO] TestAgent: shutdown complete -46:33.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=tcp -46:33.594 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=udp -46:33.594 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28387 network=tcp -46:34.095 [INFO] TestAgent: Waiting for endpoints to shut down -46:34.095 [INFO] TestAgent: Endpoints down ---- PASS: TestLeaveCommand_FailOnNonFlagArgs (0.94s) -2022-11-07T22:46:33.190Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:33.200Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:33.227 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4736dc9f-571d-d02b-783d-baf0ddbbda81 Address:127.0.0.1:28384}]" -46:33.228 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4736dc9f-571d-d02b-783d-baf0ddbbda81.dc1 127.0.0.1 -46:33.228 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 127.0.0.1 -2022-11-07T22:46:33.228Z [INFO] agent.router: Initializing LAN area manager -46:33.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=udp -46:33.229 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28384 [Follower]" leader= -46:33.236 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=tcp -46:33.236 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81.dc1 area=wan -46:33.236 [INFO] TestAgent.server: Adding LAN server: server="Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" -46:33.237 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28380 network=tcp -46:33.237 [INFO] TestAgent: started state syncer -46:33.237 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28385 network=tcp -46:33.298 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:33.298 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28384 [Candidate]" term=2 -46:33.307 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:33.307 [DEBUG] TestAgent.server.raft: vote granted: from=4736dc9f-571d-d02b-783d-baf0ddbbda81 term=2 tally=1 -46:33.307 [INFO] TestAgent.server.raft: election won: tally=1 -46:33.307 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28384 [Leader]" -46:33.307 [INFO] TestAgent.server: cluster leadership acquired -46:33.307 [INFO] TestAgent.server: New leader elected: payload=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 -46:33.312 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28384 -46:33.328 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:33.349 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:33.349 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:33.349 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:33.349 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:33.349 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:33.349 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 -46:33.349 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 -46:33.359 [INFO] TestAgent.server: federation state anti-entropy synced -46:33.542 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:33.545 [INFO] TestAgent: Synced node info -46:33.545 [DEBUG] TestAgent: Node info in sync -46:33.598 [INFO] TestAgent.server: server starting leave -46:33.598 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-4736dc9f-571d-d02b-783d-baf0ddbbda81.dc1 127.0.0.1 -46:33.598 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81.dc1 area=wan -2022-11-07T22:46:33.598Z [INFO] agent.router.manager: shutting down -46:36.538 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:36.538 [DEBUG] TestAgent: Node info in sync -46:36.599 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 127.0.0.1 -46:36.599 [INFO] TestAgent.server: Removing LAN server: server="Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" -46:36.599 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-4736dc9f-571d-d02b-783d-baf0ddbbda81 -46:37.318 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:39.318 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:39.599 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s -46:41.318 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:43.318 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" -46:43.318 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" -46:44.599 [INFO] TestAgent: Requesting shutdown -46:44.599 [INFO] TestAgent.server: shutting down server -46:44.600 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:44.600 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:44.600 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:44.600 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:44.600 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:44.600 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:44.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:44.600 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:44.600 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -2022-11-07T22:46:44.609Z [INFO] agent.router.manager: shutting down -46:44.609 [INFO] TestAgent: consul server down -46:44.609 [INFO] TestAgent: shutdown complete -46:44.609 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/leave from=127.0.0.1:38888 latency=11.011627308s -46:44.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=tcp -46:44.610 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=udp -46:44.610 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28380 network=tcp -46:45.110 [INFO] TestAgent: Waiting for endpoints to shut down -46:45.110 [INFO] TestAgent: Endpoints down ---- PASS: TestLeaveCommand (11.95s) +2021-10-05T16:33:39.555Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:39.602Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:39.688 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bbc3cdd1-3d60-8589-09f6-82c53014594d Address:127.0.0.1:14090}]" +33:39.689 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bbc3cdd1-3d60-8589-09f6-82c53014594d.dc1 127.0.0.1 +33:39.689 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +33:39.689 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bbc3cdd1-3d60-8589-09f6-82c53014594d 127.0.0.1 +2021-10-05T16:33:39.690Z [INFO] agent.router: Initializing LAN area manager +33:39.690 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +33:39.690 [INFO] TestAgent.server: Adding LAN server: server="Node-bbc3cdd1-3d60-8589-09f6-82c53014594d (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +33:39.690 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bbc3cdd1-3d60-8589-09f6-82c53014594d.dc1 area=wan +33:39.691 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +33:39.691 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +33:39.691 [INFO] TestAgent: started state syncer +33:39.691 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +33:39.736 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:39.736 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +33:39.761 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:39.761 [DEBUG] TestAgent.server.raft: vote granted: from=bbc3cdd1-3d60-8589-09f6-82c53014594d term=2 tally=1 +33:39.761 [INFO] TestAgent.server.raft: election won: tally=1 +33:39.761 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +33:39.761 [INFO] TestAgent.server: cluster leadership acquired +33:39.761 [INFO] TestAgent.server: New leader elected: payload=Node-bbc3cdd1-3d60-8589-09f6-82c53014594d +33:39.767 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 +33:39.824 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:39.873 [INFO] TestAgent: Synced node info +33:39.873 [DEBUG] TestAgent: Node info in sync +33:39.897 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:39.897 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:39.897 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:39.897 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:39.897 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:39.897 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bbc3cdd1-3d60-8589-09f6-82c53014594d +33:39.897 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bbc3cdd1-3d60-8589-09f6-82c53014594d +33:39.917 [INFO] TestAgent.server: federation state anti-entropy synced +33:40.028 [INFO] TestAgent: Requesting shutdown +33:40.029 [INFO] TestAgent.server: shutting down server +33:40.029 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:40.029 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:40.029 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:40.029 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:40.029 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:40.029 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:40.029 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:40.029 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:40.029 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:40.029 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:40.040 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:40.040Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:40.040Z [INFO] agent.router.manager: shutting down +33:40.040 [INFO] TestAgent: consul server down +33:40.040 [INFO] TestAgent: shutdown complete +33:40.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +33:40.040 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +33:40.040 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +33:40.544 [INFO] TestAgent: Waiting for endpoints to shut down +33:40.544 [INFO] TestAgent: Endpoints down +--- PASS: TestLeaveCommand_FailOnNonFlagArgs (1.09s) +2021-10-05T16:33:39.562Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:39.630Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:39.712 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:09828261-2155-52e8-91d8-8f6111edbb2c Address:127.0.0.1:14097}]" +33:39.712 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-09828261-2155-52e8-91d8-8f6111edbb2c.dc1 127.0.0.1 +33:39.713 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-09828261-2155-52e8-91d8-8f6111edbb2c 127.0.0.1 +2021-10-05T16:33:39.713Z [INFO] agent.router: Initializing LAN area manager +33:39.713 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=udp +33:39.713 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14097 [Follower]" leader= +33:39.713 [INFO] TestAgent.server: Adding LAN server: server="Node-09828261-2155-52e8-91d8-8f6111edbb2c (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +33:39.713 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-09828261-2155-52e8-91d8-8f6111edbb2c.dc1 area=wan +33:39.714 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14092 network=tcp +33:39.714 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14093 network=tcp +33:39.714 [INFO] TestAgent: started state syncer +33:39.714 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14098 network=tcp +33:39.782 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:39.782 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14097 [Candidate]" term=2 +33:39.791 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:39.791 [DEBUG] TestAgent.server.raft: vote granted: from=09828261-2155-52e8-91d8-8f6111edbb2c term=2 tally=1 +33:39.791 [INFO] TestAgent.server.raft: election won: tally=1 +33:39.791 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14097 [Leader]" +33:39.794 [INFO] TestAgent.server: cluster leadership acquired +33:39.796 [INFO] TestAgent.server: New leader elected: payload=Node-09828261-2155-52e8-91d8-8f6111edbb2c +33:39.796 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14097 +33:39.811 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:39.839 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:39.839 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:39.839 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:39.839 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:39.839 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:39.839 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-09828261-2155-52e8-91d8-8f6111edbb2c +33:39.839 [INFO] TestAgent.server: member joined, marking health alive: member=Node-09828261-2155-52e8-91d8-8f6111edbb2c +33:39.872 [INFO] TestAgent: Synced node info +33:39.872 [DEBUG] TestAgent: Node info in sync +33:39.890 [INFO] TestAgent.server: federation state anti-entropy synced +33:40.049 [INFO] TestAgent.server: server starting leave +33:40.049 [INFO] TestAgent.server.serf.wan: serf: EventMemberLeave: Node-09828261-2155-52e8-91d8-8f6111edbb2c.dc1 127.0.0.1 +33:40.049 [INFO] TestAgent.server: Handled event for server in area: event=member-leave server=Node-09828261-2155-52e8-91d8-8f6111edbb2c.dc1 area=wan +2021-10-05T16:33:40.049Z [INFO] agent.router.manager: shutting down +33:41.328 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:41.328 [DEBUG] TestAgent: Node info in sync +33:43.052 [INFO] TestAgent.server.serf.lan: serf: EventMemberLeave: Node-09828261-2155-52e8-91d8-8f6111edbb2c 127.0.0.1 +33:43.052 [INFO] TestAgent.server: Removing LAN server: server="Node-09828261-2155-52e8-91d8-8f6111edbb2c (Addr: tcp/127.0.0.1:14097) (DC: dc1)" +33:43.052 [WARN] TestAgent.server: deregistering self should be done by follower: name=Node-09828261-2155-52e8-91d8-8f6111edbb2c +33:43.801 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +33:45.801 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +33:46.056 [INFO] TestAgent.server: Waiting to drain RPC traffic: drain_time=5s +33:47.800 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +33:49.804 [ERROR] TestAgent.server.autopilot: Error promoting servers: error="error getting server raft protocol versions: No servers found" +33:49.804 [ERROR] TestAgent.server.autopilot: Error updating cluster health: error="error getting server raft protocol versions: No servers found" +33:51.060 [INFO] TestAgent: Requesting shutdown +33:51.060 [INFO] TestAgent.server: shutting down server +33:51.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:51.060 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:51.060 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:51.060 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:51.060 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:51.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:51.060 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:51.060 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:51.060 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +2021-10-05T16:33:51.073Z [INFO] agent.router.manager: shutting down +33:51.074 [INFO] TestAgent: consul server down +33:51.074 [INFO] TestAgent: shutdown complete +33:51.074 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/leave from=127.0.0.1:32836 latency=11.024774113s +33:51.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=tcp +33:51.076 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14092 network=udp +33:51.076 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14093 network=tcp +33:51.576 [INFO] TestAgent: Waiting for endpoints to shut down +33:51.576 [INFO] TestAgent: Endpoints down +--- PASS: TestLeaveCommand (12.12s) PASS -ok github.com/hashicorp/consul/command/leave 11.990s +ok github.com/hashicorp/consul/command/leave 12.216s === RUN TestLockCommand_noTabs === PAUSE TestLockCommand_noTabs === RUN TestLockCommand_BadArgs @@ -87874,660 +88029,663 @@ === RUN TestLockCommand_ChildExitCode === PAUSE TestLockCommand_ChildExitCode === CONT TestLockCommand_noTabs -=== CONT TestLockCommand_MonitorRetry_Lock_Default -=== CONT TestLockCommand_MonitorRetry_Semaphore_Arg -=== CONT TestLockCommand_NoShell +=== CONT TestLockCommand_MonitorRetry_Semaphore_Default +=== CONT TestLockCommand_TryLock === CONT TestLockCommand +=== CONT TestLockCommand_NoShell +=== CONT TestLockCommand_ChildExitCode [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestLockCommand_TrySemaphore -=== CONT TestLockCommand_TryLock +--- PASS: TestLockCommand_noTabs (0.00s) === CONT TestLockCommand_BadArgs +=== CONT TestLockCommand_MonitorRetry_Lock_Default +=== CONT TestLockCommand_TrySemaphore +=== CONT TestLockCommand_MonitorRetry_Semaphore_Arg === CONT TestLockCommand_MonitorRetry_Lock_Arg -=== CONT TestLockCommand_MonitorRetry_Semaphore_Default -=== CONT TestLockCommand_ChildExitCode ---- PASS: TestLockCommand_noTabs (0.00s) ---- PASS: TestLockCommand_BadArgs (0.00s) +--- PASS: TestLockCommand_BadArgs (0.02s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestLockCommand_ChildExitCode/clean_exit === RUN TestLockCommand_ChildExitCode/error_exit === RUN TestLockCommand_ChildExitCode/not_propagated -2022-11-07T22:46:34.931Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.969Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9053de81-ad39-24fe-fdb8-8043cc2fe1ea Address:127.0.0.1:20251}]" -46:35.018 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20251 [Follower]" leader= -46:35.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea.dc1 127.0.0.1 -46:35.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea 127.0.0.1 -2022-11-07T22:46:35.021Z [INFO] agent.router: Initializing LAN area manager -46:35.021 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20246 network=udp -46:35.021 [INFO] TestAgent.server: Adding LAN server: server="Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea (Addr: tcp/127.0.0.1:20251) (DC: dc1)" -46:35.021 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea.dc1 area=wan -46:35.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20246 network=tcp -46:35.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20247 network=tcp -46:35.043 [INFO] TestAgent: started state syncer -46:35.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20252 network=tcp -46:35.085 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.085 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20251 [Candidate]" term=2 -46:35.099 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.099 [DEBUG] TestAgent.server.raft: vote granted: from=9053de81-ad39-24fe-fdb8-8043cc2fe1ea term=2 tally=1 -46:35.099 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.099 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20251 [Leader]" -46:35.099 [INFO] TestAgent.server: cluster leadership acquired -46:35.099 [INFO] TestAgent.server: New leader elected: payload=Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea -46:35.121 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20251 -46:35.150 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.198 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.198 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.198 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.198 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.198 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.198 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea -46:35.198 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9053de81-ad39-24fe-fdb8-8043cc2fe1ea -46:35.206 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.231 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59704 latency=2.071389ms -46:35.240 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:59704 latency=6.60432ms -46:35.243 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/372f9b8b-b3ed-4b9e-1384-de68ba34cf72?acquire=372f9b8b-b3ed-4b9e-1384-de68ba34cf72&flags=16210313421097356768 from=127.0.0.1:59704 latency=2.835885ms -46:35.244 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=10000ms from=127.0.0.1:59704 latency=866.897µs -46:35.250 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:59704 latency=5.614143ms -46:35.252 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:59704 latency=669.109µs -46:35.278 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:59716 latency=251.412µs -46:35.282 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=14&flags=16210313421097356768 from=127.0.0.1:59716 latency=3.816118ms -46:35.285 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=14&recurse= from=127.0.0.1:59704 latency=32.55149ms -46:35.292 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:35.299 [INFO] TestAgent: Synced node info -46:35.300 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/372f9b8b-b3ed-4b9e-1384-de68ba34cf72 from=127.0.0.1:59716 latency=13.497855ms -46:35.300 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:59716 latency=118.884µs -46:35.306 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/372f9b8b-b3ed-4b9e-1384-de68ba34cf72 from=127.0.0.1:59704 latency=5.784632ms -46:35.309 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:59716 latency=8.294989ms -46:35.309 [INFO] TestAgent: Requesting shutdown -46:35.309 [INFO] TestAgent.server: shutting down server -46:35.309 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.309 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.309 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.309 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.309 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.309 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.309 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.309 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.309 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.309 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.314 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.321Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.321Z [INFO] agent.router.manager: shutting down -46:35.321 [INFO] TestAgent: consul server down -46:35.321 [INFO] TestAgent: shutdown complete -46:35.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20246 network=tcp -46:35.321 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20246 network=udp -46:35.321 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20247 network=tcp -46:35.821 [INFO] TestAgent: Waiting for endpoints to shut down -46:35.821 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_TrySemaphore (0.94s) -2022-11-07T22:46:34.945Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.974Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.042 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0596f670-9eba-6c59-044e-a6090b82742c Address:127.0.0.1:20258}]" -46:35.042 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20258 [Follower]" leader= -46:35.043 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0596f670-9eba-6c59-044e-a6090b82742c.dc1 127.0.0.1 -46:35.044 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0596f670-9eba-6c59-044e-a6090b82742c 127.0.0.1 -2022-11-07T22:46:35.044Z [INFO] agent.router: Initializing LAN area manager -46:35.044 [INFO] TestAgent.server: Adding LAN server: server="Node-0596f670-9eba-6c59-044e-a6090b82742c (Addr: tcp/127.0.0.1:20258) (DC: dc1)" -46:35.046 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20253 network=udp -46:35.046 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20253 network=tcp -46:35.046 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0596f670-9eba-6c59-044e-a6090b82742c.dc1 area=wan -46:35.046 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20254 network=tcp -46:35.047 [INFO] TestAgent: started state syncer -46:35.047 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20259 network=tcp -46:35.086 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.086 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20258 [Candidate]" term=2 -46:35.098 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.098 [DEBUG] TestAgent.server.raft: vote granted: from=0596f670-9eba-6c59-044e-a6090b82742c term=2 tally=1 -46:35.098 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.098 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20258 [Leader]" -46:35.098 [INFO] TestAgent.server: cluster leadership acquired -46:35.098 [INFO] TestAgent.server: New leader elected: payload=Node-0596f670-9eba-6c59-044e-a6090b82742c -46:35.111 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20258 -46:35.144 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.177 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.177 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.177 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.177 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.177 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.177 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0596f670-9eba-6c59-044e-a6090b82742c -46:35.177 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0596f670-9eba-6c59-044e-a6090b82742c -46:35.197 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.277 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:60724 latency=1.291986ms -46:35.289 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:60724 latency=8.179071ms -46:35.291 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:60724 latency=45.025µs -46:35.296 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=dfe55407-97f6-dff1-cc59-7a0e55236ea4&flags=3304740253564472344 from=127.0.0.1:60724 latency=4.82849ms -46:35.300 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:60724 latency=2.131929ms -46:35.306 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=dfe55407-97f6-dff1-cc59-7a0e55236ea4 from=127.0.0.1:60734 latency=5.907311ms -46:35.307 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:60734 latency=117.785µs -46:35.307 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:60724 latency=6.313743ms -46:35.311 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/dfe55407-97f6-dff1-cc59-7a0e55236ea4 from=127.0.0.1:60736 latency=3.953792ms -46:35.319 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:60734 latency=12.039058ms -46:35.319 [INFO] TestAgent: Requesting shutdown -46:35.320 [INFO] TestAgent.server: shutting down server -46:35.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.320 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.320 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.320 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.320 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.320 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:35.320 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.320 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.320 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.320 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.323 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.326Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.326Z [INFO] agent.router.manager: shutting down -46:35.327 [INFO] TestAgent: consul server down -46:35.327 [INFO] TestAgent: shutdown complete -46:35.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20253 network=tcp -46:35.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20253 network=udp -46:35.327 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20254 network=tcp -46:35.827 [INFO] TestAgent: Waiting for endpoints to shut down -46:35.827 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_MonitorRetry_Lock_Arg (0.95s) -2022-11-07T22:46:34.937Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.969Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.018 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 Address:127.0.0.1:20237}]" -46:35.018 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20237 [Follower]" leader= -46:35.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050.dc1 127.0.0.1 -46:35.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 127.0.0.1 -2022-11-07T22:46:35.021Z [INFO] agent.router: Initializing LAN area manager -46:35.021 [INFO] TestAgent.server: Adding LAN server: server="Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 (Addr: tcp/127.0.0.1:20237) (DC: dc1)" -46:35.022 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050.dc1 area=wan -46:35.043 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20232 network=udp -46:35.043 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20232 network=tcp -46:35.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20233 network=tcp -46:35.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20238 network=tcp -46:35.043 [INFO] TestAgent: started state syncer -46:35.064 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.064 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20237 [Candidate]" term=2 -46:35.075 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.075 [DEBUG] TestAgent.server.raft: vote granted: from=8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 term=2 tally=1 -46:35.076 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.076 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20237 [Leader]" -46:35.076 [INFO] TestAgent.server: cluster leadership acquired -46:35.076 [INFO] TestAgent.server: New leader elected: payload=Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 -46:35.088 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20237 -46:35.111 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.151 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.151 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.151 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.151 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.151 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.151 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 -46:35.151 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8ed46b53-05cb-7ee8-5eb3-c0249ca4e050 -46:35.158 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.322 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:32952 latency=889.79µs -46:35.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:32952 latency=3.583153ms -46:35.328 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:32952 latency=28.883µs -46:35.330 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=b6ee1f02-ab2f-eb58-34c6-467c1983f7b6&flags=3304740253564472344 from=127.0.0.1:32952 latency=2.626494ms -46:35.333 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:32952 latency=1.915843ms -46:35.339 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=b6ee1f02-ab2f-eb58-34c6-467c1983f7b6 from=127.0.0.1:32956 latency=3.134495ms -46:35.339 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:32952 latency=5.64655ms -46:35.340 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:32956 latency=306.799µs -46:35.342 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/b6ee1f02-ab2f-eb58-34c6-467c1983f7b6 from=127.0.0.1:32952 latency=2.807242ms -46:35.348 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:32956 latency=8.358456ms -46:35.348 [INFO] TestAgent: Requesting shutdown -46:35.349 [INFO] TestAgent.server: shutting down server -46:35.349 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.349 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.349 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.349 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.349 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.349 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.349 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.349 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:35.349 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.349 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.349 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.352 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.357Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.357Z [INFO] agent.router.manager: shutting down -46:35.357 [INFO] TestAgent: consul server down -46:35.357 [INFO] TestAgent: shutdown complete -46:35.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20232 network=tcp -46:35.357 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20232 network=udp -46:35.357 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20233 network=tcp -46:35.857 [INFO] TestAgent: Waiting for endpoints to shut down -46:35.857 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand (0.98s) -2022-11-07T22:46:34.932Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.965Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.017 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e333088d-ddbb-f42d-bb0d-49758b5a46a5 Address:127.0.0.1:20223}]" -46:35.017 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20223 [Follower]" leader= -46:35.019 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5.dc1 127.0.0.1 -46:35.021 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5 127.0.0.1 -2022-11-07T22:46:35.021Z [INFO] agent.router: Initializing LAN area manager -46:35.021 [INFO] TestAgent.server: Adding LAN server: server="Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5 (Addr: tcp/127.0.0.1:20223) (DC: dc1)" -46:35.022 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5.dc1 area=wan -46:35.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=udp -46:35.042 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=tcp -46:35.043 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20219 network=tcp -46:35.043 [INFO] TestAgent: started state syncer -46:35.043 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20224 network=tcp -46:35.076 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.076 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20223 [Candidate]" term=2 -46:35.098 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.098 [DEBUG] TestAgent.server.raft: vote granted: from=e333088d-ddbb-f42d-bb0d-49758b5a46a5 term=2 tally=1 -46:35.098 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.098 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20223 [Leader]" -46:35.098 [INFO] TestAgent.server: cluster leadership acquired -46:35.098 [INFO] TestAgent.server: New leader elected: payload=Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5 -46:35.111 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20223 -46:35.143 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.175 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.175 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.175 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.175 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.175 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.175 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5 -46:35.175 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e333088d-ddbb-f42d-bb0d-49758b5a46a5 -46:35.268 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.398 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:37406 latency=1.498408ms -46:35.404 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:37406 latency=4.560587ms -46:35.404 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:37406 latency=67.291µs -46:35.408 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=4ed18731-c7ab-101f-7fd1-c4d6c38b8242&flags=3304740253564472344 from=127.0.0.1:37406 latency=3.371836ms -46:35.410 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:37406 latency=512.068µs -46:35.413 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:35.415 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=4ed18731-c7ab-101f-7fd1-c4d6c38b8242 from=127.0.0.1:37410 latency=3.245723ms -46:35.415 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:37410 latency=99.936µs -46:35.419 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:37406 latency=8.946797ms -46:35.420 [INFO] TestAgent: Synced node info -46:35.423 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:37410 latency=7.066618ms -46:35.423 [INFO] TestAgent: Requesting shutdown -46:35.423 [INFO] TestAgent.server: shutting down server -46:35.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.423 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.423 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.423 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.423 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.423 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/4ed18731-c7ab-101f-7fd1-c4d6c38b8242 from=127.0.0.1:37412 latency=7.580446ms -46:35.423 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.423 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.423 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.423 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.423 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.428 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.432Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.432Z [INFO] agent.router.manager: shutting down -46:35.432 [INFO] TestAgent: consul server down -46:35.432 [INFO] TestAgent: shutdown complete -46:35.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=tcp -46:35.432 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=udp -46:35.432 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20219 network=tcp -46:35.934 [INFO] TestAgent: Waiting for endpoints to shut down -46:35.934 [INFO] TestAgent: Endpoints down -2022-11-07T22:46:34.925Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.951Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:34.996 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9e7fd78d-b377-cff5-548f-c269e3911cd6 Address:127.0.0.1:20265}]" -46:34.996 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20265 [Follower]" leader= -46:34.997 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9e7fd78d-b377-cff5-548f-c269e3911cd6.dc1 127.0.0.1 -46:34.998 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9e7fd78d-b377-cff5-548f-c269e3911cd6 127.0.0.1 -2022-11-07T22:46:34.998Z [INFO] agent.router: Initializing LAN area manager -46:34.998 [INFO] TestAgent.server: Adding LAN server: server="Node-9e7fd78d-b377-cff5-548f-c269e3911cd6 (Addr: tcp/127.0.0.1:20265) (DC: dc1)" -46:34.998 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9e7fd78d-b377-cff5-548f-c269e3911cd6.dc1 area=wan -46:34.998 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20260 network=tcp -46:34.998 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20260 network=udp -46:34.999 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20261 network=tcp -46:34.999 [INFO] TestAgent: started state syncer -46:34.999 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20266 network=tcp -46:35.038 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.038 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20265 [Candidate]" term=2 -46:35.047 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.047 [DEBUG] TestAgent.server.raft: vote granted: from=9e7fd78d-b377-cff5-548f-c269e3911cd6 term=2 tally=1 -46:35.047 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.047 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20265 [Leader]" -46:35.047 [INFO] TestAgent.server: cluster leadership acquired -46:35.047 [INFO] TestAgent.server: New leader elected: payload=Node-9e7fd78d-b377-cff5-548f-c269e3911cd6 -46:35.052 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20265 -46:35.067 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.097 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.097 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.097 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.097 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.097 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.097 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9e7fd78d-b377-cff5-548f-c269e3911cd6 -46:35.097 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9e7fd78d-b377-cff5-548f-c269e3911cd6 -46:35.110 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.232 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:35.238 [INFO] TestAgent: Synced node info -46:35.385 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:47044 latency=1.243602ms -46:35.391 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:47044 latency=3.468393ms -46:35.395 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/62558a34-02e3-293d-b1d6-2334a9fd2227?acquire=62558a34-02e3-293d-b1d6-2334a9fd2227&flags=16210313421097356768 from=127.0.0.1:47044 latency=3.090688ms -46:35.395 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=15000ms from=127.0.0.1:47044 latency=110.202µs -46:35.400 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:47044 latency=4.391591ms -46:35.402 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:47044 latency=423.723µs -46:35.405 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:47048 latency=344.088µs -46:35.410 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=15&recurse= from=127.0.0.1:47044 latency=7.038378ms -46:35.410 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=15&flags=16210313421097356768 from=127.0.0.1:47048 latency=4.886782ms -46:35.414 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/62558a34-02e3-293d-b1d6-2334a9fd2227 from=127.0.0.1:47048 latency=3.568236ms -46:35.415 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:47048 latency=408.261µs -46:35.421 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=16 from=127.0.0.1:47048 latency=5.580784ms -46:35.421 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/62558a34-02e3-293d-b1d6-2334a9fd2227 from=127.0.0.1:47044 latency=6.404801ms -46:35.422 [INFO] TestAgent: Requesting shutdown -46:35.422 [INFO] TestAgent.server: shutting down server -46:35.422 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.422 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.422 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.422 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.422 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.422 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.422 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.422 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.422 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.425 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.430Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.430Z [INFO] agent.router.manager: shutting down -46:35.430 [INFO] TestAgent: consul server down -46:35.430 [INFO] TestAgent: shutdown complete -46:35.430 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20260 network=tcp -46:35.430 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20260 network=udp -46:35.430 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20261 network=tcp -46:35.934 [INFO] TestAgent: Waiting for endpoints to shut down -46:35.934 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_MonitorRetry_Lock_Default (1.06s) ---- PASS: TestLockCommand_MonitorRetry_Semaphore_Default (1.05s) -2022-11-07T22:46:34.945Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.972Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.042 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cacd6e5e-5625-ed33-bc44-9e6710211f36 Address:127.0.0.1:20216}]" -46:35.042 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= -46:35.043 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cacd6e5e-5625-ed33-bc44-9e6710211f36.dc1 127.0.0.1 -46:35.044 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cacd6e5e-5625-ed33-bc44-9e6710211f36 127.0.0.1 -2022-11-07T22:46:35.044Z [INFO] agent.router: Initializing LAN area manager -46:35.044 [INFO] TestAgent.server: Adding LAN server: server="Node-cacd6e5e-5625-ed33-bc44-9e6710211f36 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -46:35.044 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cacd6e5e-5625-ed33-bc44-9e6710211f36.dc1 area=wan -46:35.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp -46:35.046 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp -46:35.047 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp -46:35.047 [INFO] TestAgent: started state syncer -46:35.047 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp -46:35.109 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.109 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 -46:35.135 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.135 [DEBUG] TestAgent.server.raft: vote granted: from=cacd6e5e-5625-ed33-bc44-9e6710211f36 term=2 tally=1 -46:35.135 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.135 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" -46:35.135 [INFO] TestAgent.server: cluster leadership acquired -46:35.135 [INFO] TestAgent.server: New leader elected: payload=Node-cacd6e5e-5625-ed33-bc44-9e6710211f36 -46:35.146 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20216 -46:35.174 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.218 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.218 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.218 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.218 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.218 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.218 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cacd6e5e-5625-ed33-bc44-9e6710211f36 -46:35.218 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cacd6e5e-5625-ed33-bc44-9e6710211f36 -46:35.232 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.287 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:35.292 [INFO] TestAgent: Synced node info -46:35.463 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44916 latency=1.388361ms -46:35.469 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44916 latency=3.252171ms -46:35.472 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/011fb5d6-1404-3af4-5c95-63a76009fe06?acquire=011fb5d6-1404-3af4-5c95-63a76009fe06&flags=16210313421097356768 from=127.0.0.1:44916 latency=3.163087ms -46:35.473 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=15000ms from=127.0.0.1:44916 latency=155.21µs -46:35.478 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:44916 latency=4.97742ms -46:35.482 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:44916 latency=498.201µs -46:35.484 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44918 latency=156.719µs -46:35.488 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=15&flags=16210313421097356768 from=127.0.0.1:44918 latency=2.943589ms -46:35.488 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=15&recurse= from=127.0.0.1:44916 latency=4.801361ms -46:35.492 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/011fb5d6-1404-3af4-5c95-63a76009fe06 from=127.0.0.1:44918 latency=2.941926ms -46:35.492 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:44918 latency=161.465µs -46:35.498 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/011fb5d6-1404-3af4-5c95-63a76009fe06 from=127.0.0.1:44916 latency=5.877417ms -46:35.498 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=16 from=127.0.0.1:44918 latency=5.472689ms -46:35.499 [INFO] TestAgent: Requesting shutdown -46:35.499 [INFO] TestAgent.server: shutting down server -46:35.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.499 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.499 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.499 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.499 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.499 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.499 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.499 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.499 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.506 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.509Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.509Z [INFO] agent.router.manager: shutting down -46:35.509 [INFO] TestAgent: consul server down -46:35.509 [INFO] TestAgent: shutdown complete -46:35.509 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp -46:35.509 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp -46:35.509 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp -46:36.009 [INFO] TestAgent: Waiting for endpoints to shut down -46:36.009 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_MonitorRetry_Semaphore_Arg (1.13s) -2022-11-07T22:46:34.944Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.974Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.042 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d3fe89fd-3951-13ce-e7ca-70eb58913286 Address:127.0.0.1:20230}]" -46:35.042 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20230 [Follower]" leader= -46:35.043 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d3fe89fd-3951-13ce-e7ca-70eb58913286.dc1 127.0.0.1 -46:35.043 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d3fe89fd-3951-13ce-e7ca-70eb58913286 127.0.0.1 -2022-11-07T22:46:35.043Z [INFO] agent.router: Initializing LAN area manager -46:35.044 [INFO] TestAgent.server: Adding LAN server: server="Node-d3fe89fd-3951-13ce-e7ca-70eb58913286 (Addr: tcp/127.0.0.1:20230) (DC: dc1)" -46:35.044 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d3fe89fd-3951-13ce-e7ca-70eb58913286.dc1 area=wan -46:35.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=udp -46:35.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=tcp -46:35.046 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20226 network=tcp -46:35.046 [INFO] TestAgent: started state syncer -46:35.047 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20231 network=tcp -46:35.080 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.080 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20230 [Candidate]" term=2 -46:35.100 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.100 [DEBUG] TestAgent.server.raft: vote granted: from=d3fe89fd-3951-13ce-e7ca-70eb58913286 term=2 tally=1 -46:35.100 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.100 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20230 [Leader]" -46:35.100 [INFO] TestAgent.server: cluster leadership acquired -46:35.100 [INFO] TestAgent.server: New leader elected: payload=Node-d3fe89fd-3951-13ce-e7ca-70eb58913286 -46:35.118 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20230 -46:35.150 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.194 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.194 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.194 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.194 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.194 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.195 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d3fe89fd-3951-13ce-e7ca-70eb58913286 -46:35.195 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d3fe89fd-3951-13ce-e7ca-70eb58913286 -46:35.203 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.231 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:35968 latency=1.684689ms -46:35.241 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:35968 latency=7.390314ms -46:35.242 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:35968 latency=38.94µs -46:35.245 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=c00757b9-5785-2a22-498d-944c39fec009&flags=3304740253564472344 from=127.0.0.1:35968 latency=2.989002ms -46:35.248 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:35968 latency=1.477492ms -46:35.253 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=c00757b9-5785-2a22-498d-944c39fec009 from=127.0.0.1:35970 latency=3.263047ms -46:35.254 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:35968 latency=2.907481ms -46:35.254 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:35970 latency=393.533µs -46:35.258 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/c00757b9-5785-2a22-498d-944c39fec009 from=127.0.0.1:35968 latency=3.540487ms -46:35.262 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:35970 latency=7.224864ms -46:35.274 [INFO] TestAgent: Requesting shutdown -46:35.274 [INFO] TestAgent.server: shutting down server -46:35.275 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.275 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.275 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.275 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.275 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.275 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.275 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:35.275 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.275 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.275 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.275 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.280 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.287Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.288Z [INFO] agent.router.manager: shutting down -46:35.288 [INFO] TestAgent: consul server down -46:35.288 [INFO] TestAgent: shutdown complete -46:35.288 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=tcp -46:35.289 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=udp -46:35.289 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20226 network=tcp -46:36.290 [WARN] TestAgent: Timeout stopping server: protocol=HTTP address=127.0.0.1:20226 network=tcp -46:36.290 [INFO] TestAgent: Waiting for endpoints to shut down -46:36.290 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_NoShell (1.41s) -2022-11-07T22:46:34.940Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.962Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.009 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c375308-f7b4-b0d5-a7e6-ce0a198a9eea Address:127.0.0.1:20272}]" -46:35.009 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20272 [Follower]" leader= -46:35.011 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea.dc1 127.0.0.1 -46:35.011 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea 127.0.0.1 -2022-11-07T22:46:35.011Z [INFO] agent.router: Initializing LAN area manager -46:35.011 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea.dc1 area=wan -46:35.011 [INFO] TestAgent.server: Adding LAN server: server="Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea (Addr: tcp/127.0.0.1:20272) (DC: dc1)" -46:35.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20267 network=tcp -46:35.012 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20267 network=udp -46:35.012 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20268 network=tcp -46:35.012 [INFO] TestAgent: started state syncer -46:35.012 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20273 network=tcp -46:35.069 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.069 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20272 [Candidate]" term=2 -46:35.078 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.078 [DEBUG] TestAgent.server.raft: vote granted: from=0c375308-f7b4-b0d5-a7e6-ce0a198a9eea term=2 tally=1 -46:35.078 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.078 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20272 [Leader]" -46:35.078 [INFO] TestAgent.server: cluster leadership acquired -46:35.078 [INFO] TestAgent.server: New leader elected: payload=Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea -46:35.083 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20272 -46:35.102 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.143 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.144 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.144 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.144 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.144 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.144 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea -46:35.144 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c375308-f7b4-b0d5-a7e6-ce0a198a9eea -46:35.159 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.160 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44594 latency=1.27966ms -46:35.174 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44594 latency=10.944097ms -46:35.177 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44594 latency=56.347µs -46:35.188 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=2b97cf73-0fff-6e08-f357-fc462b038465&flags=3304740253564472344 from=127.0.0.1:44594 latency=10.486831ms -46:35.192 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44594 latency=600.171µs -46:35.201 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=2b97cf73-0fff-6e08-f357-fc462b038465 from=127.0.0.1:44596 latency=6.922251ms -46:35.201 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:44594 latency=8.810547ms -46:35.203 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44596 latency=1.123452ms -46:35.218 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:44596 latency=14.68878ms -46:35.220 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/2b97cf73-0fff-6e08-f357-fc462b038465 from=127.0.0.1:44594 latency=17.528201ms -46:35.226 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44598 latency=5.003021ms -46:35.241 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44598 latency=11.472685ms -46:35.242 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44598 latency=50.409µs -46:35.248 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=02a93501-3d60-d830-7e08-801f20f88568&flags=3304740253564472344 from=127.0.0.1:44598 latency=5.569491ms -46:35.250 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44598 latency=432.906µs -46:35.279 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=02a93501-3d60-d830-7e08-801f20f88568 from=127.0.0.1:44606 latency=3.554801ms -46:35.279 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=18 from=127.0.0.1:44598 latency=28.466027ms -46:35.280 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44606 latency=401.093µs -46:35.287 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=19 from=127.0.0.1:44606 latency=7.235257ms -46:35.289 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/02a93501-3d60-d830-7e08-801f20f88568 from=127.0.0.1:44598 latency=9.72373ms -46:35.292 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44616 latency=771.569µs -46:35.299 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44616 latency=5.415026ms -46:35.299 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44616 latency=68.511µs -46:35.304 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=828c1854-a55f-5382-8148-c54a7b2167b0&flags=3304740253564472344 from=127.0.0.1:44616 latency=4.232872ms -46:35.306 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44616 latency=497.255µs -46:35.312 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=828c1854-a55f-5382-8148-c54a7b2167b0 from=127.0.0.1:44622 latency=3.979824ms -46:35.312 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=23 from=127.0.0.1:44616 latency=4.999766ms -46:35.312 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44622 latency=86.994µs -46:35.319 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=24 from=127.0.0.1:44622 latency=6.371048ms -46:35.319 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/828c1854-a55f-5382-8148-c54a7b2167b0 from=127.0.0.1:44616 latency=6.734651ms -46:35.319 [INFO] TestAgent: Requesting shutdown -46:35.319 [INFO] TestAgent.server: shutting down server -46:35.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.319 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.319 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.319 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.319 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.319 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.319 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.319 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:35.319 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.319 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.319 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.323 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.326Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.326Z [INFO] agent.router.manager: shutting down -46:35.327 [INFO] TestAgent: consul server down -46:35.327 [INFO] TestAgent: shutdown complete -46:35.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20267 network=tcp -46:35.327 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20267 network=udp -46:35.327 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20268 network=tcp -46:36.327 [WARN] TestAgent: Timeout stopping server: protocol=HTTP address=127.0.0.1:20268 network=tcp -46:36.327 [INFO] TestAgent: Waiting for endpoints to shut down -46:36.327 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_ChildExitCode (1.44s) - --- PASS: TestLockCommand_ChildExitCode/clean_exit (0.06s) - --- PASS: TestLockCommand_ChildExitCode/error_exit (0.07s) - --- PASS: TestLockCommand_ChildExitCode/not_propagated (0.03s) -2022-11-07T22:46:34.941Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:34.971Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:35.042 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40c4daf2-277b-f0f6-c0df-aadfc4d1d51f Address:127.0.0.1:20244}]" -46:35.042 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20244 [Follower]" leader= -46:35.043 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f.dc1 127.0.0.1 -46:35.043 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f 127.0.0.1 -2022-11-07T22:46:35.043Z [INFO] agent.router: Initializing LAN area manager -46:35.044 [INFO] TestAgent.server: Adding LAN server: server="Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f (Addr: tcp/127.0.0.1:20244) (DC: dc1)" -46:35.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20239 network=udp -46:35.044 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f.dc1 area=wan -46:35.044 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20239 network=tcp -46:35.047 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20240 network=tcp -46:35.047 [INFO] TestAgent: started state syncer -46:35.047 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20245 network=tcp -46:35.109 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:35.109 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20244 [Candidate]" term=2 -46:35.131 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:35.131 [DEBUG] TestAgent.server.raft: vote granted: from=40c4daf2-277b-f0f6-c0df-aadfc4d1d51f term=2 tally=1 -46:35.131 [INFO] TestAgent.server.raft: election won: tally=1 -46:35.131 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20244 [Leader]" -46:35.132 [INFO] TestAgent.server: cluster leadership acquired -46:35.132 [INFO] TestAgent.server: New leader elected: payload=Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f -46:35.141 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20244 -46:35.163 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:35.205 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:35.205 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:35.205 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:35.205 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:35.205 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:35.206 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f -46:35.206 [INFO] TestAgent.server: member joined, marking health alive: member=Node-40c4daf2-277b-f0f6-c0df-aadfc4d1d51f -46:35.223 [INFO] TestAgent.server: federation state anti-entropy synced -46:35.224 [INFO] TestAgent: Synced node info -46:35.224 [DEBUG] TestAgent: Node info in sync -46:35.325 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:46374 latency=709.688µs -46:35.330 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:46374 latency=3.016757ms -46:35.330 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=10000ms from=127.0.0.1:46374 latency=33.924µs -46:35.334 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=882a9aae-9a23-1e69-801c-adb26f372d24&flags=3304740253564472344 from=127.0.0.1:46374 latency=3.544609ms -46:35.336 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:46374 latency=583.583µs -46:35.340 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=882a9aae-9a23-1e69-801c-adb26f372d24 from=127.0.0.1:46378 latency=2.709917ms -46:35.340 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=14 from=127.0.0.1:46374 latency=3.768901ms -46:35.340 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:46378 latency=69.891µs -46:35.349 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/882a9aae-9a23-1e69-801c-adb26f372d24 from=127.0.0.1:46374 latency=8.768535ms -46:35.349 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:46378 latency=8.879178ms -46:35.350 [INFO] TestAgent: Requesting shutdown -46:35.350 [INFO] TestAgent.server: shutting down server -46:35.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:35.350 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:35.350 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:35.350 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:35.350 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:35.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:35.350 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:35.350 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:35.350 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:35.350 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:35.354 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:35.359Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:35.359Z [INFO] agent.router.manager: shutting down -46:35.359 [INFO] TestAgent: consul server down -46:35.359 [INFO] TestAgent: shutdown complete -46:35.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20239 network=tcp -46:35.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20239 network=udp -46:35.359 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20240 network=tcp -46:36.360 [WARN] TestAgent: Timeout stopping server: protocol=HTTP address=127.0.0.1:20240 network=tcp -46:36.360 [INFO] TestAgent: Waiting for endpoints to shut down -46:36.360 [INFO] TestAgent: Endpoints down ---- PASS: TestLockCommand_TryLock (1.48s) +2021-10-05T16:33:43.131Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.249Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.413 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:54bed6f3-748e-af67-8b07-d7a6cd9d6c14 Address:127.0.0.1:30426}]" +33:43.414 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14.dc1 127.0.0.1 +33:43.415 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14 127.0.0.1 +2021-10-05T16:33:43.415Z [INFO] agent.router: Initializing LAN area manager +33:43.428 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp +33:43.428 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= +33:43.429 [INFO] TestAgent.server: Adding LAN server: server="Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" +33:43.429 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14.dc1 area=wan +33:43.429 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp +33:43.430 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp +33:43.430 [INFO] TestAgent: started state syncer +33:43.432 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp +33:43.500 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.500 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 +33:43.520 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.520 [DEBUG] TestAgent.server.raft: vote granted: from=54bed6f3-748e-af67-8b07-d7a6cd9d6c14 term=2 tally=1 +33:43.520 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.520 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" +33:43.521 [INFO] TestAgent.server: cluster leadership acquired +33:43.521 [INFO] TestAgent.server: New leader elected: payload=Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14 +33:43.527 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30426 +33:43.568 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.639 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.639 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.639 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.639 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.639 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.639 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14 +33:43.639 [INFO] TestAgent.server: member joined, marking health alive: member=Node-54bed6f3-748e-af67-8b07-d7a6cd9d6c14 +33:43.680 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:55718 latency=1.905695ms +33:43.684 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.700 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:55718 latency=17.214908ms +33:43.710 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:43.718 [INFO] TestAgent: Synced node info +33:43.730 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/68043c89-232f-e79f-af6e-d7deb215de45?acquire=68043c89-232f-e79f-af6e-d7deb215de45&flags=16210313421097356768 from=127.0.0.1:55718 latency=17.825566ms +33:43.731 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=15000ms from=127.0.0.1:55718 latency=338.365µs +33:43.750 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:55718 latency=14.577235ms +33:43.758 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:55718 latency=6.812617ms +33:43.761 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:55718 latency=1.352036ms +33:43.778 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=15&flags=16210313421097356768 from=127.0.0.1:55718 latency=12.354382ms +33:43.778 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=15&recurse= from=127.0.0.1:55720 latency=18.821933ms +33:43.794 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/68043c89-232f-e79f-af6e-d7deb215de45 from=127.0.0.1:55718 latency=8.567678ms +33:43.796 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:55718 latency=610.527µs +33:43.821 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=16 from=127.0.0.1:55718 latency=22.289242ms +33:43.822 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/68043c89-232f-e79f-af6e-d7deb215de45 from=127.0.0.1:55720 latency=27.733461ms +33:43.823 [INFO] TestAgent: Requesting shutdown +33:43.823 [INFO] TestAgent.server: shutting down server +33:43.823 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:43.823 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:43.823 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:43.823 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:43.823 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:43.823 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:43.823 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:43.823 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:43.823 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:43.823 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:43.831 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:43.839Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:43.839Z [INFO] agent.router.manager: shutting down +33:43.839 [INFO] TestAgent: consul server down +33:43.839 [INFO] TestAgent: shutdown complete +33:43.839 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp +33:43.839 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp +33:43.839 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp +33:44.340 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.340 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_MonitorRetry_Semaphore_Default (1.31s) +2021-10-05T16:33:43.254Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.299Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.490 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:95204445-fc8e-d776-cb30-43827310c65d Address:127.0.0.1:30447}]" +33:43.512 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30447 [Follower]" leader= +33:43.547 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-95204445-fc8e-d776-cb30-43827310c65d.dc1 127.0.0.1 +33:43.547 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-95204445-fc8e-d776-cb30-43827310c65d 127.0.0.1 +2021-10-05T16:33:43.547Z [INFO] agent.router: Initializing LAN area manager +33:43.547 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30442 network=udp +33:43.564 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30442 network=tcp +33:43.565 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30443 network=tcp +33:43.565 [INFO] TestAgent: started state syncer +33:43.565 [INFO] TestAgent.server: Adding LAN server: server="Node-95204445-fc8e-d776-cb30-43827310c65d (Addr: tcp/127.0.0.1:30447) (DC: dc1)" +33:43.565 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30448 network=tcp +33:43.565 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-95204445-fc8e-d776-cb30-43827310c65d.dc1 area=wan +33:43.580 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.580 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30447 [Candidate]" term=2 +33:43.636 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.636 [DEBUG] TestAgent.server.raft: vote granted: from=95204445-fc8e-d776-cb30-43827310c65d term=2 tally=1 +33:43.636 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.636 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30447 [Leader]" +33:43.638 [INFO] TestAgent.server: cluster leadership acquired +33:43.639 [INFO] TestAgent.server: New leader elected: payload=Node-95204445-fc8e-d776-cb30-43827310c65d +33:43.655 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30447 +33:43.707 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.786 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.786 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.786 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.786 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.786 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.786 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-95204445-fc8e-d776-cb30-43827310c65d +33:43.786 [INFO] TestAgent.server: member joined, marking health alive: member=Node-95204445-fc8e-d776-cb30-43827310c65d +33:43.826 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.854 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:51630 latency=13.008577ms +33:43.869 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:51630 latency=9.170188ms +33:43.872 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=10000ms from=127.0.0.1:51630 latency=57.699µs +33:43.887 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=dcbd7bc6-ec81-a342-022d-d0d65c6ef340&flags=3304740253564472344 from=127.0.0.1:51630 latency=14.80847ms +33:43.895 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:51630 latency=4.851578ms +33:43.912 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:43.917 [INFO] TestAgent: Synced node info +33:43.942 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:51630 latency=31.978932ms +33:43.944 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=dcbd7bc6-ec81-a342-022d-d0d65c6ef340 from=127.0.0.1:51640 latency=22.466466ms +33:43.957 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:51640 latency=11.069267ms +33:43.964 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/dcbd7bc6-ec81-a342-022d-d0d65c6ef340 from=127.0.0.1:51630 latency=19.178056ms +33:44.010 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:51640 latency=49.301695ms +33:44.014 [INFO] TestAgent: Requesting shutdown +33:44.014 [INFO] TestAgent.server: shutting down server +33:44.014 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.014 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.014 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.014 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.014 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.014 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.014 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.014 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.014 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.014 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.024 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.036Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.040Z [INFO] agent.router.manager: shutting down +33:44.044 [INFO] TestAgent: consul server down +33:44.044 [INFO] TestAgent: shutdown complete +33:44.044 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30442 network=tcp +33:44.045 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30442 network=udp +33:44.045 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30443 network=tcp +33:44.548 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.548 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_TryLock (1.52s) +2021-10-05T16:33:43.279Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.436Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.496 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:78fabf2e-c1d9-8be0-9909-d0f7196e1f24 Address:127.0.0.1:30440}]" +33:43.496 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24.dc1 127.0.0.1 +33:43.497 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30440 [Follower]" leader= +33:43.535 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.535 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30440 [Candidate]" term=2 +33:43.545 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24 127.0.0.1 +2021-10-05T16:33:43.545Z [INFO] agent.router: Initializing LAN area manager +33:43.546 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30435 network=udp +33:43.549 [INFO] TestAgent.server: Adding LAN server: server="Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24 (Addr: tcp/127.0.0.1:30440) (DC: dc1)" +33:43.564 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24.dc1 area=wan +33:43.566 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30435 network=tcp +33:43.567 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30436 network=tcp +33:43.567 [INFO] TestAgent: started state syncer +33:43.567 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30441 network=tcp +33:43.582 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.582 [DEBUG] TestAgent.server.raft: vote granted: from=78fabf2e-c1d9-8be0-9909-d0f7196e1f24 term=2 tally=1 +33:43.582 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.582 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30440 [Leader]" +33:43.584 [INFO] TestAgent.server: cluster leadership acquired +33:43.584 [INFO] TestAgent.server: New leader elected: payload=Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24 +33:43.608 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30440 +33:43.660 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.718 [INFO] TestAgent: Synced node info +33:43.737 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.737 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.737 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.737 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.737 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.737 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24 +33:43.737 [INFO] TestAgent.server: member joined, marking health alive: member=Node-78fabf2e-c1d9-8be0-9909-d0f7196e1f24 +33:43.765 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.874 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:51770 latency=1.57632ms +33:43.899 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:51770 latency=21.083131ms +33:43.911 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:51770 latency=267.402µs +33:43.956 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=757bd25c-1299-2405-360a-ade057884950&flags=3304740253564472344 from=127.0.0.1:51770 latency=44.553822ms +33:44.005 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:51770 latency=23.334578ms +33:44.025 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=757bd25c-1299-2405-360a-ade057884950 from=127.0.0.1:51788 latency=12.169547ms +33:44.025 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=14 from=127.0.0.1:51770 latency=17.5974ms +33:44.049 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:51788 latency=14.459852ms +33:44.078 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/757bd25c-1299-2405-360a-ade057884950 from=127.0.0.1:51770 latency=51.998224ms +33:44.080 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:51788 latency=30.22639ms +33:44.082 [INFO] TestAgent: Requesting shutdown +33:44.082 [INFO] TestAgent.server: shutting down server +33:44.082 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.082 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.082 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.082 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.082 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.082 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.082 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.082 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.082 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.082 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.094 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.106Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.106Z [INFO] agent.router.manager: shutting down +33:44.107 [INFO] TestAgent: consul server down +33:44.107 [INFO] TestAgent: shutdown complete +33:44.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30435 network=tcp +33:44.107 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30435 network=udp +33:44.108 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30436 network=tcp +33:44.612 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.612 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_NoShell (1.58s) +2021-10-05T16:33:43.429Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.547Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.655 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:da279459-eba7-14a0-300c-c74738528af9 Address:127.0.0.1:30482}]" +33:43.673 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-da279459-eba7-14a0-300c-c74738528af9.dc1 127.0.0.1 +33:43.673 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-da279459-eba7-14a0-300c-c74738528af9 127.0.0.1 +2021-10-05T16:33:43.673Z [INFO] agent.router: Initializing LAN area manager +33:43.674 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30477 network=udp +33:43.674 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30482 [Follower]" leader= +33:43.674 [INFO] TestAgent.server: Adding LAN server: server="Node-da279459-eba7-14a0-300c-c74738528af9 (Addr: tcp/127.0.0.1:30482) (DC: dc1)" +33:43.674 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-da279459-eba7-14a0-300c-c74738528af9.dc1 area=wan +33:43.674 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30477 network=tcp +33:43.675 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30478 network=tcp +33:43.675 [INFO] TestAgent: started state syncer +33:43.675 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30483 network=tcp +33:43.711 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.711 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30482 [Candidate]" term=2 +33:43.735 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.736 [DEBUG] TestAgent.server.raft: vote granted: from=da279459-eba7-14a0-300c-c74738528af9 term=2 tally=1 +33:43.736 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.736 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30482 [Leader]" +33:43.736 [INFO] TestAgent.server: cluster leadership acquired +33:43.736 [INFO] TestAgent.server: New leader elected: payload=Node-da279459-eba7-14a0-300c-c74738528af9 +33:43.763 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30482 +33:43.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.844 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.844 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.845 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.845 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.845 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.845 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-da279459-eba7-14a0-300c-c74738528af9 +33:43.845 [INFO] TestAgent.server: member joined, marking health alive: member=Node-da279459-eba7-14a0-300c-c74738528af9 +33:43.871 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.884 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:39526 latency=1.683261ms +33:43.918 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:39526 latency=28.975144ms +33:43.923 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:39526 latency=52.357µs +33:43.957 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=c0cd0381-429a-eb69-617f-10f05622690a&flags=3304740253564472344 from=127.0.0.1:39526 latency=33.046936ms +33:44.006 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:39526 latency=15.048447ms +33:44.039 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=13 from=127.0.0.1:39526 latency=31.933746ms +33:44.058 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=c0cd0381-429a-eb69-617f-10f05622690a from=127.0.0.1:39538 latency=51.748844ms +33:44.059 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:39538 latency=509.119µs +33:44.076 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:44.099 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=14 from=127.0.0.1:39538 latency=38.693168ms +33:44.099 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/c0cd0381-429a-eb69-617f-10f05622690a from=127.0.0.1:39526 latency=40.429774ms +33:44.101 [INFO] TestAgent: Requesting shutdown +33:44.101 [INFO] TestAgent.server: shutting down server +33:44.101 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.101 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.101 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.101 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.101 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.101 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.101 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.101 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.101 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.101 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.116 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +33:44.120 [INFO] TestAgent: Synced node info +33:44.120 [DEBUG] TestAgent: Node info in sync +2021-10-05T16:33:44.120Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.120Z [INFO] agent.router.manager: shutting down +33:44.120 [INFO] TestAgent: consul server down +33:44.121 [INFO] TestAgent: shutdown complete +33:44.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30477 network=tcp +33:44.121 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30477 network=udp +33:44.121 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30478 network=tcp +33:44.636 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.636 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_MonitorRetry_Lock_Arg (1.59s) +2021-10-05T16:33:43.499Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.537Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.645 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:24f45052-4bf6-0c9e-273d-d9b253d13b6a Address:127.0.0.1:30461}]" +33:43.651 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30461 [Follower]" leader= +33:43.677 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a.dc1 127.0.0.1 +33:43.678 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a 127.0.0.1 +2021-10-05T16:33:43.678Z [INFO] agent.router: Initializing LAN area manager +33:43.678 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30456 network=udp +33:43.684 [INFO] TestAgent.server: Adding LAN server: server="Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a (Addr: tcp/127.0.0.1:30461) (DC: dc1)" +33:43.684 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a.dc1 area=wan +33:43.684 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30456 network=tcp +33:43.685 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30457 network=tcp +33:43.685 [INFO] TestAgent: started state syncer +33:43.685 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30462 network=tcp +33:43.720 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.720 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30461 [Candidate]" term=2 +33:43.744 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.744 [DEBUG] TestAgent.server.raft: vote granted: from=24f45052-4bf6-0c9e-273d-d9b253d13b6a term=2 tally=1 +33:43.744 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.744 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30461 [Leader]" +33:43.745 [INFO] TestAgent.server: cluster leadership acquired +33:43.745 [INFO] TestAgent.server: New leader elected: payload=Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a +33:43.762 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30461 +33:43.800 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.872 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.872 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.872 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.872 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.872 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a +33:43.872 [INFO] TestAgent.server: member joined, marking health alive: member=Node-24f45052-4bf6-0c9e-273d-d9b253d13b6a +33:43.898 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.914 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44244 latency=4.192923ms +33:43.945 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44244 latency=24.746189ms +33:43.955 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:43.957 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44244 latency=65.859µs +33:43.995 [INFO] TestAgent: Synced node info +33:44.011 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=04851cc2-b78b-97d9-691d-1d9b95746e65&flags=3304740253564472344 from=127.0.0.1:44244 latency=24.301308ms +33:44.019 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44244 latency=4.066691ms +33:44.083 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=14 from=127.0.0.1:44244 latency=62.937753ms +33:44.083 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=04851cc2-b78b-97d9-691d-1d9b95746e65 from=127.0.0.1:44260 latency=24.535902ms +33:44.085 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44260 latency=211.506µs +33:44.110 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/04851cc2-b78b-97d9-691d-1d9b95746e65 from=127.0.0.1:44244 latency=25.737344ms +33:44.124 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:44260 latency=39.013404ms +33:44.126 [INFO] TestAgent: Requesting shutdown +33:44.126 [INFO] TestAgent.server: shutting down server +33:44.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.126 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.126 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.126 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.126 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.127 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.127 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.127 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.127 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.127 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.136 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.142Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.142Z [INFO] agent.router.manager: shutting down +33:44.142 [INFO] TestAgent: consul server down +33:44.142 [INFO] TestAgent: shutdown complete +33:44.142 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30456 network=tcp +33:44.142 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30456 network=udp +33:44.142 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30457 network=tcp +33:44.643 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.643 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_MonitorRetry_Lock_Default (1.60s) +2021-10-05T16:33:43.355Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.460Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.648 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f Address:127.0.0.1:30468}]" +33:43.649 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f.dc1 127.0.0.1 +33:43.651 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f 127.0.0.1 +2021-10-05T16:33:43.651Z [INFO] agent.router: Initializing LAN area manager +33:43.651 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30463 network=udp +33:43.651 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30468 [Follower]" leader= +33:43.674 [INFO] TestAgent.server: Adding LAN server: server="Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f (Addr: tcp/127.0.0.1:30468) (DC: dc1)" +33:43.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30463 network=tcp +33:43.683 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f.dc1 area=wan +33:43.688 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30464 network=tcp +33:43.688 [INFO] TestAgent: started state syncer +33:43.689 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30469 network=tcp +33:43.714 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.714 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30468 [Candidate]" term=2 +33:43.742 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.742 [DEBUG] TestAgent.server.raft: vote granted: from=3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f term=2 tally=1 +33:43.742 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.742 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30468 [Leader]" +33:43.742 [INFO] TestAgent.server: cluster leadership acquired +33:43.742 [INFO] TestAgent.server: New leader elected: payload=Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f +33:43.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30468 +33:43.820 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.860 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.860 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.860 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.860 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.860 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.860 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f +33:43.860 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3ba2c7c9-8f4b-5e10-e4a6-bc328b5be62f +33:43.883 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:48130 latency=1.781167ms +33:43.884 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.919 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:48130 latency=4.189207ms +33:43.954 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/1cc0ed47-2b90-cb46-4233-50be73f71423?acquire=1cc0ed47-2b90-cb46-4233-50be73f71423&flags=16210313421097356768 from=127.0.0.1:48130 latency=25.59269ms +33:44.013 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=10000ms from=127.0.0.1:48130 latency=15.349598ms +33:44.026 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:48130 latency=12.164074ms +33:44.046 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:44.060 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:48146 latency=2.23063ms +33:44.081 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:48130 latency=33.371273ms +33:44.082 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=14&flags=16210313421097356768 from=127.0.0.1:48146 latency=21.063907ms +33:44.082 [INFO] TestAgent: Synced node info +33:44.089 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=14&recurse= from=127.0.0.1:48130 latency=6.05369ms +33:44.089 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/1cc0ed47-2b90-cb46-4233-50be73f71423 from=127.0.0.1:48146 latency=6.538591ms +33:44.094 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:48146 latency=171.087µs +33:44.112 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/1cc0ed47-2b90-cb46-4233-50be73f71423 from=127.0.0.1:48130 latency=18.268874ms +33:44.125 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=16 from=127.0.0.1:48146 latency=24.351665ms +33:44.125 [INFO] TestAgent: Requesting shutdown +33:44.125 [INFO] TestAgent.server: shutting down server +33:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.125 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.125 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.126 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.126 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.126 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.126 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.136 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.148Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.148Z [INFO] agent.router.manager: shutting down +33:44.152 [INFO] TestAgent: consul server down +33:44.152 [INFO] TestAgent: shutdown complete +33:44.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30463 network=tcp +33:44.152 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30463 network=udp +33:44.152 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30464 network=tcp +33:44.653 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.653 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_TrySemaphore (1.61s) +2021-10-05T16:33:43.171Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.381Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.472 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:152b5795-c054-dfab-0950-c33e52f4a2d1 Address:127.0.0.1:30433}]" +33:43.502 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30433 [Follower]" leader= +33:43.545 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-152b5795-c054-dfab-0950-c33e52f4a2d1.dc1 127.0.0.1 +33:43.546 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.546 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30433 [Candidate]" term=2 +33:43.556 [WARN] TestAgent.server.raft: unable to get address for sever, using fallback address: id=152b5795-c054-dfab-0950-c33e52f4a2d1 fallback=127.0.0.1:30433 error="Could not find address for server id 152b5795-c054-dfab-0950-c33e52f4a2d1" +33:43.564 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-152b5795-c054-dfab-0950-c33e52f4a2d1 127.0.0.1 +2021-10-05T16:33:43.565Z [INFO] agent.router: Initializing LAN area manager +33:43.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=udp +33:43.565 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-152b5795-c054-dfab-0950-c33e52f4a2d1.dc1 area=wan +33:43.565 [INFO] TestAgent.server: Adding LAN server: server="Node-152b5795-c054-dfab-0950-c33e52f4a2d1 (Addr: tcp/127.0.0.1:30433) (DC: dc1)" +33:43.565 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=tcp +33:43.566 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30429 network=tcp +33:43.566 [INFO] TestAgent: started state syncer +33:43.566 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30434 network=tcp +33:43.589 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.589 [DEBUG] TestAgent.server.raft: vote granted: from=152b5795-c054-dfab-0950-c33e52f4a2d1 term=2 tally=1 +33:43.589 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.589 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30433 [Leader]" +33:43.594 [INFO] TestAgent.server: cluster leadership acquired +33:43.594 [INFO] TestAgent.server: New leader elected: payload=Node-152b5795-c054-dfab-0950-c33e52f4a2d1 +33:43.637 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30433 +33:43.740 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.741 [INFO] TestAgent: Synced node info +33:43.812 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.812 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.812 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.812 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.812 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.812 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-152b5795-c054-dfab-0950-c33e52f4a2d1 +33:43.812 [INFO] TestAgent.server: member joined, marking health alive: member=Node-152b5795-c054-dfab-0950-c33e52f4a2d1 +33:43.860 [INFO] TestAgent.server: federation state anti-entropy synced +33:44.017 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:38626 latency=7.872698ms +33:44.052 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:38626 latency=29.538669ms +33:44.053 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:38626 latency=84.097µs +33:44.064 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=0f8fc3e9-3128-0c99-2c18-48c1f0ae43de&flags=3304740253564472344 from=127.0.0.1:38626 latency=11.132848ms +33:44.085 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:38626 latency=14.061868ms +33:44.088 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=0f8fc3e9-3128-0c99-2c18-48c1f0ae43de from=127.0.0.1:38636 latency=6.937491ms +33:44.093 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=14 from=127.0.0.1:38626 latency=375.101µs +33:44.093 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:38636 latency=347.154µs +33:44.132 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:38636 latency=38.195423ms +33:44.132 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/0f8fc3e9-3128-0c99-2c18-48c1f0ae43de from=127.0.0.1:38638 latency=39.067056ms +33:44.133 [INFO] TestAgent: Requesting shutdown +33:44.133 [INFO] TestAgent.server: shutting down server +33:44.133 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.133 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.133 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.133 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.133 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.133 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.133 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.134 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.134 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.134 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.148 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.156Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.156Z [INFO] agent.router.manager: shutting down +33:44.156 [INFO] TestAgent: consul server down +33:44.156 [INFO] TestAgent: shutdown complete +33:44.156 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=tcp +33:44.156 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=udp +33:44.156 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30429 network=tcp +33:44.658 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.658 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand (1.63s) +2021-10-05T16:33:43.547Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.583Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.656 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c4fe0992-47ff-0a14-b583-67024e7909cc Address:127.0.0.1:30475}]" +33:43.684 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c4fe0992-47ff-0a14-b583-67024e7909cc.dc1 127.0.0.1 +33:43.684 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30475 [Follower]" leader= +33:43.684 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c4fe0992-47ff-0a14-b583-67024e7909cc 127.0.0.1 +2021-10-05T16:33:43.684Z [INFO] agent.router: Initializing LAN area manager +33:43.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30470 network=udp +33:43.685 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c4fe0992-47ff-0a14-b583-67024e7909cc.dc1 area=wan +33:43.685 [INFO] TestAgent.server: Adding LAN server: server="Node-c4fe0992-47ff-0a14-b583-67024e7909cc (Addr: tcp/127.0.0.1:30475) (DC: dc1)" +33:43.685 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30470 network=tcp +33:43.685 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30471 network=tcp +33:43.686 [INFO] TestAgent: started state syncer +33:43.686 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30476 network=tcp +33:43.749 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.749 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30475 [Candidate]" term=2 +33:43.784 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.785 [DEBUG] TestAgent.server.raft: vote granted: from=c4fe0992-47ff-0a14-b583-67024e7909cc term=2 tally=1 +33:43.785 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.785 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30475 [Leader]" +33:43.786 [INFO] TestAgent.server: cluster leadership acquired +33:43.786 [INFO] TestAgent.server: New leader elected: payload=Node-c4fe0992-47ff-0a14-b583-67024e7909cc +33:43.813 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30475 +33:43.845 [INFO] TestAgent: Synced node info +33:43.845 [DEBUG] TestAgent: Node info in sync +33:43.872 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.942 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.942 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.942 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.942 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.942 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.943 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c4fe0992-47ff-0a14-b583-67024e7909cc +33:43.943 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c4fe0992-47ff-0a14-b583-67024e7909cc +33:43.963 [INFO] TestAgent.server: federation state anti-entropy synced +33:44.013 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:50486 latency=6.655732ms +33:44.046 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:50486 latency=30.326381ms +33:44.061 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/d30ded77-67bb-e139-d49d-d8051f131e06?acquire=d30ded77-67bb-e139-d49d-d8051f131e06&flags=16210313421097356768 from=127.0.0.1:50486 latency=9.124347ms +33:44.089 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse=&wait=15000ms from=127.0.0.1:50486 latency=11.370338ms +33:44.100 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=0&flags=16210313421097356768 from=127.0.0.1:50486 latency=7.755598ms +33:44.112 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&recurse= from=127.0.0.1:50486 latency=4.85039ms +33:44.125 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:50504 latency=954.243µs +33:44.137 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?cas=15&flags=16210313421097356768 from=127.0.0.1:50504 latency=10.264694ms +33:44.137 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?consistent=&index=15&recurse= from=127.0.0.1:50486 latency=20.875278ms +33:44.147 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/d30ded77-67bb-e139-d49d-d8051f131e06 from=127.0.0.1:50504 latency=7.030172ms +33:44.157 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix?recurse= from=127.0.0.1:50504 latency=666.582µs +33:44.165 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/d30ded77-67bb-e139-d49d-d8051f131e06 from=127.0.0.1:50486 latency=9.423932ms +33:44.166 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=16 from=127.0.0.1:50504 latency=5.21845ms +33:44.167 [INFO] TestAgent: Requesting shutdown +33:44.167 [INFO] TestAgent.server: shutting down server +33:44.167 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.167 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.167 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.167 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.167 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.167 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.167 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.167 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.167 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.167 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.171 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.176Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.176Z [INFO] agent.router.manager: shutting down +33:44.176 [INFO] TestAgent: consul server down +33:44.176 [INFO] TestAgent: shutdown complete +33:44.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30470 network=tcp +33:44.176 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30470 network=udp +33:44.176 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30471 network=tcp +33:44.677 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.677 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_MonitorRetry_Semaphore_Arg (1.63s) +2021-10-05T16:33:43.387Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:43.501Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:43.620 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1d46da6e-9190-0492-409b-b1317a09a603 Address:127.0.0.1:30454}]" +33:43.621 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1d46da6e-9190-0492-409b-b1317a09a603.dc1 127.0.0.1 +33:43.621 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1d46da6e-9190-0492-409b-b1317a09a603 127.0.0.1 +2021-10-05T16:33:43.621Z [INFO] agent.router: Initializing LAN area manager +33:43.641 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30454 [Follower]" leader= +33:43.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30449 network=udp +33:43.644 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30449 network=tcp +33:43.644 [INFO] TestAgent.server: Adding LAN server: server="Node-1d46da6e-9190-0492-409b-b1317a09a603 (Addr: tcp/127.0.0.1:30454) (DC: dc1)" +33:43.645 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1d46da6e-9190-0492-409b-b1317a09a603.dc1 area=wan +33:43.646 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30450 network=tcp +33:43.646 [INFO] TestAgent: started state syncer +33:43.675 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30455 network=tcp +33:43.704 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:43.704 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30454 [Candidate]" term=2 +33:43.749 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:43.749 [DEBUG] TestAgent.server.raft: vote granted: from=1d46da6e-9190-0492-409b-b1317a09a603 term=2 tally=1 +33:43.749 [INFO] TestAgent.server.raft: election won: tally=1 +33:43.749 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30454 [Leader]" +33:43.749 [INFO] TestAgent.server: cluster leadership acquired +33:43.749 [INFO] TestAgent.server: New leader elected: payload=Node-1d46da6e-9190-0492-409b-b1317a09a603 +33:43.774 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30454 +33:43.819 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:43.876 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:43.877 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:43.877 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:43.877 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:43.877 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:43.877 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1d46da6e-9190-0492-409b-b1317a09a603 +33:43.877 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1d46da6e-9190-0492-409b-b1317a09a603 +33:43.890 [INFO] TestAgent.server: federation state anti-entropy synced +33:43.940 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44080 latency=14.163023ms +33:44.003 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44080 latency=29.476493ms +33:44.016 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:44.020 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44080 latency=37.877µs +33:44.038 [INFO] TestAgent: Synced node info +33:44.045 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:44.045 [DEBUG] TestAgent: Node info in sync +33:44.045 [DEBUG] TestAgent: Node info in sync +33:44.053 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=44c51ca8-382e-7827-f3f7-860ce5b61aeb&flags=3304740253564472344 from=127.0.0.1:44080 latency=31.787842ms +33:44.060 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44080 latency=354.837µs +33:44.078 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=44c51ca8-382e-7827-f3f7-860ce5b61aeb from=127.0.0.1:44094 latency=10.21408ms +33:44.079 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=14 from=127.0.0.1:44080 latency=18.151565ms +33:44.079 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44080 latency=188.07µs +33:44.124 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=15 from=127.0.0.1:44080 latency=43.903877ms +33:44.125 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/44c51ca8-382e-7827-f3f7-860ce5b61aeb from=127.0.0.1:44094 latency=45.304809ms +33:44.130 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44102 latency=1.704003ms +33:44.145 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44102 latency=10.763107ms +33:44.145 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44102 latency=46.699µs +33:44.152 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=87f0291f-ac30-bae4-a849-50b974d12384&flags=3304740253564472344 from=127.0.0.1:44102 latency=6.867035ms +33:44.176 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=87f0291f-ac30-bae4-a849-50b974d12384 from=127.0.0.1:44102 latency=9.930749ms +33:44.176 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44104 latency=10.18713ms +33:44.178 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44102 latency=543.325µs +33:44.186 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/87f0291f-ac30-bae4-a849-50b974d12384 from=127.0.0.1:44104 latency=8.662713ms +33:44.186 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=20 from=127.0.0.1:44102 latency=7.967126ms +33:44.189 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:44106 latency=1.492689ms +33:44.203 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/create from=127.0.0.1:44106 latency=10.915316ms +33:44.206 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?wait=15000ms from=127.0.0.1:44106 latency=67.529µs +33:44.210 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?acquire=703ae6b1-c3f6-5678-9202-7e5f0be2ffc2&flags=3304740253564472344 from=127.0.0.1:44106 latency=3.844162ms +33:44.215 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent= from=127.0.0.1:44106 latency=1.744838ms +33:44.234 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/test/prefix/.lock?flags=3304740253564472344&release=703ae6b1-c3f6-5678-9202-7e5f0be2ffc2 from=127.0.0.1:44108 latency=3.991973ms +33:44.235 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock?consistent=&index=24 from=127.0.0.1:44106 latency=17.96711ms +33:44.236 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/kv/test/prefix/.lock from=127.0.0.1:44108 latency=560.964µs +33:44.242 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/kv/test/prefix/.lock?cas=25 from=127.0.0.1:44108 latency=5.783668ms +33:44.242 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/session/destroy/703ae6b1-c3f6-5678-9202-7e5f0be2ffc2 from=127.0.0.1:44106 latency=7.283986ms +33:44.243 [INFO] TestAgent: Requesting shutdown +33:44.243 [INFO] TestAgent.server: shutting down server +33:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:44.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:44.243 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:44.243 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:44.243 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:44.246 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:44.262Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:44.262Z [INFO] agent.router.manager: shutting down +33:44.262 [INFO] TestAgent: consul server down +33:44.262 [INFO] TestAgent: shutdown complete +33:44.262 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30449 network=tcp +33:44.262 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30449 network=udp +33:44.262 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30450 network=tcp +33:44.764 [INFO] TestAgent: Waiting for endpoints to shut down +33:44.764 [INFO] TestAgent: Endpoints down +--- PASS: TestLockCommand_ChildExitCode (1.74s) + --- PASS: TestLockCommand_ChildExitCode/clean_exit (0.20s) + --- PASS: TestLockCommand_ChildExitCode/error_exit (0.06s) + --- PASS: TestLockCommand_ChildExitCode/not_propagated (0.06s) PASS -ok github.com/hashicorp/consul/command/lock 1.516s +ok github.com/hashicorp/consul/command/lock 1.966s === RUN TestLoginCommand_noTabs === PAUSE TestLoginCommand_noTabs === RUN TestLoginCommand @@ -88537,296 +88695,293 @@ === RUN TestLoginCommand_jwt === PAUSE TestLoginCommand_jwt === CONT TestLoginCommand_noTabs +--- PASS: TestLoginCommand_noTabs (0.00s) === CONT TestLoginCommand_jwt +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestLoginCommand_k8s === CONT TestLoginCommand -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestLoginCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== RUN TestLoginCommand_k8s/try_login_with_method_configured_and_binding_rules -=== RUN TestLoginCommand_jwt/success_-_jwt_static_keys -=== RUN TestLoginCommand_jwt/success_-_jwt_jwks -=== RUN TestLoginCommand_jwt/success_-_jwt_oidc_discovery === RUN TestLoginCommand/method_is_required === RUN TestLoginCommand/token-sink-file_is_required === RUN TestLoginCommand/bearer-token-file_is_required === RUN TestLoginCommand/bearer-token-file_is_empty === RUN TestLoginCommand/try_login_with_no_method_configured +=== RUN TestLoginCommand_jwt/success_-_jwt_static_keys === RUN TestLoginCommand/try_login_with_method_configured_but_no_binding_rules === RUN TestLoginCommand/try_login_with_method_configured_and_binding_rules -2022-11-07T22:46:36.878Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:36.891Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:36.920 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e6e4df4a-3298-54c2-81f3-241dce108ace Address:127.0.0.1:18181}]" -46:36.920 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18181 [Follower]" leader= -46:36.921 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e6e4df4a-3298-54c2-81f3-241dce108ace.dc1 127.0.0.1 -46:36.921 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e6e4df4a-3298-54c2-81f3-241dce108ace 127.0.0.1 -2022-11-07T22:46:36.921Z [INFO] agent.router: Initializing LAN area manager -46:36.921 [INFO] TestAgent.server: Adding LAN server: server="Node-e6e4df4a-3298-54c2-81f3-241dce108ace (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -46:36.921 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e6e4df4a-3298-54c2-81f3-241dce108ace.dc1 area=wan -46:36.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=udp -46:36.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18176 network=tcp -46:36.922 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18177 network=tcp -46:36.922 [INFO] TestAgent: started state syncer -46:36.922 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18182 network=tcp -46:36.960 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:36.961 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18181 [Candidate]" term=2 -46:36.972 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:36.977 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:36.977 [DEBUG] TestAgent.server.raft: vote granted: from=e6e4df4a-3298-54c2-81f3-241dce108ace term=2 tally=1 -46:36.977 [INFO] TestAgent.server.raft: election won: tally=1 -46:36.977 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18181 [Leader]" -46:36.977 [INFO] TestAgent.server: cluster leadership acquired -46:36.978 [INFO] TestAgent.server: New leader elected: payload=Node-e6e4df4a-3298-54c2-81f3-241dce108ace -46:36.985 [INFO] TestAgent.server: initializing acls -46:37.006 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:37.006 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:37.008 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="ACL not found" -46:37.015 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:37.027 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:37.027 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:37.027 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:37.027 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-e6e4df4a-3298-54c2-81f3-241dce108ace -46:37.028 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-e6e4df4a-3298-54c2-81f3-241dce108ace.dc1 -46:37.028 [INFO] TestAgent.server: Updating LAN server: server="Node-e6e4df4a-3298-54c2-81f3-241dce108ace (Addr: tcp/127.0.0.1:18181) (DC: dc1)" -46:37.028 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-e6e4df4a-3298-54c2-81f3-241dce108ace.dc1 area=wan -46:37.044 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:37.064 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:37.064 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:37.064 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:37.064 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:37.064 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:37.064 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e6e4df4a-3298-54c2-81f3-241dce108ace -46:37.064 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e6e4df4a-3298-54c2-81f3-241dce108ace -46:37.089 [INFO] TestAgent.server: federation state anti-entropy synced -46:37.089 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e6e4df4a-3298-54c2-81f3-241dce108ace -46:37.106 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:55814 latency=14.182724ms -46:37.113 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:55814 latency=4.745153ms -46:37.133 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:55816 latency=18.005473ms -46:37.138 [INFO] TestAgent: Requesting shutdown -46:37.138 [INFO] TestAgent.server: shutting down server -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:37.138 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:37.138 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:37.138 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:37.138 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:37.141 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:37.144Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:37.144Z [INFO] agent.router.manager: shutting down -46:37.144 [INFO] TestAgent: consul server down -46:37.144 [INFO] TestAgent: shutdown complete -46:37.144 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=tcp -46:37.144 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18176 network=udp -46:37.145 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18177 network=tcp -46:37.645 [INFO] TestAgent: Waiting for endpoints to shut down -46:37.645 [INFO] TestAgent: Endpoints down ---- PASS: TestLoginCommand_k8s (0.82s) - --- PASS: TestLoginCommand_k8s/try_login_with_method_configured_and_binding_rules (0.02s) -2022-11-07T22:46:36.878Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:36.892Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:36.920 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ee3cd016-4f55-2955-80c1-3b07cc5873ca Address:127.0.0.1:18188}]" -46:36.920 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18188 [Follower]" leader= -46:36.920 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 127.0.0.1 -46:36.921 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca 127.0.0.1 -2022-11-07T22:46:36.921Z [INFO] agent.router: Initializing LAN area manager -46:36.921 [INFO] TestAgent.server: Adding LAN server: server="Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca (Addr: tcp/127.0.0.1:18188) (DC: dc1)" -46:36.921 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 area=wan -46:36.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18183 network=udp -46:36.921 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18183 network=tcp -46:36.921 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18184 network=tcp -46:36.921 [INFO] TestAgent: started state syncer -46:36.922 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18189 network=tcp -46:36.957 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:36.957 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18188 [Candidate]" term=2 -46:36.968 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:36.968 [DEBUG] TestAgent.server.raft: vote granted: from=ee3cd016-4f55-2955-80c1-3b07cc5873ca term=2 tally=1 -46:36.968 [INFO] TestAgent.server.raft: election won: tally=1 -46:36.968 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18188 [Leader]" -46:36.968 [INFO] TestAgent.server: cluster leadership acquired -46:36.968 [INFO] TestAgent.server: New leader elected: payload=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:36.972 [INFO] TestAgent.server: initializing acls -46:36.981 [INFO] TestAgent.server: initializing acls -46:36.982 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:36.982 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:37.008 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:37.008 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:37.027 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:37.027 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:37.038 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:37.038 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:37.038 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:37.038 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:37.038 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 -46:37.039 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 area=wan -46:37.039 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:37.039 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 -46:37.040 [INFO] TestAgent.server: Updating LAN server: server="Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca (Addr: tcp/127.0.0.1:18188) (DC: dc1)" -46:37.040 [INFO] TestAgent.server: Updating LAN server: server="Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca (Addr: tcp/127.0.0.1:18188) (DC: dc1)" -46:37.040 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca.dc1 area=wan -46:37.052 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:37.082 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:37.082 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:37.082 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:37.082 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:37.082 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:37.083 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.083 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.091 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.091 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ee3cd016-4f55-2955-80c1-3b07cc5873ca -46:37.095 [INFO] TestAgent.server: federation state anti-entropy synced -46:37.108 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:37.114 [INFO] TestAgent: Synced node info -46:37.114 [DEBUG] TestAgent: Node info in sync -46:37.306 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:46632 error="ACL not found" -46:37.306 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:46632 latency=242.513µs -46:37.310 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:46634 latency=3.44956ms -46:37.312 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:46636 error="Permission denied" -46:37.312 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:46636 latency=207.077µs -46:37.317 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:46634 latency=4.922517ms -46:37.329 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:46640 latency=9.611732ms -46:37.334 [INFO] TestAgent: Requesting shutdown -46:37.334 [INFO] TestAgent.server: shutting down server -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:37.334 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:37.334 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:37.334 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:37.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:37.334 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:37.334 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:37.334 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:37.334 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:37.335 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:37.338 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:37.341Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:37.341Z [INFO] agent.router.manager: shutting down -46:37.341 [INFO] TestAgent: consul server down -46:37.341 [INFO] TestAgent: shutdown complete -46:37.341 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18183 network=tcp -46:37.342 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18183 network=udp -46:37.342 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18184 network=tcp -46:37.842 [INFO] TestAgent: Waiting for endpoints to shut down -46:37.842 [INFO] TestAgent: Endpoints down ---- PASS: TestLoginCommand (1.01s) +=== RUN TestLoginCommand_jwt/success_-_jwt_jwks +=== RUN TestLoginCommand_k8s/try_login_with_method_configured_and_binding_rules +=== RUN TestLoginCommand_jwt/success_-_jwt_oidc_discovery +2021-10-05T16:33:45.347Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:45.402Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:45.504 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 Address:127.0.0.1:27377}]" +33:45.504 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27377 [Follower]" leader= +33:45.505 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1.dc1 127.0.0.1 +33:45.505 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 127.0.0.1 +2021-10-05T16:33:45.505Z [INFO] agent.router: Initializing LAN area manager +33:45.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27372 network=udp +33:45.506 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1.dc1 area=wan +33:45.506 [INFO] TestAgent.server: Adding LAN server: server="Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 (Addr: tcp/127.0.0.1:27377) (DC: dc1)" +33:45.506 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27372 network=tcp +33:45.506 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27373 network=tcp +33:45.506 [INFO] TestAgent: started state syncer +33:45.506 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27378 network=tcp +33:45.556 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +33:45.556 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:45.556 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27377 [Candidate]" term=2 +33:45.573 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:45.573 [DEBUG] TestAgent.server.raft: vote granted: from=5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 term=2 tally=1 +33:45.573 [INFO] TestAgent.server.raft: election won: tally=1 +33:45.573 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27377 [Leader]" +33:45.573 [INFO] TestAgent.server: cluster leadership acquired +33:45.573 [INFO] TestAgent.server: New leader elected: payload=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 +33:45.583 [INFO] TestAgent.server: initializing acls +33:45.587 [INFO] TestAgent.server: Created ACL 'global-management' policy +33:45.587 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.593 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.608 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.608 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +33:45.608 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +33:45.608 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 +33:45.608 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1.dc1 +33:45.608 [INFO] TestAgent.server: Updating LAN server: server="Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 (Addr: tcp/127.0.0.1:27377) (DC: dc1)" +33:45.608 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1.dc1 area=wan +33:45.624 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:45.664 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:45.664 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:45.664 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:45.664 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:45.664 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:45.664 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 +33:45.664 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 +33:45.674 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5b406c34-2b52-602f-1c1c-dda5fa5d3bb1 +33:45.682 [INFO] TestAgent.server: federation state anti-entropy synced +33:45.703 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:37542 error="ACL not found" +33:45.703 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:37542 latency=328.978µs +33:45.715 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:37544 latency=7.190473ms +33:45.717 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/login from=127.0.0.1:37548 error="Permission denied" +33:45.717 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:37548 latency=275.276µs +33:45.723 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:37544 latency=5.451188ms +33:45.743 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:37550 latency=8.130841ms +33:45.749 [INFO] TestAgent: Requesting shutdown +33:45.749 [INFO] TestAgent.server: shutting down server +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:45.749 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:45.749 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:45.749 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:45.749 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:45.749 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:45.759 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:45.767Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:45.767Z [INFO] agent.router.manager: shutting down +33:45.767 [INFO] TestAgent: consul server down +33:45.767 [INFO] TestAgent: shutdown complete +33:45.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27372 network=tcp +33:45.767 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27372 network=udp +33:45.767 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27373 network=tcp +33:46.268 [INFO] TestAgent: Waiting for endpoints to shut down +33:46.268 [INFO] TestAgent: Endpoints down +--- PASS: TestLoginCommand (1.06s) --- PASS: TestLoginCommand/method_is_required (0.00s) --- PASS: TestLoginCommand/token-sink-file_is_required (0.00s) --- PASS: TestLoginCommand/bearer-token-file_is_required (0.00s) --- PASS: TestLoginCommand/bearer-token-file_is_empty (0.00s) --- PASS: TestLoginCommand/try_login_with_no_method_configured (0.00s) --- PASS: TestLoginCommand/try_login_with_method_configured_but_no_binding_rules (0.00s) - --- PASS: TestLoginCommand/try_login_with_method_configured_and_binding_rules (0.02s) -2022-11-07T22:46:36.866Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:36.890Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:36.918 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 Address:127.0.0.1:18174}]" -46:36.918 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:18174 [Follower]" leader= -46:36.919 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 127.0.0.1 -46:36.919 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 127.0.0.1 -2022-11-07T22:46:36.919Z [INFO] agent.router: Initializing LAN area manager -46:36.919 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 area=wan -46:36.919 [INFO] TestAgent.server: Adding LAN server: server="Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -46:36.920 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=udp -46:36.920 [INFO] TestAgent: Started DNS server: address=127.0.0.1:18169 network=tcp -46:36.920 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:18170 network=tcp -46:36.920 [INFO] TestAgent: started state syncer -46:36.920 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:18175 network=tcp -46:36.957 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:36.957 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:18174 [Candidate]" term=2 -46:36.968 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:36.968 [DEBUG] TestAgent.server.raft: vote granted: from=a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 term=2 tally=1 -46:36.968 [INFO] TestAgent.server.raft: election won: tally=1 -46:36.968 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:18174 [Leader]" -46:36.968 [INFO] TestAgent.server: cluster leadership acquired -46:36.968 [INFO] TestAgent.server: New leader elected: payload=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:36.973 [INFO] TestAgent.server: initializing acls -46:36.982 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:36.982 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:36.982 [INFO] TestAgent.server: initializing acls -46:36.982 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:37.011 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:37.026 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:37.029 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:37.029 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:37.029 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:37.029 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode -46:37.029 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.029 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 -46:37.029 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 area=wan -46:37.029 [INFO] TestAgent.server: Updating LAN server: server="Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -46:37.033 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:37.033 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.033 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 -46:37.033 [INFO] TestAgent.server: Updating LAN server: server="Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 (Addr: tcp/127.0.0.1:18174) (DC: dc1)" -46:37.033 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2.dc1 area=wan -46:37.050 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:37.071 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:37.071 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:37.071 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:37.071 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:37.071 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:37.071 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.071 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.075 [INFO] TestAgent.server: federation state anti-entropy synced -46:37.087 [INFO] TestAgent: Synced node info -46:37.087 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.087 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a141ab47-fc2e-cbbd-7b90-3c1c878c0bc2 -46:37.190 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48816 latency=4.085412ms -46:37.203 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:48816 latency=12.627479ms -46:37.213 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:48818 latency=6.023233ms -46:37.234 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48816 latency=15.028696ms -46:37.247 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:37.247 [DEBUG] TestAgent: Node info in sync -46:37.247 [DEBUG] TestAgent: Node info in sync -46:37.248 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:48816 latency=13.293023ms -46:37.269 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:48824 latency=16.657922ms -46:37.304 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:48816 latency=29.943472ms -46:37.334 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:48816 latency=28.799743ms -46:37.355 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:48844 latency=18.312422ms -46:37.360 [INFO] TestAgent: Requesting shutdown -46:37.360 [INFO] TestAgent.server: shutting down server -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:37.360 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:37.360 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:37.360 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:37.360 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:37.364 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:37.367Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:37.367Z [INFO] agent.router.manager: shutting down -46:37.367 [INFO] TestAgent: consul server down -46:37.367 [INFO] TestAgent: shutdown complete -46:37.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=tcp -46:37.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:18169 network=udp -46:37.367 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:18170 network=tcp -46:37.867 [INFO] TestAgent: Waiting for endpoints to shut down -46:37.867 [INFO] TestAgent: Endpoints down ---- PASS: TestLoginCommand_jwt (1.04s) - --- PASS: TestLoginCommand_jwt/success_-_jwt_static_keys (0.03s) - --- PASS: TestLoginCommand_jwt/success_-_jwt_jwks (0.06s) - --- PASS: TestLoginCommand_jwt/success_-_jwt_oidc_discovery (0.09s) + --- PASS: TestLoginCommand/try_login_with_method_configured_and_binding_rules (0.03s) +2021-10-05T16:33:45.297Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:45.381Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:45.439 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7dff90bc-1d65-f82c-5684-093745dff02b Address:127.0.0.1:27370}]" +33:45.440 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 127.0.0.1 +33:45.440 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27370 [Follower]" leader= +33:45.457 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7dff90bc-1d65-f82c-5684-093745dff02b 127.0.0.1 +2021-10-05T16:33:45.457Z [INFO] agent.router: Initializing LAN area manager +33:45.458 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=udp +33:45.458 [INFO] TestAgent.server: Adding LAN server: server="Node-7dff90bc-1d65-f82c-5684-093745dff02b (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +33:45.458 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 area=wan +33:45.458 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=tcp +33:45.460 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27366 network=tcp +33:45.489 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:45.489 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27370 [Candidate]" term=2 +33:45.505 [INFO] TestAgent: started state syncer +33:45.505 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27371 network=tcp +33:45.506 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:45.506 [DEBUG] TestAgent.server.raft: vote granted: from=7dff90bc-1d65-f82c-5684-093745dff02b term=2 tally=1 +33:45.506 [INFO] TestAgent.server.raft: election won: tally=1 +33:45.506 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27370 [Leader]" +33:45.506 [INFO] TestAgent.server: cluster leadership acquired +33:45.506 [INFO] TestAgent.server: New leader elected: payload=Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.512 [INFO] TestAgent.server: initializing acls +33:45.520 [INFO] TestAgent.server: Created ACL 'global-management' policy +33:45.520 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.520 [INFO] TestAgent.server: initializing acls +33:45.520 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.526 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.538 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.538 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +33:45.538 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +33:45.538 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +33:45.538 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.538 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 +33:45.538 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 area=wan +33:45.538 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.538 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.538 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 +33:45.539 [INFO] TestAgent.server: Updating LAN server: server="Node-7dff90bc-1d65-f82c-5684-093745dff02b (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +33:45.539 [INFO] TestAgent.server: Updating LAN server: server="Node-7dff90bc-1d65-f82c-5684-093745dff02b (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +33:45.539 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-7dff90bc-1d65-f82c-5684-093745dff02b.dc1 area=wan +33:45.549 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:45.574 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:45.574 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:45.574 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:45.574 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:45.574 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:45.574 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.574 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.580 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.580 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7dff90bc-1d65-f82c-5684-093745dff02b +33:45.587 [INFO] TestAgent.server: federation state anti-entropy synced +33:45.639 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:45.654 [INFO] TestAgent: Synced node info +33:45.654 [DEBUG] TestAgent: Node info in sync +33:45.840 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:36456 latency=17.7665ms +33:45.856 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:36456 latency=15.562714ms +33:45.925 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:36460 latency=57.29204ms +33:45.933 [INFO] TestAgent: Requesting shutdown +33:45.933 [INFO] TestAgent.server: shutting down server +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:45.933 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:45.933 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:45.933 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:45.933 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:45.939 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:45.947Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:45.947Z [INFO] agent.router.manager: shutting down +33:45.947 [INFO] TestAgent: consul server down +33:45.947 [INFO] TestAgent: shutdown complete +33:45.947 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=tcp +33:45.947 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=udp +33:45.947 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27366 network=tcp +33:46.449 [INFO] TestAgent: Waiting for endpoints to shut down +33:46.449 [INFO] TestAgent: Endpoints down +--- PASS: TestLoginCommand_k8s (1.25s) + --- PASS: TestLoginCommand_k8s/try_login_with_method_configured_and_binding_rules (0.07s) +2021-10-05T16:33:45.299Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:45.347Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:45.422 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 Address:127.0.0.1:27363}]" +33:45.423 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 127.0.0.1 +33:45.424 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 127.0.0.1 +2021-10-05T16:33:45.424Z [INFO] agent.router: Initializing LAN area manager +33:45.424 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +33:45.439 [INFO] TestAgent.server: Adding LAN server: server="Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +33:45.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +33:45.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +33:45.440 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 area=wan +33:45.457 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +33:45.457 [INFO] TestAgent: started state syncer +33:45.458 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +33:45.489 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +33:45.493 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:45.493 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +33:45.520 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:45.520 [DEBUG] TestAgent.server.raft: vote granted: from=1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 term=2 tally=1 +33:45.520 [INFO] TestAgent.server.raft: election won: tally=1 +33:45.520 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +33:45.520 [INFO] TestAgent.server: cluster leadership acquired +33:45.520 [INFO] TestAgent.server: New leader elected: payload=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.556 [INFO] TestAgent.server: initializing acls +33:45.568 [INFO] TestAgent.server: Created ACL 'global-management' policy +33:45.568 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.590 [INFO] TestAgent.server: initializing acls +33:45.590 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.605 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.605 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.610 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.610 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +33:45.610 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +33:45.610 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode +33:45.611 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.612 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 +33:45.612 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 area=wan +33:45.612 [INFO] TestAgent.server: Updating LAN server: server="Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +33:45.616 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.616 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.616 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 +33:45.617 [INFO] TestAgent.server: Updating LAN server: server="Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +33:45.617 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7.dc1 area=wan +33:45.640 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:45.682 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:45.682 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:45.682 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:45.682 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:45.682 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:45.682 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.682 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.688 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.688 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1e53b862-ecba-a9c9-0bbc-529c22b6e9c7 +33:45.696 [INFO] TestAgent.server: federation state anti-entropy synced +33:45.739 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:60160 latency=26.40912ms +33:45.759 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:60160 latency=11.795717ms +33:45.764 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:45.784 [INFO] TestAgent: Synced node info +33:45.785 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:60166 latency=20.176725ms +33:45.829 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:60160 latency=29.038895ms +33:45.859 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:60160 latency=28.964037ms +33:45.901 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:60176 latency=32.082026ms +33:45.942 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:60160 latency=33.503724ms +33:45.979 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:60160 latency=36.345465ms +33:45.993 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:60198 latency=9.898714ms +33:46.004 [INFO] TestAgent: Requesting shutdown +33:46.004 [INFO] TestAgent.server: shutting down server +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:46.004 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:46.004 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:46.004 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:46.004 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:46.016 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:46.028Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:46.028Z [INFO] agent.router.manager: shutting down +33:46.028 [INFO] TestAgent: consul server down +33:46.028 [INFO] TestAgent: shutdown complete +33:46.028 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +33:46.028 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +33:46.028 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +33:46.532 [INFO] TestAgent: Waiting for endpoints to shut down +33:46.532 [INFO] TestAgent: Endpoints down +--- PASS: TestLoginCommand_jwt (1.33s) + --- PASS: TestLoginCommand_jwt/success_-_jwt_static_keys (0.09s) + --- PASS: TestLoginCommand_jwt/success_-_jwt_jwks (0.11s) + --- PASS: TestLoginCommand_jwt/success_-_jwt_oidc_discovery (0.10s) PASS -ok github.com/hashicorp/consul/command/login 1.070s +ok github.com/hashicorp/consul/command/login 1.432s === RUN TestLogout_noTabs === PAUSE TestLogout_noTabs === RUN TestLogoutCommand @@ -88834,198 +88989,198 @@ === RUN TestLogoutCommand_k8s === PAUSE TestLogoutCommand_k8s === CONT TestLogout_noTabs +--- PASS: TestLogout_noTabs (0.00s) === CONT TestLogoutCommand_k8s -=== CONT TestLogoutCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestLogout_noTabs (0.00s) +=== CONT TestLogoutCommand [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -=== RUN TestLogoutCommand_k8s/no_token_specified -=== RUN TestLogoutCommand_k8s/logout_of_deleted_token -=== RUN TestLogoutCommand_k8s/logout_of_ordinary_token -=== RUN TestLogoutCommand_k8s/logout_of_login_token === RUN TestLogoutCommand/no_token_specified === RUN TestLogoutCommand/logout_of_deleted_token === RUN TestLogoutCommand/logout_of_ordinary_token +=== RUN TestLogoutCommand_k8s/no_token_specified === RUN TestLogoutCommand/logout_of_login_token -2022-11-07T22:46:40.702Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:40.717Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:40.741 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:787866ec-8c1f-9706-7bef-683269c7ff1a Address:127.0.0.1:21244}]" -46:40.741 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21244 [Follower]" leader= -46:40.743 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-787866ec-8c1f-9706-7bef-683269c7ff1a.dc1 127.0.0.1 -46:40.744 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-787866ec-8c1f-9706-7bef-683269c7ff1a 127.0.0.1 -2022-11-07T22:46:40.744Z [INFO] agent.router: Initializing LAN area manager -46:40.744 [INFO] TestAgent.server: Adding LAN server: server="Node-787866ec-8c1f-9706-7bef-683269c7ff1a (Addr: tcp/127.0.0.1:21244) (DC: dc1)" -46:40.744 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=tcp -46:40.744 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-787866ec-8c1f-9706-7bef-683269c7ff1a.dc1 area=wan -46:40.744 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21239 network=udp -46:40.744 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21240 network=tcp -46:40.744 [INFO] TestAgent: started state syncer -46:40.745 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21245 network=tcp -46:40.794 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:40.812 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:40.812 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21244 [Candidate]" term=2 -46:40.821 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:40.821 [DEBUG] TestAgent.server.raft: vote granted: from=787866ec-8c1f-9706-7bef-683269c7ff1a term=2 tally=1 -46:40.821 [INFO] TestAgent.server.raft: election won: tally=1 -46:40.821 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21244 [Leader]" -46:40.821 [INFO] TestAgent.server: cluster leadership acquired -46:40.821 [INFO] TestAgent.server: New leader elected: payload=Node-787866ec-8c1f-9706-7bef-683269c7ff1a -46:40.828 [INFO] TestAgent.server: initializing acls -46:40.831 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:40.831 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:40.837 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:40.845 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:40.845 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:40.845 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:40.846 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-787866ec-8c1f-9706-7bef-683269c7ff1a -46:40.846 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-787866ec-8c1f-9706-7bef-683269c7ff1a.dc1 -46:40.846 [INFO] TestAgent.server: Updating LAN server: server="Node-787866ec-8c1f-9706-7bef-683269c7ff1a (Addr: tcp/127.0.0.1:21244) (DC: dc1)" -46:40.846 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-787866ec-8c1f-9706-7bef-683269c7ff1a.dc1 area=wan -46:40.855 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:40.871 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:40.871 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:40.871 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:40.871 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:40.872 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:40.872 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-787866ec-8c1f-9706-7bef-683269c7ff1a -46:40.872 [INFO] TestAgent.server: member joined, marking health alive: member=Node-787866ec-8c1f-9706-7bef-683269c7ff1a -46:40.877 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-787866ec-8c1f-9706-7bef-683269c7ff1a -46:41.025 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:41.030 [INFO] TestAgent: Synced node info -46:41.031 [DEBUG] TestAgent: Node info in sync -46:41.065 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:40202 error="ACL not found" -46:41.066 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:40202 latency=1.089594ms -46:41.069 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:40204 error="ACL not found" -46:41.069 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:40204 latency=168.646µs -46:41.076 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:40206 latency=6.542648ms -46:41.077 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:40208 error="Permission denied" -46:41.077 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:40208 latency=134.516µs -46:41.083 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:40206 latency=4.417105ms -46:41.087 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:40206 latency=3.6111ms -46:41.100 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:40206 latency=12.275911ms -46:41.107 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:40212 latency=3.311275ms -46:41.107 [INFO] TestAgent: Requesting shutdown -46:41.107 [INFO] TestAgent.server: shutting down server -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:41.107 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:41.107 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:41.107 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:41.107 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:41.111 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:41.114Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:41.114Z [INFO] agent.router.manager: shutting down -46:41.114 [INFO] TestAgent: consul server down -46:41.114 [INFO] TestAgent: shutdown complete -46:41.114 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=tcp -46:41.114 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21239 network=udp -46:41.114 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21240 network=tcp -46:41.614 [INFO] TestAgent: Waiting for endpoints to shut down -46:41.615 [INFO] TestAgent: Endpoints down ---- PASS: TestLogoutCommand_k8s (0.95s) - --- PASS: TestLogoutCommand_k8s/no_token_specified (0.00s) - --- PASS: TestLogoutCommand_k8s/logout_of_deleted_token (0.00s) - --- PASS: TestLogoutCommand_k8s/logout_of_ordinary_token (0.00s) - --- PASS: TestLogoutCommand_k8s/logout_of_login_token (0.00s) -2022-11-07T22:46:40.701Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:40.717Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:40.741 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4cd1d7a6-9994-0a68-d556-9d67473a8527 Address:127.0.0.1:21237}]" -46:40.741 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:21237 [Follower]" leader= -46:40.742 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-4cd1d7a6-9994-0a68-d556-9d67473a8527.dc1 127.0.0.1 -46:40.743 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 127.0.0.1 -2022-11-07T22:46:40.743Z [INFO] agent.router: Initializing LAN area manager -46:40.744 [INFO] TestAgent.server: Adding LAN server: server="Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -46:40.744 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527.dc1 area=wan -46:40.744 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=udp -46:40.744 [INFO] TestAgent: Started DNS server: address=127.0.0.1:21232 network=tcp -46:40.744 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:21233 network=tcp -46:40.744 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:21238 network=tcp -46:40.744 [INFO] TestAgent: started state syncer -46:40.794 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= -46:40.800 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:40.800 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:21237 [Candidate]" term=2 -46:40.808 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:40.808 [DEBUG] TestAgent.server.raft: vote granted: from=4cd1d7a6-9994-0a68-d556-9d67473a8527 term=2 tally=1 -46:40.808 [INFO] TestAgent.server.raft: election won: tally=1 -46:40.808 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:21237 [Leader]" -46:40.808 [INFO] TestAgent.server: cluster leadership acquired -46:40.808 [INFO] TestAgent.server: New leader elected: payload=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 -46:40.814 [INFO] TestAgent.server: initializing acls -46:40.818 [INFO] TestAgent.server: Created ACL 'global-management' policy -46:40.818 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated -46:40.824 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration -46:40.832 [INFO] TestAgent.server: Created ACL anonymous token from configuration -46:40.832 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" -46:40.832 [INFO] TestAgent.leader: started routine: routine="acl token reaping" -46:40.833 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 -46:40.833 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-4cd1d7a6-9994-0a68-d556-9d67473a8527.dc1 -46:40.833 [INFO] TestAgent.server: Updating LAN server: server="Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 (Addr: tcp/127.0.0.1:21237) (DC: dc1)" -46:40.833 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527.dc1 area=wan -46:40.843 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:40.858 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:40.858 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:40.858 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:40.858 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:40.858 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:40.858 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 -46:40.858 [INFO] TestAgent.server: member joined, marking health alive: member=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 -46:40.868 [INFO] TestAgent.server: federation state anti-entropy synced -46:40.868 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-4cd1d7a6-9994-0a68-d556-9d67473a8527 -46:41.159 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:41.165 [INFO] TestAgent: Synced node info -46:41.183 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:41870 error="ACL not found" -46:41.183 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:41870 latency=140.867µs -46:41.184 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:41872 error="ACL not found" -46:41.184 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:41872 latency=117.183µs -46:41.188 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:41874 latency=3.791399ms -46:41.190 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:41876 error="Permission denied" -46:41.190 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:41876 latency=167.736µs -46:41.193 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:41874 latency=3.081272ms -46:41.197 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:41874 latency=3.467168ms -46:41.204 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:41874 latency=6.279335ms -46:41.207 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:41878 latency=2.821923ms -46:41.208 [INFO] TestAgent: Requesting shutdown -46:41.208 [INFO] TestAgent.server: shutting down server -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:41.208 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:41.208 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:41.208 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" -46:41.208 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:41.211 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:41.214Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:41.214Z [INFO] agent.router.manager: shutting down -46:41.215 [INFO] TestAgent: consul server down -46:41.215 [INFO] TestAgent: shutdown complete -46:41.215 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=tcp -46:41.215 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:21232 network=udp -46:41.215 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:21233 network=tcp -46:41.715 [INFO] TestAgent: Waiting for endpoints to shut down -46:41.715 [INFO] TestAgent: Endpoints down ---- PASS: TestLogoutCommand (1.06s) - --- PASS: TestLogoutCommand/no_token_specified (0.00s) +=== RUN TestLogoutCommand_k8s/logout_of_deleted_token +=== RUN TestLogoutCommand_k8s/logout_of_ordinary_token +=== RUN TestLogoutCommand_k8s/logout_of_login_token +2021-10-05T16:33:45.606Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:45.621Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:45.667 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9791eff9-d60d-2b19-313d-921a922d3603 Address:127.0.0.1:26349}]" +33:45.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= +33:45.668 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9791eff9-d60d-2b19-313d-921a922d3603.dc1 127.0.0.1 +33:45.669 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9791eff9-d60d-2b19-313d-921a922d3603 127.0.0.1 +2021-10-05T16:33:45.669Z [INFO] agent.router: Initializing LAN area manager +33:45.670 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp +33:45.670 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9791eff9-d60d-2b19-313d-921a922d3603.dc1 area=wan +33:45.670 [INFO] TestAgent.server: Adding LAN server: server="Node-9791eff9-d60d-2b19-313d-921a922d3603 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" +33:45.670 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp +33:45.671 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp +33:45.671 [INFO] TestAgent: started state syncer +33:45.671 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp +33:45.710 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:45.710 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 +33:45.720 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +33:45.738 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:45.738 [DEBUG] TestAgent.server.raft: vote granted: from=9791eff9-d60d-2b19-313d-921a922d3603 term=2 tally=1 +33:45.738 [INFO] TestAgent.server.raft: election won: tally=1 +33:45.738 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" +33:45.738 [INFO] TestAgent.server: cluster leadership acquired +33:45.738 [INFO] TestAgent.server: New leader elected: payload=Node-9791eff9-d60d-2b19-313d-921a922d3603 +33:45.746 [INFO] TestAgent.server: initializing acls +33:45.751 [INFO] TestAgent.server: Created ACL 'global-management' policy +33:45.751 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.768 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.788 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.788 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +33:45.788 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +33:45.788 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-9791eff9-d60d-2b19-313d-921a922d3603 +33:45.788 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-9791eff9-d60d-2b19-313d-921a922d3603.dc1 +33:45.791 [INFO] TestAgent.server: Updating LAN server: server="Node-9791eff9-d60d-2b19-313d-921a922d3603 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" +33:45.791 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-9791eff9-d60d-2b19-313d-921a922d3603.dc1 area=wan +33:45.807 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:45.880 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:45.880 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:45.880 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:45.880 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:45.880 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:45.880 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9791eff9-d60d-2b19-313d-921a922d3603 +33:45.880 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9791eff9-d60d-2b19-313d-921a922d3603 +33:45.896 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9791eff9-d60d-2b19-313d-921a922d3603 +33:45.914 [INFO] TestAgent.server: federation state anti-entropy synced +33:45.936 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:43162 error="ACL not found" +33:45.939 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:43162 latency=2.250982ms +33:45.941 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:43168 error="ACL not found" +33:45.942 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:43168 latency=301.276µs +33:45.972 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:43170 latency=24.931303ms +33:45.977 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:43176 error="Permission denied" +33:45.980 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:43176 latency=2.664693ms +33:45.999 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:43170 latency=17.548429ms +33:46.020 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:43170 latency=20.076861ms +33:46.038 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:43170 latency=16.316355ms +33:46.048 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:46.080 [INFO] TestAgent: Synced node info +33:46.089 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:43182 latency=46.579923ms +33:46.092 [INFO] TestAgent: Requesting shutdown +33:46.092 [INFO] TestAgent.server: shutting down server +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:46.092 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:46.092 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:46.092 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:46.092 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:46.102 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:46.111Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:46.111Z [INFO] agent.router.manager: shutting down +33:46.111 [INFO] TestAgent: consul server down +33:46.111 [INFO] TestAgent: shutdown complete +33:46.111 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp +33:46.111 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp +33:46.111 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp +33:46.612 [INFO] TestAgent: Waiting for endpoints to shut down +33:46.612 [INFO] TestAgent: Endpoints down +--- PASS: TestLogoutCommand (1.11s) + --- PASS: TestLogoutCommand/no_token_specified (0.04s) --- PASS: TestLogoutCommand/logout_of_deleted_token (0.00s) --- PASS: TestLogoutCommand/logout_of_ordinary_token (0.00s) - --- PASS: TestLogoutCommand/logout_of_login_token (0.00s) + --- PASS: TestLogoutCommand/logout_of_login_token (0.05s) +2021-10-05T16:33:45.574Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:45.640Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:45.690 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:df457432-5f5b-c484-4f36-13fa1c576200 Address:127.0.0.1:26342}]" +33:45.691 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-df457432-5f5b-c484-4f36-13fa1c576200.dc1 127.0.0.1 +33:45.692 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= +33:45.692 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-df457432-5f5b-c484-4f36-13fa1c576200 127.0.0.1 +2021-10-05T16:33:45.693Z [INFO] agent.router: Initializing LAN area manager +33:45.693 [INFO] TestAgent.server: Adding LAN server: server="Node-df457432-5f5b-c484-4f36-13fa1c576200 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +33:45.693 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-df457432-5f5b-c484-4f36-13fa1c576200.dc1 area=wan +33:45.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp +33:45.693 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp +33:45.694 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp +33:45.694 [INFO] TestAgent: started state syncer +33:45.694 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp +33:45.738 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:45.738 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 +33:45.743 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader= +33:45.782 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:45.782 [DEBUG] TestAgent.server.raft: vote granted: from=df457432-5f5b-c484-4f36-13fa1c576200 term=2 tally=1 +33:45.782 [INFO] TestAgent.server.raft: election won: tally=1 +33:45.782 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" +33:45.784 [INFO] TestAgent.server: cluster leadership acquired +33:45.784 [INFO] TestAgent.server: New leader elected: payload=Node-df457432-5f5b-c484-4f36-13fa1c576200 +33:45.792 [INFO] TestAgent.server: initializing acls +33:45.795 [INFO] TestAgent.server: Created ACL 'global-management' policy +33:45.795 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated +33:45.801 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration +33:45.816 [INFO] TestAgent.server: Created ACL anonymous token from configuration +33:45.816 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade" +33:45.816 [INFO] TestAgent.leader: started routine: routine="acl token reaping" +33:45.816 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-df457432-5f5b-c484-4f36-13fa1c576200 +33:45.816 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-df457432-5f5b-c484-4f36-13fa1c576200.dc1 +33:45.817 [INFO] TestAgent.server: Updating LAN server: server="Node-df457432-5f5b-c484-4f36-13fa1c576200 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +33:45.817 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-df457432-5f5b-c484-4f36-13fa1c576200.dc1 area=wan +33:45.827 [INFO] TestAgent: Synced node info +33:45.827 [DEBUG] TestAgent: Node info in sync +33:45.847 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:45.874 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:45.874 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:45.874 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:45.874 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:45.874 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:45.874 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df457432-5f5b-c484-4f36-13fa1c576200 +33:45.874 [INFO] TestAgent.server: member joined, marking health alive: member=Node-df457432-5f5b-c484-4f36-13fa1c576200 +33:45.885 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-df457432-5f5b-c484-4f36-13fa1c576200 +33:45.900 [INFO] TestAgent.server: federation state anti-entropy synced +33:46.041 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:59102 error="ACL not found" +33:46.041 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:59102 latency=147.845µs +33:46.043 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:59106 error="ACL not found" +33:46.043 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:59106 latency=373.505µs +33:46.073 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/token from=127.0.0.1:59108 latency=21.302603ms +33:46.093 [ERROR] TestAgent.http: Request error: method=POST url=/v1/acl/logout from=127.0.0.1:59110 error="Permission denied" +33:46.095 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:59110 latency=1.331054ms +33:46.128 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/auth-method from=127.0.0.1:59108 latency=16.613478ms +33:46.152 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/acl/binding-rule from=127.0.0.1:59108 latency=19.871814ms +33:46.212 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/login from=127.0.0.1:59108 latency=56.35583ms +33:46.240 [DEBUG] TestAgent.http: Request finished: method=POST url=/v1/acl/logout from=127.0.0.1:59114 latency=16.389008ms +33:46.244 [INFO] TestAgent: Requesting shutdown +33:46.244 [INFO] TestAgent.server: shutting down server +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade" +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping" +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:46.244 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:46.244 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:46.244 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:46.244 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:46.260 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:46.269Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:46.269Z [INFO] agent.router.manager: shutting down +33:46.269 [INFO] TestAgent: consul server down +33:46.269 [INFO] TestAgent: shutdown complete +33:46.269 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp +33:46.269 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp +33:46.269 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp +33:46.772 [INFO] TestAgent: Waiting for endpoints to shut down +33:46.772 [INFO] TestAgent: Endpoints down +--- PASS: TestLogoutCommand_k8s (1.27s) + --- PASS: TestLogoutCommand_k8s/no_token_specified (0.01s) + --- PASS: TestLogoutCommand_k8s/logout_of_deleted_token (0.00s) + --- PASS: TestLogoutCommand_k8s/logout_of_ordinary_token (0.01s) + --- PASS: TestLogoutCommand_k8s/logout_of_login_token (0.03s) PASS -ok github.com/hashicorp/consul/command/logout 1.084s +ok github.com/hashicorp/consul/command/logout 1.381s === RUN TestMaintCommand_noTabs === PAUSE TestMaintCommand_noTabs === RUN TestMaintCommand_ConflictingArgs @@ -89043,373 +89198,361 @@ === RUN TestMaintCommand_ServiceMaintenance_NoService === PAUSE TestMaintCommand_ServiceMaintenance_NoService === CONT TestMaintCommand_noTabs -=== CONT TestMaintCommand_EnableServiceMaintenance -=== CONT TestMaintCommand_EnableNodeMaintenance -=== CONT TestMaintCommand_NoArgs -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestMaintCommand_ConflictingArgs +--- PASS: TestMaintCommand_noTabs (0.00s) === CONT TestMaintCommand_ServiceMaintenance_NoService +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestMaintCommand_DisableNodeMaintenance ---- PASS: TestMaintCommand_noTabs (0.00s) === CONT TestMaintCommand_DisableServiceMaintenance +=== CONT TestMaintCommand_NoArgs +=== CONT TestMaintCommand_EnableServiceMaintenance +=== CONT TestMaintCommand_EnableNodeMaintenance +=== CONT TestMaintCommand_ConflictingArgs +--- PASS: TestMaintCommand_ConflictingArgs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestMaintCommand_ConflictingArgs (0.00s) -2022-11-07T22:46:41.999Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.013Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.054 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba7f0606-5bbe-ceb6-56ca-971cdc10aac8 Address:127.0.0.1:17153}]" -46:42.054 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= -46:42.054 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba7f0606-5bbe-ceb6-56ca-971cdc10aac8.dc1 127.0.0.1 -46:42.055 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba7f0606-5bbe-ceb6-56ca-971cdc10aac8 127.0.0.1 -2022-11-07T22:46:42.055Z [INFO] agent.router: Initializing LAN area manager -46:42.056 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba7f0606-5bbe-ceb6-56ca-971cdc10aac8.dc1 area=wan -46:42.056 [INFO] TestAgent.server: Adding LAN server: server="Node-ba7f0606-5bbe-ceb6-56ca-971cdc10aac8 (Addr: tcp/127.0.0.1:17153) (DC: dc1)" -46:42.098 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=tcp -46:42.098 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=udp -46:42.099 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17149 network=tcp -46:42.099 [INFO] TestAgent: started state syncer -46:42.099 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17154 network=tcp -46:42.103 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.103 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 -46:42.117 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.117 [DEBUG] TestAgent.server.raft: vote granted: from=ba7f0606-5bbe-ceb6-56ca-971cdc10aac8 term=2 tally=1 -46:42.117 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.117 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" -46:42.117 [INFO] TestAgent.server: cluster leadership acquired -46:42.117 [INFO] TestAgent.server: New leader elected: payload=Node-ba7f0606-5bbe-ceb6-56ca-971cdc10aac8 -46:42.125 [INFO] TestAgent: Service entered maintenance mode: service=test -46:42.134 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17153 -46:42.134 [INFO] TestAgent: Synced node info -46:42.161 [INFO] TestAgent: Synced service: service=test -46:42.161 [DEBUG] TestAgent: Check in sync: check=_service_maintenance:test -46:42.161 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/test?enable=true&reason=broken from=127.0.0.1:37284 latency=42.594787ms -46:42.161 [INFO] TestAgent: Requesting shutdown -46:42.161 [INFO] TestAgent.server: shutting down server -46:42.161 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.161 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:42.165 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.168 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:42.176Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.176Z [INFO] agent.router.manager: shutting down -46:42.180 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: leadership lost while committing log" -46:42.180 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" -46:42.180 [INFO] TestAgent: consul server down -46:42.180 [INFO] TestAgent: shutdown complete -46:42.180 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp -46:42.181 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp -46:42.181 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp -46:42.681 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.681 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_EnableServiceMaintenance (0.73s) -2022-11-07T22:46:41.997Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.012Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.050 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:29852b10-63f2-043c-ad1a-b125ea342b4f Address:127.0.0.1:17181}]" -46:42.050 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17181 [Follower]" leader= -46:42.050 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-29852b10-63f2-043c-ad1a-b125ea342b4f.dc1 127.0.0.1 -46:42.051 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-29852b10-63f2-043c-ad1a-b125ea342b4f 127.0.0.1 -2022-11-07T22:46:42.051Z [INFO] agent.router: Initializing LAN area manager -46:42.051 [INFO] TestAgent.server: Adding LAN server: server="Node-29852b10-63f2-043c-ad1a-b125ea342b4f (Addr: tcp/127.0.0.1:17181) (DC: dc1)" -46:42.051 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-29852b10-63f2-043c-ad1a-b125ea342b4f.dc1 area=wan -46:42.051 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17176 network=udp -46:42.051 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17176 network=tcp -46:42.051 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17177 network=tcp -46:42.052 [INFO] TestAgent: started state syncer -46:42.052 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17182 network=tcp -46:42.108 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.108 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17181 [Candidate]" term=2 -46:42.121 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.121 [DEBUG] TestAgent.server.raft: vote granted: from=29852b10-63f2-043c-ad1a-b125ea342b4f term=2 tally=1 -46:42.121 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.121 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17181 [Leader]" -46:42.121 [INFO] TestAgent.server: cluster leadership acquired -46:42.121 [INFO] TestAgent.server: New leader elected: payload=Node-29852b10-63f2-043c-ad1a-b125ea342b4f -46:42.135 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17181 -46:42.153 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.179 [INFO] TestAgent: Synced node info -46:42.190 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:42.190 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:42.190 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:42.190 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:42.190 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:42.190 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-29852b10-63f2-043c-ad1a-b125ea342b4f -46:42.190 [INFO] TestAgent.server: member joined, marking health alive: member=Node-29852b10-63f2-043c-ad1a-b125ea342b4f -46:42.191 [INFO] TestAgent: Synced service: service=test -46:42.191 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/test?enable=false from=127.0.0.1:43060 latency=21.56486ms -46:42.191 [INFO] TestAgent: Requesting shutdown -46:42.191 [INFO] TestAgent.server: shutting down server -46:42.191 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:42.191 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:42.191 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:42.191 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:42.191 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.191 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:42.191 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:42.191 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:42.191 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:42.198 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:42.201 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:42.201 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -2022-11-07T22:46:42.205Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.205Z [INFO] agent.router.manager: shutting down -46:42.205 [INFO] TestAgent: consul server down -46:42.205 [INFO] TestAgent: shutdown complete -46:42.205 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17176 network=tcp -46:42.205 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17176 network=udp -46:42.205 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17177 network=tcp -46:42.705 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.705 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_DisableServiceMaintenance (0.75s) -2022-11-07T22:46:41.996Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.008Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.052 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8ad4ce88-9eca-24c3-d8f9-880d8301c68d Address:127.0.0.1:17160}]" -46:42.052 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17160 [Follower]" leader= -46:42.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d.dc1 127.0.0.1 -46:42.055 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d 127.0.0.1 -2022-11-07T22:46:42.055Z [INFO] agent.router: Initializing LAN area manager -46:42.055 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d.dc1 area=wan -46:42.055 [INFO] TestAgent.server: Adding LAN server: server="Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d (Addr: tcp/127.0.0.1:17160) (DC: dc1)" -46:42.098 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=tcp -46:42.098 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17155 network=udp -46:42.099 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17156 network=tcp -46:42.099 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17161 network=tcp -46:42.099 [INFO] TestAgent: started state syncer -46:42.120 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.120 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17160 [Candidate]" term=2 -46:42.140 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.140 [DEBUG] TestAgent.server.raft: vote granted: from=8ad4ce88-9eca-24c3-d8f9-880d8301c68d term=2 tally=1 -46:42.140 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.140 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17160 [Leader]" -46:42.140 [INFO] TestAgent.server: cluster leadership acquired -46:42.140 [INFO] TestAgent.server: New leader elected: payload=Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d -46:42.150 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17160 -46:42.167 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.201 [INFO] TestAgent: Node entered maintenance mode -46:42.201 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:42.201 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:42.201 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:42.201 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:42.201 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:42.201 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d -46:42.201 [INFO] TestAgent.server: member joined, marking health alive: member=Node-8ad4ce88-9eca-24c3-d8f9-880d8301c68d -46:42.213 [INFO] TestAgent: Synced node info -46:42.215 [INFO] TestAgent.server: federation state anti-entropy synced -46:42.218 [INFO] TestAgent: Synced check: check=_node_maintenance -46:42.218 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance?enable=true&reason=broken from=127.0.0.1:45520 latency=23.122036ms -46:42.218 [INFO] TestAgent: Requesting shutdown -46:42.218 [INFO] TestAgent.server: shutting down server -46:42.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:42.218 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:42.218 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:42.218 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:42.218 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.218 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:42.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:42.218 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:42.218 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:42.218 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:42.218 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:42.221 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:42.224Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.224Z [INFO] agent.router.manager: shutting down -46:42.224 [INFO] TestAgent: consul server down -46:42.224 [INFO] TestAgent: shutdown complete -46:42.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=tcp -46:42.224 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17155 network=udp -46:42.224 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17156 network=tcp -46:42.724 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.725 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_EnableNodeMaintenance (0.77s) -2022-11-07T22:46:41.995Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.007Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.052 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0524ea5e-a5ff-ec9f-1637-458c8393754f Address:127.0.0.1:17188}]" -46:42.052 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17188 [Follower]" leader= -46:42.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0524ea5e-a5ff-ec9f-1637-458c8393754f.dc1 127.0.0.1 -46:42.053 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0524ea5e-a5ff-ec9f-1637-458c8393754f 127.0.0.1 -2022-11-07T22:46:42.053Z [INFO] agent.router: Initializing LAN area manager -46:42.054 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0524ea5e-a5ff-ec9f-1637-458c8393754f.dc1 area=wan -46:42.055 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17183 network=udp -46:42.055 [INFO] TestAgent.server: Adding LAN server: server="Node-0524ea5e-a5ff-ec9f-1637-458c8393754f (Addr: tcp/127.0.0.1:17188) (DC: dc1)" -46:42.055 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17183 network=tcp -46:42.098 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17184 network=tcp -46:42.099 [INFO] TestAgent: started state syncer -46:42.099 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17189 network=tcp -46:42.100 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.100 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17188 [Candidate]" term=2 -46:42.110 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.110 [DEBUG] TestAgent.server.raft: vote granted: from=0524ea5e-a5ff-ec9f-1637-458c8393754f term=2 tally=1 -46:42.110 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.110 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17188 [Leader]" -46:42.111 [INFO] TestAgent.server: cluster leadership acquired -46:42.111 [INFO] TestAgent.server: New leader elected: payload=Node-0524ea5e-a5ff-ec9f-1637-458c8393754f -46:42.119 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17188 -46:42.141 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.173 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:42.173 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:42.173 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:42.173 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:42.173 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:42.173 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0524ea5e-a5ff-ec9f-1637-458c8393754f -46:42.173 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0524ea5e-a5ff-ec9f-1637-458c8393754f -46:42.182 [INFO] TestAgent.server: federation state anti-entropy synced -46:42.285 [INFO] TestAgent: Synced node info -46:42.285 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance?enable=false from=127.0.0.1:48600 latency=3.79082ms -46:42.285 [INFO] TestAgent: Requesting shutdown -46:42.285 [INFO] TestAgent.server: shutting down server -46:42.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:42.285 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:42.285 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:42.285 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:42.285 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:42.285 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:42.285 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:42.285 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:42.285 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:42.285 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:42.289 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:42.292Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.292Z [INFO] agent.router.manager: shutting down -46:42.293 [INFO] TestAgent: consul server down -46:42.293 [INFO] TestAgent: shutdown complete -46:42.293 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17183 network=tcp -46:42.293 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17183 network=udp -46:42.293 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17184 network=tcp -46:42.793 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.793 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_DisableNodeMaintenance (0.84s) -2022-11-07T22:46:41.995Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.009Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.052 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c5a7b7e0-5563-1a36-0108-b68e3491a005 Address:127.0.0.1:17174}]" -46:42.052 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17174 [Follower]" leader= -46:42.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c5a7b7e0-5563-1a36-0108-b68e3491a005.dc1 127.0.0.1 -46:42.054 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c5a7b7e0-5563-1a36-0108-b68e3491a005 127.0.0.1 -2022-11-07T22:46:42.054Z [INFO] agent.router: Initializing LAN area manager -46:42.054 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=udp -46:42.054 [INFO] TestAgent.server: Adding LAN server: server="Node-c5a7b7e0-5563-1a36-0108-b68e3491a005 (Addr: tcp/127.0.0.1:17174) (DC: dc1)" -46:42.055 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c5a7b7e0-5563-1a36-0108-b68e3491a005.dc1 area=wan -46:42.098 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17169 network=tcp -46:42.099 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17170 network=tcp -46:42.099 [INFO] TestAgent: started state syncer -46:42.099 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17175 network=tcp -46:42.113 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.113 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17174 [Candidate]" term=2 -46:42.129 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.129 [DEBUG] TestAgent.server.raft: vote granted: from=c5a7b7e0-5563-1a36-0108-b68e3491a005 term=2 tally=1 -46:42.129 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.129 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17174 [Leader]" -46:42.129 [INFO] TestAgent.server: cluster leadership acquired -46:42.129 [INFO] TestAgent.server: New leader elected: payload=Node-c5a7b7e0-5563-1a36-0108-b68e3491a005 -46:42.141 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17174 -46:42.159 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.188 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:42.188 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:42.188 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:42.188 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:42.188 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:42.188 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c5a7b7e0-5563-1a36-0108-b68e3491a005 -46:42.188 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c5a7b7e0-5563-1a36-0108-b68e3491a005 -46:42.204 [INFO] TestAgent.server: federation state anti-entropy synced -46:42.353 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/redis?enable=true&reason=broken from=127.0.0.1:42520 latency=221.781µs -46:42.354 [INFO] TestAgent: Requesting shutdown -46:42.354 [INFO] TestAgent.server: shutting down server -46:42.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:42.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:42.354 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:42.354 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:42.354 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.354 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:42.354 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:42.354 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:42.354 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:42.354 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:42.354 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:42.357 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:42.360Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.360Z [INFO] agent.router.manager: shutting down -46:42.360 [INFO] TestAgent: consul server down -46:42.360 [INFO] TestAgent: shutdown complete -46:42.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=tcp -46:42.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17169 network=udp -46:42.360 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17170 network=tcp -46:42.861 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.861 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_ServiceMaintenance_NoService (0.91s) -2022-11-07T22:46:41.996Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:42.011Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:42.052 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:940efd99-6b72-b676-c634-54c54d0900f6 Address:127.0.0.1:17167}]" -46:42.052 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17167 [Follower]" leader= -46:42.053 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-940efd99-6b72-b676-c634-54c54d0900f6.dc1 127.0.0.1 -46:42.054 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-940efd99-6b72-b676-c634-54c54d0900f6 127.0.0.1 -2022-11-07T22:46:42.054Z [INFO] agent.router: Initializing LAN area manager -46:42.054 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=udp -46:42.054 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-940efd99-6b72-b676-c634-54c54d0900f6.dc1 area=wan -46:42.055 [INFO] TestAgent.server: Adding LAN server: server="Node-940efd99-6b72-b676-c634-54c54d0900f6 (Addr: tcp/127.0.0.1:17167) (DC: dc1)" -46:42.055 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17162 network=tcp -46:42.098 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17163 network=tcp -46:42.098 [INFO] TestAgent: started state syncer -46:42.098 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17168 network=tcp -46:42.114 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:42.114 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17167 [Candidate]" term=2 -46:42.128 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:42.128 [DEBUG] TestAgent.server.raft: vote granted: from=940efd99-6b72-b676-c634-54c54d0900f6 term=2 tally=1 -46:42.128 [INFO] TestAgent.server.raft: election won: tally=1 -46:42.128 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17167 [Leader]" -46:42.128 [INFO] TestAgent.server: cluster leadership acquired -46:42.128 [INFO] TestAgent.server: New leader elected: payload=Node-940efd99-6b72-b676-c634-54c54d0900f6 -46:42.141 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17167 -46:42.158 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:42.188 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:42.188 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:42.188 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:42.188 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:42.188 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:42.188 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-940efd99-6b72-b676-c634-54c54d0900f6 -46:42.188 [INFO] TestAgent.server: member joined, marking health alive: member=Node-940efd99-6b72-b676-c634-54c54d0900f6 -46:42.206 [INFO] TestAgent.server: federation state anti-entropy synced -46:42.337 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:42.340 [INFO] TestAgent: Synced node info -46:42.340 [DEBUG] TestAgent: Node info in sync -46:42.354 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:42.354 [DEBUG] TestAgent: Node info in sync -46:42.457 [DEBUG] TestAgent: Node info in sync -46:42.461 [INFO] TestAgent: Service entered maintenance mode: service=test -46:42.463 [INFO] TestAgent: Synced service: service=test -46:42.463 [DEBUG] TestAgent: Check in sync: check=_service_maintenance:test -46:42.463 [DEBUG] TestAgent: Node info in sync -46:42.463 [DEBUG] TestAgent: Service in sync: service=test -46:42.463 [DEBUG] TestAgent: Check in sync: check=_service_maintenance:test -46:42.463 [DEBUG] TestAgent: Node info in sync -46:42.463 [DEBUG] TestAgent: Service in sync: service=test -46:42.463 [DEBUG] TestAgent: Check in sync: check=_service_maintenance:test -46:42.466 [INFO] TestAgent: Synced check: check=_node_maintenance -46:42.467 [INFO] TestAgent: Node entered maintenance mode -46:42.468 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:51120 latency=1.015324ms -46:42.471 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:51120 latency=673.025µs -46:42.471 [INFO] TestAgent: Requesting shutdown -46:42.471 [INFO] TestAgent.server: shutting down server -46:42.471 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:42.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:42.471 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:42.471 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:42.471 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:42.471 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:42.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:42.471 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:42.471 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:42.471 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:42.474 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:42.477Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:42.477Z [INFO] agent.router.manager: shutting down -46:42.477 [INFO] TestAgent: consul server down -46:42.477 [INFO] TestAgent: shutdown complete -46:42.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=tcp -46:42.477 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17162 network=udp -46:42.477 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17163 network=tcp -46:42.978 [INFO] TestAgent: Waiting for endpoints to shut down -46:42.978 [INFO] TestAgent: Endpoints down ---- PASS: TestMaintCommand_NoArgs (1.02s) +2021-10-05T16:33:56.099Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.123Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.238 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7aac7243-27ba-ed5d-781c-ef9322cd3251 Address:127.0.0.1:26377}]" +33:56.238 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26377 [Follower]" leader= +33:56.244 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7aac7243-27ba-ed5d-781c-ef9322cd3251.dc1 127.0.0.1 +33:56.250 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7aac7243-27ba-ed5d-781c-ef9322cd3251 127.0.0.1 +2021-10-05T16:33:56.250Z [INFO] agent.router: Initializing LAN area manager +33:56.301 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26372 network=udp +33:56.301 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26372 network=tcp +33:56.301 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26373 network=tcp +33:56.302 [INFO] TestAgent: started state syncer +33:56.302 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26378 network=tcp +33:56.302 [INFO] TestAgent.server: Adding LAN server: server="Node-7aac7243-27ba-ed5d-781c-ef9322cd3251 (Addr: tcp/127.0.0.1:26377) (DC: dc1)" +33:56.302 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7aac7243-27ba-ed5d-781c-ef9322cd3251.dc1 area=wan +33:56.307 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.307 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26377 [Candidate]" term=2 +33:56.344 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.344 [DEBUG] TestAgent.server.raft: vote granted: from=7aac7243-27ba-ed5d-781c-ef9322cd3251 term=2 tally=1 +33:56.344 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.344 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26377 [Leader]" +33:56.348 [INFO] TestAgent.server: cluster leadership acquired +33:56.348 [INFO] TestAgent.server: New leader elected: payload=Node-7aac7243-27ba-ed5d-781c-ef9322cd3251 +33:56.396 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26377 +33:56.428 [INFO] TestAgent: Synced node info +33:56.456 [INFO] TestAgent: Node entered maintenance mode +33:56.456 [DEBUG] TestAgent: Node info in sync +33:56.464 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.468 [INFO] TestAgent: Synced check: check=_node_maintenance +33:56.468 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance?enable=true&reason=broken from=127.0.0.1:50038 latency=26.926654ms +33:56.468 [INFO] TestAgent: Requesting shutdown +33:56.468 [INFO] TestAgent.server: shutting down server +33:56.468 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.484 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.506Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.506Z [INFO] agent.router.manager: shutting down +33:56.516 [INFO] TestAgent: consul server down +33:56.516 [INFO] TestAgent: shutdown complete +33:56.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26372 network=tcp +33:56.516 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26372 network=udp +33:56.516 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26373 network=tcp +33:56.516 [ERROR] TestAgent.server.connect: Raft apply failed: error="leadership lost while committing log" +33:56.516 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +33:56.516 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:57.017 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.017 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_EnableNodeMaintenance (1.14s) +2021-10-05T16:33:56.157Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.218Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.338 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d880131c-e480-a291-b4a4-00b178b047ac Address:127.0.0.1:26363}]" +33:56.339 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d880131c-e480-a291-b4a4-00b178b047ac.dc1 127.0.0.1 +33:56.340 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d880131c-e480-a291-b4a4-00b178b047ac 127.0.0.1 +2021-10-05T16:33:56.340Z [INFO] agent.router: Initializing LAN area manager +33:56.340 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26363 [Follower]" leader= +33:56.340 [INFO] TestAgent.server: Adding LAN server: server="Node-d880131c-e480-a291-b4a4-00b178b047ac (Addr: tcp/127.0.0.1:26363) (DC: dc1)" +33:56.340 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=udp +33:56.340 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d880131c-e480-a291-b4a4-00b178b047ac.dc1 area=wan +33:56.341 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26358 network=tcp +33:56.342 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26359 network=tcp +33:56.342 [INFO] TestAgent: started state syncer +33:56.342 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26364 network=tcp +33:56.387 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.387 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26363 [Candidate]" term=2 +33:56.405 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.405 [DEBUG] TestAgent.server.raft: vote granted: from=d880131c-e480-a291-b4a4-00b178b047ac term=2 tally=1 +33:56.405 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.405 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26363 [Leader]" +33:56.408 [INFO] TestAgent.server: cluster leadership acquired +33:56.408 [INFO] TestAgent.server: New leader elected: payload=Node-d880131c-e480-a291-b4a4-00b178b047ac +33:56.416 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26363 +33:56.445 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.468 [INFO] TestAgent: Service entered maintenance mode: service=test +33:56.484 [INFO] TestAgent: Node entered maintenance mode +33:56.490 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:56.490 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:56.490 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:56.490 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:56.490 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:56.490 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d880131c-e480-a291-b4a4-00b178b047ac +33:56.490 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d880131c-e480-a291-b4a4-00b178b047ac +33:56.499 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:42904 latency=8.474588ms +33:56.511 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:42904 latency=452.857µs +33:56.512 [INFO] TestAgent: Requesting shutdown +33:56.512 [INFO] TestAgent.server: shutting down server +33:56.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:56.512 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:56.512 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:56.512 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:56.512 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.512 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:56.512 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:56.512 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:56.512 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:56.515 [INFO] TestAgent.server: federation state anti-entropy synced +33:56.515 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:56.518 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.525Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.525Z [INFO] agent.router.manager: shutting down +33:56.525 [INFO] TestAgent: consul server down +33:56.525 [INFO] TestAgent: shutdown complete +33:56.525 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=tcp +33:56.525 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26358 network=udp +33:56.525 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26359 network=tcp +33:57.028 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.028 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_NoArgs (1.15s) +2021-10-05T16:33:56.079Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.209Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.322 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 Address:127.0.0.1:26349}]" +33:56.322 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3.dc1 127.0.0.1 +33:56.323 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 127.0.0.1 +2021-10-05T16:33:56.323Z [INFO] agent.router: Initializing LAN area manager +33:56.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=udp +33:56.323 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26349 [Follower]" leader= +33:56.323 [INFO] TestAgent.server: Adding LAN server: server="Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 (Addr: tcp/127.0.0.1:26349) (DC: dc1)" +33:56.323 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3.dc1 area=wan +33:56.323 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26344 network=tcp +33:56.337 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26345 network=tcp +33:56.337 [INFO] TestAgent: started state syncer +33:56.337 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26350 network=tcp +33:56.380 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.380 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26349 [Candidate]" term=2 +33:56.398 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.398 [DEBUG] TestAgent.server.raft: vote granted: from=c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 term=2 tally=1 +33:56.398 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.398 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26349 [Leader]" +33:56.400 [INFO] TestAgent.server: cluster leadership acquired +33:56.400 [INFO] TestAgent.server: New leader elected: payload=Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 +33:56.406 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26349 +33:56.428 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.475 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:56.475 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:56.475 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:56.475 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:56.475 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:56.475 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 +33:56.475 [INFO] TestAgent.server: member joined, marking health alive: member=Node-c3e9a17f-f1aa-e059-6a1e-ac595c303ab3 +33:56.516 [INFO] TestAgent.server: federation state anti-entropy synced +33:56.534 [INFO] TestAgent: Synced node info +33:56.534 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/maintenance?enable=false from=127.0.0.1:43242 latency=3.636772ms +33:56.535 [INFO] TestAgent: Requesting shutdown +33:56.535 [INFO] TestAgent.server: shutting down server +33:56.535 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:56.535 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:56.535 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:56.535 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:56.535 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.535 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:56.535 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:56.535 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:56.535 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:56.535 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:56.535 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:56.541 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.557Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.557Z [INFO] agent.router.manager: shutting down +33:56.557 [INFO] TestAgent: consul server down +33:56.557 [INFO] TestAgent: shutdown complete +33:56.557 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=tcp +33:56.557 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26344 network=udp +33:56.557 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26345 network=tcp +33:57.057 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.057 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_DisableNodeMaintenance (1.18s) +2021-10-05T16:33:56.073Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.124Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.237 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 Address:127.0.0.1:26370}]" +33:56.238 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26370 [Follower]" leader= +33:56.253 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5.dc1 127.0.0.1 +33:56.253 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 127.0.0.1 +2021-10-05T16:33:56.253Z [INFO] agent.router: Initializing LAN area manager +33:56.262 [INFO] TestAgent.server: Adding LAN server: server="Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 (Addr: tcp/127.0.0.1:26370) (DC: dc1)" +33:56.262 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5.dc1 area=wan +33:56.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26365 network=udp +33:56.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26365 network=tcp +33:56.305 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26366 network=tcp +33:56.305 [INFO] TestAgent: started state syncer +33:56.305 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26371 network=tcp +33:56.307 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.307 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26370 [Candidate]" term=2 +33:56.343 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.343 [DEBUG] TestAgent.server.raft: vote granted: from=88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 term=2 tally=1 +33:56.343 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.343 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26370 [Leader]" +33:56.343 [INFO] TestAgent.server: cluster leadership acquired +33:56.343 [INFO] TestAgent.server: New leader elected: payload=Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 +33:56.350 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26370 +33:56.372 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.424 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:56.424 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:56.424 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:56.424 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:56.424 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:56.424 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 +33:56.424 [INFO] TestAgent.server: member joined, marking health alive: member=Node-88d88f4b-bd75-29c5-d126-6fa72f6ed3b5 +33:56.460 [INFO] TestAgent.server: federation state anti-entropy synced +33:56.557 [INFO] TestAgent: Service entered maintenance mode: service=test +33:56.561 [INFO] TestAgent: Synced node info +33:56.568 [INFO] TestAgent: Synced service: service=test +33:56.568 [DEBUG] TestAgent: Check in sync: check=_service_maintenance:test +33:56.568 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/test?enable=true&reason=broken from=127.0.0.1:45924 latency=27.90238ms +33:56.568 [INFO] TestAgent: Requesting shutdown +33:56.568 [INFO] TestAgent.server: shutting down server +33:56.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:56.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:56.568 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:56.568 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:56.568 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.568 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:56.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:56.568 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:56.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:56.568 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:56.568 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:56.576 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.584Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.584Z [INFO] agent.router.manager: shutting down +33:56.584 [INFO] TestAgent: consul server down +33:56.584 [INFO] TestAgent: shutdown complete +33:56.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26365 network=tcp +33:56.584 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26365 network=udp +33:56.584 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26366 network=tcp +33:57.088 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.088 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_EnableServiceMaintenance (1.21s) +2021-10-05T16:33:56.097Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.185Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.253 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:acb1be2b-2b35-7f88-f05a-c2750281225e Address:127.0.0.1:26356}]" +33:56.254 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26356 [Follower]" leader= +33:56.293 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.293 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26356 [Candidate]" term=2 +33:56.301 [WARN] TestAgent.server.raft: unable to get address for sever, using fallback address: id=acb1be2b-2b35-7f88-f05a-c2750281225e fallback=127.0.0.1:26356 error="Could not find address for server id acb1be2b-2b35-7f88-f05a-c2750281225e" +33:56.303 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-acb1be2b-2b35-7f88-f05a-c2750281225e.dc1 127.0.0.1 +33:56.303 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-acb1be2b-2b35-7f88-f05a-c2750281225e 127.0.0.1 +2021-10-05T16:33:56.303Z [INFO] agent.router: Initializing LAN area manager +33:56.303 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=udp +33:56.304 [INFO] TestAgent.server: Adding LAN server: server="Node-acb1be2b-2b35-7f88-f05a-c2750281225e (Addr: tcp/127.0.0.1:26356) (DC: dc1)" +33:56.304 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-acb1be2b-2b35-7f88-f05a-c2750281225e.dc1 area=wan +33:56.304 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26351 network=tcp +33:56.305 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26352 network=tcp +33:56.305 [INFO] TestAgent: started state syncer +33:56.305 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26357 network=tcp +33:56.316 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.316 [DEBUG] TestAgent.server.raft: vote granted: from=acb1be2b-2b35-7f88-f05a-c2750281225e term=2 tally=1 +33:56.316 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.316 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26356 [Leader]" +33:56.320 [INFO] TestAgent.server: cluster leadership acquired +33:56.320 [INFO] TestAgent.server: New leader elected: payload=Node-acb1be2b-2b35-7f88-f05a-c2750281225e +33:56.348 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26356 +33:56.370 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.404 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:56.404 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:56.404 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:56.404 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:56.404 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:56.404 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-acb1be2b-2b35-7f88-f05a-c2750281225e +33:56.404 [INFO] TestAgent.server: member joined, marking health alive: member=Node-acb1be2b-2b35-7f88-f05a-c2750281225e +33:56.425 [INFO] TestAgent.server: federation state anti-entropy synced +33:56.636 [INFO] TestAgent: Synced node info +33:56.652 [INFO] TestAgent: Synced service: service=test +33:56.652 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/test?enable=false from=127.0.0.1:43500 latency=31.945332ms +33:56.656 [INFO] TestAgent: Requesting shutdown +33:56.656 [INFO] TestAgent.server: shutting down server +33:56.656 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:56.656 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:56.656 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:56.656 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:56.656 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.656 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:56.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:56.656 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:56.656 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:56.656 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:56.656 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:56.664 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.672Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.672Z [INFO] agent.router.manager: shutting down +33:56.672 [INFO] TestAgent: consul server down +33:56.672 [INFO] TestAgent: shutdown complete +33:56.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=tcp +33:56.672 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26351 network=udp +33:56.672 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26352 network=tcp +33:57.176 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.176 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_DisableServiceMaintenance (1.30s) +2021-10-05T16:33:56.125Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:56.217Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:56.317 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0d378823-ee0f-452d-ff46-ad160e7533a1 Address:127.0.0.1:26342}]" +33:56.318 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0d378823-ee0f-452d-ff46-ad160e7533a1.dc1 127.0.0.1 +33:56.318 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0d378823-ee0f-452d-ff46-ad160e7533a1 127.0.0.1 +2021-10-05T16:33:56.318Z [INFO] agent.router: Initializing LAN area manager +33:56.320 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= +33:56.320 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0d378823-ee0f-452d-ff46-ad160e7533a1.dc1 area=wan +33:56.320 [INFO] TestAgent.server: Adding LAN server: server="Node-0d378823-ee0f-452d-ff46-ad160e7533a1 (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +33:56.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp +33:56.321 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp +33:56.321 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp +33:56.321 [INFO] TestAgent: started state syncer +33:56.324 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp +33:56.383 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:56.384 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 +33:56.416 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:56.416 [DEBUG] TestAgent.server.raft: vote granted: from=0d378823-ee0f-452d-ff46-ad160e7533a1 term=2 tally=1 +33:56.416 [INFO] TestAgent.server.raft: election won: tally=1 +33:56.416 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" +33:56.416 [INFO] TestAgent.server: cluster leadership acquired +33:56.416 [INFO] TestAgent.server: New leader elected: payload=Node-0d378823-ee0f-452d-ff46-ad160e7533a1 +33:56.456 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26342 +33:56.508 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:56.537 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:56.537 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:56.537 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:56.537 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:56.537 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:56.537 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0d378823-ee0f-452d-ff46-ad160e7533a1 +33:56.537 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0d378823-ee0f-452d-ff46-ad160e7533a1 +33:56.568 [INFO] TestAgent.server: federation state anti-entropy synced +33:56.749 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/maintenance/redis?enable=true&reason=broken from=127.0.0.1:59170 latency=280.052µs +33:56.749 [INFO] TestAgent: Requesting shutdown +33:56.749 [INFO] TestAgent.server: shutting down server +33:56.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:56.749 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:56.749 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:56.749 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:56.749 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:56.749 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:56.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:56.749 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:56.749 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:56.749 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:56.749 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:56.764 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:56.772Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:56.772Z [INFO] agent.router.manager: shutting down +33:56.772 [INFO] TestAgent: consul server down +33:56.772 [INFO] TestAgent: shutdown complete +33:56.772 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp +33:56.772 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp +33:56.772 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp +33:57.276 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.276 [INFO] TestAgent: Endpoints down +--- PASS: TestMaintCommand_ServiceMaintenance_NoService (1.40s) PASS -ok github.com/hashicorp/consul/command/maint 1.053s +ok github.com/hashicorp/consul/command/maint 1.526s === RUN TestMembersCommand_noTabs === PAUSE TestMembersCommand_noTabs === RUN TestMembersCommand @@ -89423,463 +89566,451 @@ === RUN TestMembersCommand_verticalBar === PAUSE TestMembersCommand_verticalBar === CONT TestMembersCommand_noTabs -=== CONT TestMembersCommand_statusFilter_failed -=== CONT TestMembersCommand_WAN -=== CONT TestMembersCommand_statusFilter +--- PASS: TestMembersCommand_noTabs (0.00s) === CONT TestMembersCommand_verticalBar [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestMembersCommand_statusFilter +=== CONT TestMembersCommand_statusFilter_failed +=== CONT TestMembersCommand_WAN === CONT TestMembersCommand ---- PASS: TestMembersCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:43.319Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:43.340Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:43.378 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:297daa49-aa11-3f72-8c86-21b53c867953 Address:127.0.0.1:20237}]" -46:43.378 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20237 [Follower]" leader= -46:43.379 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-297daa49-aa11-3f72-8c86-21b53c867953.dc1 127.0.0.1 -46:43.379 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-297daa49-aa11-3f72-8c86-21b53c867953 127.0.0.1 -2022-11-07T22:46:43.379Z [INFO] agent.router: Initializing LAN area manager -46:43.379 [INFO] TestAgent.server: Adding LAN server: server="Node-297daa49-aa11-3f72-8c86-21b53c867953 (Addr: tcp/127.0.0.1:20237) (DC: dc1)" -46:43.379 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-297daa49-aa11-3f72-8c86-21b53c867953.dc1 area=wan -46:43.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20232 network=udp -46:43.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20232 network=tcp -46:43.380 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20233 network=tcp -46:43.402 [INFO] TestAgent: started state syncer -46:43.402 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20238 network=tcp -46:43.448 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:43.448 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20237 [Candidate]" term=2 -46:43.465 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:43.466 [DEBUG] TestAgent.server.raft: vote granted: from=297daa49-aa11-3f72-8c86-21b53c867953 term=2 tally=1 -46:43.466 [INFO] TestAgent.server.raft: election won: tally=1 -46:43.466 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20237 [Leader]" -46:43.466 [INFO] TestAgent.server: cluster leadership acquired -46:43.466 [INFO] TestAgent.server: New leader elected: payload=Node-297daa49-aa11-3f72-8c86-21b53c867953 -46:43.474 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20237 -46:43.491 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:43.519 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:43.519 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:43.519 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:43.519 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:43.519 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:43.519 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-297daa49-aa11-3f72-8c86-21b53c867953 -46:43.519 [INFO] TestAgent.server: member joined, marking health alive: member=Node-297daa49-aa11-3f72-8c86-21b53c867953 -46:43.528 [INFO] TestAgent.server: federation state anti-entropy synced -46:43.557 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:33070 latency=374.223µs -46:43.558 [INFO] TestAgent: Requesting shutdown -46:43.558 [INFO] TestAgent.server: shutting down server -46:43.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:43.558 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:43.558 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:43.558 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:43.558 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:43.558 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:43.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:43.558 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:43.558 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:43.558 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:43.558 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:43.561 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:43.564Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:43.564Z [INFO] agent.router.manager: shutting down -46:43.564 [INFO] TestAgent: consul server down -46:43.564 [INFO] TestAgent: shutdown complete -46:43.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20232 network=tcp -46:43.564 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20232 network=udp -46:43.564 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20233 network=tcp -46:44.064 [INFO] TestAgent: Waiting for endpoints to shut down -46:44.065 [INFO] TestAgent: Endpoints down ---- PASS: TestMembersCommand_statusFilter (0.78s) -2022-11-07T22:46:43.326Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:43.344Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:43.379 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1b6d43a6-a9ff-91ad-5de6-34461e71abc4 Address:127.0.0.1:20230}]" -46:43.379 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20230 [Follower]" leader= -46:43.379 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4.dc1 127.0.0.1 -46:43.380 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4 127.0.0.1 -2022-11-07T22:46:43.380Z [INFO] agent.router: Initializing LAN area manager -46:43.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=udp -46:43.380 [INFO] TestAgent.server: Adding LAN server: server="Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4 (Addr: tcp/127.0.0.1:20230) (DC: dc1)" -46:43.380 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4.dc1 area=wan -46:43.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=tcp -46:43.402 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20226 network=tcp -46:43.402 [INFO] TestAgent: started state syncer -46:43.402 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20231 network=tcp -46:43.434 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:43.434 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20230 [Candidate]" term=2 -46:43.454 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:43.454 [DEBUG] TestAgent.server.raft: vote granted: from=1b6d43a6-a9ff-91ad-5de6-34461e71abc4 term=2 tally=1 -46:43.454 [INFO] TestAgent.server.raft: election won: tally=1 -46:43.454 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20230 [Leader]" -46:43.454 [INFO] TestAgent.server: cluster leadership acquired -46:43.454 [INFO] TestAgent.server: New leader elected: payload=Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4 -46:43.462 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20230 -46:43.475 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:43.500 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:43.500 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:43.500 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:43.500 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:43.500 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:43.500 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4 -46:43.500 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1b6d43a6-a9ff-91ad-5de6-34461e71abc4 -46:43.516 [INFO] TestAgent.server: federation state anti-entropy synced -46:43.593 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:36110 latency=259.084µs -46:43.594 [INFO] TestAgent: Requesting shutdown -46:43.594 [INFO] TestAgent.server: shutting down server -46:43.594 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:43.594 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:43.594 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:43.594 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:43.594 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:43.594 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:43.594 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:43.594 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:43.594 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:43.594 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:43.594 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:43.598 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:43.600Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:43.600Z [INFO] agent.router.manager: shutting down -46:43.600 [INFO] TestAgent: consul server down -46:43.600 [INFO] TestAgent: shutdown complete -46:43.600 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=tcp -46:43.600 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=udp -46:43.600 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20226 network=tcp -46:44.101 [INFO] TestAgent: Waiting for endpoints to shut down -46:44.101 [INFO] TestAgent: Endpoints down ---- PASS: TestMembersCommand (0.82s) -2022-11-07T22:46:43.323Z [WARN] agent: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:43.323Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:43.337Z [WARN] agent.auto_config: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:43.337Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:43.375 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:4f34a158-7980-95a9-0905-50675d3ea217 Address:127.0.0.1:20244}]" -46:43.375 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20244 [Follower]" leader= -46:43.375 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: name|with|bars.dc1 127.0.0.1 -46:43.377 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: name|with|bars 127.0.0.1 -2022-11-07T22:46:43.377Z [INFO] agent.router: Initializing LAN area manager -46:43.377 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20239 network=udp -46:43.377 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=name|with|bars.dc1 area=wan -46:43.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20239 network=tcp -46:43.378 [INFO] TestAgent.server: Adding LAN server: server="name|with|bars (Addr: tcp/127.0.0.1:20244) (DC: dc1)" -46:43.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20240 network=tcp -46:43.379 [INFO] TestAgent: started state syncer -46:43.379 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20245 network=tcp -46:43.438 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:43.438 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20244 [Candidate]" term=2 -46:43.453 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:43.453 [DEBUG] TestAgent.server.raft: vote granted: from=4f34a158-7980-95a9-0905-50675d3ea217 term=2 tally=1 -46:43.453 [INFO] TestAgent.server.raft: election won: tally=1 -46:43.453 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20244 [Leader]" -46:43.453 [INFO] TestAgent.server: cluster leadership acquired -46:43.453 [INFO] TestAgent.server: New leader elected: payload=name|with|bars -46:43.465 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20244 -46:43.479 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:43.506 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:43.506 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:43.506 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:43.506 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:43.506 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:43.506 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=name|with|bars -46:43.506 [INFO] TestAgent.server: member joined, marking health alive: member=name|with|bars -46:43.519 [INFO] TestAgent.server: federation state anti-entropy synced -46:43.639 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:46494 latency=293.34µs -46:43.641 [INFO] TestAgent: Requesting shutdown -46:43.641 [INFO] TestAgent.server: shutting down server -46:43.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:43.641 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:43.641 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:43.641 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:43.641 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:43.641 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:43.641 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:43.641 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:43.641 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:43.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:43.641 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:43.644 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:43.647Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:43.647Z [INFO] agent.router.manager: shutting down -46:43.647 [INFO] TestAgent: consul server down -46:43.647 [INFO] TestAgent: shutdown complete -46:43.647 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20239 network=tcp -46:43.647 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20239 network=udp -46:43.647 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20240 network=tcp -46:44.148 [INFO] TestAgent: Waiting for endpoints to shut down -46:44.148 [INFO] TestAgent: Endpoints down ---- PASS: TestMembersCommand_verticalBar (0.87s) -2022-11-07T22:46:43.320Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:43.338Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:43.375 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7b72f9e0-affb-64cf-dc49-1b5fbdc733aa Address:127.0.0.1:20223}]" -46:43.375 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20223 [Follower]" leader= -46:43.377 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa.dc1 127.0.0.1 -46:43.379 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa 127.0.0.1 -2022-11-07T22:46:43.379Z [INFO] agent.router: Initializing LAN area manager -46:43.379 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa.dc1 area=wan -46:43.379 [INFO] TestAgent.server: Adding LAN server: server="Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa (Addr: tcp/127.0.0.1:20223) (DC: dc1)" -46:43.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=tcp -46:43.379 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=udp -46:43.380 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20219 network=tcp -46:43.380 [INFO] TestAgent: started state syncer -46:43.380 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20224 network=tcp -46:43.427 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:43.427 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20223 [Candidate]" term=2 -46:43.437 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:43.437 [DEBUG] TestAgent.server.raft: vote granted: from=7b72f9e0-affb-64cf-dc49-1b5fbdc733aa term=2 tally=1 -46:43.437 [INFO] TestAgent.server.raft: election won: tally=1 -46:43.437 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20223 [Leader]" -46:43.437 [INFO] TestAgent.server: cluster leadership acquired -46:43.437 [INFO] TestAgent.server: New leader elected: payload=Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa -46:43.453 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20223 -46:43.470 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:43.506 [INFO] TestAgent: Synced node info -46:43.506 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:43.506 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:43.506 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:43.506 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:43.506 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:43.506 [DEBUG] TestAgent: Node info in sync -46:43.506 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa -46:43.506 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7b72f9e0-affb-64cf-dc49-1b5fbdc733aa -46:43.511 [INFO] TestAgent.server: federation state anti-entropy synced -46:43.745 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:37516 latency=221.722µs -46:43.746 [INFO] TestAgent: Requesting shutdown -46:43.746 [INFO] TestAgent.server: shutting down server -46:43.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:43.746 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:43.746 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:43.746 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:43.746 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:43.746 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:43.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:43.746 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:43.746 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:43.746 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:43.749 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:43.753Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:43.753Z [INFO] agent.router.manager: shutting down -46:43.753 [INFO] TestAgent: consul server down -46:43.753 [INFO] TestAgent: shutdown complete -46:43.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=tcp -46:43.753 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=udp -46:43.753 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20219 network=tcp -46:44.253 [INFO] TestAgent: Waiting for endpoints to shut down -46:44.253 [INFO] TestAgent: Endpoints down ---- PASS: TestMembersCommand_statusFilter_failed (0.97s) -2022-11-07T22:46:43.323Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:43.337Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:43.375 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:632c3242-2224-6787-b1ad-5740c7a7f9d0 Address:127.0.0.1:20216}]" -46:43.375 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= -46:43.376 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-632c3242-2224-6787-b1ad-5740c7a7f9d0.dc1 127.0.0.1 -46:43.377 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-632c3242-2224-6787-b1ad-5740c7a7f9d0 127.0.0.1 -2022-11-07T22:46:43.377Z [INFO] agent.router: Initializing LAN area manager -46:43.377 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp -46:43.378 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-632c3242-2224-6787-b1ad-5740c7a7f9d0.dc1 area=wan -46:43.378 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp -46:43.379 [INFO] TestAgent.server: Adding LAN server: server="Node-632c3242-2224-6787-b1ad-5740c7a7f9d0 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" -46:43.379 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp -46:43.379 [INFO] TestAgent: started state syncer -46:43.379 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp -46:43.411 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:43.411 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 -46:43.419 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:43.419 [DEBUG] TestAgent.server.raft: vote granted: from=632c3242-2224-6787-b1ad-5740c7a7f9d0 term=2 tally=1 -46:43.419 [INFO] TestAgent.server.raft: election won: tally=1 -46:43.419 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" -46:43.419 [INFO] TestAgent.server: cluster leadership acquired -46:43.419 [INFO] TestAgent.server: New leader elected: payload=Node-632c3242-2224-6787-b1ad-5740c7a7f9d0 -46:43.424 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20216 -46:43.435 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:43.448 [INFO] TestAgent: Synced node info -46:43.448 [DEBUG] TestAgent: Node info in sync -46:43.472 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:43.472 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:43.472 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:43.472 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:43.472 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:43.472 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-632c3242-2224-6787-b1ad-5740c7a7f9d0 -46:43.472 [INFO] TestAgent.server: member joined, marking health alive: member=Node-632c3242-2224-6787-b1ad-5740c7a7f9d0 -46:43.488 [INFO] TestAgent.server: federation state anti-entropy synced -46:43.756 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all&wan=1 from=127.0.0.1:45020 latency=208.927µs -46:43.757 [INFO] TestAgent: Requesting shutdown -46:43.757 [INFO] TestAgent.server: shutting down server -46:43.757 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:43.757 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:43.757 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:43.757 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:43.757 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:43.757 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:43.757 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:43.757 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:43.757 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:43.757 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:43.760 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:43.763Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:43.763Z [INFO] agent.router.manager: shutting down -46:43.763 [INFO] TestAgent: consul server down -46:43.763 [INFO] TestAgent: shutdown complete -46:43.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp -46:43.763 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp -46:43.764 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp -46:44.264 [INFO] TestAgent: Waiting for endpoints to shut down -46:44.264 [INFO] TestAgent: Endpoints down ---- PASS: TestMembersCommand_WAN (0.98s) +2021-10-05T16:33:57.120Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:57.235Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:57.341 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dec05263-99c9-23c0-de97-7d417c0ec504 Address:127.0.0.1:15118}]" +33:57.356 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= +33:57.389 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dec05263-99c9-23c0-de97-7d417c0ec504.dc1 127.0.0.1 +33:57.389 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dec05263-99c9-23c0-de97-7d417c0ec504 127.0.0.1 +2021-10-05T16:33:57.390Z [INFO] agent.router: Initializing LAN area manager +33:57.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp +33:57.390 [INFO] TestAgent.server: Adding LAN server: server="Node-dec05263-99c9-23c0-de97-7d417c0ec504 (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +33:57.390 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dec05263-99c9-23c0-de97-7d417c0ec504.dc1 area=wan +33:57.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp +33:57.391 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp +33:57.391 [INFO] TestAgent: started state syncer +33:57.391 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp +33:57.397 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:57.397 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 +33:57.418 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:57.418 [DEBUG] TestAgent.server.raft: vote granted: from=dec05263-99c9-23c0-de97-7d417c0ec504 term=2 tally=1 +33:57.418 [INFO] TestAgent.server.raft: election won: tally=1 +33:57.418 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" +33:57.418 [INFO] TestAgent.server: cluster leadership acquired +33:57.418 [INFO] TestAgent.server: New leader elected: payload=Node-dec05263-99c9-23c0-de97-7d417c0ec504 +33:57.436 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15118 +33:57.464 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:39096 latency=17.143376ms +33:57.473 [INFO] TestAgent: Requesting shutdown +33:57.473 [INFO] TestAgent.server: shutting down server +33:57.473 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:57.473 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:57.481 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:57.482Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:57.482Z [INFO] agent.router.manager: shutting down +33:57.492 [ERROR] TestAgent.server: failed to establish leadership: error="leadership lost while committing log" +33:57.492 [INFO] TestAgent: consul server down +33:57.492 [INFO] TestAgent: shutdown complete +33:57.492 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +33:57.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp +33:57.492 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +33:57.492 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=2 retry_limit=3 error="raft is already shutdown" +33:57.492 [ERROR] TestAgent.server: failed to transfer leadership: error="failed to transfer leadership in 3 attempts" +33:57.492 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp +33:57.492 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp +33:57.996 [INFO] TestAgent: Waiting for endpoints to shut down +33:57.996 [INFO] TestAgent: Endpoints down +--- PASS: TestMembersCommand_statusFilter (1.05s) +2021-10-05T16:33:57.158Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:57.222Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:57.305 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:02c682d0-bde3-939f-dd34-7e24463cf3a0 Address:127.0.0.1:15132}]" +33:57.306 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-02c682d0-bde3-939f-dd34-7e24463cf3a0.dc1 127.0.0.1 +33:57.307 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-02c682d0-bde3-939f-dd34-7e24463cf3a0 127.0.0.1 +2021-10-05T16:33:57.307Z [INFO] agent.router: Initializing LAN area manager +33:57.307 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=udp +33:57.307 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15132 [Follower]" leader= +33:57.307 [INFO] TestAgent.server: Adding LAN server: server="Node-02c682d0-bde3-939f-dd34-7e24463cf3a0 (Addr: tcp/127.0.0.1:15132) (DC: dc1)" +33:57.307 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-02c682d0-bde3-939f-dd34-7e24463cf3a0.dc1 area=wan +33:57.308 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15127 network=tcp +33:57.308 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15128 network=tcp +33:57.308 [INFO] TestAgent: started state syncer +33:57.308 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15133 network=tcp +33:57.356 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:57.356 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15132 [Candidate]" term=2 +33:57.381 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:57.381 [DEBUG] TestAgent.server.raft: vote granted: from=02c682d0-bde3-939f-dd34-7e24463cf3a0 term=2 tally=1 +33:57.381 [INFO] TestAgent.server.raft: election won: tally=1 +33:57.381 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15132 [Leader]" +33:57.389 [INFO] TestAgent.server: cluster leadership acquired +33:57.390 [INFO] TestAgent.server: New leader elected: payload=Node-02c682d0-bde3-939f-dd34-7e24463cf3a0 +33:57.401 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15132 +33:57.417 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:57.466 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:57.466 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:57.466 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:57.466 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:57.466 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:57.466 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-02c682d0-bde3-939f-dd34-7e24463cf3a0 +33:57.466 [INFO] TestAgent.server: member joined, marking health alive: member=Node-02c682d0-bde3-939f-dd34-7e24463cf3a0 +33:57.496 [INFO] TestAgent.server: federation state anti-entropy synced +33:57.596 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +33:57.601 [INFO] TestAgent: Synced node info +33:57.601 [DEBUG] TestAgent: Node info in sync +33:57.640 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:51594 latency=269.345µs +33:57.642 [INFO] TestAgent: Requesting shutdown +33:57.642 [INFO] TestAgent.server: shutting down server +33:57.642 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:57.642 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:57.642 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:57.642 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:57.642 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:57.642 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:57.642 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:57.642 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:57.642 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:57.642 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:57.650 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:57.656Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:57.656Z [INFO] agent.router.manager: shutting down +33:57.656 [INFO] TestAgent: consul server down +33:57.656 [INFO] TestAgent: shutdown complete +33:57.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=tcp +33:57.656 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15127 network=udp +33:57.656 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15128 network=tcp +33:58.160 [INFO] TestAgent: Waiting for endpoints to shut down +33:58.160 [INFO] TestAgent: Endpoints down +--- PASS: TestMembersCommand (1.21s) +2021-10-05T16:33:57.155Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:57.284Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:57.385 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2d49d776-f98e-2c91-2bca-dd66d14593c1 Address:127.0.0.1:15125}]" +33:57.386 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2d49d776-f98e-2c91-2bca-dd66d14593c1.dc1 127.0.0.1 +33:57.386 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2d49d776-f98e-2c91-2bca-dd66d14593c1 127.0.0.1 +2021-10-05T16:33:57.386Z [INFO] agent.router: Initializing LAN area manager +33:57.386 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=udp +33:57.388 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15125 [Follower]" leader= +33:57.388 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15120 network=tcp +33:57.388 [INFO] TestAgent.server: Adding LAN server: server="Node-2d49d776-f98e-2c91-2bca-dd66d14593c1 (Addr: tcp/127.0.0.1:15125) (DC: dc1)" +33:57.388 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2d49d776-f98e-2c91-2bca-dd66d14593c1.dc1 area=wan +33:57.389 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15121 network=tcp +33:57.389 [INFO] TestAgent: started state syncer +33:57.389 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15126 network=tcp +33:57.440 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:57.440 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15125 [Candidate]" term=2 +33:57.492 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:57.492 [DEBUG] TestAgent.server.raft: vote granted: from=2d49d776-f98e-2c91-2bca-dd66d14593c1 term=2 tally=1 +33:57.493 [INFO] TestAgent.server.raft: election won: tally=1 +33:57.493 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15125 [Leader]" +33:57.493 [INFO] TestAgent.server: cluster leadership acquired +33:57.493 [INFO] TestAgent.server: New leader elected: payload=Node-2d49d776-f98e-2c91-2bca-dd66d14593c1 +33:57.510 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15125 +33:57.526 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:57.556 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:57.556 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:57.556 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:57.556 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:57.556 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:57.556 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2d49d776-f98e-2c91-2bca-dd66d14593c1 +33:57.556 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2d49d776-f98e-2c91-2bca-dd66d14593c1 +33:57.599 [INFO] TestAgent.server: federation state anti-entropy synced +33:57.672 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all&wan=1 from=127.0.0.1:57804 latency=1.264567ms +33:57.672 [INFO] TestAgent: Requesting shutdown +33:57.672 [INFO] TestAgent.server: shutting down server +33:57.672 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:57.672 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:57.672 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:57.672 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:57.672 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:57.672 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:57.672 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:57.673 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:57.673 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:57.673 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:57.673 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:57.677 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:57.684Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:57.684Z [INFO] agent.router.manager: shutting down +33:57.684 [INFO] TestAgent: consul server down +33:57.684 [INFO] TestAgent: shutdown complete +33:57.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=tcp +33:57.684 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15120 network=udp +33:57.684 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15121 network=tcp +33:58.188 [INFO] TestAgent: Waiting for endpoints to shut down +33:58.188 [INFO] TestAgent: Endpoints down +--- PASS: TestMembersCommand_WAN (1.24s) +2021-10-05T16:33:57.186Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:57.253Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:57.387 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ba1bb00b-a098-e4e4-373f-993f86f574f6 Address:127.0.0.1:15139}]" +33:57.387 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ba1bb00b-a098-e4e4-373f-993f86f574f6.dc1 127.0.0.1 +33:57.388 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ba1bb00b-a098-e4e4-373f-993f86f574f6 127.0.0.1 +33:57.389 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15139 [Follower]" leader= +2021-10-05T16:33:57.388Z [INFO] agent.router: Initializing LAN area manager +33:57.389 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=udp +33:57.390 [INFO] TestAgent.server: Adding LAN server: server="Node-ba1bb00b-a098-e4e4-373f-993f86f574f6 (Addr: tcp/127.0.0.1:15139) (DC: dc1)" +33:57.390 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ba1bb00b-a098-e4e4-373f-993f86f574f6.dc1 area=wan +33:57.390 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15134 network=tcp +33:57.390 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15135 network=tcp +33:57.391 [INFO] TestAgent: started state syncer +33:57.391 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15140 network=tcp +33:57.444 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:57.444 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15139 [Candidate]" term=2 +33:57.475 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:57.475 [DEBUG] TestAgent.server.raft: vote granted: from=ba1bb00b-a098-e4e4-373f-993f86f574f6 term=2 tally=1 +33:57.475 [INFO] TestAgent.server.raft: election won: tally=1 +33:57.475 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15139 [Leader]" +33:57.476 [INFO] TestAgent.server: cluster leadership acquired +33:57.477 [INFO] TestAgent.server: New leader elected: payload=Node-ba1bb00b-a098-e4e4-373f-993f86f574f6 +33:57.520 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15139 +33:57.556 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:57.628 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:57.628 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:57.628 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:57.628 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:57.628 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:57.628 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ba1bb00b-a098-e4e4-373f-993f86f574f6 +33:57.628 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ba1bb00b-a098-e4e4-373f-993f86f574f6 +33:57.658 [INFO] TestAgent: Synced node info +33:57.658 [DEBUG] TestAgent: Node info in sync +33:57.658 [INFO] TestAgent.server: federation state anti-entropy synced +33:57.713 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:59666 latency=1.263121ms +33:57.716 [INFO] TestAgent: Requesting shutdown +33:57.716 [INFO] TestAgent.server: shutting down server +33:57.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:57.716 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:57.716 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:57.716 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:57.716 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:57.716 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:57.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:57.716 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:57.716 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:57.716 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:57.724 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:57.728Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:57.728Z [INFO] agent.router.manager: shutting down +33:57.728 [INFO] TestAgent: consul server down +33:57.728 [INFO] TestAgent: shutdown complete +33:57.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=tcp +33:57.728 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15134 network=udp +33:57.728 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15135 network=tcp +33:58.232 [INFO] TestAgent: Waiting for endpoints to shut down +33:58.232 [INFO] TestAgent: Endpoints down +--- PASS: TestMembersCommand_statusFilter_failed (1.28s) +2021-10-05T16:33:57.158Z [WARN] agent: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:33:57.158Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:33:57.226Z [WARN] agent.auto_config: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:33:57.226Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +33:57.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6ca7f9a3-0934-71ca-db9b-9d905953e467 Address:127.0.0.1:15111}]" +33:57.333 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: name|with|bars.dc1 127.0.0.1 +33:57.340 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +33:57.374 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: name|with|bars 127.0.0.1 +2021-10-05T16:33:57.374Z [INFO] agent.router: Initializing LAN area manager +33:57.375 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +33:57.375 [INFO] TestAgent.server: Adding LAN server: server="name|with|bars (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +33:57.375 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=name|with|bars.dc1 area=wan +33:57.384 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +33:57.385 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +33:57.385 [INFO] TestAgent: started state syncer +33:57.386 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +33:57.399 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +33:57.399 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +33:57.413 [DEBUG] TestAgent.server.raft: votes: needed=1 +33:57.413 [DEBUG] TestAgent.server.raft: vote granted: from=6ca7f9a3-0934-71ca-db9b-9d905953e467 term=2 tally=1 +33:57.413 [INFO] TestAgent.server.raft: election won: tally=1 +33:57.413 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +33:57.413 [INFO] TestAgent.server: cluster leadership acquired +33:57.413 [INFO] TestAgent.server: New leader elected: payload=name|with|bars +33:57.429 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 +33:57.472 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +33:57.512 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +33:57.512 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +33:57.512 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +33:57.512 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +33:57.512 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +33:57.512 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=name|with|bars +33:57.512 [INFO] TestAgent.server: member joined, marking health alive: member=name|with|bars +33:57.519 [INFO] TestAgent.server: federation state anti-entropy synced +33:57.737 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/members?segment=_all from=127.0.0.1:57096 latency=446.377µs +33:57.737 [INFO] TestAgent: Requesting shutdown +33:57.737 [INFO] TestAgent.server: shutting down server +33:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +33:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +33:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +33:57.737 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +33:57.737 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +33:57.737 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +33:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +33:57.737 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +33:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +33:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +33:57.737 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +33:57.748 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:33:57.752Z [INFO] agent.router.manager: shutting down +2021-10-05T16:33:57.752Z [INFO] agent.router.manager: shutting down +33:57.752 [INFO] TestAgent: consul server down +33:57.752 [INFO] TestAgent: shutdown complete +33:57.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +33:57.752 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +33:57.752 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +33:58.256 [INFO] TestAgent: Waiting for endpoints to shut down +33:58.256 [INFO] TestAgent: Endpoints down +--- PASS: TestMembersCommand_verticalBar (1.31s) PASS -ok github.com/hashicorp/consul/command/members 1.016s +ok github.com/hashicorp/consul/command/members 1.393s === RUN TestOperatorCommand_noTabs === PAUSE TestOperatorCommand_noTabs === CONT TestOperatorCommand_noTabs --- PASS: TestOperatorCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/operator 0.005s +ok github.com/hashicorp/consul/command/operator 0.011s === RUN TestOperatorAutopilotCommand_noTabs === PAUSE TestOperatorAutopilotCommand_noTabs === CONT TestOperatorAutopilotCommand_noTabs --- PASS: TestOperatorAutopilotCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/operator/autopilot 0.005s +ok github.com/hashicorp/consul/command/operator/autopilot 0.043s === RUN TestOperatorAutopilotGetConfigCommand_noTabs === PAUSE TestOperatorAutopilotGetConfigCommand_noTabs === RUN TestOperatorAutopilotGetConfigCommand === PAUSE TestOperatorAutopilotGetConfigCommand === CONT TestOperatorAutopilotGetConfigCommand_noTabs +--- PASS: TestOperatorAutopilotGetConfigCommand_noTabs (0.00s) === CONT TestOperatorAutopilotGetConfigCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestOperatorAutopilotGetConfigCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:46.401Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:46.413Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:46.437 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 Address:127.0.0.1:31447}]" -46:46.437 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:31447 [Follower]" leader= -46:46.437 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6.dc1 127.0.0.1 -46:46.438 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 127.0.0.1 -2022-11-07T22:46:46.438Z [INFO] agent.router: Initializing LAN area manager -46:46.438 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=udp -46:46.438 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6.dc1 area=wan -46:46.439 [INFO] TestAgent.server: Adding LAN server: server="Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 (Addr: tcp/127.0.0.1:31447) (DC: dc1)" -46:46.439 [INFO] TestAgent: Started DNS server: address=127.0.0.1:31442 network=tcp -46:46.439 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:31443 network=tcp -46:46.439 [INFO] TestAgent: started state syncer -46:46.439 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:31448 network=tcp -46:46.475 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:46.475 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:31447 [Candidate]" term=2 -46:46.482 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:46.482 [DEBUG] TestAgent.server.raft: vote granted: from=ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 term=2 tally=1 -46:46.482 [INFO] TestAgent.server.raft: election won: tally=1 -46:46.482 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:31447 [Leader]" -46:46.482 [INFO] TestAgent.server: cluster leadership acquired -46:46.482 [INFO] TestAgent.server: New leader elected: payload=Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 -46:46.487 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:31447 -46:46.499 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:46.521 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:46.521 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:46.521 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:46.521 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:46.521 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:46.521 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 -46:46.521 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ff9b9c22-d6b7-387f-9e6c-4dd4359b64f6 -46:46.525 [INFO] TestAgent.server: federation state anti-entropy synced -46:46.813 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:57802 latency=501.253µs -46:46.814 [INFO] TestAgent: Requesting shutdown -46:46.814 [INFO] TestAgent.server: shutting down server -46:46.814 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:46.814 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:46.814 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:46.814 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:46.814 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:46.814 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:46.814 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:46.814 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:46.814 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:46.814 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:46.814 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:46.818 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:46.821Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:46.821Z [INFO] agent.router.manager: shutting down -46:46.821 [INFO] TestAgent: consul server down -46:46.821 [INFO] TestAgent: shutdown complete -46:46.821 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=tcp -46:46.821 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:31442 network=udp -46:46.821 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:31443 network=tcp -46:47.321 [INFO] TestAgent: Waiting for endpoints to shut down -46:47.321 [INFO] TestAgent: Endpoints down ---- PASS: TestOperatorAutopilotGetConfigCommand (0.95s) +2021-10-05T16:34:00.287Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:00.312Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:00.471 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b832338e-3121-b1fd-4720-826a6f3aea38 Address:127.0.0.1:29405}]" +34:00.472 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= +34:00.472 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b832338e-3121-b1fd-4720-826a6f3aea38.dc1 127.0.0.1 +34:00.473 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b832338e-3121-b1fd-4720-826a6f3aea38 127.0.0.1 +2021-10-05T16:34:00.473Z [INFO] agent.router: Initializing LAN area manager +34:00.474 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp +34:00.475 [INFO] TestAgent.server: Adding LAN server: server="Node-b832338e-3121-b1fd-4720-826a6f3aea38 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" +34:00.475 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b832338e-3121-b1fd-4720-826a6f3aea38.dc1 area=wan +34:00.475 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp +34:00.475 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp +34:00.475 [INFO] TestAgent: started state syncer +34:00.476 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp +34:00.511 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:00.511 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 +34:00.523 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:00.523 [DEBUG] TestAgent.server.raft: vote granted: from=b832338e-3121-b1fd-4720-826a6f3aea38 term=2 tally=1 +34:00.523 [INFO] TestAgent.server.raft: election won: tally=1 +34:00.523 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" +34:00.523 [INFO] TestAgent.server: cluster leadership acquired +34:00.523 [INFO] TestAgent.server: New leader elected: payload=Node-b832338e-3121-b1fd-4720-826a6f3aea38 +34:00.532 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 +34:00.546 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:00.620 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:00.620 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:00.620 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:00.620 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:00.620 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:00.620 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b832338e-3121-b1fd-4720-826a6f3aea38 +34:00.620 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b832338e-3121-b1fd-4720-826a6f3aea38 +34:00.632 [INFO] TestAgent.server: federation state anti-entropy synced +34:00.642 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:35312 latency=3.439666ms +34:00.649 [INFO] TestAgent: Requesting shutdown +34:00.649 [INFO] TestAgent.server: shutting down server +34:00.649 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:00.649 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:00.649 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:00.649 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:00.649 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:00.649 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:00.649 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:00.649 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:00.649 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:00.649 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:00.649 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:00.661 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:00.668Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:00.668Z [INFO] agent.router.manager: shutting down +34:00.668 [INFO] TestAgent: consul server down +34:00.668 [INFO] TestAgent: shutdown complete +34:00.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp +34:00.668 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp +34:00.668 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp +34:01.172 [INFO] TestAgent: Waiting for endpoints to shut down +34:01.172 [INFO] TestAgent: Endpoints down +--- PASS: TestOperatorAutopilotGetConfigCommand (0.94s) PASS -ok github.com/hashicorp/consul/command/operator/autopilot/get 0.981s +ok github.com/hashicorp/consul/command/operator/autopilot/get 1.124s === RUN TestOperatorAutopilotSetConfigCommand_noTabs === PAUSE TestOperatorAutopilotSetConfigCommand_noTabs === RUN TestOperatorAutopilotSetConfigCommand === PAUSE TestOperatorAutopilotSetConfigCommand === CONT TestOperatorAutopilotSetConfigCommand_noTabs +--- PASS: TestOperatorAutopilotSetConfigCommand_noTabs (0.00s) === CONT TestOperatorAutopilotSetConfigCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestOperatorAutopilotSetConfigCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:46.998Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:47.009Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:47.030 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7684be8b-f915-692f-a30d-26ac62c6d70e Address:127.0.0.1:10006}]" -46:47.030 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:10006 [Follower]" leader= -46:47.031 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7684be8b-f915-692f-a30d-26ac62c6d70e.dc1 127.0.0.1 -46:47.032 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7684be8b-f915-692f-a30d-26ac62c6d70e 127.0.0.1 -2022-11-07T22:46:47.032Z [INFO] agent.router: Initializing LAN area manager -46:47.032 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7684be8b-f915-692f-a30d-26ac62c6d70e.dc1 area=wan -46:47.032 [INFO] TestAgent.server: Adding LAN server: server="Node-7684be8b-f915-692f-a30d-26ac62c6d70e (Addr: tcp/127.0.0.1:10006) (DC: dc1)" -46:47.032 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=tcp -46:47.032 [INFO] TestAgent: Started DNS server: address=127.0.0.1:10001 network=udp -46:47.033 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:10002 network=tcp -46:47.033 [INFO] TestAgent: started state syncer -46:47.033 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:10007 network=tcp -46:47.085 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:47.085 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:10006 [Candidate]" term=2 -46:47.093 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:47.093 [DEBUG] TestAgent.server.raft: vote granted: from=7684be8b-f915-692f-a30d-26ac62c6d70e term=2 tally=1 -46:47.093 [INFO] TestAgent.server.raft: election won: tally=1 -46:47.093 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:10006 [Leader]" -46:47.093 [INFO] TestAgent.server: cluster leadership acquired -46:47.093 [INFO] TestAgent.server: New leader elected: payload=Node-7684be8b-f915-692f-a30d-26ac62c6d70e -46:47.098 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:10006 -46:47.110 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:47.127 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:47.127 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:47.127 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:47.127 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:47.128 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:47.128 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7684be8b-f915-692f-a30d-26ac62c6d70e -46:47.128 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7684be8b-f915-692f-a30d-26ac62c6d70e -46:47.137 [INFO] TestAgent.server: federation state anti-entropy synced -46:47.166 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:38478 latency=366.603µs -46:47.172 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/configuration?cas=4 from=127.0.0.1:38478 latency=4.992823ms -46:47.173 [INFO] TestAgent: Requesting shutdown -46:47.173 [INFO] TestAgent.server: shutting down server -46:47.173 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:47.173 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:47.173 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:47.173 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:47.173 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:47.173 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:47.173 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:47.173 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:47.173 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:47.173 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:47.173 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:47.176 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:47.179Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:47.179Z [INFO] agent.router.manager: shutting down -46:47.179 [INFO] TestAgent: consul server down -46:47.179 [INFO] TestAgent: shutdown complete -46:47.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=tcp -46:47.179 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:10001 network=udp -46:47.179 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:10002 network=tcp -46:47.680 [INFO] TestAgent: Waiting for endpoints to shut down -46:47.680 [INFO] TestAgent: Endpoints down ---- PASS: TestOperatorAutopilotSetConfigCommand (0.71s) +2021-10-05T16:34:02.844Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:02.905Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:02.988 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cb8ccc42-266c-a3d2-1d62-f244470cad1b Address:127.0.0.1:26342}]" +34:02.989 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b.dc1 127.0.0.1 +34:02.990 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b 127.0.0.1 +2021-10-05T16:34:02.990Z [INFO] agent.router: Initializing LAN area manager +34:02.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=udp +34:02.990 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:26342 [Follower]" leader= +34:02.990 [INFO] TestAgent.server: Adding LAN server: server="Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b (Addr: tcp/127.0.0.1:26342) (DC: dc1)" +34:02.991 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b.dc1 area=wan +34:02.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:26337 network=tcp +34:02.991 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:26338 network=tcp +34:02.991 [INFO] TestAgent: started state syncer +34:02.991 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:26343 network=tcp +34:03.027 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:03.027 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:26342 [Candidate]" term=2 +34:03.072 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:03.072 [DEBUG] TestAgent.server.raft: vote granted: from=cb8ccc42-266c-a3d2-1d62-f244470cad1b term=2 tally=1 +34:03.072 [INFO] TestAgent.server.raft: election won: tally=1 +34:03.072 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:26342 [Leader]" +34:03.076 [INFO] TestAgent.server: cluster leadership acquired +34:03.076 [INFO] TestAgent.server: New leader elected: payload=Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b +34:03.108 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:26342 +34:03.158 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:03.250 [INFO] TestAgent: Synced node info +34:03.250 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:03.250 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:03.250 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:03.250 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:03.250 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:03.250 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b +34:03.250 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cb8ccc42-266c-a3d2-1d62-f244470cad1b +34:03.262 [INFO] TestAgent.server: federation state anti-entropy synced +34:03.427 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/autopilot/configuration from=127.0.0.1:59204 latency=1.629888ms +34:03.434 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/operator/autopilot/configuration?cas=4 from=127.0.0.1:59204 latency=5.887554ms +34:03.435 [INFO] TestAgent: Requesting shutdown +34:03.435 [INFO] TestAgent.server: shutting down server +34:03.435 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:03.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:03.435 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:03.435 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:03.435 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:03.435 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:03.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:03.435 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:03.435 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:03.435 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:03.443 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:03.456Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:03.456Z [INFO] agent.router.manager: shutting down +34:03.456 [INFO] TestAgent: consul server down +34:03.456 [INFO] TestAgent: shutdown complete +34:03.456 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=tcp +34:03.456 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:26337 network=udp +34:03.456 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:26338 network=tcp +34:03.960 [INFO] TestAgent: Waiting for endpoints to shut down +34:03.960 [INFO] TestAgent: Endpoints down +--- PASS: TestOperatorAutopilotSetConfigCommand (1.24s) PASS -ok github.com/hashicorp/consul/command/operator/autopilot/set 0.743s +ok github.com/hashicorp/consul/command/operator/autopilot/set 1.491s === RUN TestOperatorRaftCommand_noTabs === PAUSE TestOperatorRaftCommand_noTabs === CONT TestOperatorRaftCommand_noTabs --- PASS: TestOperatorRaftCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/operator/raft 0.005s +ok github.com/hashicorp/consul/command/operator/raft 0.016s === RUN TestOperatorRaftListPeersCommand_noTabs === PAUSE TestOperatorRaftListPeersCommand_noTabs === RUN TestOperatorRaftListPeersCommand @@ -89887,270 +90018,250 @@ === RUN TestOperatorRaftListPeersCommand_verticalBar === PAUSE TestOperatorRaftListPeersCommand_verticalBar === CONT TestOperatorRaftListPeersCommand_noTabs +--- PASS: TestOperatorRaftListPeersCommand_noTabs (0.00s) === CONT TestOperatorRaftListPeersCommand_verticalBar -=== CONT TestOperatorRaftListPeersCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestOperatorRaftListPeersCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:48.731Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:48.743Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:48.771 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0c51fba0-80db-972b-c38d-188f1b3cc5e6 Address:127.0.0.1:13069}]" -46:48.771 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13069 [Follower]" leader= -46:48.773 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6.dc1 127.0.0.1 -46:48.773 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6 127.0.0.1 -2022-11-07T22:46:48.773Z [INFO] agent.router: Initializing LAN area manager -46:48.774 [INFO] TestAgent.server: Adding LAN server: server="Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6 (Addr: tcp/127.0.0.1:13069) (DC: dc1)" -46:48.774 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6.dc1 area=wan -46:48.774 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=udp -46:48.774 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=tcp -46:48.775 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13065 network=tcp -46:48.775 [INFO] TestAgent: started state syncer -46:48.775 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13070 network=tcp -46:48.831 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:48.832 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13069 [Candidate]" term=2 -46:48.841 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:48.841 [DEBUG] TestAgent.server.raft: vote granted: from=0c51fba0-80db-972b-c38d-188f1b3cc5e6 term=2 tally=1 -46:48.841 [INFO] TestAgent.server.raft: election won: tally=1 -46:48.841 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13069 [Leader]" -46:48.841 [INFO] TestAgent.server: cluster leadership acquired -46:48.841 [INFO] TestAgent.server: New leader elected: payload=Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6 -46:48.846 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13069 -46:48.857 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:48.878 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:48.878 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:48.878 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:48.878 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:48.878 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:48.878 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6 -46:48.878 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0c51fba0-80db-972b-c38d-188f1b3cc5e6 -46:48.882 [INFO] TestAgent.server: federation state anti-entropy synced -46:49.063 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:36332 latency=488.068µs -46:49.063 [INFO] TestAgent: Requesting shutdown -46:49.063 [INFO] TestAgent.server: shutting down server -46:49.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:49.063 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:49.063 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:49.063 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:49.063 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:49.063 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:49.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:49.063 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:49.063 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:49.063 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:49.063 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:49.066 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:49.070Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:49.070Z [INFO] agent.router.manager: shutting down -46:49.070 [INFO] TestAgent: consul server down -46:49.070 [INFO] TestAgent: shutdown complete -46:49.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=tcp -46:49.070 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=udp -46:49.070 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13065 network=tcp -46:49.570 [INFO] TestAgent: Waiting for endpoints to shut down -46:49.570 [INFO] TestAgent: Endpoints down ---- PASS: TestOperatorRaftListPeersCommand (0.87s) -2022-11-07T22:46:48.732Z [WARN] agent: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:48.732Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:48.744Z [WARN] agent.auto_config: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. -2022-11-07T22:46:48.744Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:48.769 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f82ce162-bec1-eea2-ef27-6c82c0de5974 Address:127.0.0.1:13076}]" -46:48.770 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13076 [Follower]" leader= -46:48.770 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: name|with|bars.dc1 127.0.0.1 -46:48.771 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: name|with|bars 127.0.0.1 -2022-11-07T22:46:48.771Z [INFO] agent.router: Initializing LAN area manager -46:48.771 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=name|with|bars.dc1 area=wan -46:48.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=tcp -46:48.771 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=udp -46:48.771 [INFO] TestAgent.server: Adding LAN server: server="name|with|bars (Addr: tcp/127.0.0.1:13076) (DC: dc1)" -46:48.771 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13072 network=tcp -46:48.771 [INFO] TestAgent: started state syncer -46:48.771 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13077 network=tcp -46:48.816 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:48.816 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13076 [Candidate]" term=2 -46:48.824 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:48.824 [DEBUG] TestAgent.server.raft: vote granted: from=f82ce162-bec1-eea2-ef27-6c82c0de5974 term=2 tally=1 -46:48.824 [INFO] TestAgent.server.raft: election won: tally=1 -46:48.824 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13076 [Leader]" -46:48.824 [INFO] TestAgent.server: cluster leadership acquired -46:48.824 [INFO] TestAgent.server: New leader elected: payload=name|with|bars -46:48.829 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13076 -46:48.842 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:48.859 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:48.859 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:48.859 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:48.859 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:48.859 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:48.859 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=name|with|bars -46:48.859 [INFO] TestAgent.server: member joined, marking health alive: member=name|with|bars -46:48.865 [INFO] TestAgent.server: federation state anti-entropy synced -46:49.027 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:49.030 [INFO] TestAgent: Synced node info -46:49.198 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:48406 latency=1.37181ms -46:49.198 [INFO] TestAgent: Requesting shutdown -46:49.198 [INFO] TestAgent.server: shutting down server -46:49.198 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:49.198 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:49.198 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:49.198 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:49.198 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:49.198 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:49.198 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:49.198 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:49.198 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:49.198 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:49.202 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:49.205Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:49.205Z [INFO] agent.router.manager: shutting down -46:49.205 [INFO] TestAgent: consul server down -46:49.205 [INFO] TestAgent: shutdown complete -46:49.205 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=tcp -46:49.205 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=udp -46:49.205 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13072 network=tcp -46:49.705 [INFO] TestAgent: Waiting for endpoints to shut down -46:49.705 [INFO] TestAgent: Endpoints down ---- PASS: TestOperatorRaftListPeersCommand_verticalBar (1.01s) +=== CONT TestOperatorRaftListPeersCommand +2021-10-05T16:34:08.865Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:08.898Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:08.986 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:af9dead1-8fe9-16a5-50e9-07dd15fc16d9 Address:127.0.0.1:13076}]" +34:08.987 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9.dc1 127.0.0.1 +34:08.987 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9 127.0.0.1 +2021-10-05T16:34:08.987Z [INFO] agent.router: Initializing LAN area manager +34:08.987 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=udp +34:08.987 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13076 [Follower]" leader= +34:08.988 [INFO] TestAgent.server: Adding LAN server: server="Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9 (Addr: tcp/127.0.0.1:13076) (DC: dc1)" +34:08.988 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9.dc1 area=wan +34:08.988 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=tcp +34:08.988 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13072 network=tcp +34:08.988 [INFO] TestAgent: started state syncer +34:08.989 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13077 network=tcp +34:09.029 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:09.029 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13076 [Candidate]" term=2 +34:09.070 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:09.070 [DEBUG] TestAgent.server.raft: vote granted: from=af9dead1-8fe9-16a5-50e9-07dd15fc16d9 term=2 tally=1 +34:09.070 [INFO] TestAgent.server.raft: election won: tally=1 +34:09.070 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13076 [Leader]" +34:09.070 [INFO] TestAgent.server: cluster leadership acquired +34:09.070 [INFO] TestAgent.server: New leader elected: payload=Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9 +34:09.082 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13076 +34:09.129 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:09.165 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:38988 latency=664.218µs +34:09.165 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:09.165 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:09.165 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:09.165 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:09.165 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:09.166 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9 +34:09.166 [INFO] TestAgent.server: member joined, marking health alive: member=Node-af9dead1-8fe9-16a5-50e9-07dd15fc16d9 +34:09.167 [INFO] TestAgent: Requesting shutdown +34:09.167 [INFO] TestAgent.server: shutting down server +34:09.167 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:09.167 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:09.167 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:09.167 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:09.167 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:09.167 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:09.167 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:09.167 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:09.167 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:09.171 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +34:09.173 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:09.173 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +2021-10-05T16:34:09.177Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:09.177Z [INFO] agent.router.manager: shutting down +34:09.177 [INFO] TestAgent: consul server down +34:09.177 [INFO] TestAgent: shutdown complete +34:09.177 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=tcp +34:09.177 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=udp +34:09.177 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13072 network=tcp +34:09.680 [INFO] TestAgent: Waiting for endpoints to shut down +34:09.680 [INFO] TestAgent: Endpoints down +--- PASS: TestOperatorRaftListPeersCommand (0.94s) +2021-10-05T16:34:08.880Z [WARN] agent: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:34:08.880Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:08.899Z [WARN] agent.auto_config: Node name "name|with|bars" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. +2021-10-05T16:34:08.899Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:08.989 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:25cc2c19-6f8a-a3e3-f4e7-478f308d8cbf Address:127.0.0.1:13069}]" +34:08.990 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: name|with|bars.dc1 127.0.0.1 +34:08.991 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: name|with|bars 127.0.0.1 +2021-10-05T16:34:08.991Z [INFO] agent.router: Initializing LAN area manager +34:08.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=udp +34:08.991 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13069 [Follower]" leader= +34:08.991 [INFO] TestAgent.server: Adding LAN server: server="name|with|bars (Addr: tcp/127.0.0.1:13069) (DC: dc1)" +34:08.991 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=name|with|bars.dc1 area=wan +34:08.991 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=tcp +34:08.992 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13065 network=tcp +34:08.992 [INFO] TestAgent: started state syncer +34:08.992 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13070 network=tcp +34:09.059 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:09.059 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13069 [Candidate]" term=2 +34:09.074 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:09.074 [DEBUG] TestAgent.server.raft: vote granted: from=25cc2c19-6f8a-a3e3-f4e7-478f308d8cbf term=2 tally=1 +34:09.074 [INFO] TestAgent.server.raft: election won: tally=1 +34:09.074 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13069 [Leader]" +34:09.074 [INFO] TestAgent.server: cluster leadership acquired +34:09.074 [INFO] TestAgent.server: New leader elected: payload=name|with|bars +34:09.092 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13069 +34:09.136 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:09.212 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:09.212 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:09.212 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:09.212 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:09.212 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:09.212 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=name|with|bars +34:09.212 [INFO] TestAgent.server: member joined, marking health alive: member=name|with|bars +34:09.241 [INFO] TestAgent.server: federation state anti-entropy synced +34:09.252 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/operator/raft/configuration from=127.0.0.1:51266 latency=320.542µs +34:09.252 [INFO] TestAgent: Requesting shutdown +34:09.252 [INFO] TestAgent.server: shutting down server +34:09.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:09.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:09.252 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:09.252 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:09.252 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:09.252 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:09.252 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:09.253 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:09.253 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:09.253 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:09.253 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:09.263 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:09.271Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:09.271Z [INFO] agent.router.manager: shutting down +34:09.271 [INFO] TestAgent: consul server down +34:09.271 [INFO] TestAgent: shutdown complete +34:09.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=tcp +34:09.271 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=udp +34:09.271 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13065 network=tcp +34:09.772 [INFO] TestAgent: Waiting for endpoints to shut down +34:09.772 [INFO] TestAgent: Endpoints down +--- PASS: TestOperatorRaftListPeersCommand_verticalBar (1.06s) PASS -ok github.com/hashicorp/consul/command/operator/raft/listpeers 1.039s +ok github.com/hashicorp/consul/command/operator/raft/listpeers 1.291s === RUN TestOperatorRaftRemovePeerCommand_noTabs === PAUSE TestOperatorRaftRemovePeerCommand_noTabs === RUN TestOperatorRaftRemovePeerCommand === PAUSE TestOperatorRaftRemovePeerCommand === CONT TestOperatorRaftRemovePeerCommand_noTabs +--- PASS: TestOperatorRaftRemovePeerCommand_noTabs (0.00s) === CONT TestOperatorRaftRemovePeerCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestOperatorRaftRemovePeerCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestOperatorRaftRemovePeerCommand/Test_the_remove-peer_subcommand_directly === RUN TestOperatorRaftRemovePeerCommand/Test_the_remove-peer_subcommand_with_-id -2022-11-07T22:46:49.197Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:49.208Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:49.228 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2c0209e7-086a-af3a-c647-5833ab0ed8d5 Address:127.0.0.1:12048}]" -46:49.228 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= -46:49.229 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5.dc1 127.0.0.1 -46:49.229 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5 127.0.0.1 -2022-11-07T22:46:49.229Z [INFO] agent.router: Initializing LAN area manager -46:49.229 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5.dc1 area=wan -46:49.229 [INFO] TestAgent.server: Adding LAN server: server="Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5 (Addr: tcp/127.0.0.1:12048) (DC: dc1)" -46:49.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp -46:49.229 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp -46:49.230 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp -46:49.230 [INFO] TestAgent: started state syncer -46:49.230 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp -46:49.277 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:49.278 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 -46:49.285 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:49.285 [DEBUG] TestAgent.server.raft: vote granted: from=2c0209e7-086a-af3a-c647-5833ab0ed8d5 term=2 tally=1 -46:49.285 [INFO] TestAgent.server.raft: election won: tally=1 -46:49.285 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" -46:49.285 [INFO] TestAgent.server: cluster leadership acquired -46:49.286 [INFO] TestAgent.server: New leader elected: payload=Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5 -46:49.290 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12048 -46:49.301 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:49.319 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:49.319 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:49.319 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:49.319 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:49.319 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:49.319 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5 -46:49.319 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2c0209e7-086a-af3a-c647-5833ab0ed8d5 -46:49.324 [INFO] TestAgent.server: federation state anti-entropy synced -46:49.456 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:49.460 [INFO] TestAgent: Synced node info -46:49.460 [DEBUG] TestAgent: Node info in sync -46:49.655 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer?address=nope from=127.0.0.1:39490 error="address "nope" was not found in the Raft configuration" -46:49.655 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer?address=nope from=127.0.0.1:39490 latency=970.925µs -46:49.657 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer?id=nope from=127.0.0.1:39492 error="id "nope" was not found in the Raft configuration" -46:49.657 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer?id=nope from=127.0.0.1:39492 latency=225.633µs -46:49.657 [INFO] TestAgent: Requesting shutdown -46:49.657 [INFO] TestAgent.server: shutting down server -46:49.657 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:49.657 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:49.657 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:49.657 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:49.657 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:49.657 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:49.657 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:49.657 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:49.657 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:49.657 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:49.663 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:49.667Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:49.667Z [INFO] agent.router.manager: shutting down -46:49.667 [INFO] TestAgent: consul server down -46:49.667 [INFO] TestAgent: shutdown complete -46:49.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp -46:49.667 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp -46:49.667 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp -46:50.167 [INFO] TestAgent: Waiting for endpoints to shut down -46:50.167 [INFO] TestAgent: Endpoints down ---- PASS: TestOperatorRaftRemovePeerCommand (1.00s) - --- PASS: TestOperatorRaftRemovePeerCommand/Test_the_remove-peer_subcommand_directly (0.00s) +2021-10-05T16:34:08.235Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:08.271Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:08.360 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7549470c-48c8-affe-c949-f23de85b0c45 Address:127.0.0.1:14090}]" +34:08.361 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7549470c-48c8-affe-c949-f23de85b0c45.dc1 127.0.0.1 +34:08.361 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7549470c-48c8-affe-c949-f23de85b0c45 127.0.0.1 +2021-10-05T16:34:08.361Z [INFO] agent.router: Initializing LAN area manager +34:08.374 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=udp +34:08.380 [INFO] TestAgent: Started DNS server: address=127.0.0.1:14085 network=tcp +34:08.374 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:14090 [Follower]" leader= +34:08.375 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7549470c-48c8-affe-c949-f23de85b0c45.dc1 area=wan +34:08.375 [INFO] TestAgent.server: Adding LAN server: server="Node-7549470c-48c8-affe-c949-f23de85b0c45 (Addr: tcp/127.0.0.1:14090) (DC: dc1)" +34:08.394 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:14086 network=tcp +34:08.394 [INFO] TestAgent: started state syncer +34:08.394 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:14091 network=tcp +34:08.432 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:08.432 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:14090 [Candidate]" term=2 +34:08.467 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:08.467 [DEBUG] TestAgent.server.raft: vote granted: from=7549470c-48c8-affe-c949-f23de85b0c45 term=2 tally=1 +34:08.467 [INFO] TestAgent.server.raft: election won: tally=1 +34:08.467 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:14090 [Leader]" +34:08.467 [INFO] TestAgent.server: cluster leadership acquired +34:08.467 [INFO] TestAgent.server: New leader elected: payload=Node-7549470c-48c8-affe-c949-f23de85b0c45 +34:08.473 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:14090 +34:08.495 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:08.500 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer?address=nope from=127.0.0.1:45972 error="address "nope" was not found in the Raft configuration" +34:08.504 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer?address=nope from=127.0.0.1:45972 latency=3.689621ms +34:08.505 [ERROR] TestAgent.http: Request error: method=DELETE url=/v1/operator/raft/peer?id=nope from=127.0.0.1:45974 error="id "nope" was not found in the Raft configuration" +34:08.506 [DEBUG] TestAgent.http: Request finished: method=DELETE url=/v1/operator/raft/peer?id=nope from=127.0.0.1:45974 latency=587.856µs +34:08.506 [INFO] TestAgent: Requesting shutdown +34:08.506 [INFO] TestAgent.server: shutting down server +34:08.506 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:08.506 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:08.516 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:08.523Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:08.523Z [INFO] agent.router.manager: shutting down +34:08.523 [INFO] TestAgent: consul server down +34:08.523 [INFO] TestAgent: shutdown complete +34:08.523 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=tcp +34:08.523 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:14085 network=udp +34:08.524 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:14086 network=tcp +34:08.528 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: raft is already shutdown" +34:08.529 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=0 retry_limit=3 error="raft is already shutdown" +34:08.529 [ERROR] TestAgent.server: failed to transfer leadership attempt, will retry: attempt=1 retry_limit=3 error="raft is already shutdown" +34:09.024 [INFO] TestAgent: Waiting for endpoints to shut down +34:09.024 [INFO] TestAgent: Endpoints down +--- PASS: TestOperatorRaftRemovePeerCommand (0.94s) + --- PASS: TestOperatorRaftRemovePeerCommand/Test_the_remove-peer_subcommand_directly (0.01s) --- PASS: TestOperatorRaftRemovePeerCommand/Test_the_remove-peer_subcommand_with_-id (0.00s) PASS -ok github.com/hashicorp/consul/command/operator/raft/removepeer 1.034s +ok github.com/hashicorp/consul/command/operator/raft/removepeer 1.245s === RUN TestReloadCommand_noTabs === PAUSE TestReloadCommand_noTabs === RUN TestReloadCommand === PAUSE TestReloadCommand === CONT TestReloadCommand_noTabs +--- PASS: TestReloadCommand_noTabs (0.00s) === CONT TestReloadCommand [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestReloadCommand_noTabs (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:51.413Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:51.423Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:51.452 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7cc4166d-96b1-5ac4-321c-e5918dfef94a Address:127.0.0.1:17153}]" -46:51.452 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= -46:51.453 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a.dc1 127.0.0.1 -46:51.454 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a 127.0.0.1 -2022-11-07T22:46:51.454Z [INFO] agent.router: Initializing LAN area manager -46:51.454 [INFO] TestAgent.server: Adding LAN server: server="Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a (Addr: tcp/127.0.0.1:17153) (DC: dc1)" -46:51.454 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a.dc1 area=wan -46:51.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=tcp -46:51.454 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17148 network=udp -46:51.455 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17149 network=tcp -46:51.455 [INFO] TestAgent: started state syncer -46:51.455 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17154 network=tcp -46:51.507 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:51.507 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 -46:51.515 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:51.515 [DEBUG] TestAgent.server.raft: vote granted: from=7cc4166d-96b1-5ac4-321c-e5918dfef94a term=2 tally=1 -46:51.515 [INFO] TestAgent.server.raft: election won: tally=1 -46:51.515 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" -46:51.515 [INFO] TestAgent.server: cluster leadership acquired -46:51.515 [INFO] TestAgent.server: New leader elected: payload=Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a -46:51.520 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17153 -46:51.531 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:51.547 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:51.548 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:51.548 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:51.548 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:51.548 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:51.548 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a -46:51.548 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7cc4166d-96b1-5ac4-321c-e5918dfef94a -46:51.557 [INFO] TestAgent.server: federation state anti-entropy synced -2022-11-07T22:46:51.882Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:51.882 [INFO] TestAgent: Requesting shutdown -46:51.882 [INFO] TestAgent.server: shutting down server -46:51.882 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:51.882 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:51.882 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:51.887 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:51.889Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:51.889Z [INFO] agent.router.manager: shutting down -46:51.890 [INFO] TestAgent: consul server down -46:51.890 [INFO] TestAgent: shutdown complete -46:51.890 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp -46:51.890 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp -46:51.890 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp -46:52.390 [INFO] TestAgent: Waiting for endpoints to shut down -46:52.390 [INFO] TestAgent: Endpoints down ---- PASS: TestReloadCommand (1.02s) +2021-10-05T16:34:12.927Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:12.950Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:13.037 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2d5007df-861d-2a0b-3911-ed60647710af Address:127.0.0.1:12048}]" +34:13.038 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2d5007df-861d-2a0b-3911-ed60647710af.dc1 127.0.0.1 +34:13.039 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2d5007df-861d-2a0b-3911-ed60647710af 127.0.0.1 +2021-10-05T16:34:13.039Z [INFO] agent.router: Initializing LAN area manager +34:13.039 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=udp +34:13.039 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:12048 [Follower]" leader= +34:13.047 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2d5007df-861d-2a0b-3911-ed60647710af.dc1 area=wan +34:13.047 [INFO] TestAgent.server: Adding LAN server: server="Node-2d5007df-861d-2a0b-3911-ed60647710af (Addr: tcp/127.0.0.1:12048) (DC: dc1)" +34:13.048 [INFO] TestAgent: Started DNS server: address=127.0.0.1:12043 network=tcp +34:13.050 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:12044 network=tcp +34:13.050 [INFO] TestAgent: started state syncer +34:13.050 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:12049 network=tcp +34:13.088 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:13.088 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:12048 [Candidate]" term=2 +34:13.123 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:13.123 [DEBUG] TestAgent.server.raft: vote granted: from=2d5007df-861d-2a0b-3911-ed60647710af term=2 tally=1 +34:13.123 [INFO] TestAgent.server.raft: election won: tally=1 +34:13.123 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:12048 [Leader]" +34:13.123 [INFO] TestAgent.server: cluster leadership acquired +34:13.123 [INFO] TestAgent.server: New leader elected: payload=Node-2d5007df-861d-2a0b-3911-ed60647710af +34:13.130 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:12048 +34:13.158 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:34:13.213Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:13.216 [INFO] TestAgent: Requesting shutdown +34:13.216 [INFO] TestAgent.server: shutting down server +34:13.216 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:13.216 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:13.220 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:13.220 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:13.220 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:13.220 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:13.220 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:13.220 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2d5007df-861d-2a0b-3911-ed60647710af +34:13.224 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:13.236Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:13.236Z [INFO] agent.router.manager: shutting down +34:13.236 [INFO] TestAgent: consul server down +34:13.236 [INFO] TestAgent: shutdown complete +34:13.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=tcp +34:13.236 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:12043 network=udp +34:13.236 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:12044 network=tcp +34:13.236 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-2d5007df-861d-2a0b-3911-ed60647710af 127.0.0.1 12046 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:2d5007df-861d-2a0b-3911-ed60647710af port:12048 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:12047] alive 1 5 2 2 5 4}" error="leadership lost while committing log" +34:13.737 [INFO] TestAgent: Waiting for endpoints to shut down +34:13.737 [INFO] TestAgent: Endpoints down +--- PASS: TestReloadCommand (0.89s) PASS -ok github.com/hashicorp/consul/command/reload 1.057s +ok github.com/hashicorp/consul/command/reload 1.100s === RUN TestRTTCommand_noTabs === PAUSE TestRTTCommand_noTabs === RUN TestRTTCommand_BadArgs @@ -90160,148 +90271,152 @@ === RUN TestRTTCommand_WAN === PAUSE TestRTTCommand_WAN === CONT TestRTTCommand_noTabs +--- PASS: TestRTTCommand_noTabs (0.00s) === CONT TestRTTCommand_WAN -=== CONT TestRTTCommand_LAN +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestRTTCommand_BadArgs === RUN TestRTTCommand_BadArgs/#00 -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... ---- PASS: TestRTTCommand_noTabs (0.00s) -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestRTTCommand_BadArgs/node1_node2_node3 +=== CONT TestRTTCommand_LAN === RUN TestRTTCommand_BadArgs/-wan_node1_node2 +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range === RUN TestRTTCommand_BadArgs/-wan_node1.dc1_node2 === RUN TestRTTCommand_BadArgs/-wan_node1_node2.dc1 ---- PASS: TestRTTCommand_BadArgs (0.01s) - --- PASS: TestRTTCommand_BadArgs/#00 (0.01s) +--- PASS: TestRTTCommand_BadArgs (0.00s) + --- PASS: TestRTTCommand_BadArgs/#00 (0.00s) --- PASS: TestRTTCommand_BadArgs/node1_node2_node3 (0.00s) --- PASS: TestRTTCommand_BadArgs/-wan_node1_node2 (0.00s) --- PASS: TestRTTCommand_BadArgs/-wan_node1.dc1_node2 (0.00s) --- PASS: TestRTTCommand_BadArgs/-wan_node1_node2.dc1 (0.00s) -2022-11-07T22:46:51.816Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:51.826Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:51.857 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3fa0b75a-3429-20b3-f499-d89a4f971b43 Address:127.0.0.1:16132}]" -46:51.857 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:16132 [Follower]" leader= -46:51.858 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3fa0b75a-3429-20b3-f499-d89a4f971b43.dc1 127.0.0.1 -46:51.858 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3fa0b75a-3429-20b3-f499-d89a4f971b43 127.0.0.1 -2022-11-07T22:46:51.858Z [INFO] agent.router: Initializing LAN area manager -46:51.858 [INFO] TestAgent.server: Adding LAN server: server="Node-3fa0b75a-3429-20b3-f499-d89a4f971b43 (Addr: tcp/127.0.0.1:16132) (DC: dc1)" -46:51.858 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3fa0b75a-3429-20b3-f499-d89a4f971b43.dc1 area=wan -46:51.858 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=tcp -46:51.858 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16127 network=udp -46:51.859 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:16128 network=tcp -46:51.859 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:16133 network=tcp -46:51.859 [INFO] TestAgent: started state syncer -46:51.917 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:51.917 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:16132 [Candidate]" term=2 -46:51.930 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:51.930 [DEBUG] TestAgent.server.raft: vote granted: from=3fa0b75a-3429-20b3-f499-d89a4f971b43 term=2 tally=1 -46:51.930 [INFO] TestAgent.server.raft: election won: tally=1 -46:51.930 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:16132 [Leader]" -46:51.930 [INFO] TestAgent.server: cluster leadership acquired -46:51.931 [INFO] TestAgent.server: New leader elected: payload=Node-3fa0b75a-3429-20b3-f499-d89a4f971b43 -46:51.939 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:16132 -46:51.950 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:51.969 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:51.969 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:51.969 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:51.969 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:51.969 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:51.969 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3fa0b75a-3429-20b3-f499-d89a4f971b43 -46:51.969 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3fa0b75a-3429-20b3-f499-d89a4f971b43 -46:51.974 [INFO] TestAgent.server: federation state anti-entropy synced -46:52.001 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:35018 latency=986.715µs -46:52.004 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:35020 latency=1.527971ms -46:52.008 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:35020 latency=145.501µs -46:52.009 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:35022 latency=115.279µs -46:52.009 [INFO] TestAgent: Requesting shutdown -46:52.010 [INFO] TestAgent.server: shutting down server -46:52.010 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:52.010 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:52.010 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:52.010 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:52.010 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:52.010 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:52.010 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:52.010 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:52.010 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:52.010 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:52.010 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:52.013 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:52.016Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:52.016Z [INFO] agent.router.manager: shutting down -46:52.016 [INFO] TestAgent: consul server down -46:52.016 [INFO] TestAgent: shutdown complete -46:52.016 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=tcp -46:52.016 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16127 network=udp -46:52.017 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:16128 network=tcp -46:52.517 [INFO] TestAgent: Waiting for endpoints to shut down -46:52.517 [INFO] TestAgent: Endpoints down ---- PASS: TestRTTCommand_WAN (0.74s) -2022-11-07T22:46:51.817Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:51.828Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:51.859 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b0440fcd-250d-3157-44f0-9e4c03105fc9 Address:127.0.0.1:16139}]" -46:51.859 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:16139 [Follower]" leader= -46:51.859 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b0440fcd-250d-3157-44f0-9e4c03105fc9.dc1 127.0.0.1 -46:51.860 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b0440fcd-250d-3157-44f0-9e4c03105fc9 127.0.0.1 -2022-11-07T22:46:51.860Z [INFO] agent.router: Initializing LAN area manager -46:51.860 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b0440fcd-250d-3157-44f0-9e4c03105fc9.dc1 area=wan -46:51.860 [INFO] TestAgent.server: Adding LAN server: server="Node-b0440fcd-250d-3157-44f0-9e4c03105fc9 (Addr: tcp/127.0.0.1:16139) (DC: dc1)" -46:51.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16134 network=udp -46:51.860 [INFO] TestAgent: Started DNS server: address=127.0.0.1:16134 network=tcp -46:51.860 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:16135 network=tcp -46:51.860 [INFO] TestAgent: started state syncer -46:51.860 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:16140 network=tcp -46:51.896 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:51.896 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:16139 [Candidate]" term=2 -46:51.904 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:51.904 [DEBUG] TestAgent.server.raft: vote granted: from=b0440fcd-250d-3157-44f0-9e4c03105fc9 term=2 tally=1 -46:51.904 [INFO] TestAgent.server.raft: election won: tally=1 -46:51.904 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:16139 [Leader]" -46:51.904 [INFO] TestAgent.server: cluster leadership acquired -46:51.904 [INFO] TestAgent.server: New leader elected: payload=Node-b0440fcd-250d-3157-44f0-9e4c03105fc9 -46:51.909 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:16139 -46:51.922 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:51.945 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:51.945 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:51.945 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:51.945 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:51.945 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:51.945 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b0440fcd-250d-3157-44f0-9e4c03105fc9 -46:51.945 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b0440fcd-250d-3157-44f0-9e4c03105fc9 -46:51.952 [INFO] TestAgent.server: federation state anti-entropy synced -46:52.187 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59664 latency=256.912µs -46:52.213 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59666 latency=184.587µs -46:52.239 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59668 latency=221.247µs -46:52.241 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:59670 latency=750.778µs -46:52.242 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59670 latency=90.963µs -46:52.243 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:59672 latency=130.748µs -46:52.243 [INFO] TestAgent: Requesting shutdown -46:52.243 [INFO] TestAgent.server: shutting down server -46:52.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:52.243 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:52.243 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:52.243 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:52.243 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:52.243 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:52.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:52.243 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:52.243 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:52.243 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:52.243 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:52.247 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:52.251Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:52.251Z [INFO] agent.router.manager: shutting down -46:52.251 [INFO] TestAgent: consul server down -46:52.251 [INFO] TestAgent: shutdown complete -46:52.251 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16134 network=tcp -46:52.251 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:16134 network=udp -46:52.251 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:16135 network=tcp -46:52.751 [INFO] TestAgent: Waiting for endpoints to shut down -46:52.751 [INFO] TestAgent: Endpoints down ---- PASS: TestRTTCommand_LAN (0.98s) +2021-10-05T16:34:13.886Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:13.913Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:13.964 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:63255a5c-81a2-ee18-987b-592e8c3f8a2b Address:127.0.0.1:27363}]" +34:13.964 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +34:13.965 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b.dc1 127.0.0.1 +34:13.970 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b 127.0.0.1 +2021-10-05T16:34:13.970Z [INFO] agent.router: Initializing LAN area manager +34:13.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp +34:13.971 [INFO] TestAgent.server: Adding LAN server: server="Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +34:13.971 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b.dc1 area=wan +34:13.971 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp +34:13.984 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp +34:13.984 [INFO] TestAgent: started state syncer +34:13.984 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp +34:14.017 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:14.017 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +34:14.031 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:14.031 [DEBUG] TestAgent.server.raft: vote granted: from=63255a5c-81a2-ee18-987b-592e8c3f8a2b term=2 tally=1 +34:14.031 [INFO] TestAgent.server.raft: election won: tally=1 +34:14.031 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +34:14.032 [INFO] TestAgent.server: cluster leadership acquired +34:14.032 [INFO] TestAgent.server: New leader elected: payload=Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b +34:14.040 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +34:14.082 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:14.148 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:14.148 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:14.148 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:14.148 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:14.148 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:14.148 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b +34:14.148 [INFO] TestAgent.server: member joined, marking health alive: member=Node-63255a5c-81a2-ee18-987b-592e8c3f8a2b +34:14.172 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:14.211 [INFO] TestAgent: Synced node info +34:14.211 [DEBUG] TestAgent: Node info in sync +34:14.211 [INFO] TestAgent.server: federation state anti-entropy synced +34:14.258 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:60354 latency=1.511142ms +34:14.269 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:60356 latency=4.772639ms +34:14.290 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:60356 latency=8.171123ms +34:14.305 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/datacenters from=127.0.0.1:60360 latency=11.877683ms +34:14.305 [INFO] TestAgent: Requesting shutdown +34:14.306 [INFO] TestAgent.server: shutting down server +34:14.306 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:14.306 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:14.306 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:14.306 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:14.306 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:14.306 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:14.306 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:14.306 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:14.306 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:14.306 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:14.311 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:14.320Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:14.320Z [INFO] agent.router.manager: shutting down +34:14.320 [INFO] TestAgent: consul server down +34:14.320 [INFO] TestAgent: shutdown complete +34:14.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +34:14.320 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +34:14.320 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +34:14.821 [INFO] TestAgent: Waiting for endpoints to shut down +34:14.821 [INFO] TestAgent: Endpoints down +--- PASS: TestRTTCommand_WAN (1.00s) +2021-10-05T16:34:13.888Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:13.913Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:13.949 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:d9004e24-52dc-f31e-6cf7-1e1550dca682 Address:127.0.0.1:27370}]" +34:13.950 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-d9004e24-52dc-f31e-6cf7-1e1550dca682.dc1 127.0.0.1 +34:13.951 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-d9004e24-52dc-f31e-6cf7-1e1550dca682 127.0.0.1 +2021-10-05T16:34:13.951Z [INFO] agent.router: Initializing LAN area manager +34:13.951 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=udp +34:13.951 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27370 [Follower]" leader= +34:13.952 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-d9004e24-52dc-f31e-6cf7-1e1550dca682.dc1 area=wan +34:13.952 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=tcp +34:13.953 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27366 network=tcp +34:13.953 [INFO] TestAgent: started state syncer +34:13.954 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27371 network=tcp +34:13.954 [INFO] TestAgent.server: Adding LAN server: server="Node-d9004e24-52dc-f31e-6cf7-1e1550dca682 (Addr: tcp/127.0.0.1:27370) (DC: dc1)" +34:13.989 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:13.989 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27370 [Candidate]" term=2 +34:13.996 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:13.996 [DEBUG] TestAgent.server.raft: vote granted: from=d9004e24-52dc-f31e-6cf7-1e1550dca682 term=2 tally=1 +34:13.996 [INFO] TestAgent.server.raft: election won: tally=1 +34:13.996 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27370 [Leader]" +34:13.997 [INFO] TestAgent.server: cluster leadership acquired +34:13.997 [INFO] TestAgent.server: New leader elected: payload=Node-d9004e24-52dc-f31e-6cf7-1e1550dca682 +34:14.001 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27370 +34:14.013 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:14.039 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:14.039 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:14.039 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:14.039 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:14.039 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:14.039 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-d9004e24-52dc-f31e-6cf7-1e1550dca682 +34:14.039 [INFO] TestAgent.server: member joined, marking health alive: member=Node-d9004e24-52dc-f31e-6cf7-1e1550dca682 +34:14.068 [INFO] TestAgent.server: federation state anti-entropy synced +34:14.144 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:14.155 [INFO] TestAgent: Synced node info +34:14.155 [DEBUG] TestAgent: Node info in sync +34:14.285 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36644 latency=391.796µs +34:14.316 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36648 latency=285.553µs +34:14.348 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36650 latency=260.834µs +34:14.351 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:36652 latency=1.268661ms +34:14.358 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36652 latency=288.419µs +34:14.364 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/coordinate/nodes from=127.0.0.1:36654 latency=249.205µs +34:14.368 [INFO] TestAgent: Requesting shutdown +34:14.368 [INFO] TestAgent.server: shutting down server +34:14.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:14.368 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:14.368 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:14.368 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:14.368 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:14.368 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:14.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:14.368 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:14.368 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:14.368 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:14.380 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:14.388Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:14.388Z [INFO] agent.router.manager: shutting down +34:14.388 [INFO] TestAgent: consul server down +34:14.388 [INFO] TestAgent: shutdown complete +34:14.388 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=tcp +34:14.388 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=udp +34:14.388 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27366 network=tcp +34:14.892 [INFO] TestAgent: Waiting for endpoints to shut down +34:14.892 [INFO] TestAgent: Endpoints down +--- PASS: TestRTTCommand_LAN (1.06s) PASS -ok github.com/hashicorp/consul/command/rtt 1.008s +ok github.com/hashicorp/consul/command/rtt 1.216s === RUN TestDevModeHasNoServices === PAUSE TestDevModeHasNoServices === RUN TestStructsToAgentService @@ -90310,6 +90425,7 @@ === PAUSE TestCommand_noTabs === CONT TestDevModeHasNoServices === CONT TestCommand_noTabs +--- PASS: TestCommand_noTabs (0.00s) === CONT TestStructsToAgentService === RUN TestStructsToAgentService/Basic_service_with_port === PAUSE TestStructsToAgentService/Basic_service_with_port @@ -90319,27 +90435,26 @@ === PAUSE TestStructsToAgentService/Service_with_an_unnamed_check === RUN TestStructsToAgentService/Service_with_a_zero-value_check === PAUSE TestStructsToAgentService/Service_with_a_zero-value_check ---- PASS: TestCommand_noTabs (0.00s) === RUN TestStructsToAgentService/Service_with_checks === PAUSE TestStructsToAgentService/Service_with_checks === RUN TestStructsToAgentService/Proxy_service === PAUSE TestStructsToAgentService/Proxy_service === CONT TestStructsToAgentService/Basic_service_with_port -=== CONT TestStructsToAgentService/Service_with_a_zero-value_check -=== CONT TestStructsToAgentService/Service_with_an_unnamed_check === CONT TestStructsToAgentService/Proxy_service === CONT TestStructsToAgentService/Service_with_checks +=== CONT TestStructsToAgentService/Service_with_a_zero-value_check +=== CONT TestStructsToAgentService/Service_with_an_unnamed_check === CONT TestStructsToAgentService/Service_with_a_check --- PASS: TestStructsToAgentService (0.00s) - --- PASS: TestStructsToAgentService/Service_with_a_zero-value_check (0.00s) - --- PASS: TestStructsToAgentService/Service_with_an_unnamed_check (0.00s) --- PASS: TestStructsToAgentService/Basic_service_with_port (0.00s) - --- PASS: TestStructsToAgentService/Service_with_a_check (0.00s) - --- PASS: TestStructsToAgentService/Service_with_checks (0.00s) --- PASS: TestStructsToAgentService/Proxy_service (0.00s) ---- PASS: TestDevModeHasNoServices (0.01s) + --- PASS: TestStructsToAgentService/Service_with_checks (0.00s) + --- PASS: TestStructsToAgentService/Service_with_a_zero-value_check (0.00s) + --- PASS: TestStructsToAgentService/Service_with_a_check (0.01s) + --- PASS: TestStructsToAgentService/Service_with_an_unnamed_check (0.01s) +--- PASS: TestDevModeHasNoServices (0.04s) PASS -ok github.com/hashicorp/consul/command/services 0.027s +ok github.com/hashicorp/consul/command/services 0.139s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -90351,228 +90466,231 @@ === RUN TestCommand_Flag === PAUSE TestCommand_Flag === CONT TestCommand_noTabs -=== CONT TestCommand_File_nameOnly +--- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand_Flag -=== CONT TestCommand_Validation [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestCommand_File_id -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCommand_noTabs (0.00s) +=== CONT TestCommand_File_nameOnly +=== CONT TestCommand_Validation === RUN TestCommand_Validation/no_args_or_id === RUN TestCommand_Validation/args_and_-id ---- PASS: TestCommand_Validation (0.00s) +--- PASS: TestCommand_Validation (0.01s) --- PASS: TestCommand_Validation/no_args_or_id (0.00s) --- PASS: TestCommand_Validation/args_and_-id (0.00s) -2022-11-07T22:46:54.198Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:54.215Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:54.246 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f Address:127.0.0.1:29419}]" -46:54.246 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29419 [Follower]" leader= -46:54.247 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f.dc1 127.0.0.1 -46:54.247 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f 127.0.0.1 -2022-11-07T22:46:54.247Z [INFO] agent.router: Initializing LAN area manager -46:54.247 [INFO] TestAgent.server: Adding LAN server: server="Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f (Addr: tcp/127.0.0.1:29419) (DC: dc1)" -46:54.247 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f.dc1 area=wan -46:54.248 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=udp -46:54.248 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29414 network=tcp -46:54.248 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29415 network=tcp -46:54.248 [INFO] TestAgent: started state syncer -46:54.248 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29420 network=tcp -46:54.304 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:54.304 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29419 [Candidate]" term=2 -46:54.316 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:54.316 [DEBUG] TestAgent.server.raft: vote granted: from=7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f term=2 tally=1 -46:54.316 [INFO] TestAgent.server.raft: election won: tally=1 -46:54.316 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29419 [Leader]" -46:54.316 [INFO] TestAgent.server: cluster leadership acquired -46:54.316 [INFO] TestAgent.server: New leader elected: payload=Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f -46:54.324 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29419 -46:54.336 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:54.354 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:54.354 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:54.354 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:54.354 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:54.354 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:54.354 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f -46:54.354 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7b0f21f3-ad7d-ba9c-d561-46906a6d3f1f -46:54.363 [INFO] TestAgent.server: federation state anti-entropy synced -46:54.391 [INFO] TestAgent: Synced node info -46:54.393 [INFO] TestAgent: Synced service: service=web -46:54.393 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59310 latency=9.582751ms -46:54.397 [DEBUG] TestAgent: Node info in sync -46:54.397 [DEBUG] TestAgent: Service in sync: service=web -46:54.402 [INFO] TestAgent: Synced service: service=db -46:54.402 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:59310 latency=8.261593ms -46:54.412 [DEBUG] TestAgent: removed service: service=web -46:54.412 [DEBUG] TestAgent: Node info in sync -46:54.415 [INFO] TestAgent: Deregistered service: service=web -46:54.415 [DEBUG] TestAgent: Service in sync: service=db -46:54.415 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:59312 latency=3.085305ms -46:54.416 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:59310 latency=238.52µs -46:54.416 [INFO] TestAgent: Requesting shutdown -46:54.416 [INFO] TestAgent.server: shutting down server -46:54.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:54.416 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:54.416 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:54.416 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:54.416 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:54.416 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:54.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:54.416 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:54.416 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:54.416 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:54.416 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:54.419 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:54.423Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:54.423Z [INFO] agent.router.manager: shutting down -46:54.423 [INFO] TestAgent: consul server down -46:54.423 [INFO] TestAgent: shutdown complete -46:54.423 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=tcp -46:54.423 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29414 network=udp -46:54.423 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29415 network=tcp -46:54.923 [INFO] TestAgent: Waiting for endpoints to shut down -46:54.923 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_File_id (0.76s) -2022-11-07T22:46:54.197Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:54.207Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:54.241 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:f5fcb2ea-23b1-ccf9-e024-0402979c18c3 Address:127.0.0.1:29412}]" -46:54.241 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= -46:54.242 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3.dc1 127.0.0.1 -46:54.244 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3 127.0.0.1 -2022-11-07T22:46:54.244Z [INFO] agent.router: Initializing LAN area manager -46:54.244 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3.dc1 area=wan -46:54.244 [INFO] TestAgent.server: Adding LAN server: server="Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3 (Addr: tcp/127.0.0.1:29412) (DC: dc1)" -46:54.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp -46:54.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp -46:54.244 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp -46:54.245 [INFO] TestAgent: started state syncer -46:54.245 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp -46:54.286 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:54.286 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 -46:54.296 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:54.296 [DEBUG] TestAgent.server.raft: vote granted: from=f5fcb2ea-23b1-ccf9-e024-0402979c18c3 term=2 tally=1 -46:54.296 [INFO] TestAgent.server.raft: election won: tally=1 -46:54.296 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" -46:54.296 [INFO] TestAgent.server: cluster leadership acquired -46:54.296 [INFO] TestAgent.server: New leader elected: payload=Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3 -46:54.303 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 -46:54.317 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:54.338 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:54.338 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:54.338 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:54.338 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:54.338 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:54.338 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3 -46:54.338 [INFO] TestAgent.server: member joined, marking health alive: member=Node-f5fcb2ea-23b1-ccf9-e024-0402979c18c3 -46:54.347 [INFO] TestAgent.server: federation state anti-entropy synced -46:54.499 [INFO] TestAgent: Synced node info -46:54.502 [INFO] TestAgent: Synced service: service=web -46:54.502 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:34388 latency=9.570439ms -46:54.505 [DEBUG] TestAgent: Node info in sync -46:54.505 [DEBUG] TestAgent: Service in sync: service=web -46:54.510 [INFO] TestAgent: Synced service: service=db -46:54.510 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:34388 latency=7.617269ms -46:54.523 [DEBUG] TestAgent: removed service: service=web -46:54.523 [DEBUG] TestAgent: Node info in sync -46:54.526 [INFO] TestAgent: Deregistered service: service=web -46:54.526 [DEBUG] TestAgent: Service in sync: service=db -46:54.526 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:34390 latency=3.190873ms -46:54.527 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:34388 latency=751.583µs -46:54.527 [INFO] TestAgent: Requesting shutdown -46:54.527 [INFO] TestAgent.server: shutting down server -46:54.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:54.527 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:54.527 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:54.527 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:54.527 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:54.527 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:54.527 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:54.528 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:54.528 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:54.528 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:54.528 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:54.531 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:54.534Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:54.534Z [INFO] agent.router.manager: shutting down -46:54.534 [INFO] TestAgent: consul server down -46:54.534 [INFO] TestAgent: shutdown complete -46:54.534 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp -46:54.534 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp -46:54.534 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp -46:55.034 [INFO] TestAgent: Waiting for endpoints to shut down -46:55.035 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_File_nameOnly (0.87s) -2022-11-07T22:46:54.199Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:54.212Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:54.243 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:67f429d8-b5a5-a678-401a-7a4a6fa3d1fa Address:127.0.0.1:29405}]" -46:54.243 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= -46:54.244 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa.dc1 127.0.0.1 -46:54.244 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa 127.0.0.1 -2022-11-07T22:46:54.244Z [INFO] agent.router: Initializing LAN area manager -46:54.244 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa.dc1 area=wan -46:54.244 [INFO] TestAgent.server: Adding LAN server: server="Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa (Addr: tcp/127.0.0.1:29405) (DC: dc1)" -46:54.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp -46:54.244 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp -46:54.245 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp -46:54.245 [INFO] TestAgent: started state syncer -46:54.245 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp -46:54.282 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:54.282 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 -46:54.292 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:54.292 [DEBUG] TestAgent.server.raft: vote granted: from=67f429d8-b5a5-a678-401a-7a4a6fa3d1fa term=2 tally=1 -46:54.292 [INFO] TestAgent.server.raft: election won: tally=1 -46:54.292 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" -46:54.292 [INFO] TestAgent.server: cluster leadership acquired -46:54.292 [INFO] TestAgent.server: New leader elected: payload=Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa -46:54.298 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 -46:54.310 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:54.331 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:54.331 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:54.331 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:54.331 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:54.331 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:54.331 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa -46:54.331 [INFO] TestAgent.server: member joined, marking health alive: member=Node-67f429d8-b5a5-a678-401a-7a4a6fa3d1fa -46:54.343 [INFO] TestAgent.server: federation state anti-entropy synced -46:54.554 [INFO] TestAgent: Synced node info -46:54.556 [INFO] TestAgent: Synced service: service=web -46:54.556 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:55580 latency=9.070599ms -46:54.561 [DEBUG] TestAgent: Node info in sync -46:54.561 [DEBUG] TestAgent: Service in sync: service=web -46:54.565 [INFO] TestAgent: Synced service: service=db -46:54.565 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:55580 latency=8.473037ms -46:54.566 [DEBUG] TestAgent: removed service: service=web -46:54.566 [DEBUG] TestAgent: Node info in sync -46:54.569 [INFO] TestAgent: Deregistered service: service=web -46:54.569 [DEBUG] TestAgent: Service in sync: service=db -46:54.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:55582 latency=2.892683ms -46:54.569 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:55580 latency=195.776µs -46:54.570 [INFO] TestAgent: Requesting shutdown -46:54.570 [INFO] TestAgent.server: shutting down server -46:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:54.570 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:54.570 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:54.570 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:54.570 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:54.570 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:54.573 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:54.576Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:54.576Z [INFO] agent.router.manager: shutting down -46:54.576 [INFO] TestAgent: consul server down -46:54.576 [INFO] TestAgent: shutdown complete -46:54.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp -46:54.576 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp -46:54.576 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp -46:55.077 [INFO] TestAgent: Waiting for endpoints to shut down -46:55.077 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_Flag (0.91s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:34:20.579Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:20.623Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:20.666 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2fa4d284-ed57-fe4b-254a-3467d475623d Address:127.0.0.1:20223}]" +34:20.667 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2fa4d284-ed57-fe4b-254a-3467d475623d.dc1 127.0.0.1 +34:20.668 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20223 [Follower]" leader= +34:20.668 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2fa4d284-ed57-fe4b-254a-3467d475623d 127.0.0.1 +2021-10-05T16:34:20.668Z [INFO] agent.router: Initializing LAN area manager +34:20.668 [INFO] TestAgent.server: Adding LAN server: server="Node-2fa4d284-ed57-fe4b-254a-3467d475623d (Addr: tcp/127.0.0.1:20223) (DC: dc1)" +34:20.668 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2fa4d284-ed57-fe4b-254a-3467d475623d.dc1 area=wan +34:20.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=udp +34:20.668 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20218 network=tcp +34:20.669 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20219 network=tcp +34:20.669 [INFO] TestAgent: started state syncer +34:20.669 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20224 network=tcp +34:20.704 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:20.704 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20223 [Candidate]" term=2 +34:20.714 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:20.714 [DEBUG] TestAgent.server.raft: vote granted: from=2fa4d284-ed57-fe4b-254a-3467d475623d term=2 tally=1 +34:20.714 [INFO] TestAgent.server.raft: election won: tally=1 +34:20.714 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20223 [Leader]" +34:20.715 [INFO] TestAgent.server: cluster leadership acquired +34:20.715 [INFO] TestAgent.server: New leader elected: payload=Node-2fa4d284-ed57-fe4b-254a-3467d475623d +34:20.723 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20223 +34:20.804 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:20.836 [INFO] TestAgent: Synced node info +34:20.872 [INFO] TestAgent: Synced service: service=web +34:20.872 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:47584 latency=56.051047ms +34:20.891 [DEBUG] TestAgent: Node info in sync +34:20.898 [INFO] TestAgent: Synced service: service=db +34:20.898 [DEBUG] TestAgent: Service in sync: service=web +34:20.898 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:47584 latency=17.945768ms +34:20.898 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:20.898 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:20.898 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:20.898 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:20.898 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:20.898 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2fa4d284-ed57-fe4b-254a-3467d475623d +34:20.898 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2fa4d284-ed57-fe4b-254a-3467d475623d +34:20.932 [DEBUG] TestAgent: removed service: service=web +34:20.932 [DEBUG] TestAgent: Node info in sync +34:20.955 [INFO] TestAgent: Deregistered service: service=web +34:20.955 [DEBUG] TestAgent: Service in sync: service=db +34:20.955 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:47588 latency=31.642773ms +34:20.955 [INFO] TestAgent.server: federation state anti-entropy synced +34:20.960 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:47584 latency=3.294304ms +34:20.962 [INFO] TestAgent: Requesting shutdown +34:20.963 [INFO] TestAgent.server: shutting down server +34:20.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:20.963 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:20.963 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:20.963 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:20.963 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:20.963 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:20.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:20.963 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:20.963 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:20.963 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:20.963 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:20.968 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:20.976Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:20.976Z [INFO] agent.router.manager: shutting down +34:20.976 [INFO] TestAgent: consul server down +34:20.976 [INFO] TestAgent: shutdown complete +34:20.976 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=tcp +34:20.976 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20218 network=udp +34:20.976 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20219 network=tcp +34:21.477 [INFO] TestAgent: Waiting for endpoints to shut down +34:21.477 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_File_id (1.02s) +2021-10-05T16:34:20.544Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:20.570Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:20.724 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9cf95e72-480a-a2c6-6e0b-986c57d57156 Address:127.0.0.1:20216}]" +34:20.729 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9cf95e72-480a-a2c6-6e0b-986c57d57156.dc1 127.0.0.1 +34:20.730 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9cf95e72-480a-a2c6-6e0b-986c57d57156 127.0.0.1 +2021-10-05T16:34:20.730Z [INFO] agent.router: Initializing LAN area manager +34:20.735 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20216 [Follower]" leader= +34:20.737 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=tcp +34:20.752 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9cf95e72-480a-a2c6-6e0b-986c57d57156.dc1 area=wan +34:20.752 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20211 network=udp +34:20.752 [INFO] TestAgent.server: Adding LAN server: server="Node-9cf95e72-480a-a2c6-6e0b-986c57d57156 (Addr: tcp/127.0.0.1:20216) (DC: dc1)" +34:20.784 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20212 network=tcp +34:20.784 [INFO] TestAgent: started state syncer +34:20.785 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20217 network=tcp +34:20.791 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:20.791 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20216 [Candidate]" term=2 +34:20.806 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:20.806 [DEBUG] TestAgent.server.raft: vote granted: from=9cf95e72-480a-a2c6-6e0b-986c57d57156 term=2 tally=1 +34:20.806 [INFO] TestAgent.server.raft: election won: tally=1 +34:20.806 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20216 [Leader]" +34:20.808 [INFO] TestAgent.server: cluster leadership acquired +34:20.808 [INFO] TestAgent.server: New leader elected: payload=Node-9cf95e72-480a-a2c6-6e0b-986c57d57156 +34:20.826 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20216 +34:20.867 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:20.932 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:20.932 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:20.932 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:20.932 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:20.932 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:20.932 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9cf95e72-480a-a2c6-6e0b-986c57d57156 +34:20.932 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9cf95e72-480a-a2c6-6e0b-986c57d57156 +34:20.940 [INFO] TestAgent: Synced node info +34:20.940 [DEBUG] TestAgent: Node info in sync +34:20.956 [INFO] TestAgent.server: federation state anti-entropy synced +34:20.976 [DEBUG] TestAgent: Node info in sync +34:20.992 [INFO] TestAgent: Synced service: service=web +34:20.992 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:34796 latency=23.673098ms +34:21.008 [DEBUG] TestAgent: Node info in sync +34:21.008 [DEBUG] TestAgent: Service in sync: service=web +34:21.024 [INFO] TestAgent: Synced service: service=db +34:21.024 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:34796 latency=27.847741ms +34:21.029 [DEBUG] TestAgent: removed service: service=web +34:21.029 [DEBUG] TestAgent: Node info in sync +34:21.039 [INFO] TestAgent: Deregistered service: service=web +34:21.039 [DEBUG] TestAgent: Service in sync: service=db +34:21.039 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:34798 latency=10.055566ms +34:21.040 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:34796 latency=174.687µs +34:21.041 [INFO] TestAgent: Requesting shutdown +34:21.041 [INFO] TestAgent.server: shutting down server +34:21.041 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:21.041 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:21.041 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:21.041 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:21.041 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:21.041 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:21.041 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:21.041 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:21.041 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:21.041 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:21.044 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:21.050Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:21.051Z [INFO] agent.router.manager: shutting down +34:21.051 [INFO] TestAgent: consul server down +34:21.051 [INFO] TestAgent: shutdown complete +34:21.051 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=tcp +34:21.051 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20211 network=udp +34:21.051 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20212 network=tcp +34:21.552 [INFO] TestAgent: Waiting for endpoints to shut down +34:21.552 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_Flag (1.10s) +2021-10-05T16:34:20.613Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:20.638Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:20.674 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dd8f845f-bdfa-9af0-8321-a38d382e4010 Address:127.0.0.1:20230}]" +34:20.674 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dd8f845f-bdfa-9af0-8321-a38d382e4010.dc1 127.0.0.1 +34:20.675 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dd8f845f-bdfa-9af0-8321-a38d382e4010 127.0.0.1 +2021-10-05T16:34:20.675Z [INFO] agent.router: Initializing LAN area manager +34:20.675 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=udp +34:20.675 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:20230 [Follower]" leader= +34:20.676 [INFO] TestAgent.server: Adding LAN server: server="Node-dd8f845f-bdfa-9af0-8321-a38d382e4010 (Addr: tcp/127.0.0.1:20230) (DC: dc1)" +34:20.676 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dd8f845f-bdfa-9af0-8321-a38d382e4010.dc1 area=wan +34:20.676 [INFO] TestAgent: Started DNS server: address=127.0.0.1:20225 network=tcp +34:20.677 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:20226 network=tcp +34:20.735 [INFO] TestAgent: started state syncer +34:20.735 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:20231 network=tcp +34:20.778 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:20.778 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:20230 [Candidate]" term=2 +34:20.799 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:20.799 [DEBUG] TestAgent.server.raft: vote granted: from=dd8f845f-bdfa-9af0-8321-a38d382e4010 term=2 tally=1 +34:20.799 [INFO] TestAgent.server.raft: election won: tally=1 +34:20.799 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:20230 [Leader]" +34:20.799 [INFO] TestAgent.server: cluster leadership acquired +34:20.799 [INFO] TestAgent.server: New leader elected: payload=Node-dd8f845f-bdfa-9af0-8321-a38d382e4010 +34:20.811 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:20230 +34:20.838 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:20.900 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:20.900 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:20.900 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:20.900 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:20.900 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:20.900 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dd8f845f-bdfa-9af0-8321-a38d382e4010 +34:20.900 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dd8f845f-bdfa-9af0-8321-a38d382e4010 +34:20.928 [INFO] TestAgent.server: federation state anti-entropy synced +34:21.020 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:21.026 [INFO] TestAgent: Synced node info +34:21.026 [DEBUG] TestAgent: Node info in sync +34:21.160 [DEBUG] TestAgent: Node info in sync +34:21.176 [INFO] TestAgent: Synced service: service=web +34:21.176 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54880 latency=27.709114ms +34:21.188 [DEBUG] TestAgent: Node info in sync +34:21.188 [DEBUG] TestAgent: Service in sync: service=web +34:21.208 [INFO] TestAgent: Synced service: service=db +34:21.208 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54880 latency=27.995517ms +34:21.230 [DEBUG] TestAgent: removed service: service=web +34:21.230 [DEBUG] TestAgent: Node info in sync +34:21.237 [INFO] TestAgent: Deregistered service: service=web +34:21.237 [DEBUG] TestAgent: Service in sync: service=db +34:21.237 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/deregister/web from=127.0.0.1:54882 latency=6.642752ms +34:21.241 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:54880 latency=315.643µs +34:21.245 [INFO] TestAgent: Requesting shutdown +34:21.252 [INFO] TestAgent.server: shutting down server +34:21.252 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:21.252 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:21.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:21.252 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:21.252 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:21.252 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:21.253 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:21.253 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:21.253 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:21.253 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:21.262 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:21.276Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:21.276Z [INFO] agent.router.manager: shutting down +34:21.276 [INFO] TestAgent: consul server down +34:21.276 [INFO] TestAgent: shutdown complete +34:21.276 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=tcp +34:21.276 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:20225 network=udp +34:21.276 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:20226 network=tcp +34:21.780 [INFO] TestAgent: Waiting for endpoints to shut down +34:21.780 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_File_nameOnly (1.32s) PASS -ok github.com/hashicorp/consul/command/services/deregister 0.942s +ok github.com/hashicorp/consul/command/services/deregister 1.515s === RUN TestCommand_noTabs === PAUSE TestCommand_noTabs === RUN TestCommand_Validation @@ -90586,270 +90704,274 @@ === RUN TestCommand_FileWithUnnamedCheck === PAUSE TestCommand_FileWithUnnamedCheck === CONT TestCommand_noTabs -=== CONT TestCommand_Flags_TaggedAddresses +--- PASS: TestCommand_noTabs (0.00s) === CONT TestCommand_FileWithUnnamedCheck [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestCommand_File === CONT TestCommand_Flags +=== CONT TestCommand_File +=== CONT TestCommand_Flags_TaggedAddresses === CONT TestCommand_Validation -[INFO] freeport: detected ephemeral port range of [32768, 60999] -[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestCommand_noTabs (0.00s) === RUN TestCommand_Validation/no_args_or_id === RUN TestCommand_Validation/args_and_-name --- PASS: TestCommand_Validation (0.00s) --- PASS: TestCommand_Validation/no_args_or_id (0.00s) --- PASS: TestCommand_Validation/args_and_-name (0.00s) -2022-11-07T22:46:56.168Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.180Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.214 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 Address:127.0.0.1:28398}]" -46:56.214 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28398 [Follower]" leader= -46:56.215 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6.dc1 127.0.0.1 -46:56.215 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 127.0.0.1 -2022-11-07T22:46:56.215Z [INFO] agent.router: Initializing LAN area manager -46:56.215 [INFO] TestAgent.server: Adding LAN server: server="Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 (Addr: tcp/127.0.0.1:28398) (DC: dc1)" -46:56.215 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6.dc1 area=wan -46:56.215 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28393 network=udp -46:56.215 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28393 network=tcp -46:56.216 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28394 network=tcp -46:56.216 [INFO] TestAgent: started state syncer -46:56.216 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28399 network=tcp -46:56.279 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.279 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28398 [Candidate]" term=2 -46:56.292 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.292 [DEBUG] TestAgent.server.raft: vote granted: from=0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 term=2 tally=1 -46:56.292 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.292 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28398 [Leader]" -46:56.292 [INFO] TestAgent.server: cluster leadership acquired -46:56.292 [INFO] TestAgent.server: New leader elected: payload=Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 -46:56.302 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28398 -46:56.333 [INFO] TestAgent: Synced node info -46:56.333 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.338 [INFO] TestAgent: Synced service: service=web -46:56.338 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:45330 latency=32.413311ms -46:56.341 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:45334 latency=324.069µs -46:56.342 [INFO] TestAgent: Requesting shutdown -46:56.342 [INFO] TestAgent.server: shutting down server -46:56.342 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:56.342 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:56.347 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -46:56.353 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.353 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.353 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.353 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.353 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.353 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 -46:56.353 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 -2022-11-07T22:46:56.354Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:56.354Z [INFO] agent.router.manager: shutting down -46:56.353 [ERROR] TestAgent.server: failed to reconcile member: member="{Node-0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 127.0.0.1 28396 map[acls:0 bootstrap:1 build:1.8.7: dc:dc1 ft_fs:1 id:0d6eff7a-1a3e-f5b1-b5cc-dfb6a87d00c6 port:28398 raft_vsn:3 role:consul segment: vsn:2 vsn_max:3 vsn_min:2 wan_join_port:28397] alive 1 5 2 2 5 4}" error="raft is already shutdown" -46:56.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:56.354 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:56.354 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:56.354 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:56.354 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:56.354 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:56.354 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:56.360 [INFO] TestAgent: consul server down -46:56.360 [INFO] TestAgent: shutdown complete -46:56.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28393 network=tcp -46:56.360 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28393 network=udp -46:56.360 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28394 network=tcp -46:56.360 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="error performing federation state anti-entropy sync: leadership lost while committing log" -46:56.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:56.860 [INFO] TestAgent: Waiting for endpoints to shut down -46:56.860 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_Flags (0.74s) -2022-11-07T22:46:56.169Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.181Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.213 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 Address:127.0.0.1:28391}]" -46:56.213 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28391 [Follower]" leader= -46:56.214 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92.dc1 127.0.0.1 -46:56.214 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 127.0.0.1 -2022-11-07T22:46:56.214Z [INFO] agent.router: Initializing LAN area manager -46:56.214 [INFO] TestAgent.server: Adding LAN server: server="Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 (Addr: tcp/127.0.0.1:28391) (DC: dc1)" -46:56.215 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92.dc1 area=wan -46:56.215 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=udp -46:56.215 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28386 network=tcp -46:56.215 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28387 network=tcp -46:56.215 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28392 network=tcp -46:56.215 [INFO] TestAgent: started state syncer -46:56.268 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.268 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28391 [Candidate]" term=2 -46:56.281 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.281 [DEBUG] TestAgent.server.raft: vote granted: from=97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 term=2 tally=1 -46:56.281 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.281 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28391 [Leader]" -46:56.282 [INFO] TestAgent.server: cluster leadership acquired -46:56.282 [INFO] TestAgent.server: New leader elected: payload=Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 -46:56.292 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28391 -46:56.311 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.337 [INFO] TestAgent: Synced node info -46:56.357 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.357 [INFO] TestAgent: Synced service: service=web -46:56.357 [DEBUG] TestAgent: Check in sync: check=service:web -46:56.357 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54722 latency=50.930612ms -46:56.357 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.357 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.357 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.357 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.357 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 -46:56.357 [INFO] TestAgent.server: member joined, marking health alive: member=Node-97ea79e7-ce96-1714-7a8e-0f60ae7bdb92 -46:56.357 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:54726 latency=160.26µs -46:56.358 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:54726 latency=426.029µs -46:56.359 [INFO] TestAgent: Requesting shutdown -46:56.359 [INFO] TestAgent.server: shutting down server -46:56.359 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:56.359 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:56.359 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:56.359 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:56.359 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:56.359 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="shutdown waiting for leader" -46:56.359 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:56.359 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:56.359 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:56.359 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:56.359 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:56.362 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:56.367Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:56.367Z [INFO] agent.router.manager: shutting down -46:56.367 [INFO] TestAgent: consul server down -46:56.367 [INFO] TestAgent: shutdown complete -46:56.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=tcp -46:56.367 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28386 network=udp -46:56.367 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28387 network=tcp -46:56.867 [INFO] TestAgent: Waiting for endpoints to shut down -46:56.867 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_FileWithUnnamedCheck (0.74s) -2022-11-07T22:46:56.167Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.181Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.216 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b0bc222-1afe-d660-df43-ba8e0d7f5f52 Address:127.0.0.1:28384}]" -46:56.216 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28384 [Follower]" leader= -46:56.219 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52.dc1 127.0.0.1 -46:56.219 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52 127.0.0.1 -2022-11-07T22:46:56.219Z [INFO] agent.router: Initializing LAN area manager -46:56.221 [INFO] TestAgent.server: Adding LAN server: server="Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52 (Addr: tcp/127.0.0.1:28384) (DC: dc1)" -46:56.221 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=udp -46:56.221 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52.dc1 area=wan -46:56.221 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28379 network=tcp -46:56.222 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28380 network=tcp -46:56.222 [INFO] TestAgent: started state syncer -46:56.222 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28385 network=tcp -46:56.267 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.267 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28384 [Candidate]" term=2 -46:56.281 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.281 [DEBUG] TestAgent.server.raft: vote granted: from=3b0bc222-1afe-d660-df43-ba8e0d7f5f52 term=2 tally=1 -46:56.281 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.281 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28384 [Leader]" -46:56.281 [INFO] TestAgent.server: cluster leadership acquired -46:56.281 [INFO] TestAgent.server: New leader elected: payload=Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52 -46:56.291 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28384 -46:56.311 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.340 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.340 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.340 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.340 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.340 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.340 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52 -46:56.340 [INFO] TestAgent.server: member joined, marking health alive: member=Node-3b0bc222-1afe-d660-df43-ba8e0d7f5f52 -46:56.353 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:56.360 [INFO] TestAgent: Synced node info -46:56.360 [INFO] TestAgent.server: federation state anti-entropy synced -46:56.365 [DEBUG] TestAgent: Node info in sync -46:56.368 [INFO] TestAgent: Synced service: service=web -46:56.368 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:39110 latency=7.420558ms -46:56.369 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:39112 latency=64.007µs -46:56.369 [INFO] TestAgent: Requesting shutdown -46:56.369 [INFO] TestAgent.server: shutting down server -46:56.369 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:56.369 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:56.369 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:56.369 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:56.369 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:56.369 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:56.369 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:56.369 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:56.369 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:56.369 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:56.372 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:56.375Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:56.375Z [INFO] agent.router.manager: shutting down -46:56.375 [INFO] TestAgent: consul server down -46:56.375 [INFO] TestAgent: shutdown complete -46:56.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=tcp -46:56.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28379 network=udp -46:56.375 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28380 network=tcp -46:56.875 [INFO] TestAgent: Waiting for endpoints to shut down -46:56.875 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_Flags_TaggedAddresses (0.75s) -2022-11-07T22:46:56.168Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.180Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.216 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a01e0979-5531-f024-9c7a-ab75e5290356 Address:127.0.0.1:28405}]" -46:56.216 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:28405 [Follower]" leader= -46:56.222 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a01e0979-5531-f024-9c7a-ab75e5290356.dc1 127.0.0.1 -46:56.222 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a01e0979-5531-f024-9c7a-ab75e5290356 127.0.0.1 -2022-11-07T22:46:56.222Z [INFO] agent.router: Initializing LAN area manager -46:56.223 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a01e0979-5531-f024-9c7a-ab75e5290356.dc1 area=wan -46:56.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28400 network=udp -46:56.223 [INFO] TestAgent: Started DNS server: address=127.0.0.1:28400 network=tcp -46:56.223 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:28401 network=tcp -46:56.223 [INFO] TestAgent.server: Adding LAN server: server="Node-a01e0979-5531-f024-9c7a-ab75e5290356 (Addr: tcp/127.0.0.1:28405) (DC: dc1)" -46:56.223 [INFO] TestAgent: started state syncer -46:56.223 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:28406 network=tcp -46:56.276 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.276 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:28405 [Candidate]" term=2 -46:56.287 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.287 [DEBUG] TestAgent.server.raft: vote granted: from=a01e0979-5531-f024-9c7a-ab75e5290356 term=2 tally=1 -46:56.287 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.287 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:28405 [Leader]" -46:56.287 [INFO] TestAgent.server: cluster leadership acquired -46:56.288 [INFO] TestAgent.server: New leader elected: payload=Node-a01e0979-5531-f024-9c7a-ab75e5290356 -46:56.297 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:28405 -46:56.314 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.342 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.342 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.342 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.342 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.342 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.342 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a01e0979-5531-f024-9c7a-ab75e5290356 -46:56.342 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a01e0979-5531-f024-9c7a-ab75e5290356 -46:56.348 [INFO] TestAgent.server: federation state anti-entropy synced -46:56.405 [INFO] TestAgent: Synced node info -46:56.408 [INFO] TestAgent: Synced service: service=web -46:56.408 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:36108 latency=9.000519ms -46:56.408 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:36110 latency=227.69µs -46:56.409 [INFO] TestAgent: Requesting shutdown -46:56.409 [INFO] TestAgent.server: shutting down server -46:56.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:56.409 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:56.409 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:56.409 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:56.409 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:56.409 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:56.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:56.409 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:56.409 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:56.409 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:56.409 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:56.412 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:56.416Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:56.416Z [INFO] agent.router.manager: shutting down -46:56.416 [INFO] TestAgent: consul server down -46:56.416 [INFO] TestAgent: shutdown complete -46:56.416 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28400 network=tcp -46:56.416 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:28400 network=udp -46:56.416 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:28401 network=tcp -46:56.916 [INFO] TestAgent: Waiting for endpoints to shut down -46:56.916 [INFO] TestAgent: Endpoints down ---- PASS: TestCommand_File (0.79s) +[INFO] freeport: detected ephemeral port range of [32768, 60999] +[INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range +2021-10-05T16:34:26.171Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.222Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.300 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:1297140b-362e-d214-28f1-7527f1d413c5 Address:127.0.0.1:30440}]" +34:26.300 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30440 [Follower]" leader= +34:26.301 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-1297140b-362e-d214-28f1-7527f1d413c5.dc1 127.0.0.1 +34:26.301 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-1297140b-362e-d214-28f1-7527f1d413c5 127.0.0.1 +2021-10-05T16:34:26.301Z [INFO] agent.router: Initializing LAN area manager +34:26.302 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30435 network=udp +34:26.302 [INFO] TestAgent.server: Adding LAN server: server="Node-1297140b-362e-d214-28f1-7527f1d413c5 (Addr: tcp/127.0.0.1:30440) (DC: dc1)" +34:26.302 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-1297140b-362e-d214-28f1-7527f1d413c5.dc1 area=wan +34:26.325 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30435 network=tcp +34:26.343 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30436 network=tcp +34:26.343 [INFO] TestAgent: started state syncer +34:26.363 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30441 network=tcp +34:26.363 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.363 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30440 [Candidate]" term=2 +34:26.394 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.394 [DEBUG] TestAgent.server.raft: vote granted: from=1297140b-362e-d214-28f1-7527f1d413c5 term=2 tally=1 +34:26.394 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.394 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30440 [Leader]" +34:26.395 [INFO] TestAgent.server: cluster leadership acquired +34:26.395 [INFO] TestAgent.server: New leader elected: payload=Node-1297140b-362e-d214-28f1-7527f1d413c5 +34:26.413 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30440 +34:26.445 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.483 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.483 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.483 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.483 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.483 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.483 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-1297140b-362e-d214-28f1-7527f1d413c5 +34:26.483 [INFO] TestAgent.server: member joined, marking health alive: member=Node-1297140b-362e-d214-28f1-7527f1d413c5 +34:26.505 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.577 [INFO] TestAgent: Synced node info +34:26.584 [INFO] TestAgent: Synced service: service=web +34:26.584 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:52090 latency=29.72219ms +34:26.585 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:26.585 [DEBUG] TestAgent: Node info in sync +34:26.599 [INFO] TestAgent: Synced service: service=web +34:26.599 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:52094 latency=12.623271ms +34:26.604 [INFO] TestAgent: Requesting shutdown +34:26.604 [INFO] TestAgent.server: shutting down server +34:26.604 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:26.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:26.604 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:26.604 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:26.604 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:26.604 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:26.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:26.604 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:26.604 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:26.604 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:26.622 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:26.633Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:26.633Z [INFO] agent.router.manager: shutting down +34:26.633 [INFO] TestAgent: consul server down +34:26.633 [INFO] TestAgent: shutdown complete +34:26.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30435 network=tcp +34:26.633 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30435 network=udp +34:26.633 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30436 network=tcp +34:27.133 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.133 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_Flags_TaggedAddresses (1.07s) +2021-10-05T16:34:26.156Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.193Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:36e654cd-02d3-82f6-8e0b-e8b061e769bb Address:127.0.0.1:30433}]" +34:26.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb.dc1 127.0.0.1 +34:26.326 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb 127.0.0.1 +2021-10-05T16:34:26.326Z [INFO] agent.router: Initializing LAN area manager +34:26.340 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=udp +34:26.340 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30433 [Follower]" leader= +34:26.341 [INFO] TestAgent.server: Adding LAN server: server="Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb (Addr: tcp/127.0.0.1:30433) (DC: dc1)" +34:26.341 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb.dc1 area=wan +34:26.341 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30428 network=tcp +34:26.342 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30429 network=tcp +34:26.342 [INFO] TestAgent: started state syncer +34:26.363 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30434 network=tcp +34:26.404 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.404 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30433 [Candidate]" term=2 +34:26.415 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.415 [DEBUG] TestAgent.server.raft: vote granted: from=36e654cd-02d3-82f6-8e0b-e8b061e769bb term=2 tally=1 +34:26.415 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.415 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30433 [Leader]" +34:26.416 [INFO] TestAgent.server: cluster leadership acquired +34:26.416 [INFO] TestAgent.server: New leader elected: payload=Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb +34:26.427 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30433 +34:26.480 [INFO] TestAgent: Synced node info +34:26.502 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.525 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.525 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.525 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.525 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.525 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.525 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb +34:26.525 [INFO] TestAgent.server: member joined, marking health alive: member=Node-36e654cd-02d3-82f6-8e0b-e8b061e769bb +34:26.552 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.587 [DEBUG] TestAgent: Node info in sync +34:26.608 [INFO] TestAgent: Synced service: service=web +34:26.608 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:38932 latency=26.803192ms +34:26.620 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:38938 latency=3.831401ms +34:26.623 [INFO] TestAgent: Requesting shutdown +34:26.623 [INFO] TestAgent.server: shutting down server +34:26.623 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:26.623 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:26.623 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:26.623 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:26.623 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:26.623 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:26.623 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:26.623 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:26.623 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:26.623 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:26.627 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:26.637Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:26.637Z [INFO] agent.router.manager: shutting down +34:26.637 [INFO] TestAgent: consul server down +34:26.637 [INFO] TestAgent: shutdown complete +34:26.637 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=tcp +34:26.637 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30428 network=udp +34:26.637 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30429 network=tcp +34:27.143 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.143 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_Flags (1.08s) +2021-10-05T16:34:26.147Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.193Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.324 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:5807623b-18a9-272d-38e1-92f9cb9fc2e7 Address:127.0.0.1:30447}]" +34:26.325 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7.dc1 127.0.0.1 +34:26.325 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7 127.0.0.1 +2021-10-05T16:34:26.325Z [INFO] agent.router: Initializing LAN area manager +34:26.326 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30442 network=udp +34:26.326 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30447 [Follower]" leader= +34:26.362 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7.dc1 area=wan +34:26.362 [INFO] TestAgent.server: Adding LAN server: server="Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7 (Addr: tcp/127.0.0.1:30447) (DC: dc1)" +34:26.362 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30442 network=tcp +34:26.362 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30443 network=tcp +34:26.362 [INFO] TestAgent: started state syncer +34:26.364 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30448 network=tcp +34:26.374 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.374 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30447 [Candidate]" term=2 +34:26.408 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.408 [DEBUG] TestAgent.server.raft: vote granted: from=5807623b-18a9-272d-38e1-92f9cb9fc2e7 term=2 tally=1 +34:26.408 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.408 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30447 [Leader]" +34:26.412 [INFO] TestAgent.server: cluster leadership acquired +34:26.412 [INFO] TestAgent.server: New leader elected: payload=Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7 +34:26.426 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30447 +34:26.446 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.508 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.508 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.508 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.508 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.508 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.508 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7 +34:26.508 [INFO] TestAgent.server: member joined, marking health alive: member=Node-5807623b-18a9-272d-38e1-92f9cb9fc2e7 +34:26.549 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.627 [INFO] TestAgent: Synced node info +34:26.638 [INFO] TestAgent: Synced service: service=web +34:26.638 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:51958 latency=41.897622ms +34:26.638 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:26.638 [DEBUG] TestAgent: Node info in sync +34:26.653 [INFO] TestAgent: Synced service: service=web +34:26.654 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:51962 latency=15.104977ms +34:26.655 [INFO] TestAgent: Requesting shutdown +34:26.655 [INFO] TestAgent.server: shutting down server +34:26.655 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:26.655 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:26.655 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:26.655 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:26.655 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:26.655 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:26.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:26.655 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:26.655 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:26.655 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:26.663 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:26.678Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:26.678Z [INFO] agent.router.manager: shutting down +34:26.678 [INFO] TestAgent: consul server down +34:26.678 [INFO] TestAgent: shutdown complete +34:26.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30442 network=tcp +34:26.678 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30442 network=udp +34:26.678 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30443 network=tcp +34:27.180 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.180 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_File (1.12s) +2021-10-05T16:34:26.175Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.213Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.268 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7146fa77-93a2-68ca-b8d5-014693546a11 Address:127.0.0.1:30426}]" +34:26.269 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:30426 [Follower]" leader= +34:26.269 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7146fa77-93a2-68ca-b8d5-014693546a11.dc1 127.0.0.1 +34:26.270 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7146fa77-93a2-68ca-b8d5-014693546a11 127.0.0.1 +2021-10-05T16:34:26.270Z [INFO] agent.router: Initializing LAN area manager +34:26.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=udp +34:26.270 [INFO] TestAgent.server: Adding LAN server: server="Node-7146fa77-93a2-68ca-b8d5-014693546a11 (Addr: tcp/127.0.0.1:30426) (DC: dc1)" +34:26.270 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7146fa77-93a2-68ca-b8d5-014693546a11.dc1 area=wan +34:26.270 [INFO] TestAgent: Started DNS server: address=127.0.0.1:30421 network=tcp +34:26.271 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:30422 network=tcp +34:26.271 [INFO] TestAgent: started state syncer +34:26.271 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:30427 network=tcp +34:26.341 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.341 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:30426 [Candidate]" term=2 +34:26.368 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.368 [DEBUG] TestAgent.server.raft: vote granted: from=7146fa77-93a2-68ca-b8d5-014693546a11 term=2 tally=1 +34:26.368 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.368 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:30426 [Leader]" +34:26.372 [INFO] TestAgent.server: cluster leadership acquired +34:26.372 [INFO] TestAgent.server: New leader elected: payload=Node-7146fa77-93a2-68ca-b8d5-014693546a11 +34:26.389 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:30426 +34:26.430 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.470 [INFO] TestAgent: Synced node info +34:26.501 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.501 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.501 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.501 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.501 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.501 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7146fa77-93a2-68ca-b8d5-014693546a11 +34:26.502 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7146fa77-93a2-68ca-b8d5-014693546a11 +34:26.522 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.718 [DEBUG] TestAgent: Node info in sync +34:26.722 [INFO] TestAgent: Synced service: service=web +34:26.722 [DEBUG] TestAgent: Check in sync: check=service:web +34:26.722 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:56056 latency=22.159973ms +34:26.723 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/services from=127.0.0.1:56058 latency=329.974µs +34:26.727 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/checks from=127.0.0.1:56058 latency=447.365µs +34:26.728 [INFO] TestAgent: Requesting shutdown +34:26.728 [INFO] TestAgent.server: shutting down server +34:26.728 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:26.728 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:26.728 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:26.728 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:26.728 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:26.728 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:26.728 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:26.728 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:26.728 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:26.728 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:26.733 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:26.739Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:26.739Z [INFO] agent.router.manager: shutting down +34:26.740 [INFO] TestAgent: consul server down +34:26.740 [INFO] TestAgent: shutdown complete +34:26.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=tcp +34:26.740 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:30421 network=udp +34:26.740 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:30422 network=tcp +34:27.242 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.242 [INFO] TestAgent: Endpoints down +--- PASS: TestCommand_FileWithUnnamedCheck (1.18s) PASS -ok github.com/hashicorp/consul/command/services/register 0.819s +ok github.com/hashicorp/consul/command/services/register 1.374s === RUN TestSnapshotCommand_noTabs === PAUSE TestSnapshotCommand_noTabs === CONT TestSnapshotCommand_noTabs --- PASS: TestSnapshotCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/snapshot 0.005s +ok github.com/hashicorp/consul/command/snapshot 0.050s === RUN TestSnapshotInspectCommand_noTabs === PAUSE TestSnapshotInspectCommand_noTabs === RUN TestSnapshotInspectCommand_Validation @@ -90857,77 +90979,78 @@ === RUN TestSnapshotInspectCommand === PAUSE TestSnapshotInspectCommand === CONT TestSnapshotInspectCommand_noTabs -=== CONT TestSnapshotInspectCommand --- PASS: TestSnapshotInspectCommand_noTabs (0.00s) +=== CONT TestSnapshotInspectCommand +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestSnapshotInspectCommand_Validation --- PASS: TestSnapshotInspectCommand_Validation (0.00s) -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range -2022-11-07T22:46:56.794Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.808Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.828 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:7460f8dd-e634-e018-5af6-98fa6c5276f2 Address:127.0.0.1:19195}]" -46:56.828 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:19195 [Follower]" leader= -46:56.829 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-7460f8dd-e634-e018-5af6-98fa6c5276f2.dc1 127.0.0.1 -46:56.830 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-7460f8dd-e634-e018-5af6-98fa6c5276f2 127.0.0.1 -2022-11-07T22:46:56.830Z [INFO] agent.router: Initializing LAN area manager -46:56.830 [INFO] TestAgent.server: Adding LAN server: server="Node-7460f8dd-e634-e018-5af6-98fa6c5276f2 (Addr: tcp/127.0.0.1:19195) (DC: dc1)" -46:56.830 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-7460f8dd-e634-e018-5af6-98fa6c5276f2.dc1 area=wan -46:56.830 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=tcp -46:56.830 [INFO] TestAgent: Started DNS server: address=127.0.0.1:19190 network=udp -46:56.831 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:19191 network=tcp -46:56.831 [INFO] TestAgent: started state syncer -46:56.831 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:19196 network=tcp -46:56.884 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.884 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:19195 [Candidate]" term=2 -46:56.892 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.892 [DEBUG] TestAgent.server.raft: vote granted: from=7460f8dd-e634-e018-5af6-98fa6c5276f2 term=2 tally=1 -46:56.892 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.892 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:19195 [Leader]" -46:56.892 [INFO] TestAgent.server: cluster leadership acquired -46:56.892 [INFO] TestAgent.server: New leader elected: payload=Node-7460f8dd-e634-e018-5af6-98fa6c5276f2 -46:56.897 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:19195 -46:56.908 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.925 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.925 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.925 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.925 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.925 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.925 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-7460f8dd-e634-e018-5af6-98fa6c5276f2 -46:56.925 [INFO] TestAgent.server: member joined, marking health alive: member=Node-7460f8dd-e634-e018-5af6-98fa6c5276f2 -46:56.936 [INFO] TestAgent.server: federation state anti-entropy synced -46:57.156 [INFO] TestAgent.server.fsm: snapshot created: duration=100.754µs -46:57.156 [INFO] TestAgent.server.raft: starting snapshot up to: index=11 -46:57.156 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotInspectCommand-TestAgent-agent876961205/raft/snapshots/2-11-1667861217156.tmp -46:57.170 [INFO] TestAgent.server.raft: snapshot complete up to: index=11 -46:57.173 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:57.175 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:54930 latency=19.191776ms -46:57.176 [INFO] TestAgent: Synced node info -46:57.177 [INFO] TestAgent: Requesting shutdown -46:57.177 [INFO] TestAgent.server: shutting down server -46:57.177 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:57.177 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:57.177 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:57.177 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:57.177 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:57.177 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:57.177 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:57.177 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:57.177 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:57.177 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:57.180 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:57.184Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:57.184Z [INFO] agent.router.manager: shutting down -46:57.184 [INFO] TestAgent: consul server down -46:57.184 [INFO] TestAgent: shutdown complete -46:57.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=tcp -46:57.184 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:19190 network=udp -46:57.184 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:19191 network=tcp -46:57.684 [INFO] TestAgent: Waiting for endpoints to shut down -46:57.684 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshotInspectCommand (0.93s) +2021-10-05T16:34:25.978Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:25.990Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.037 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:dfe0e081-bb22-dbc9-f502-857f4dcd2331 Address:127.0.0.1:22258}]" +34:26.038 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= +34:26.039 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331.dc1 127.0.0.1 +34:26.040 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331 127.0.0.1 +2021-10-05T16:34:26.040Z [INFO] agent.router: Initializing LAN area manager +34:26.049 [INFO] TestAgent.server: Adding LAN server: server="Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331 (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +34:26.049 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331.dc1 area=wan +34:26.049 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=udp +34:26.049 [INFO] TestAgent: Started DNS server: address=127.0.0.1:22253 network=tcp +34:26.050 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:22254 network=tcp +34:26.050 [INFO] TestAgent: started state syncer +34:26.050 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:22259 network=tcp +34:26.108 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.108 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 +34:26.128 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.128 [DEBUG] TestAgent.server.raft: vote granted: from=dfe0e081-bb22-dbc9-f502-857f4dcd2331 term=2 tally=1 +34:26.128 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.128 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" +34:26.134 [INFO] TestAgent.server: cluster leadership acquired +34:26.135 [INFO] TestAgent.server: New leader elected: payload=Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331 +34:26.160 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 +34:26.205 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.268 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.268 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.268 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.268 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.268 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.268 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331 +34:26.268 [INFO] TestAgent.server: member joined, marking health alive: member=Node-dfe0e081-bb22-dbc9-f502-857f4dcd2331 +34:26.296 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:26.304 [INFO] TestAgent.server.fsm: snapshot created: duration=92.089µs +34:26.312 [INFO] TestAgent.server.raft: starting snapshot up to: index=10 +34:26.312 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotInspectCommand-TestAgent-agent515351129/raft/snapshots/2-10-1633451666312.tmp +34:26.312 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.326 [INFO] TestAgent: Synced node info +34:26.326 [DEBUG] TestAgent: Node info in sync +34:26.342 [INFO] TestAgent.server.raft: snapshot complete up to: index=10 +34:26.358 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:46916 latency=218.74556ms +34:26.360 [INFO] TestAgent: Requesting shutdown +34:26.360 [INFO] TestAgent.server: shutting down server +34:26.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:26.360 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:26.360 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:26.360 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:26.360 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:26.360 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:26.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:26.360 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:26.360 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:26.360 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:26.369 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:26.375Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:26.375Z [INFO] agent.router.manager: shutting down +34:26.375 [INFO] TestAgent: consul server down +34:26.375 [INFO] TestAgent: shutdown complete +34:26.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp +34:26.375 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp +34:26.375 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp +34:26.880 [INFO] TestAgent: Waiting for endpoints to shut down +34:26.880 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshotInspectCommand (0.99s) PASS -ok github.com/hashicorp/consul/command/snapshot/inspect 0.961s +ok github.com/hashicorp/consul/command/snapshot/inspect 1.091s === RUN TestSnapshotRestoreCommand_noTabs === PAUSE TestSnapshotRestoreCommand_noTabs === RUN TestSnapshotRestoreCommand_Validation @@ -90937,185 +91060,185 @@ === RUN TestSnapshotRestoreCommand_TruncatedSnapshot === PAUSE TestSnapshotRestoreCommand_TruncatedSnapshot === CONT TestSnapshotRestoreCommand_noTabs -=== CONT TestSnapshotRestoreCommand -=== CONT TestSnapshotRestoreCommand_Validation -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === CONT TestSnapshotRestoreCommand_TruncatedSnapshot --- PASS: TestSnapshotRestoreCommand_noTabs (0.00s) +=== CONT TestSnapshotRestoreCommand +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... +=== CONT TestSnapshotRestoreCommand_Validation +--- PASS: TestSnapshotRestoreCommand_Validation (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestSnapshotRestoreCommand_Validation (0.00s) === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_200_bytes_from_end === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_16_bytes_from_end === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_8_bytes_from_end === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_4_bytes_from_end === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_2_bytes_from_end === RUN TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_1_bytes_from_end -2022-11-07T22:46:56.836Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.845Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.873 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a49cb985-c162-789f-c5d1-bd6c621b119f Address:127.0.0.1:27370}]" -46:56.873 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27370 [Follower]" leader= -46:56.873 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-a49cb985-c162-789f-c5d1-bd6c621b119f.dc1 127.0.0.1 -46:56.876 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-a49cb985-c162-789f-c5d1-bd6c621b119f 127.0.0.1 -2022-11-07T22:46:56.876Z [INFO] agent.router: Initializing LAN area manager -46:56.876 [INFO] TestAgent.server: Adding LAN server: server="Node-a49cb985-c162-789f-c5d1-bd6c621b119f (Addr: tcp/127.0.0.1:27370) (DC: dc1)" -46:56.876 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=udp -46:56.876 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-a49cb985-c162-789f-c5d1-bd6c621b119f.dc1 area=wan -46:56.877 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27365 network=tcp -46:56.877 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27366 network=tcp -46:56.877 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27371 network=tcp -46:56.877 [INFO] TestAgent: started state syncer -46:56.927 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.927 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27370 [Candidate]" term=2 -46:56.937 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.937 [DEBUG] TestAgent.server.raft: vote granted: from=a49cb985-c162-789f-c5d1-bd6c621b119f term=2 tally=1 -46:56.937 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.937 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27370 [Leader]" -46:56.937 [INFO] TestAgent.server: cluster leadership acquired -46:56.937 [INFO] TestAgent.server: New leader elected: payload=Node-a49cb985-c162-789f-c5d1-bd6c621b119f -46:56.941 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27370 -46:56.954 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.973 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.973 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.973 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.973 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.973 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.973 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-a49cb985-c162-789f-c5d1-bd6c621b119f -46:56.973 [INFO] TestAgent.server: member joined, marking health alive: member=Node-a49cb985-c162-789f-c5d1-bd6c621b119f -46:56.982 [INFO] TestAgent.server: federation state anti-entropy synced -46:56.992 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/blob from=127.0.0.1:33060 latency=7.784536ms -46:56.992 [INFO] TestAgent.server.fsm: snapshot created: duration=36.933µs -46:56.992 [INFO] TestAgent.server.raft: starting snapshot up to: index=12 -46:56.992 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand_TruncatedSnapshot-TestAgent-agent570001897/raft/snapshots/2-12-1667861216992.tmp -46:57.006 [INFO] TestAgent.server.raft: snapshot complete up to: index=12 -46:57.017 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:33060 latency=24.182178ms -46:57.019 [ERROR] TestAgent.server: Failed to close snapshot decompressor: error="unexpected EOF" -46:57.019 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33062 error="failed to read snapshot file: failed reading snapshot: unexpected EOF" -46:57.020 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33062 latency=2.197868ms -46:57.024 [ERROR] TestAgent.server: Failed to close snapshot decompressor: error="unexpected EOF" -46:57.024 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33064 error="failed to read snapshot file: failed reading snapshot: unexpected EOF" -46:57.024 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33064 latency=1.536958ms -46:57.026 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33066 error="unexpected EOF" -46:57.026 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33066 latency=1.620556ms -46:57.029 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33068 error="unexpected EOF" -46:57.029 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33068 latency=1.481251ms -46:57.031 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33070 error="unexpected EOF" -46:57.031 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33070 latency=1.439231ms -46:57.034 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:33072 error="unexpected EOF" -46:57.034 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:33072 latency=1.988326ms -46:57.034 [INFO] TestAgent: Requesting shutdown -46:57.034 [INFO] TestAgent.server: shutting down server -46:57.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:57.034 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:57.034 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:57.034 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:57.034 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:57.034 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:57.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:57.034 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -46:57.034 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:57.034 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:57.034 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:57.038 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:57.041Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:57.041Z [INFO] agent.router.manager: shutting down -46:57.041 [INFO] TestAgent: consul server down -46:57.041 [INFO] TestAgent: shutdown complete -46:57.041 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=tcp -46:57.041 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27365 network=udp -46:57.041 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27366 network=tcp -46:57.541 [INFO] TestAgent: Waiting for endpoints to shut down -46:57.541 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot (0.74s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_200_bytes_from_end (0.00s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_16_bytes_from_end (0.00s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_8_bytes_from_end (0.00s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_4_bytes_from_end (0.00s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_2_bytes_from_end (0.00s) - --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_1_bytes_from_end (0.00s) -2022-11-07T22:46:56.837Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:56.847Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:56.876 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:2e484bac-072a-242d-07d5-1bba3000fc97 Address:127.0.0.1:27363}]" -46:56.876 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= -46:56.877 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-2e484bac-072a-242d-07d5-1bba3000fc97.dc1 127.0.0.1 -46:56.877 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-2e484bac-072a-242d-07d5-1bba3000fc97 127.0.0.1 -2022-11-07T22:46:56.877Z [INFO] agent.router: Initializing LAN area manager -46:56.877 [INFO] TestAgent.server: Adding LAN server: server="Node-2e484bac-072a-242d-07d5-1bba3000fc97 (Addr: tcp/127.0.0.1:27363) (DC: dc1)" -46:56.877 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=udp -46:56.877 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-2e484bac-072a-242d-07d5-1bba3000fc97.dc1 area=wan -46:56.877 [INFO] TestAgent: Started DNS server: address=127.0.0.1:27358 network=tcp -46:56.878 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:27359 network=tcp -46:56.878 [INFO] TestAgent: started state syncer -46:56.878 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:27364 network=tcp -46:56.940 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:56.940 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 -46:56.949 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:56.949 [DEBUG] TestAgent.server.raft: vote granted: from=2e484bac-072a-242d-07d5-1bba3000fc97 term=2 tally=1 -46:56.949 [INFO] TestAgent.server.raft: election won: tally=1 -46:56.949 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" -46:56.949 [INFO] TestAgent.server: cluster leadership acquired -46:56.949 [INFO] TestAgent.server: New leader elected: payload=Node-2e484bac-072a-242d-07d5-1bba3000fc97 -46:56.955 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 -46:56.966 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:56.984 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:56.984 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:56.984 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:56.984 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:56.984 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:56.984 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-2e484bac-072a-242d-07d5-1bba3000fc97 -46:56.984 [INFO] TestAgent.server: member joined, marking health alive: member=Node-2e484bac-072a-242d-07d5-1bba3000fc97 -46:56.995 [INFO] TestAgent.server: federation state anti-entropy synced -46:57.131 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:57.134 [INFO] TestAgent: Synced node info -46:57.134 [DEBUG] TestAgent: Node info in sync -46:57.308 [INFO] TestAgent.server.fsm: snapshot created: duration=71.208µs -46:57.308 [INFO] TestAgent.server.raft: starting snapshot up to: index=12 -46:57.308 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand-TestAgent-agent216161330/raft/snapshots/2-12-1667861217308.tmp -46:57.322 [INFO] TestAgent.server.raft: snapshot complete up to: index=12 -46:57.327 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:38522 latency=19.107544ms -46:57.332 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand-TestAgent-agent216161330/raft/snapshots/2-13-1667861217332.tmp -46:57.346 [INFO] TestAgent.server.raft: copied to local snapshot: bytes=5146 -46:57.347 [INFO] TestAgent.server.raft: restored user snapshot: index=1 -46:57.352 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:57.352 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:57.352 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:57.352 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:57.352 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:57.352 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:57.352 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:57.352 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 -46:57.353 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:57.353 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:57.353 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:57.353 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:57.353 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:38524 latency=24.647895ms -46:57.353 [INFO] TestAgent: Requesting shutdown -46:57.353 [INFO] TestAgent.server: shutting down server -46:57.353 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:57.353 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:57.353 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:57.353 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:57.353 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:57.353 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:57.353 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:57.353 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:57.353 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:57.353 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:57.353 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:57.353 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:57.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:57.359Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:57.359Z [INFO] agent.router.manager: shutting down -46:57.359 [INFO] TestAgent: consul server down -46:57.359 [INFO] TestAgent: shutdown complete -46:57.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp -46:57.359 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp -46:57.359 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp -46:57.859 [INFO] TestAgent: Waiting for endpoints to shut down -46:57.860 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshotRestoreCommand (1.06s) +2021-10-05T16:34:26.572Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.614Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.679 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 Address:127.0.0.1:15111}]" +34:26.680 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15111 [Follower]" leader= +34:26.680 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1.dc1 127.0.0.1 +34:26.680 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 127.0.0.1 +2021-10-05T16:34:26.681Z [INFO] agent.router: Initializing LAN area manager +34:26.681 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=udp +34:26.681 [INFO] TestAgent.server: Adding LAN server: server="Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 (Addr: tcp/127.0.0.1:15111) (DC: dc1)" +34:26.681 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1.dc1 area=wan +34:26.681 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15106 network=tcp +34:26.682 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15107 network=tcp +34:26.682 [INFO] TestAgent: started state syncer +34:26.682 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15112 network=tcp +34:26.720 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.720 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15111 [Candidate]" term=2 +34:26.741 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.741 [DEBUG] TestAgent.server.raft: vote granted: from=ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 term=2 tally=1 +34:26.741 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.741 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15111 [Leader]" +34:26.743 [INFO] TestAgent.server: cluster leadership acquired +34:26.743 [INFO] TestAgent.server: New leader elected: payload=Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 +34:26.750 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 +34:26.769 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.832 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.832 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.832 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.832 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.832 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.832 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 +34:26.832 [INFO] TestAgent.server: member joined, marking health alive: member=Node-ed65b7b3-8f4d-67bb-76ab-66348dfe15f1 +34:26.845 [INFO] TestAgent.server: federation state anti-entropy synced +34:26.919 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:26.925 [INFO] TestAgent: Synced node info +34:26.925 [DEBUG] TestAgent: Node info in sync +34:27.128 [INFO] TestAgent.server.fsm: snapshot created: duration=152.394µs +34:27.129 [INFO] TestAgent.server.raft: starting snapshot up to: index=12 +34:27.129 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand-TestAgent-agent716578905/raft/snapshots/2-12-1633451667129.tmp +34:27.183 [INFO] TestAgent.server.raft: snapshot complete up to: index=12 +34:27.226 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:57272 latency=97.54318ms +34:27.244 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand-TestAgent-agent716578905/raft/snapshots/2-13-1633451667244.tmp +34:27.272 [INFO] TestAgent.server.raft: copied to local snapshot: bytes=5146 +34:27.273 [INFO] TestAgent.server.raft: restored user snapshot: index=1 +34:27.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:27.308 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:27.308 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:27.308 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:27.308 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:27.308 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:27.308 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:27.308 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15111 +34:27.309 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:27.309 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:27.309 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:27.309 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:27.309 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:57274 latency=81.37818ms +34:27.310 [INFO] TestAgent: Requesting shutdown +34:27.310 [INFO] TestAgent.server: shutting down server +34:27.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:27.310 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:27.310 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:27.310 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:27.310 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:27.310 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:27.310 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:27.310 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:27.312 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:27.312 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:27.312 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:27.312 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:27.319 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:27.338Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:27.338Z [INFO] agent.router.manager: shutting down +34:27.338 [INFO] TestAgent: consul server down +34:27.338 [INFO] TestAgent: shutdown complete +34:27.338 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=tcp +34:27.338 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15106 network=udp +34:27.338 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15107 network=tcp +34:27.840 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.840 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshotRestoreCommand (1.37s) +2021-10-05T16:34:26.572Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:26.614Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:26.660 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:45e73622-238a-6eb9-7ede-7f98ca64ca6e Address:127.0.0.1:15118}]" +34:26.661 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e.dc1 127.0.0.1 +34:26.662 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e 127.0.0.1 +2021-10-05T16:34:26.662Z [INFO] agent.router: Initializing LAN area manager +34:26.662 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=udp +34:26.662 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:15118 [Follower]" leader= +34:26.663 [INFO] TestAgent.server: Adding LAN server: server="Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e (Addr: tcp/127.0.0.1:15118) (DC: dc1)" +34:26.663 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e.dc1 area=wan +34:26.663 [INFO] TestAgent: Started DNS server: address=127.0.0.1:15113 network=tcp +34:26.663 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:15114 network=tcp +34:26.663 [INFO] TestAgent: started state syncer +34:26.663 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:15119 network=tcp +34:26.723 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:26.723 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:15118 [Candidate]" term=2 +34:26.740 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:26.740 [DEBUG] TestAgent.server.raft: vote granted: from=45e73622-238a-6eb9-7ede-7f98ca64ca6e term=2 tally=1 +34:26.740 [INFO] TestAgent.server.raft: election won: tally=1 +34:26.740 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:15118 [Leader]" +34:26.743 [INFO] TestAgent.server: cluster leadership acquired +34:26.744 [INFO] TestAgent.server: New leader elected: payload=Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e +34:26.752 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:15118 +34:26.758 [INFO] TestAgent: Synced node info +34:26.787 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:26.805 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:26.805 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:26.805 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:26.805 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:26.805 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:26.805 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e +34:26.806 [INFO] TestAgent.server: member joined, marking health alive: member=Node-45e73622-238a-6eb9-7ede-7f98ca64ca6e +34:26.824 [INFO] TestAgent.server: federation state anti-entropy synced +34:27.144 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/blob from=127.0.0.1:39278 latency=24.085405ms +34:27.153 [INFO] TestAgent.server.fsm: snapshot created: duration=71.869µs +34:27.153 [INFO] TestAgent.server.raft: starting snapshot up to: index=13 +34:27.153 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotRestoreCommand_TruncatedSnapshot-TestAgent-agent227043812/raft/snapshots/2-13-1633451667153.tmp +34:27.192 [INFO] TestAgent.server.raft: snapshot complete up to: index=13 +34:27.258 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:39278 latency=104.846251ms +34:27.276 [ERROR] TestAgent.server: Failed to close snapshot decompressor: error="unexpected EOF" +34:27.276 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39284 error="failed to read snapshot file: failed reading snapshot: unexpected EOF" +34:27.276 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39284 latency=13.085588ms +34:27.286 [ERROR] TestAgent.server: Failed to close snapshot decompressor: error="unexpected EOF" +34:27.286 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39286 error="failed to read snapshot file: failed reading snapshot: unexpected EOF" +34:27.287 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39286 latency=2.638631ms +34:27.293 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39288 error="unexpected EOF" +34:27.293 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39288 latency=3.865512ms +34:27.303 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39290 error="unexpected EOF" +34:27.303 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39290 latency=2.455997ms +34:27.326 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39292 error="unexpected EOF" +34:27.327 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39292 latency=2.558589ms +34:27.340 [ERROR] TestAgent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:39294 error="unexpected EOF" +34:27.340 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:39294 latency=2.453752ms +34:27.344 [INFO] TestAgent: Requesting shutdown +34:27.344 [INFO] TestAgent.server: shutting down server +34:27.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:27.345 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:27.345 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:27.345 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:27.345 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:27.345 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:27.345 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:27.345 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:27.345 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:27.345 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:27.356 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:27.368Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:27.368Z [INFO] agent.router.manager: shutting down +34:27.368 [INFO] TestAgent: consul server down +34:27.368 [INFO] TestAgent: shutdown complete +34:27.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=tcp +34:27.368 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:15113 network=udp +34:27.368 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:15114 network=tcp +34:27.872 [INFO] TestAgent: Waiting for endpoints to shut down +34:27.872 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot (1.40s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_200_bytes_from_end (0.02s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_16_bytes_from_end (0.01s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_8_bytes_from_end (0.01s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_4_bytes_from_end (0.01s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_2_bytes_from_end (0.02s) + --- PASS: TestSnapshotRestoreCommand_TruncatedSnapshot/truncate_1_bytes_from_end (0.02s) PASS -ok github.com/hashicorp/consul/command/snapshot/restore 1.097s +ok github.com/hashicorp/consul/command/snapshot/restore 1.468s === RUN TestSnapshotSaveCommand_noTabs === PAUSE TestSnapshotSaveCommand_noTabs === RUN TestSnapshotSaveCommand_Validation @@ -91125,171 +91248,174 @@ === RUN TestSnapshotSaveCommand_TruncatedStream === PAUSE TestSnapshotSaveCommand_TruncatedStream === CONT TestSnapshotSaveCommand_noTabs +=== CONT TestSnapshotSaveCommand_TruncatedStream === CONT TestSnapshotSaveCommand -=== CONT TestSnapshotSaveCommand_Validation [INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... -=== CONT TestSnapshotSaveCommand_TruncatedStream +=== CONT TestSnapshotSaveCommand_Validation --- PASS: TestSnapshotSaveCommand_noTabs (0.00s) +--- PASS: TestSnapshotSaveCommand_Validation (0.00s) [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestSnapshotSaveCommand_Validation (0.00s) === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_200_bytes_from_end === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_16_bytes_from_end === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_8_bytes_from_end === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_4_bytes_from_end === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_2_bytes_from_end === RUN TestSnapshotSaveCommand_TruncatedStream/truncate_1_bytes_from_end -2022-11-07T22:46:59.215Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:59.230Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:59.297 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:95ade37b-de28-5086-a3cb-8dcf58249d8b Address:127.0.0.1:13069}]" -46:59.298 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13069 [Follower]" leader= -46:59.298 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-95ade37b-de28-5086-a3cb-8dcf58249d8b.dc1 127.0.0.1 -46:59.299 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-95ade37b-de28-5086-a3cb-8dcf58249d8b 127.0.0.1 -2022-11-07T22:46:59.299Z [INFO] agent.router: Initializing LAN area manager -46:59.299 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-95ade37b-de28-5086-a3cb-8dcf58249d8b.dc1 area=wan -46:59.299 [INFO] TestAgent.server: Adding LAN server: server="Node-95ade37b-de28-5086-a3cb-8dcf58249d8b (Addr: tcp/127.0.0.1:13069) (DC: dc1)" -46:59.299 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=udp -46:59.299 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13064 network=tcp -46:59.299 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13065 network=tcp -46:59.299 [INFO] TestAgent: started state syncer -46:59.299 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13070 network=tcp -46:59.348 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:59.348 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13069 [Candidate]" term=2 -46:59.358 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:59.358 [DEBUG] TestAgent.server.raft: vote granted: from=95ade37b-de28-5086-a3cb-8dcf58249d8b term=2 tally=1 -46:59.358 [INFO] TestAgent.server.raft: election won: tally=1 -46:59.358 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13069 [Leader]" -46:59.358 [INFO] TestAgent.server: cluster leadership acquired -46:59.358 [INFO] TestAgent.server: New leader elected: payload=Node-95ade37b-de28-5086-a3cb-8dcf58249d8b -46:59.367 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13069 -46:59.379 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:59.406 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:59.406 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:59.406 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:59.406 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:59.406 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:59.406 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-95ade37b-de28-5086-a3cb-8dcf58249d8b -46:59.406 [INFO] TestAgent.server: member joined, marking health alive: member=Node-95ade37b-de28-5086-a3cb-8dcf58249d8b -46:59.415 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:59.419 [INFO] TestAgent: Synced node info -46:59.419 [INFO] TestAgent.server: federation state anti-entropy synced -46:59.503 [INFO] TestAgent.server.fsm: snapshot created: duration=63.191µs -46:59.503 [INFO] TestAgent.server.raft: starting snapshot up to: index=12 -46:59.503 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand-TestAgent-agent804397987/raft/snapshots/2-12-1667861219503.tmp -46:59.516 [INFO] TestAgent.server.raft: snapshot complete up to: index=12 -46:59.521 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:36406 latency=18.283825ms -46:59.534 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand-TestAgent-agent804397987/raft/snapshots/2-13-1667861219534.tmp -46:59.545 [INFO] TestAgent.server.raft: copied to local snapshot: bytes=5154 -46:59.546 [INFO] TestAgent.server.raft: restored user snapshot: index=1 -46:59.551 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:59.551 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:59.551 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:59.551 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:59.551 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:59.551 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:59.551 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13069 -46:59.551 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:59.552 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:59.552 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:59.552 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:59.552 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:59.555 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:36408 latency=24.784672ms -46:59.556 [INFO] TestAgent: Requesting shutdown -46:59.556 [INFO] TestAgent.server: shutting down server -46:59.556 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:59.556 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:59.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:59.556 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:59.556 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:59.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:59.556 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:59.556 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:59.556 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:59.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:59.556 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:59.556 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:59.559 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:59.562Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:59.562Z [INFO] agent.router.manager: shutting down -46:59.562 [INFO] TestAgent: consul server down -46:59.562 [INFO] TestAgent: shutdown complete -46:59.562 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=tcp -46:59.562 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13064 network=udp -46:59.562 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13065 network=tcp -47:00.062 [INFO] TestAgent: Waiting for endpoints to shut down -47:00.062 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshotSaveCommand (0.88s) -2022-11-07T22:46:59.215Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:46:59.229Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -46:59.305 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:632573c6-b2e8-7a9d-e988-13dc6264d768 Address:127.0.0.1:13076}]" -46:59.305 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:13076 [Follower]" leader= -46:59.305 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-632573c6-b2e8-7a9d-e988-13dc6264d768.dc1 127.0.0.1 -46:59.306 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-632573c6-b2e8-7a9d-e988-13dc6264d768 127.0.0.1 -2022-11-07T22:46:59.306Z [INFO] agent.router: Initializing LAN area manager -46:59.306 [INFO] TestAgent.server: Adding LAN server: server="Node-632573c6-b2e8-7a9d-e988-13dc6264d768 (Addr: tcp/127.0.0.1:13076) (DC: dc1)" -46:59.306 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-632573c6-b2e8-7a9d-e988-13dc6264d768.dc1 area=wan -46:59.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=udp -46:59.306 [INFO] TestAgent: Started DNS server: address=127.0.0.1:13071 network=tcp -46:59.306 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:13072 network=tcp -46:59.307 [INFO] TestAgent: started state syncer -46:59.307 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:13077 network=tcp -46:59.344 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -46:59.344 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:13076 [Candidate]" term=2 -46:59.354 [DEBUG] TestAgent.server.raft: votes: needed=1 -46:59.354 [DEBUG] TestAgent.server.raft: vote granted: from=632573c6-b2e8-7a9d-e988-13dc6264d768 term=2 tally=1 -46:59.354 [INFO] TestAgent.server.raft: election won: tally=1 -46:59.354 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:13076 [Leader]" -46:59.354 [INFO] TestAgent.server: cluster leadership acquired -46:59.354 [INFO] TestAgent.server: New leader elected: payload=Node-632573c6-b2e8-7a9d-e988-13dc6264d768 -46:59.361 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:13076 -46:59.376 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -46:59.405 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -46:59.405 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -46:59.405 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -46:59.405 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -46:59.405 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -46:59.405 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-632573c6-b2e8-7a9d-e988-13dc6264d768 -46:59.405 [INFO] TestAgent.server: member joined, marking health alive: member=Node-632573c6-b2e8-7a9d-e988-13dc6264d768 -46:59.415 [INFO] TestAgent.server: federation state anti-entropy synced -46:59.573 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -46:59.576 [INFO] TestAgent: Synced node info -46:59.723 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/blob from=127.0.0.1:48482 latency=8.927708ms -46:59.724 [INFO] TestAgent.server.fsm: snapshot created: duration=53.2µs -46:59.724 [INFO] TestAgent.server.raft: starting snapshot up to: index=13 -46:59.724 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand_TruncatedStream-TestAgent-agent698556582/raft/snapshots/2-13-1667861219724.tmp -46:59.736 [INFO] TestAgent.server.raft: snapshot complete up to: index=13 -46:59.747 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:48482 latency=23.068949ms -46:59.798 [INFO] TestAgent: Requesting shutdown -46:59.798 [INFO] TestAgent.server: shutting down server -46:59.798 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -46:59.798 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -46:59.798 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -46:59.798 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -46:59.798 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -46:59.798 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -46:59.798 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -46:59.798 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -46:59.798 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -46:59.798 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -46:59.801 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:46:59.804Z [INFO] agent.router.manager: shutting down -2022-11-07T22:46:59.804Z [INFO] agent.router.manager: shutting down -46:59.804 [INFO] TestAgent: consul server down -46:59.804 [INFO] TestAgent: shutdown complete -46:59.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=tcp -46:59.804 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:13071 network=udp -46:59.804 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:13072 network=tcp -47:00.305 [INFO] TestAgent: Waiting for endpoints to shut down -47:00.305 [INFO] TestAgent: Endpoints down ---- PASS: TestSnapshotSaveCommand_TruncatedStream (1.12s) - --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_200_bytes_from_end (0.01s) - --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_16_bytes_from_end (0.01s) - --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_8_bytes_from_end (0.01s) - --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_4_bytes_from_end (0.01s) +2021-10-05T16:34:33.092Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:33.119Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:33.158 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:947f970f-87de-c220-8703-22c0b22135d2 Address:127.0.0.1:29412}]" +34:33.159 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29412 [Follower]" leader= +34:33.159 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-947f970f-87de-c220-8703-22c0b22135d2.dc1 127.0.0.1 +34:33.160 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-947f970f-87de-c220-8703-22c0b22135d2 127.0.0.1 +2021-10-05T16:34:33.160Z [INFO] agent.router: Initializing LAN area manager +34:33.160 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-947f970f-87de-c220-8703-22c0b22135d2.dc1 area=wan +34:33.160 [INFO] TestAgent.server: Adding LAN server: server="Node-947f970f-87de-c220-8703-22c0b22135d2 (Addr: tcp/127.0.0.1:29412) (DC: dc1)" +34:33.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=udp +34:33.161 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29407 network=tcp +34:33.161 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29408 network=tcp +34:33.161 [INFO] TestAgent: started state syncer +34:33.161 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29413 network=tcp +34:33.204 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:33.204 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29412 [Candidate]" term=2 +34:33.234 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:33.234 [DEBUG] TestAgent.server.raft: vote granted: from=947f970f-87de-c220-8703-22c0b22135d2 term=2 tally=1 +34:33.234 [INFO] TestAgent.server.raft: election won: tally=1 +34:33.234 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29412 [Leader]" +34:33.236 [INFO] TestAgent.server: cluster leadership acquired +34:33.236 [INFO] TestAgent.server: New leader elected: payload=Node-947f970f-87de-c220-8703-22c0b22135d2 +34:33.249 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 +34:33.284 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:33.330 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:33.330 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:33.330 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:33.330 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:33.330 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:33.330 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-947f970f-87de-c220-8703-22c0b22135d2 +34:33.330 [INFO] TestAgent.server: member joined, marking health alive: member=Node-947f970f-87de-c220-8703-22c0b22135d2 +34:33.373 [INFO] TestAgent.server: federation state anti-entropy synced +34:33.452 [INFO] TestAgent.server.fsm: snapshot created: duration=101.631µs +34:33.452 [INFO] TestAgent.server.raft: starting snapshot up to: index=11 +34:33.452 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand-TestAgent-agent784167669/raft/snapshots/2-11-1633451673452.tmp +34:33.479 [INFO] TestAgent.server.raft: snapshot complete up to: index=11 +34:33.487 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:45010 latency=35.148063ms +34:33.495 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:33.500 [INFO] TestAgent: Synced node info +34:33.500 [DEBUG] TestAgent: Node info in sync +34:33.512 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand-TestAgent-agent784167669/raft/snapshots/2-13-1633451673512.tmp +34:33.550 [INFO] TestAgent.server.raft: copied to local snapshot: bytes=4876 +34:33.552 [INFO] TestAgent.server.raft: restored user snapshot: index=1 +34:33.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:33.568 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:33.568 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:33.568 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:33.568 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:33.568 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:33.568 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:33.568 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29412 +34:33.569 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:33.569 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:33.569 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:33.569 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:33.569 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:45014 latency=64.663958ms +34:33.572 [INFO] TestAgent: Requesting shutdown +34:33.572 [INFO] TestAgent.server: shutting down server +34:33.572 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:33.572 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:33.572 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:33.572 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:33.572 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:33.572 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:33.572 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:33.572 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:33.572 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:33.572 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:33.572 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:33.572 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:33.576 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:33.581Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:33.581Z [INFO] agent.router.manager: shutting down +34:33.581 [INFO] TestAgent: consul server down +34:33.581 [INFO] TestAgent: shutdown complete +34:33.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=tcp +34:33.581 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29407 network=udp +34:33.581 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29408 network=tcp +34:34.084 [INFO] TestAgent: Waiting for endpoints to shut down +34:34.084 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshotSaveCommand (1.08s) +2021-10-05T16:34:33.077Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:33.101Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:33.161 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ac72770-fb2f-1d9c-8846-680ec417fa07 Address:127.0.0.1:29405}]" +34:33.161 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:29405 [Follower]" leader= +34:33.162 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0ac72770-fb2f-1d9c-8846-680ec417fa07.dc1 127.0.0.1 +34:33.163 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0ac72770-fb2f-1d9c-8846-680ec417fa07 127.0.0.1 +2021-10-05T16:34:33.163Z [INFO] agent.router: Initializing LAN area manager +34:33.163 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=udp +34:33.163 [INFO] TestAgent.server: Adding LAN server: server="Node-0ac72770-fb2f-1d9c-8846-680ec417fa07 (Addr: tcp/127.0.0.1:29405) (DC: dc1)" +34:33.163 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0ac72770-fb2f-1d9c-8846-680ec417fa07.dc1 area=wan +34:33.163 [INFO] TestAgent: Started DNS server: address=127.0.0.1:29400 network=tcp +34:33.164 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:29401 network=tcp +34:33.164 [INFO] TestAgent: started state syncer +34:33.164 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:29406 network=tcp +34:33.208 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:33.208 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:29405 [Candidate]" term=2 +34:33.233 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:33.233 [DEBUG] TestAgent.server.raft: vote granted: from=0ac72770-fb2f-1d9c-8846-680ec417fa07 term=2 tally=1 +34:33.233 [INFO] TestAgent.server.raft: election won: tally=1 +34:33.233 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:29405 [Leader]" +34:33.236 [INFO] TestAgent.server: cluster leadership acquired +34:33.236 [INFO] TestAgent.server: New leader elected: payload=Node-0ac72770-fb2f-1d9c-8846-680ec417fa07 +34:33.243 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:29405 +34:33.257 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:33.312 [INFO] TestAgent: Synced node info +34:33.343 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:33.343 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:33.343 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:33.343 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:33.343 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:33.343 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0ac72770-fb2f-1d9c-8846-680ec417fa07 +34:33.343 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0ac72770-fb2f-1d9c-8846-680ec417fa07 +34:33.380 [INFO] TestAgent.server: federation state anti-entropy synced +34:33.423 [DEBUG] TestAgent.http: Request finished: method=PUT url=/v1/kv/blob from=127.0.0.1:35510 latency=10.792316ms +34:33.424 [INFO] TestAgent.server.fsm: snapshot created: duration=60.071µs +34:33.424 [INFO] TestAgent.server.raft: starting snapshot up to: index=13 +34:33.424 [INFO] TestAgent.server.snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshotSaveCommand_TruncatedStream-TestAgent-agent680610414/raft/snapshots/2-13-1633451673424.tmp +34:33.444 [INFO] TestAgent.server.raft: snapshot complete up to: index=13 +34:33.468 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:33.468 [DEBUG] TestAgent: Node info in sync +34:33.468 [DEBUG] TestAgent: Node info in sync +34:33.475 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/snapshot from=127.0.0.1:35510 latency=51.112381ms +34:33.618 [INFO] TestAgent: Requesting shutdown +34:33.618 [INFO] TestAgent.server: shutting down server +34:33.618 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:33.618 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:33.618 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:33.618 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:33.618 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:33.618 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:33.618 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:33.618 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:33.618 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:33.618 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:33.627 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:33.644Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:33.644Z [INFO] agent.router.manager: shutting down +34:33.644 [INFO] TestAgent: consul server down +34:33.644 [INFO] TestAgent: shutdown complete +34:33.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=tcp +34:33.644 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:29400 network=udp +34:33.644 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:29401 network=tcp +34:34.144 [INFO] TestAgent: Waiting for endpoints to shut down +34:34.144 [INFO] TestAgent: Endpoints down +--- PASS: TestSnapshotSaveCommand_TruncatedStream (1.14s) + --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_200_bytes_from_end (0.02s) + --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_16_bytes_from_end (0.02s) + --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_8_bytes_from_end (0.02s) + --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_4_bytes_from_end (0.02s) --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_2_bytes_from_end (0.01s) - --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_1_bytes_from_end (0.01s) + --- PASS: TestSnapshotSaveCommand_TruncatedStream/truncate_1_bytes_from_end (0.02s) PASS -ok github.com/hashicorp/consul/command/snapshot/save 1.152s +ok github.com/hashicorp/consul/command/snapshot/save 1.236s === RUN TestValidateCommand_noTabs === PAUSE TestValidateCommand_noTabs === RUN TestValidateCommand_FailOnEmptyFile @@ -91309,37 +91435,37 @@ === RUN TestValidateCommand_Quiet === PAUSE TestValidateCommand_Quiet === CONT TestValidateCommand_noTabs -=== CONT TestValidateCommand_Quiet -=== CONT TestValidateCommand_SucceedWithMinimalHCLConfigFormat -=== CONT TestValidateCommand_FailForInvalidJSONConfigFormat === CONT TestValidateCommand_SucceedWithJSONAsHCL --- PASS: TestValidateCommand_noTabs (0.00s) +=== CONT TestValidateCommand_SucceedWithMinimalHCLConfigFormat === CONT TestValidateCommand_SucceedWithMinimalJSONConfigFormat === CONT TestValidateCommand_SucceedOnMinimalConfigFile === CONT TestValidateCommand_FailOnEmptyFile +=== CONT TestValidateCommand_Quiet +=== CONT TestValidateCommand_FailForInvalidJSONConfigFormat === CONT TestValidateCommand_SucceedOnMinimalConfigDir ---- PASS: TestValidateCommand_FailForInvalidJSONConfigFormat (0.01s) ---- PASS: TestValidateCommand_SucceedOnMinimalConfigDir (0.02s) ---- PASS: TestValidateCommand_FailOnEmptyFile (0.02s) ---- PASS: TestValidateCommand_SucceedWithJSONAsHCL (0.02s) ---- PASS: TestValidateCommand_SucceedWithMinimalHCLConfigFormat (0.02s) ---- PASS: TestValidateCommand_SucceedWithMinimalJSONConfigFormat (0.02s) ---- PASS: TestValidateCommand_SucceedOnMinimalConfigFile (0.02s) ---- PASS: TestValidateCommand_Quiet (0.02s) +--- PASS: TestValidateCommand_FailForInvalidJSONConfigFormat (0.03s) +--- PASS: TestValidateCommand_SucceedOnMinimalConfigDir (0.05s) +--- PASS: TestValidateCommand_Quiet (0.06s) +--- PASS: TestValidateCommand_SucceedOnMinimalConfigFile (0.07s) +--- PASS: TestValidateCommand_FailOnEmptyFile (0.07s) +--- PASS: TestValidateCommand_SucceedWithJSONAsHCL (0.07s) +--- PASS: TestValidateCommand_SucceedWithMinimalJSONConfigFormat (0.07s) +--- PASS: TestValidateCommand_SucceedWithMinimalHCLConfigFormat (0.08s) PASS -ok github.com/hashicorp/consul/command/validate 0.039s +ok github.com/hashicorp/consul/command/validate 0.120s === RUN TestFormat === RUN TestFormat/pretty === RUN TestFormat/json ---- PASS: TestFormat (0.00s) - --- PASS: TestFormat/pretty (0.00s) - --- PASS: TestFormat/json (0.00s) +--- PASS: TestFormat (0.01s) + --- PASS: TestFormat/pretty (0.01s) + --- PASS: TestFormat/json (0.01s) === RUN TestVersionCommand_noTabs === PAUSE TestVersionCommand_noTabs === CONT TestVersionCommand_noTabs --- PASS: TestVersionCommand_noTabs (0.00s) PASS -ok github.com/hashicorp/consul/command/version 0.017s +ok github.com/hashicorp/consul/command/version 0.077s === RUN TestWatchCommand_noTabs === PAUSE TestWatchCommand_noTabs === RUN TestWatchCommand @@ -91352,62 +91478,64 @@ === RUN TestWatchCommand_loadToken/token_env === RUN TestWatchCommand_loadToken/token_file_arg === RUN TestWatchCommand_loadToken/token_file_env -2022-11-07T22:47:02.177Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:02.188Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:02.210 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 Address:127.0.0.1:25321}]" -47:02.210 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= -47:02.211 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3.dc1 127.0.0.1 -47:02.212 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 127.0.0.1 -2022-11-07T22:47:02.212Z [INFO] agent.router: Initializing LAN area manager -47:02.212 [INFO] TestAgent.server: Adding LAN server: server="Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" -47:02.213 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3.dc1 area=wan -47:02.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=tcp -47:02.213 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25316 network=udp -47:02.213 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25317 network=tcp -47:02.213 [INFO] TestAgent: started state syncer -47:02.213 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25322 network=tcp -47:02.272 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -47:02.272 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 -47:02.280 [DEBUG] TestAgent.server.raft: votes: needed=1 -47:02.280 [DEBUG] TestAgent.server.raft: vote granted: from=cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 term=2 tally=1 -47:02.280 [INFO] TestAgent.server.raft: election won: tally=1 -47:02.280 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" -47:02.280 [INFO] TestAgent.server: cluster leadership acquired -47:02.280 [INFO] TestAgent.server: New leader elected: payload=Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 -47:02.286 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25321 -47:02.296 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:02.313 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -47:02.314 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -47:02.314 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -47:02.314 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -47:02.314 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -47:02.314 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 -47:02.314 [INFO] TestAgent.server: member joined, marking health alive: member=Node-cdeb0107-6cb9-87f5-b1d3-0c46eebdf0b3 -47:02.318 [INFO] TestAgent.server: federation state anti-entropy synced -47:02.505 [INFO] TestAgent: Requesting shutdown -47:02.505 [INFO] TestAgent.server: shutting down server -47:02.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -47:02.505 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -47:02.505 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -47:02.505 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -47:02.505 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -47:02.505 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -47:02.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -47:02.505 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -47:02.505 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -47:02.505 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -47:02.505 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" -47:02.508 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:02.511Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:02.511Z [INFO] agent.router.manager: shutting down -47:02.512 [INFO] TestAgent: consul server down -47:02.512 [INFO] TestAgent: shutdown complete -47:02.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp -47:02.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp -47:02.512 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp -47:03.012 [INFO] TestAgent: Waiting for endpoints to shut down -47:03.012 [INFO] TestAgent: Endpoints down ---- PASS: TestWatchCommand_loadToken (0.87s) +2021-10-05T16:34:38.811Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:38.863Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:38.931 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9c764cc1-2a31-50db-e236-540311d0e28c Address:127.0.0.1:23279}]" +34:38.932 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23279 [Follower]" leader= +34:38.932 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-9c764cc1-2a31-50db-e236-540311d0e28c.dc1 127.0.0.1 +34:38.933 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-9c764cc1-2a31-50db-e236-540311d0e28c 127.0.0.1 +2021-10-05T16:34:38.933Z [INFO] agent.router: Initializing LAN area manager +34:38.933 [INFO] TestAgent.server: Adding LAN server: server="Node-9c764cc1-2a31-50db-e236-540311d0e28c (Addr: tcp/127.0.0.1:23279) (DC: dc1)" +34:38.933 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-9c764cc1-2a31-50db-e236-540311d0e28c.dc1 area=wan +34:38.933 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=udp +34:38.937 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23274 network=tcp +34:38.937 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23275 network=tcp +34:38.937 [INFO] TestAgent: started state syncer +34:38.937 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23280 network=tcp +34:38.970 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:38.970 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23279 [Candidate]" term=2 +34:38.979 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:38.979 [DEBUG] TestAgent.server.raft: vote granted: from=9c764cc1-2a31-50db-e236-540311d0e28c term=2 tally=1 +34:38.979 [INFO] TestAgent.server.raft: election won: tally=1 +34:38.979 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23279 [Leader]" +34:38.980 [INFO] TestAgent.server: cluster leadership acquired +34:38.980 [INFO] TestAgent.server: New leader elected: payload=Node-9c764cc1-2a31-50db-e236-540311d0e28c +34:38.991 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23279 +34:39.040 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:39.062 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:39.062 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:39.062 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:39.062 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:39.062 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:39.062 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-9c764cc1-2a31-50db-e236-540311d0e28c +34:39.062 [INFO] TestAgent.server: member joined, marking health alive: member=Node-9c764cc1-2a31-50db-e236-540311d0e28c +34:39.098 [INFO] TestAgent.server: federation state anti-entropy synced +34:39.192 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:39.200 [INFO] TestAgent: Synced node info +34:39.200 [DEBUG] TestAgent: Node info in sync +34:39.315 [INFO] TestAgent: Requesting shutdown +34:39.315 [INFO] TestAgent.server: shutting down server +34:39.315 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:39.315 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:39.315 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:39.315 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:39.315 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:39.315 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:39.315 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:39.315 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:39.315 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:39.315 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:39.329 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:39.336Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:39.336Z [INFO] agent.router.manager: shutting down +34:39.336 [INFO] TestAgent: consul server down +34:39.336 [INFO] TestAgent: shutdown complete +34:39.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=tcp +34:39.336 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23274 network=udp +34:39.336 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23275 network=tcp +34:39.840 [INFO] TestAgent: Waiting for endpoints to shut down +34:39.840 [INFO] TestAgent: Endpoints down +--- PASS: TestWatchCommand_loadToken (1.17s) --- PASS: TestWatchCommand_loadToken/token_arg (0.00s) --- PASS: TestWatchCommand_loadToken/token_env (0.00s) --- PASS: TestWatchCommand_loadToken/token_file_arg (0.00s) @@ -91417,186 +91545,169 @@ === RUN TestWatchCommandNoAgentService === PAUSE TestWatchCommandNoAgentService === CONT TestWatchCommand_noTabs -=== CONT TestWatchCommandNoAgentService --- PASS: TestWatchCommand_noTabs (0.00s) +=== CONT TestWatchCommandNoAgentService === CONT TestWatchCommandNoConnect === CONT TestWatchCommand -2022-11-07T22:47:03.031Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:03.047Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:03.073 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:0fac2d5b-3222-0e38-9c9e-d1764af58eb6 Address:127.0.0.1:25342}]" -47:03.074 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6.dc1 127.0.0.1 -47:03.074 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6 127.0.0.1 -2022-11-07T22:47:03.074Z [INFO] agent.router: Initializing LAN area manager -47:03.076 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25342 [Follower]" leader= -47:03.076 [INFO] TestAgent.server: Adding LAN server: server="Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6 (Addr: tcp/127.0.0.1:25342) (DC: dc1)" -47:03.076 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6.dc1 area=wan -47:03.076 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25337 network=tcp -47:03.076 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25337 network=udp -47:03.077 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25338 network=tcp -47:03.078 [INFO] TestAgent: started state syncer -47:03.078 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25343 network=tcp -47:03.122 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -47:03.122 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25342 [Candidate]" term=2 -47:03.132 [DEBUG] TestAgent.server.raft: votes: needed=1 -47:03.132 [DEBUG] TestAgent.server.raft: vote granted: from=0fac2d5b-3222-0e38-9c9e-d1764af58eb6 term=2 tally=1 -47:03.132 [INFO] TestAgent.server.raft: election won: tally=1 -47:03.132 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25342 [Leader]" -47:03.132 [INFO] TestAgent.server: cluster leadership acquired -47:03.133 [INFO] TestAgent.server: New leader elected: payload=Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6 -47:03.140 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25342 -47:03.156 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:03.176 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -47:03.176 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -47:03.176 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -47:03.176 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -47:03.176 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -47:03.176 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6 -47:03.176 [INFO] TestAgent.server: member joined, marking health alive: member=Node-0fac2d5b-3222-0e38-9c9e-d1764af58eb6 -47:03.190 [INFO] TestAgent.server: federation state anti-entropy synced -47:03.232 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -47:03.235 [INFO] TestAgent: Synced node info -47:03.269 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:54466 latency=1.674362ms -47:03.274 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:54468 latency=270.613µs -47:03.275 [INFO] TestAgent: Requesting shutdown -47:03.275 [INFO] TestAgent.server: shutting down server -47:03.275 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -47:03.275 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -47:03.275 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -47:03.275 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -47:03.275 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -47:03.275 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -47:03.275 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -47:03.275 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -47:03.275 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -47:03.275 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -47:03.278 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:03.281Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:03.281Z [INFO] agent.router.manager: shutting down -47:03.281 [INFO] TestAgent: consul server down -47:03.281 [INFO] TestAgent: shutdown complete -47:03.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25337 network=tcp -47:03.281 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25337 network=udp -47:03.281 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25338 network=tcp -47:03.782 [INFO] TestAgent: Waiting for endpoints to shut down -47:03.782 [INFO] TestAgent: Endpoints down ---- PASS: TestWatchCommand (0.77s) -2022-11-07T22:47:03.030Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:03.046Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:03.072 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:e2e159f5-cac3-bf09-7833-724d744de671 Address:127.0.0.1:25335}]" -47:03.072 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25335 [Follower]" leader= -47:03.073 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-e2e159f5-cac3-bf09-7833-724d744de671.dc1 127.0.0.1 -47:03.076 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-e2e159f5-cac3-bf09-7833-724d744de671 127.0.0.1 -2022-11-07T22:47:03.076Z [INFO] agent.router: Initializing LAN area manager -47:03.077 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25330 network=udp -47:03.077 [INFO] TestAgent.server: Adding LAN server: server="Node-e2e159f5-cac3-bf09-7833-724d744de671 (Addr: tcp/127.0.0.1:25335) (DC: dc1)" -47:03.077 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-e2e159f5-cac3-bf09-7833-724d744de671.dc1 area=wan -47:03.077 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25330 network=tcp -47:03.078 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25331 network=tcp -47:03.078 [INFO] TestAgent: started state syncer -47:03.078 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25336 network=tcp -47:03.132 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -47:03.132 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25335 [Candidate]" term=2 -47:03.147 [DEBUG] TestAgent.server.raft: votes: needed=1 -47:03.147 [DEBUG] TestAgent.server.raft: vote granted: from=e2e159f5-cac3-bf09-7833-724d744de671 term=2 tally=1 -47:03.147 [INFO] TestAgent.server.raft: election won: tally=1 -47:03.147 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25335 [Leader]" -47:03.147 [INFO] TestAgent.server: cluster leadership acquired -47:03.147 [INFO] TestAgent.server: New leader elected: payload=Node-e2e159f5-cac3-bf09-7833-724d744de671 -47:03.153 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25335 -47:03.169 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:03.192 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -47:03.192 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -47:03.192 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -47:03.192 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -47:03.192 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -47:03.192 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-e2e159f5-cac3-bf09-7833-724d744de671 -47:03.192 [INFO] TestAgent.server: member joined, marking health alive: member=Node-e2e159f5-cac3-bf09-7833-724d744de671 -47:03.201 [INFO] TestAgent.server: federation state anti-entropy synced -47:03.261 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -47:03.264 [INFO] TestAgent: Synced node info -47:03.343 [INFO] TestAgent: Requesting shutdown -47:03.343 [INFO] TestAgent.server: shutting down server -47:03.343 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -47:03.343 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -47:03.343 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -47:03.343 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -47:03.343 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -47:03.343 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -47:03.343 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -47:03.343 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -47:03.343 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -47:03.343 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -47:03.347 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:03.350Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:03.350Z [INFO] agent.router.manager: shutting down -47:03.350 [INFO] TestAgent: consul server down -47:03.350 [INFO] TestAgent: shutdown complete -47:03.350 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25330 network=tcp -47:03.350 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25330 network=udp -47:03.350 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25331 network=tcp -47:03.850 [INFO] TestAgent: Waiting for endpoints to shut down -47:03.850 [INFO] TestAgent: Endpoints down ---- PASS: TestWatchCommandNoConnect (0.84s) -2022-11-07T22:47:03.026Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:03.040Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:03.067 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:553fb96c-66ca-ecdb-4227-8bccc2e328bd Address:127.0.0.1:25328}]" -47:03.067 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:25328 [Follower]" leader= -47:03.068 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd.dc1 127.0.0.1 -47:03.068 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd 127.0.0.1 -2022-11-07T22:47:03.068Z [INFO] agent.router: Initializing LAN area manager -47:03.068 [INFO] TestAgent.server: Adding LAN server: server="Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd (Addr: tcp/127.0.0.1:25328) (DC: dc1)" -47:03.068 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd.dc1 area=wan -47:03.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=udp -47:03.068 [INFO] TestAgent: Started DNS server: address=127.0.0.1:25323 network=tcp -47:03.069 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:25324 network=tcp -47:03.069 [INFO] TestAgent: started state syncer -47:03.069 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:25329 network=tcp -47:03.118 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= -47:03.118 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:25328 [Candidate]" term=2 -47:03.127 [DEBUG] TestAgent.server.raft: votes: needed=1 -47:03.127 [DEBUG] TestAgent.server.raft: vote granted: from=553fb96c-66ca-ecdb-4227-8bccc2e328bd term=2 tally=1 -47:03.127 [INFO] TestAgent.server.raft: election won: tally=1 -47:03.127 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:25328 [Leader]" -47:03.127 [INFO] TestAgent.server: cluster leadership acquired -47:03.127 [INFO] TestAgent.server: New leader elected: payload=Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd -47:03.133 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25328 -47:03.147 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:03.172 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul -47:03.172 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" -47:03.172 [INFO] TestAgent.leader: started routine: routine="federation state pruning" -47:03.172 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" -47:03.172 [INFO] TestAgent.leader: started routine: routine="CA root pruning" -47:03.172 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd -47:03.172 [INFO] TestAgent.server: member joined, marking health alive: member=Node-553fb96c-66ca-ecdb-4227-8bccc2e328bd -47:03.188 [INFO] TestAgent.server: federation state anti-entropy synced -47:03.306 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth -47:03.309 [INFO] TestAgent: Synced node info -47:03.309 [DEBUG] TestAgent: Node info in sync -47:03.387 [INFO] TestAgent: Requesting shutdown -47:03.387 [INFO] TestAgent.server: shutting down server -47:03.387 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" -47:03.387 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" -47:03.387 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" -47:03.387 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" -47:03.387 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave -47:03.387 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" -47:03.387 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" -47:03.387 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" -47:03.387 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" -47:03.387 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" -47:03.390 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:03.394Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:03.394Z [INFO] agent.router.manager: shutting down -47:03.394 [INFO] TestAgent: consul server down -47:03.394 [INFO] TestAgent: shutdown complete -47:03.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=tcp -47:03.394 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:25323 network=udp -47:03.394 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:25324 network=tcp -47:03.894 [INFO] TestAgent: Waiting for endpoints to shut down -47:03.894 [INFO] TestAgent: Endpoints down ---- PASS: TestWatchCommandNoAgentService (0.88s) +2021-10-05T16:34:39.902Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:39.961Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:39.988 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:29af1436-d9ec-ff96-b4cc-8d78d4bc02da Address:127.0.0.1:23286}]" +34:39.989 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-29af1436-d9ec-ff96-b4cc-8d78d4bc02da.dc1 127.0.0.1 +34:39.989 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-29af1436-d9ec-ff96-b4cc-8d78d4bc02da 127.0.0.1 +2021-10-05T16:34:39.989Z [INFO] agent.router: Initializing LAN area manager +34:39.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=udp +34:39.990 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23286 [Follower]" leader= +34:39.990 [INFO] TestAgent.server: Adding LAN server: server="Node-29af1436-d9ec-ff96-b4cc-8d78d4bc02da (Addr: tcp/127.0.0.1:23286) (DC: dc1)" +34:39.990 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-29af1436-d9ec-ff96-b4cc-8d78d4bc02da.dc1 area=wan +34:39.990 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23281 network=tcp +34:39.991 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23282 network=tcp +34:39.991 [INFO] TestAgent: started state syncer +34:39.991 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23287 network=tcp +34:40.038 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:40.038 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23286 [Candidate]" term=2 +34:40.063 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:40.063 [DEBUG] TestAgent.server.raft: vote granted: from=29af1436-d9ec-ff96-b4cc-8d78d4bc02da term=2 tally=1 +34:40.063 [INFO] TestAgent.server.raft: election won: tally=1 +34:40.063 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23286 [Leader]" +34:40.063 [INFO] TestAgent.server: cluster leadership acquired +34:40.084 [INFO] TestAgent.server: New leader elected: payload=Node-29af1436-d9ec-ff96-b4cc-8d78d4bc02da +34:40.108 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23286 +34:40.148 [INFO] TestAgent: Requesting shutdown +34:40.148 [INFO] TestAgent.server: shutting down server +34:40.148 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:40.148 [ERROR] TestAgent.anti_entropy: failed to sync remote state: error="No cluster leader" +34:40.154 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +34:40.160 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +2021-10-05T16:34:40.162Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:40.162Z [INFO] agent.router.manager: shutting down +34:40.189 [INFO] TestAgent: consul server down +34:40.189 [INFO] TestAgent: shutdown complete +34:40.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=tcp +34:40.189 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23281 network=udp +34:40.189 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23282 network=tcp +34:40.189 [ERROR] TestAgent.server: failed to establish leadership: error="error generating CA root certificate: error computing next serial number: leadership lost while committing log" +34:40.692 [INFO] TestAgent: Waiting for endpoints to shut down +34:40.692 [INFO] TestAgent: Endpoints down +--- PASS: TestWatchCommandNoAgentService (0.85s) +2021-10-05T16:34:39.899Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:39.923Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:40.011 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 Address:127.0.0.1:23300}]" +34:40.011 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23300 [Follower]" leader= +34:40.012 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72.dc1 127.0.0.1 +34:40.013 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 127.0.0.1 +2021-10-05T16:34:40.013Z [INFO] agent.router: Initializing LAN area manager +34:40.013 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72.dc1 area=wan +34:40.013 [INFO] TestAgent.server: Adding LAN server: server="Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 (Addr: tcp/127.0.0.1:23300) (DC: dc1)" +34:40.013 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=udp +34:40.013 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23295 network=tcp +34:40.014 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23296 network=tcp +34:40.060 [INFO] TestAgent: started state syncer +34:40.060 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23301 network=tcp +34:40.084 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:40.084 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23300 [Candidate]" term=2 +34:40.132 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:40.132 [DEBUG] TestAgent.server.raft: vote granted: from=22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 term=2 tally=1 +34:40.132 [INFO] TestAgent.server.raft: election won: tally=1 +34:40.132 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23300 [Leader]" +34:40.132 [INFO] TestAgent.server: cluster leadership acquired +34:40.132 [INFO] TestAgent.server: New leader elected: payload=Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 +34:40.159 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23300 +34:40.194 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:40.230 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:40.230 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:40.230 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:40.230 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:40.230 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:40.230 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 +34:40.230 [INFO] TestAgent.server: member joined, marking health alive: member=Node-22d7fe81-c1db-7a1d-d3a7-8a2696f44e72 +34:40.260 [INFO] TestAgent.server: federation state anti-entropy synced +34:40.464 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:40.480 [INFO] TestAgent: Synced node info +34:40.480 [DEBUG] TestAgent: Node info in sync +34:40.482 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/agent/self from=127.0.0.1:47712 latency=8.356702ms +34:40.490 [DEBUG] TestAgent.http: Request finished: method=GET url=/v1/catalog/nodes from=127.0.0.1:47714 latency=489.373µs +34:40.491 [INFO] TestAgent: Requesting shutdown +34:40.491 [INFO] TestAgent.server: shutting down server +34:40.491 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:40.491 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:40.491 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:40.491 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:40.491 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:40.491 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:40.491 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:40.491 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:40.491 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:40.491 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:40.497 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:40.512Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:40.512Z [INFO] agent.router.manager: shutting down +34:40.512 [INFO] TestAgent: consul server down +34:40.512 [INFO] TestAgent: shutdown complete +34:40.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=tcp +34:40.512 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23295 network=udp +34:40.512 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23296 network=tcp +34:41.013 [INFO] TestAgent: Waiting for endpoints to shut down +34:41.013 [INFO] TestAgent: Endpoints down +--- PASS: TestWatchCommand (1.17s) +2021-10-05T16:34:39.902Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:39.945Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:40.048 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bb157f4e-f046-89e8-0343-2318537658f4 Address:127.0.0.1:23293}]" +34:40.049 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:23293 [Follower]" leader= +34:40.049 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-bb157f4e-f046-89e8-0343-2318537658f4.dc1 127.0.0.1 +34:40.050 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-bb157f4e-f046-89e8-0343-2318537658f4 127.0.0.1 +2021-10-05T16:34:40.050Z [INFO] agent.router: Initializing LAN area manager +34:40.060 [INFO] TestAgent.server: Adding LAN server: server="Node-bb157f4e-f046-89e8-0343-2318537658f4 (Addr: tcp/127.0.0.1:23293) (DC: dc1)" +34:40.060 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=udp +34:40.061 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-bb157f4e-f046-89e8-0343-2318537658f4.dc1 area=wan +34:40.062 [INFO] TestAgent: Started DNS server: address=127.0.0.1:23288 network=tcp +34:40.063 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:23289 network=tcp +34:40.063 [INFO] TestAgent: started state syncer +34:40.063 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:23294 network=tcp +34:40.100 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader= +34:40.100 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:23293 [Candidate]" term=2 +34:40.148 [DEBUG] TestAgent.server.raft: votes: needed=1 +34:40.148 [DEBUG] TestAgent.server.raft: vote granted: from=bb157f4e-f046-89e8-0343-2318537658f4 term=2 tally=1 +34:40.148 [INFO] TestAgent.server.raft: election won: tally=1 +34:40.148 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:23293 [Leader]" +34:40.152 [INFO] TestAgent.server: cluster leadership acquired +34:40.152 [INFO] TestAgent.server: New leader elected: payload=Node-bb157f4e-f046-89e8-0343-2318537658f4 +34:40.169 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:23293 +34:40.188 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:40.229 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul +34:40.229 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy" +34:40.229 [INFO] TestAgent.leader: started routine: routine="federation state pruning" +34:40.229 [INFO] TestAgent.leader: started routine: routine="intermediate cert renew watch" +34:40.229 [INFO] TestAgent.leader: started routine: routine="CA root pruning" +34:40.229 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-bb157f4e-f046-89e8-0343-2318537658f4 +34:40.229 [INFO] TestAgent.server: member joined, marking health alive: member=Node-bb157f4e-f046-89e8-0343-2318537658f4 +34:40.256 [INFO] TestAgent.server: federation state anti-entropy synced +34:40.444 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth +34:40.460 [INFO] TestAgent: Synced node info +34:40.494 [INFO] TestAgent: Requesting shutdown +34:40.494 [INFO] TestAgent.server: shutting down server +34:40.494 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning" +34:40.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy" +34:40.494 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning" +34:40.494 [DEBUG] TestAgent.leader: stopping routine: routine="intermediate cert renew watch" +34:40.494 [ERROR] TestAgent.server: error performing anti-entropy sync of federation state: error="context canceled" +34:40.494 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave +34:40.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state anti-entropy" +34:40.494 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning" +34:40.494 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning" +34:40.494 [DEBUG] TestAgent.leader: stopped routine: routine="intermediate cert renew watch" +34:40.520 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:40.527Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:40.527Z [INFO] agent.router.manager: shutting down +34:40.527 [INFO] TestAgent: consul server down +34:40.527 [INFO] TestAgent: shutdown complete +34:40.527 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=tcp +34:40.527 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:23288 network=udp +34:40.527 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:23289 network=tcp +34:41.028 [INFO] TestAgent: Waiting for endpoints to shut down +34:41.028 [INFO] TestAgent: Endpoints down +--- PASS: TestWatchCommandNoConnect (1.26s) PASS -ok github.com/hashicorp/consul/command/watch 1.790s +ok github.com/hashicorp/consul/command/watch 2.621s === RUN TestStaticResolver_Resolve === RUN TestStaticResolver_Resolve/simples --- PASS: TestStaticResolver_Resolve (0.00s) @@ -91612,97 +91723,99 @@ === RUN TestConsulResolver_Resolve/timeout_errors === RUN TestConsulResolver_Resolve/prepared_query_by_id === RUN TestConsulResolver_Resolve/prepared_query_by_name -2022-11-07T22:47:03.599Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:03.610Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:03.631 [INFO] test-consul.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:40bfd201-0089-fc4c-433f-2749324ae534 Address:127.0.0.1:17153}]" -47:03.631 [INFO] test-consul.server.raft: entering follower state: follower="Node at 127.0.0.1:17153 [Follower]" leader= -47:03.631 [INFO] test-consul.server.serf.wan: serf: EventMemberJoin: Node-40bfd201-0089-fc4c-433f-2749324ae534.dc1 127.0.0.1 -47:03.632 [INFO] test-consul.server.serf.lan: serf: EventMemberJoin: Node-40bfd201-0089-fc4c-433f-2749324ae534 127.0.0.1 -2022-11-07T22:47:03.632Z [INFO] agent.router: Initializing LAN area manager -47:03.632 [INFO] test-consul.server: Adding LAN server: server="Node-40bfd201-0089-fc4c-433f-2749324ae534 (Addr: tcp/127.0.0.1:17153) (DC: dc1)" -47:03.632 [INFO] test-consul: Started DNS server: address=127.0.0.1:17148 network=udp -47:03.632 [INFO] test-consul: Started DNS server: address=127.0.0.1:17148 network=tcp -47:03.632 [INFO] test-consul.server: Handled event for server in area: event=member-join server=Node-40bfd201-0089-fc4c-433f-2749324ae534.dc1 area=wan -47:03.633 [INFO] test-consul: Started HTTP server: address=127.0.0.1:17149 network=tcp -47:03.633 [INFO] test-consul: started state syncer -47:03.633 [INFO] test-consul: Started gRPC server: address=127.0.0.1:17154 network=tcp -47:03.692 [WARN] test-consul.server.raft: heartbeat timeout reached, starting election: last-leader= -47:03.692 [INFO] test-consul.server.raft: entering candidate state: node="Node at 127.0.0.1:17153 [Candidate]" term=2 -47:03.700 [DEBUG] test-consul.server.raft: votes: needed=1 -47:03.700 [DEBUG] test-consul.server.raft: vote granted: from=40bfd201-0089-fc4c-433f-2749324ae534 term=2 tally=1 -47:03.700 [INFO] test-consul.server.raft: election won: tally=1 -47:03.700 [INFO] test-consul.server.raft: entering leader state: leader="Node at 127.0.0.1:17153 [Leader]" -47:03.700 [INFO] test-consul.server: cluster leadership acquired -47:03.701 [INFO] test-consul.server: New leader elected: payload=Node-40bfd201-0089-fc4c-433f-2749324ae534 -47:03.705 [DEBUG] test-consul.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17153 -47:03.716 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:03.732 [INFO] test-consul.server.connect: initialized primary datacenter CA with provider: provider=consul -47:03.732 [INFO] test-consul.leader: started routine: routine="federation state anti-entropy" -47:03.732 [INFO] test-consul.leader: started routine: routine="federation state pruning" -47:03.732 [INFO] test-consul.leader: started routine: routine="intermediate cert renew watch" -47:03.732 [INFO] test-consul.leader: started routine: routine="CA root pruning" -47:03.732 [DEBUG] test-consul.server: Skipping self join check for node since the cluster is too small: node=Node-40bfd201-0089-fc4c-433f-2749324ae534 -47:03.732 [INFO] test-consul.server: member joined, marking health alive: member=Node-40bfd201-0089-fc4c-433f-2749324ae534 -47:03.739 [INFO] test-consul.server: federation state anti-entropy synced -47:03.852 [DEBUG] test-consul: Skipping remote check since it is managed automatically: check=serfHealth -47:03.855 [INFO] test-consul: Synced node info -47:03.992 [DEBUG] test-consul: Node info in sync -47:03.994 [INFO] test-consul: Synced service: service=web -47:03.994 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37912 latency=7.425456ms -47:03.995 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:03.995 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:03.995 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:03.995 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:03.998 [DEBUG] test-consul: Node info in sync -47:03.998 [DEBUG] test-consul: Service in sync: service=web -47:04.002 [INFO] test-consul: Synced service: service=web-proxy -47:04.002 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37912 latency=7.595056ms -47:04.003 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:04.003 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:04.003 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:04.003 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" -47:04.006 [DEBUG] test-consul: Node info in sync -47:04.011 [INFO] test-consul: Synced service: service=web-proxy-2 -47:04.011 [DEBUG] test-consul: Service in sync: service=web -47:04.011 [DEBUG] test-consul: Service in sync: service=web-proxy -47:04.011 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37912 latency=8.223468ms -47:04.015 [DEBUG] test-consul: Node info in sync -47:04.017 [INFO] test-consul: Synced service: service=db -47:04.017 [DEBUG] test-consul: Service in sync: service=web -47:04.017 [DEBUG] test-consul: Service in sync: service=web-proxy -47:04.017 [DEBUG] test-consul: Service in sync: service=web-proxy-2 -47:04.017 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:37912 latency=6.24ms -47:04.022 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/query from=127.0.0.1:37912 latency=4.67832ms -47:04.024 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/web?connect=true&passing=1&stale= from=127.0.0.1:37912 latency=1.101455ms -47:04.028 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/db?connect=true&passing=1&stale= from=127.0.0.1:37912 latency=331.707µs -47:04.029 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/foo?connect=true&passing=1&stale= from=127.0.0.1:37912 latency=302.824µs -47:04.031 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/query/c2e52a69-639f-a74f-f19a-f61c858379a8/execute?connect=true&stale= from=127.0.0.1:37912 latency=1.097544ms -47:04.032 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/query/test-query/execute?connect=true&stale= from=127.0.0.1:37912 latency=221.577µs -47:04.033 [INFO] test-consul: Requesting shutdown -47:04.033 [INFO] test-consul.server: shutting down server -47:04.033 [DEBUG] test-consul.leader: stopping routine: routine="federation state pruning" -47:04.033 [DEBUG] test-consul.leader: stopping routine: routine="intermediate cert renew watch" -47:04.033 [DEBUG] test-consul.leader: stopping routine: routine="CA root pruning" -47:04.033 [DEBUG] test-consul.leader: stopping routine: routine="federation state anti-entropy" -47:04.033 [WARN] test-consul.server.serf.lan: serf: Shutdown without a Leave -47:04.033 [DEBUG] test-consul.leader: stopped routine: routine="federation state pruning" -47:04.033 [ERROR] test-consul.server: error performing anti-entropy sync of federation state: error="context canceled" -47:04.033 [DEBUG] test-consul.leader: stopped routine: routine="federation state anti-entropy" -47:04.033 [DEBUG] test-consul.leader: stopped routine: routine="intermediate cert renew watch" -47:04.033 [DEBUG] test-consul.leader: stopped routine: routine="CA root pruning" -47:04.036 [WARN] test-consul.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:04.040Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:04.040Z [INFO] agent.router.manager: shutting down -47:04.040 [INFO] test-consul: consul server down -47:04.040 [INFO] test-consul: shutdown complete -47:04.040 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:17148 network=tcp -47:04.040 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:17148 network=udp -47:04.040 [INFO] test-consul: Stopping server: protocol=HTTP address=127.0.0.1:17149 network=tcp -47:04.541 [INFO] test-consul: Waiting for endpoints to shut down -47:04.541 [INFO] test-consul: Endpoints down ---- PASS: TestConsulResolver_Resolve (0.98s) - --- PASS: TestConsulResolver_Resolve/basic_service_discovery (0.00s) - --- PASS: TestConsulResolver_Resolve/basic_service_with_native_service (0.00s) +2021-10-05T16:34:45.567Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:45.586Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:45.621 [INFO] test-consul.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:3ec88015-d183-bcde-b264-236e8f1161cf Address:127.0.0.1:27363}]" +34:45.622 [INFO] test-consul.server.serf.wan: serf: EventMemberJoin: Node-3ec88015-d183-bcde-b264-236e8f1161cf.dc1 127.0.0.1 +34:45.622 [INFO] test-consul.server.serf.lan: serf: EventMemberJoin: Node-3ec88015-d183-bcde-b264-236e8f1161cf 127.0.0.1 +2021-10-05T16:34:45.623Z [INFO] agent.router: Initializing LAN area manager +34:45.623 [INFO] test-consul.server.raft: entering follower state: follower="Node at 127.0.0.1:27363 [Follower]" leader= +34:45.626 [INFO] test-consul.server: Adding LAN server: server="Node-3ec88015-d183-bcde-b264-236e8f1161cf (Addr: tcp/127.0.0.1:27363) (DC: dc1)" +34:45.626 [INFO] test-consul.server: Handled event for server in area: event=member-join server=Node-3ec88015-d183-bcde-b264-236e8f1161cf.dc1 area=wan +34:45.626 [INFO] test-consul: Started DNS server: address=127.0.0.1:27358 network=udp +34:45.626 [INFO] test-consul: Started DNS server: address=127.0.0.1:27358 network=tcp +34:45.627 [INFO] test-consul: Started HTTP server: address=127.0.0.1:27359 network=tcp +34:45.627 [INFO] test-consul: started state syncer +34:45.627 [INFO] test-consul: Started gRPC server: address=127.0.0.1:27364 network=tcp +34:45.664 [WARN] test-consul.server.raft: heartbeat timeout reached, starting election: last-leader= +34:45.664 [INFO] test-consul.server.raft: entering candidate state: node="Node at 127.0.0.1:27363 [Candidate]" term=2 +34:45.689 [DEBUG] test-consul.server.raft: votes: needed=1 +34:45.689 [DEBUG] test-consul.server.raft: vote granted: from=3ec88015-d183-bcde-b264-236e8f1161cf term=2 tally=1 +34:45.689 [INFO] test-consul.server.raft: election won: tally=1 +34:45.689 [INFO] test-consul.server.raft: entering leader state: leader="Node at 127.0.0.1:27363 [Leader]" +34:45.690 [INFO] test-consul.server: cluster leadership acquired +34:45.690 [INFO] test-consul.server: New leader elected: payload=Node-3ec88015-d183-bcde-b264-236e8f1161cf +34:45.709 [DEBUG] test-consul.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27363 +34:45.726 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:45.766 [INFO] test-consul.server.connect: initialized primary datacenter CA with provider: provider=consul +34:45.766 [INFO] test-consul.leader: started routine: routine="federation state anti-entropy" +34:45.766 [INFO] test-consul.leader: started routine: routine="federation state pruning" +34:45.766 [INFO] test-consul.leader: started routine: routine="intermediate cert renew watch" +34:45.766 [INFO] test-consul.leader: started routine: routine="CA root pruning" +34:45.766 [DEBUG] test-consul.server: Skipping self join check for node since the cluster is too small: node=Node-3ec88015-d183-bcde-b264-236e8f1161cf +34:45.766 [INFO] test-consul.server: member joined, marking health alive: member=Node-3ec88015-d183-bcde-b264-236e8f1161cf +34:45.785 [INFO] test-consul.server: federation state anti-entropy synced +34:45.919 [INFO] test-consul: Synced node info +34:45.935 [INFO] test-consul: Synced service: service=web +34:45.935 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:32838 latency=27.459309ms +34:45.938 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.938 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.938 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.938 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.960 [DEBUG] test-consul: Node info in sync +34:45.960 [DEBUG] test-consul: Service in sync: service=web +34:45.982 [INFO] test-consul: Synced service: service=web-proxy +34:45.982 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:32838 latency=45.749867ms +34:45.984 [DEBUG] test-consul: Skipping remote check since it is managed automatically: check=serfHealth +34:45.984 [DEBUG] test-consul: Node info in sync +34:45.984 [DEBUG] test-consul: Service in sync: service=web +34:45.991 [INFO] test-consul: Synced service: service=web-proxy +34:45.992 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.992 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.992 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.993 [ERROR] test-consul.proxycfg: watch error: service_id=web-proxy-2 id=service-http-checks: error="error filling agent cache: Internal cache failure: service '' not in agent state" +34:45.995 [DEBUG] test-consul: Node info in sync +34:45.995 [DEBUG] test-consul: Service in sync: service=web +34:45.995 [DEBUG] test-consul: Service in sync: service=web-proxy +34:46.001 [INFO] test-consul: Synced service: service=web-proxy-2 +34:46.001 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:32838 latency=16.619119ms +34:46.010 [DEBUG] test-consul: Node info in sync +34:46.025 [INFO] test-consul: Synced service: service=db +34:46.025 [DEBUG] test-consul: Service in sync: service=web +34:46.025 [DEBUG] test-consul: Service in sync: service=web-proxy +34:46.025 [DEBUG] test-consul: Service in sync: service=web-proxy-2 +34:46.025 [DEBUG] test-consul.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:32838 latency=20.533682ms +34:46.049 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/query from=127.0.0.1:32838 latency=17.019856ms +34:46.061 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/web?connect=true&passing=1&stale= from=127.0.0.1:32838 latency=4.441464ms +34:46.089 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/db?connect=true&passing=1&stale= from=127.0.0.1:32838 latency=1.514703ms +34:46.090 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/health/connect/foo?connect=true&passing=1&stale= from=127.0.0.1:32838 latency=225.967µs +34:46.091 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/query/5037624d-a413-2f19-c6a3-886d475a8c65/execute?connect=true&stale= from=127.0.0.1:32838 latency=372.975µs +34:46.096 [DEBUG] test-consul.http: Request finished: method=GET url=/v1/query/test-query/execute?connect=true&stale= from=127.0.0.1:32838 latency=437.189µs +34:46.097 [INFO] test-consul: Requesting shutdown +34:46.097 [INFO] test-consul.server: shutting down server +34:46.097 [DEBUG] test-consul.leader: stopping routine: routine="CA root pruning" +34:46.097 [DEBUG] test-consul.leader: stopping routine: routine="federation state anti-entropy" +34:46.097 [DEBUG] test-consul.leader: stopping routine: routine="federation state pruning" +34:46.097 [DEBUG] test-consul.leader: stopping routine: routine="intermediate cert renew watch" +34:46.097 [WARN] test-consul.server.serf.lan: serf: Shutdown without a Leave +34:46.097 [ERROR] test-consul.server: error performing anti-entropy sync of federation state: error="context canceled" +34:46.097 [DEBUG] test-consul.leader: stopped routine: routine="federation state anti-entropy" +34:46.097 [DEBUG] test-consul.leader: stopped routine: routine="CA root pruning" +34:46.097 [DEBUG] test-consul.leader: stopped routine: routine="federation state pruning" +34:46.097 [DEBUG] test-consul.leader: stopped routine: routine="intermediate cert renew watch" +34:46.108 [WARN] test-consul.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:46.116Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:46.116Z [INFO] agent.router.manager: shutting down +34:46.116 [INFO] test-consul: consul server down +34:46.116 [INFO] test-consul: shutdown complete +34:46.116 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:27358 network=tcp +34:46.116 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:27358 network=udp +34:46.116 [INFO] test-consul: Stopping server: protocol=HTTP address=127.0.0.1:27359 network=tcp +34:46.619 [INFO] test-consul: Waiting for endpoints to shut down +34:46.619 [INFO] test-consul: Endpoints down +--- PASS: TestConsulResolver_Resolve (1.15s) + --- PASS: TestConsulResolver_Resolve/basic_service_discovery (0.02s) + --- PASS: TestConsulResolver_Resolve/basic_service_with_native_service (0.01s) --- PASS: TestConsulResolver_Resolve/Bad_Type_errors (0.00s) --- PASS: TestConsulResolver_Resolve/Non-existent_service_errors (0.00s) --- PASS: TestConsulResolver_Resolve/timeout_errors (0.00s) @@ -91741,14 +91854,14 @@ service_test.go:129: DM-skipped --- SKIP: TestService_ServerTLSConfig (0.00s) === RUN TestService_HTTPClient -2022/11/07 22:47:04 starting test connect HTTPS server on 127.0.0.1:17155 -2022/11/07 22:47:04 test connect service listening on 127.0.0.1:17155 ---- PASS: TestService_HTTPClient (0.02s) +2021/10/05 16:34:46 starting test connect HTTPS server on 127.0.0.1:27365 +2021/10/05 16:34:46 test connect service listening on 127.0.0.1:27365 +--- PASS: TestService_HTTPClient (0.06s) === RUN TestService_HasDefaultHTTPResolverFromAddr --- PASS: TestService_HasDefaultHTTPResolverFromAddr (0.00s) === RUN Test_verifyServerCertMatchesURI -2022-11-07T22:47:04.573Z [ERROR] connect.watch: Watch errored: service=foo type=connect_roots error="Get "http://127.0.0.1:8500/v1/agent/connect/ca/roots": dial tcp 127.0.0.1:8500: connect: connection refused" retry=5s -2022-11-07T22:47:04.573Z [ERROR] connect.watch: Watch errored: service=foo type=connect_leaf error="Get "http://127.0.0.1:8500/v1/agent/connect/ca/leaf/foo": dial tcp 127.0.0.1:8500: connect: connection refused" retry=5s +2021-10-05T16:34:46.692Z [ERROR] connect.watch: Watch errored: service=foo type=connect_roots error="Get "http://127.0.0.1:8500/v1/agent/connect/ca/roots": dial tcp 127.0.0.1:8500: connect: connection refused" retry=5s +2021-10-05T16:34:46.692Z [ERROR] connect.watch: Watch errored: service=foo type=connect_leaf error="Get "http://127.0.0.1:8500/v1/agent/connect/ca/leaf/foo": dial tcp 127.0.0.1:8500: connect: connection refused" retry=5s === RUN Test_verifyServerCertMatchesURI/simple_match === RUN Test_verifyServerCertMatchesURI/different_trust-domain_allowed === RUN Test_verifyServerCertMatchesURI/mismatch @@ -91765,97 +91878,98 @@ === RUN TestClientSideVerifier/untrusted_CA === RUN TestClientSideVerifier/cross_signed_intermediate === RUN TestClientSideVerifier/cross_signed_without_intermediate ---- PASS: TestClientSideVerifier (0.02s) +--- PASS: TestClientSideVerifier (0.04s) --- PASS: TestClientSideVerifier/ok_service_ca1 (0.00s) --- PASS: TestClientSideVerifier/untrusted_CA (0.00s) - --- PASS: TestClientSideVerifier/cross_signed_intermediate (0.00s) + --- PASS: TestClientSideVerifier/cross_signed_intermediate (0.01s) --- PASS: TestClientSideVerifier/cross_signed_without_intermediate (0.00s) === RUN TestServerSideVerifier === RUN TestServerSideVerifier/ok_service_ca1,_allow === RUN TestServerSideVerifier/untrusted_CA -2022-11-07T22:47:05.047Z [ERROR] TestServerSideVerifier/untrusted_CA: failed TLS verification: error="x509: certificate signed by unknown authority" +2021-10-05T16:34:47.298Z [ERROR] TestServerSideVerifier/untrusted_CA: failed TLS verification: error="x509: certificate signed by unknown authority" === RUN TestServerSideVerifier/cross_signed_intermediate,_allow === RUN TestServerSideVerifier/cross_signed_without_intermediate -2022-11-07T22:47:05.051Z [ERROR] TestServerSideVerifier/cross_signed_without_intermediate: failed TLS verification: error="x509: certificate signed by unknown authority" +2021-10-05T16:34:47.328Z [ERROR] TestServerSideVerifier/cross_signed_without_intermediate: failed TLS verification: error="x509: certificate signed by unknown authority" === RUN TestServerSideVerifier/ok_service_ca1,_deny -2022-11-07T22:47:05.053Z [ERROR] TestServerSideVerifier/ok_service_ca1,_deny: authz call denied: reason="Matched intention: DENY default/* => default/db (ID: ad158449-cb6d-3bdd-ab4d-7923fb6182f1, Precedence: 8)" +2021-10-05T16:34:47.353Z [ERROR] TestServerSideVerifier/ok_service_ca1,_deny: authz call denied: reason="Matched intention: DENY default/* => default/db (ID: 7f9a2049-55b7-51d1-e993-c55eed0193ab, Precedence: 8)" === RUN TestServerSideVerifier/cross_signed_intermediate,_deny -2022-11-07T22:47:05.057Z [ERROR] TestServerSideVerifier/cross_signed_intermediate,_deny: authz call denied: reason="Matched intention: DENY default/* => default/db (ID: ad158449-cb6d-3bdd-ab4d-7923fb6182f1, Precedence: 8)" -2022-11-07T22:47:04.627Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:04.636Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:04.659 [INFO] test-consul.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:bf376796-0eaf-fbf8-120e-56de003cb019 Address:127.0.0.1:17161}]" -47:04.659 [INFO] test-consul.server.raft: entering follower state: follower="Node at 127.0.0.1:17161 [Follower]" leader= -47:04.660 [INFO] test-consul.server.serf.wan: serf: EventMemberJoin: Node-bf376796-0eaf-fbf8-120e-56de003cb019.dc1 127.0.0.1 -47:04.660 [INFO] test-consul.server.serf.lan: serf: EventMemberJoin: Node-bf376796-0eaf-fbf8-120e-56de003cb019 127.0.0.1 -2022-11-07T22:47:04.660Z [INFO] agent.router: Initializing LAN area manager -47:04.660 [INFO] test-consul.server: Adding LAN server: server="Node-bf376796-0eaf-fbf8-120e-56de003cb019 (Addr: tcp/127.0.0.1:17161) (DC: dc1)" -47:04.660 [INFO] test-consul.server: Handled event for server in area: event=member-join server=Node-bf376796-0eaf-fbf8-120e-56de003cb019.dc1 area=wan -47:04.660 [INFO] test-consul: Started DNS server: address=127.0.0.1:17156 network=udp -47:04.660 [INFO] test-consul: Started DNS server: address=127.0.0.1:17156 network=tcp -47:04.661 [INFO] test-consul: Started HTTP server: address=127.0.0.1:17157 network=tcp -47:04.661 [INFO] test-consul: started state syncer -47:04.661 [INFO] test-consul: Started gRPC server: address=127.0.0.1:17162 network=tcp -47:04.706 [WARN] test-consul.server.raft: heartbeat timeout reached, starting election: last-leader= -47:04.706 [INFO] test-consul.server.raft: entering candidate state: node="Node at 127.0.0.1:17161 [Candidate]" term=2 -47:04.714 [DEBUG] test-consul.server.raft: votes: needed=1 -47:04.714 [DEBUG] test-consul.server.raft: vote granted: from=bf376796-0eaf-fbf8-120e-56de003cb019 term=2 tally=1 -47:04.714 [INFO] test-consul.server.raft: election won: tally=1 -47:04.714 [INFO] test-consul.server.raft: entering leader state: leader="Node at 127.0.0.1:17161 [Leader]" -47:04.714 [INFO] test-consul.server: cluster leadership acquired -47:04.715 [INFO] test-consul.server: New leader elected: payload=Node-bf376796-0eaf-fbf8-120e-56de003cb019 -47:04.720 [DEBUG] test-consul.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:17161 -47:04.738 [INFO] test-consul: Synced node info -47:04.738 [DEBUG] test-consul: Node info in sync -47:04.738 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:04.753 [INFO] test-consul.server.connect: initialized primary datacenter CA with provider: provider=consul -47:04.753 [INFO] test-consul.leader: started routine: routine="federation state anti-entropy" -47:04.753 [INFO] test-consul.leader: started routine: routine="federation state pruning" -47:04.753 [INFO] test-consul.leader: started routine: routine="intermediate cert renew watch" -47:04.753 [INFO] test-consul.leader: started routine: routine="CA root pruning" -47:04.753 [DEBUG] test-consul.server: Skipping self join check for node since the cluster is too small: node=Node-bf376796-0eaf-fbf8-120e-56de003cb019 -47:04.753 [INFO] test-consul.server: member joined, marking health alive: member=Node-bf376796-0eaf-fbf8-120e-56de003cb019 -47:04.763 [INFO] test-consul.server: federation state anti-entropy synced -47:05.026 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:60118 latency=3.671464ms -47:05.029 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:60118 latency=3.166943ms -47:05.047 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:60118 latency=549.099µs -47:05.051 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:60118 latency=116.186µs -47:05.053 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:60118 latency=202.237µs -47:05.057 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:60118 latency=119.974µs -47:05.057 [INFO] test-consul: Requesting shutdown -47:05.057 [INFO] test-consul.server: shutting down server -47:05.057 [DEBUG] test-consul.leader: stopping routine: routine="intermediate cert renew watch" -47:05.057 [DEBUG] test-consul.leader: stopping routine: routine="CA root pruning" -47:05.057 [DEBUG] test-consul.leader: stopping routine: routine="federation state anti-entropy" -47:05.057 [DEBUG] test-consul.leader: stopping routine: routine="federation state pruning" -47:05.057 [WARN] test-consul.server.serf.lan: serf: Shutdown without a Leave -47:05.057 [ERROR] test-consul.server: error performing anti-entropy sync of federation state: error="context canceled" -47:05.057 [DEBUG] test-consul.leader: stopped routine: routine="federation state anti-entropy" -47:05.057 [DEBUG] test-consul.leader: stopped routine: routine="intermediate cert renew watch" -47:05.057 [DEBUG] test-consul.leader: stopped routine: routine="CA root pruning" -47:05.057 [DEBUG] test-consul.leader: stopped routine: routine="federation state pruning" -47:05.061 [WARN] test-consul.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:05.065Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:05.065Z [INFO] agent.router.manager: shutting down -47:05.065 [INFO] test-consul: consul server down -47:05.065 [INFO] test-consul: shutdown complete -47:05.065 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:17156 network=tcp -47:05.065 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:17156 network=udp -47:05.065 [INFO] test-consul: Stopping server: protocol=HTTP address=127.0.0.1:17157 network=tcp -47:05.565 [INFO] test-consul: Waiting for endpoints to shut down -47:05.565 [INFO] test-consul: Endpoints down ---- PASS: TestServerSideVerifier (0.96s) - --- PASS: TestServerSideVerifier/ok_service_ca1,_allow (0.00s) +2021-10-05T16:34:47.384Z [ERROR] TestServerSideVerifier/cross_signed_intermediate,_deny: authz call denied: reason="Matched intention: DENY default/* => default/db (ID: 7f9a2049-55b7-51d1-e993-c55eed0193ab, Precedence: 8)" +2021-10-05T16:34:46.815Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:46.835Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:46.885 [INFO] test-consul.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:62cca5be-25fa-37e1-f032-1eae13ae23de Address:127.0.0.1:27371}]" +34:46.886 [INFO] test-consul.server.serf.wan: serf: EventMemberJoin: Node-62cca5be-25fa-37e1-f032-1eae13ae23de.dc1 127.0.0.1 +34:46.886 [INFO] test-consul.server.serf.lan: serf: EventMemberJoin: Node-62cca5be-25fa-37e1-f032-1eae13ae23de 127.0.0.1 +2021-10-05T16:34:46.886Z [INFO] agent.router: Initializing LAN area manager +34:46.887 [INFO] test-consul: Started DNS server: address=127.0.0.1:27366 network=udp +34:46.887 [INFO] test-consul.server.raft: entering follower state: follower="Node at 127.0.0.1:27371 [Follower]" leader= +34:46.887 [INFO] test-consul.server: Adding LAN server: server="Node-62cca5be-25fa-37e1-f032-1eae13ae23de (Addr: tcp/127.0.0.1:27371) (DC: dc1)" +34:46.887 [INFO] test-consul.server: Handled event for server in area: event=member-join server=Node-62cca5be-25fa-37e1-f032-1eae13ae23de.dc1 area=wan +34:46.887 [INFO] test-consul: Started DNS server: address=127.0.0.1:27366 network=tcp +34:46.888 [INFO] test-consul: Started HTTP server: address=127.0.0.1:27367 network=tcp +34:46.888 [INFO] test-consul: started state syncer +34:46.888 [INFO] test-consul: Started gRPC server: address=127.0.0.1:27372 network=tcp +34:46.951 [WARN] test-consul.server.raft: heartbeat timeout reached, starting election: last-leader= +34:46.951 [INFO] test-consul.server.raft: entering candidate state: node="Node at 127.0.0.1:27371 [Candidate]" term=2 +34:46.962 [DEBUG] test-consul.server.raft: votes: needed=1 +34:46.962 [DEBUG] test-consul.server.raft: vote granted: from=62cca5be-25fa-37e1-f032-1eae13ae23de term=2 tally=1 +34:46.962 [INFO] test-consul.server.raft: election won: tally=1 +34:46.962 [INFO] test-consul.server.raft: entering leader state: leader="Node at 127.0.0.1:27371 [Leader]" +34:46.964 [INFO] test-consul.server: cluster leadership acquired +34:46.964 [INFO] test-consul.server: New leader elected: payload=Node-62cca5be-25fa-37e1-f032-1eae13ae23de +34:46.967 [DEBUG] test-consul.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:27371 +34:46.979 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:47.023 [INFO] test-consul.server.connect: initialized primary datacenter CA with provider: provider=consul +34:47.023 [INFO] test-consul.leader: started routine: routine="federation state anti-entropy" +34:47.023 [INFO] test-consul.leader: started routine: routine="federation state pruning" +34:47.023 [INFO] test-consul.leader: started routine: routine="intermediate cert renew watch" +34:47.024 [INFO] test-consul.leader: started routine: routine="CA root pruning" +34:47.024 [DEBUG] test-consul.server: Skipping self join check for node since the cluster is too small: node=Node-62cca5be-25fa-37e1-f032-1eae13ae23de +34:47.024 [INFO] test-consul.server: member joined, marking health alive: member=Node-62cca5be-25fa-37e1-f032-1eae13ae23de +34:47.036 [INFO] test-consul.server: federation state anti-entropy synced +34:47.171 [DEBUG] test-consul: Skipping remote check since it is managed automatically: check=serfHealth +34:47.175 [INFO] test-consul: Synced node info +34:47.175 [DEBUG] test-consul: Node info in sync +34:47.196 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:35072 latency=11.460664ms +34:47.216 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/connect/intentions from=127.0.0.1:35072 latency=16.03238ms +34:47.297 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:35072 latency=766.991µs +34:47.314 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:35072 latency=500.029µs +34:47.352 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:35072 latency=8.546987ms +34:47.384 [DEBUG] test-consul.http: Request finished: method=POST url=/v1/agent/connect/authorize from=127.0.0.1:35072 latency=4.002034ms +34:47.385 [INFO] test-consul: Requesting shutdown +34:47.385 [INFO] test-consul.server: shutting down server +34:47.385 [DEBUG] test-consul.leader: stopping routine: routine="federation state pruning" +34:47.385 [DEBUG] test-consul.leader: stopping routine: routine="intermediate cert renew watch" +34:47.385 [DEBUG] test-consul.leader: stopping routine: routine="CA root pruning" +34:47.385 [DEBUG] test-consul.leader: stopping routine: routine="federation state anti-entropy" +34:47.385 [WARN] test-consul.server.serf.lan: serf: Shutdown without a Leave +34:47.385 [ERROR] test-consul.server: error performing anti-entropy sync of federation state: error="context canceled" +34:47.385 [DEBUG] test-consul.leader: stopped routine: routine="federation state anti-entropy" +34:47.385 [DEBUG] test-consul.leader: stopped routine: routine="federation state pruning" +34:47.385 [DEBUG] test-consul.leader: stopped routine: routine="intermediate cert renew watch" +34:47.385 [DEBUG] test-consul.leader: stopped routine: routine="CA root pruning" +34:47.400 [WARN] test-consul.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:47.408Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:47.408Z [INFO] agent.router.manager: shutting down +34:47.408 [INFO] test-consul: consul server down +34:47.408 [INFO] test-consul: shutdown complete +34:47.408 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:27366 network=tcp +34:47.408 [INFO] test-consul: Stopping server: protocol=DNS address=127.0.0.1:27366 network=udp +34:47.408 [INFO] test-consul: Stopping server: protocol=HTTP address=127.0.0.1:27367 network=tcp +34:47.909 [INFO] test-consul: Waiting for endpoints to shut down +34:47.909 [INFO] test-consul: Endpoints down +--- PASS: TestServerSideVerifier (1.16s) + --- PASS: TestServerSideVerifier/ok_service_ca1,_allow (0.02s) --- PASS: TestServerSideVerifier/untrusted_CA (0.00s) - --- PASS: TestServerSideVerifier/cross_signed_intermediate,_allow (0.00s) + --- PASS: TestServerSideVerifier/cross_signed_intermediate,_allow (0.02s) --- PASS: TestServerSideVerifier/cross_signed_without_intermediate (0.00s) - --- PASS: TestServerSideVerifier/ok_service_ca1,_deny (0.00s) - --- PASS: TestServerSideVerifier/cross_signed_intermediate,_deny (0.00s) + --- PASS: TestServerSideVerifier/ok_service_ca1,_deny (0.02s) + --- PASS: TestServerSideVerifier/cross_signed_intermediate,_deny (0.03s) === RUN TestDynamicTLSConfig --- PASS: TestDynamicTLSConfig (0.02s) === RUN TestDynamicTLSConfig_Ready ---- PASS: TestDynamicTLSConfig_Ready (0.01s) +--- PASS: TestDynamicTLSConfig_Ready (0.04s) PASS -ok github.com/hashicorp/consul/connect 2.060s +ok github.com/hashicorp/consul/connect 2.795s ? github.com/hashicorp/consul/connect/certgen [no test files] === RUN TestUpstreamResolverFuncFromClient === PAUSE TestUpstreamResolverFuncFromClient @@ -91878,237 +91992,247 @@ proxy_test.go:21: DM-skipped --- SKIP: TestProxy_public (0.00s) === CONT TestUpstreamResolverFuncFromClient -=== CONT TestConnDstClosing -=== CONT TestAgentConfigWatcherSidecarProxy -=== CONT TestConnSrcClosing === RUN TestUpstreamResolverFuncFromClient/service -[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... === RUN TestUpstreamResolverFuncFromClient/prepared_query === RUN TestUpstreamResolverFuncFromClient/unknown_behaves_like_service --- PASS: TestUpstreamResolverFuncFromClient (0.00s) --- PASS: TestUpstreamResolverFuncFromClient/service (0.00s) --- PASS: TestUpstreamResolverFuncFromClient/prepared_query (0.00s) --- PASS: TestUpstreamResolverFuncFromClient/unknown_behaves_like_service (0.00s) +=== CONT TestConnDstClosing +=== CONT TestConnSrcClosing +=== CONT TestAgentConfigWatcherSidecarProxy +[INFO] freeport: blockSize 1500 too big for system limit 1024. Adjusting... [INFO] freeport: detected ephemeral port range of [32768, 60999] [INFO] freeport: reducing max blocks from 30 to 22 to avoid the ephemeral port range ---- PASS: TestConnDstClosing (0.01s) ---- PASS: TestConnSrcClosing (0.01s) -2022-11-07T22:47:04.820Z [ERROR] TestAgentConfigWatcherSidecarProxy.watch: Watch errored: service_id=web-sidecar-proxy type=agent_service error="Get "http://127.0.0.1:25317/v1/agent/service/web-sidecar-proxy?hash=790f39b13effbdf2": dial tcp 127.0.0.1:25317: connect: connection refused" retry=5s -2022-11-07T22:47:04.517Z [WARN] agent: bootstrap = true: do not enable unless necessary -2022-11-07T22:47:04.527Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary -47:04.549 [INFO] agent_smith.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:9c166bf2-8fff-4a62-7812-44dcfda8e2e3 Address:127.0.0.1:25321}]" -47:04.549 [INFO] agent_smith.server.raft: entering follower state: follower="Node at 127.0.0.1:25321 [Follower]" leader= -47:04.550 [INFO] agent_smith.server.serf.wan: serf: EventMemberJoin: Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3.dc1 127.0.0.1 -47:04.550 [INFO] agent_smith.server.serf.lan: serf: EventMemberJoin: Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3 127.0.0.1 -2022-11-07T22:47:04.551Z [INFO] agent.router: Initializing LAN area manager -47:04.551 [INFO] agent_smith.server: Adding LAN server: server="Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3 (Addr: tcp/127.0.0.1:25321) (DC: dc1)" -47:04.551 [INFO] agent_smith.server: Handled event for server in area: event=member-join server=Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3.dc1 area=wan -47:04.551 [INFO] agent_smith: Started DNS server: address=127.0.0.1:25316 network=udp -47:04.551 [INFO] agent_smith: Started DNS server: address=127.0.0.1:25316 network=tcp -47:04.551 [INFO] agent_smith: Started HTTP server: address=127.0.0.1:25317 network=tcp -47:04.551 [INFO] agent_smith: started state syncer -47:04.551 [INFO] agent_smith: Started gRPC server: address=127.0.0.1:25322 network=tcp -47:04.619 [WARN] agent_smith.server.raft: heartbeat timeout reached, starting election: last-leader= -47:04.619 [INFO] agent_smith.server.raft: entering candidate state: node="Node at 127.0.0.1:25321 [Candidate]" term=2 -47:04.632 [DEBUG] agent_smith.server.raft: votes: needed=1 -47:04.632 [DEBUG] agent_smith.server.raft: vote granted: from=9c166bf2-8fff-4a62-7812-44dcfda8e2e3 term=2 tally=1 -47:04.632 [INFO] agent_smith.server.raft: election won: tally=1 -47:04.632 [INFO] agent_smith.server.raft: entering leader state: leader="Node at 127.0.0.1:25321 [Leader]" -47:04.632 [INFO] agent_smith.server: cluster leadership acquired -47:04.632 [INFO] agent_smith.server: New leader elected: payload=Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3 -47:04.637 [DEBUG] agent_smith.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:25321 -47:04.651 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true -47:04.671 [INFO] agent_smith.server.connect: initialized primary datacenter CA with provider: provider=consul -47:04.671 [INFO] agent_smith.leader: started routine: routine="federation state anti-entropy" -47:04.671 [INFO] agent_smith.leader: started routine: routine="federation state pruning" -47:04.671 [INFO] agent_smith.leader: started routine: routine="intermediate cert renew watch" -47:04.671 [INFO] agent_smith.leader: started routine: routine="CA root pruning" -47:04.671 [DEBUG] agent_smith.server: Skipping self join check for node since the cluster is too small: node=Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3 -47:04.671 [INFO] agent_smith.server: member joined, marking health alive: member=Node-9c166bf2-8fff-4a62-7812-44dcfda8e2e3 -47:04.676 [INFO] agent_smith.server: federation state anti-entropy synced -47:04.754 [INFO] agent_smith: Synced node info -47:04.760 [INFO] agent_smith: Synced service: service=web-sidecar-proxy -47:04.765 [INFO] agent_smith: Synced service: service=web -47:04.765 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 -47:04.765 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 -47:04.765 [DEBUG] agent_smith.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54010 latency=31.775536ms -47:04.766 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy from=127.0.0.1:54010 latency=505.362µs -47:04.803 [DEBUG] agent_smith: Node info in sync -47:04.806 [INFO] agent_smith: Synced service: service=web -47:04.810 [INFO] agent_smith: Synced service: service=web-sidecar-proxy -47:04.810 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 -47:04.810 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 -47:04.810 [DEBUG] agent_smith.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:54012 latency=20.817804ms -47:04.811 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy?hash=e79fad2be70b72e8 from=127.0.0.1:54010 latency=43.792057ms -47:04.811 [INFO] agent_smith: Requesting shutdown -47:04.811 [INFO] agent_smith.server: shutting down server -47:04.811 [DEBUG] agent_smith.leader: stopping routine: routine="intermediate cert renew watch" -47:04.811 [DEBUG] agent_smith.leader: stopping routine: routine="CA root pruning" -47:04.811 [DEBUG] agent_smith.leader: stopping routine: routine="federation state anti-entropy" -47:04.811 [DEBUG] agent_smith.leader: stopping routine: routine="federation state pruning" -47:04.811 [WARN] agent_smith.server.serf.lan: serf: Shutdown without a Leave -47:04.811 [ERROR] agent_smith.server: error performing anti-entropy sync of federation state: error="context canceled" -47:04.811 [DEBUG] agent_smith.leader: stopped routine: routine="federation state anti-entropy" -47:04.811 [ERROR] agent_smith.anti_entropy: failed to sync remote state: error="No cluster leader" -47:04.811 [DEBUG] agent_smith.leader: stopped routine: routine="intermediate cert renew watch" -47:04.811 [DEBUG] agent_smith.leader: stopped routine: routine="CA root pruning" -47:04.811 [DEBUG] agent_smith.leader: stopped routine: routine="federation state pruning" -47:04.815 [WARN] agent_smith.server.serf.wan: serf: Shutdown without a Leave -2022-11-07T22:47:04.818Z [INFO] agent.router.manager: shutting down -2022-11-07T22:47:04.818Z [INFO] agent.router.manager: shutting down -47:04.818 [INFO] agent_smith: consul server down -47:04.818 [INFO] agent_smith: shutdown complete -47:04.818 [INFO] agent_smith: Stopping server: protocol=DNS address=127.0.0.1:25316 network=tcp -47:04.818 [INFO] agent_smith: Stopping server: protocol=DNS address=127.0.0.1:25316 network=udp -47:04.818 [INFO] agent_smith: Stopping server: protocol=HTTP address=127.0.0.1:25317 network=tcp -47:04.819 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy?hash=790f39b13effbdf2 from=127.0.0.1:54010 latency=7.404836ms -47:05.318 [INFO] agent_smith: Waiting for endpoints to shut down -47:05.318 [INFO] agent_smith: Endpoints down ---- PASS: TestAgentConfigWatcherSidecarProxy (0.84s) +--- PASS: TestConnDstClosing (0.03s) +--- PASS: TestConnSrcClosing (0.03s) +2021-10-05T16:34:44.990Z [ERROR] TestAgentConfigWatcherSidecarProxy.watch: Watch errored: service_id=web-sidecar-proxy type=agent_service error="Get "http://127.0.0.1:22254/v1/agent/service/web-sidecar-proxy?hash=790f39b13effbdf2": dial tcp 127.0.0.1:22254: connect: connection refused" retry=5s +2021-10-05T16:34:44.359Z [WARN] agent: bootstrap = true: do not enable unless necessary +2021-10-05T16:34:44.375Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary +34:44.419 [INFO] agent_smith.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f Address:127.0.0.1:22258}]" +34:44.420 [INFO] agent_smith.server.raft: entering follower state: follower="Node at 127.0.0.1:22258 [Follower]" leader= +34:44.420 [INFO] agent_smith.server.serf.wan: serf: EventMemberJoin: Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f.dc1 127.0.0.1 +34:44.421 [INFO] agent_smith.server.serf.lan: serf: EventMemberJoin: Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f 127.0.0.1 +2021-10-05T16:34:44.421Z [INFO] agent.router: Initializing LAN area manager +34:44.421 [INFO] agent_smith: Started DNS server: address=127.0.0.1:22253 network=udp +34:44.421 [INFO] agent_smith.server: Adding LAN server: server="Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f (Addr: tcp/127.0.0.1:22258) (DC: dc1)" +34:44.421 [INFO] agent_smith.server: Handled event for server in area: event=member-join server=Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f.dc1 area=wan +34:44.422 [INFO] agent_smith: Started DNS server: address=127.0.0.1:22253 network=tcp +34:44.422 [INFO] agent_smith: Started HTTP server: address=127.0.0.1:22254 network=tcp +34:44.422 [INFO] agent_smith: started state syncer +34:44.422 [INFO] agent_smith: Started gRPC server: address=127.0.0.1:22259 network=tcp +34:44.468 [WARN] agent_smith.server.raft: heartbeat timeout reached, starting election: last-leader= +34:44.468 [INFO] agent_smith.server.raft: entering candidate state: node="Node at 127.0.0.1:22258 [Candidate]" term=2 +34:44.490 [DEBUG] agent_smith.server.raft: votes: needed=1 +34:44.490 [DEBUG] agent_smith.server.raft: vote granted: from=a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f term=2 tally=1 +34:44.490 [INFO] agent_smith.server.raft: election won: tally=1 +34:44.490 [INFO] agent_smith.server.raft: entering leader state: leader="Node at 127.0.0.1:22258 [Leader]" +34:44.491 [INFO] agent_smith.server: cluster leadership acquired +34:44.491 [INFO] agent_smith.server: New leader elected: payload=Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f +34:44.508 [DEBUG] agent_smith.server: Cannot upgrade to new ACLs: leaderMode=0 mode=0 found=true leader=127.0.0.1:22258 +34:44.572 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true +34:44.640 [INFO] agent_smith.server.connect: initialized primary datacenter CA with provider: provider=consul +34:44.640 [INFO] agent_smith.leader: started routine: routine="federation state anti-entropy" +34:44.640 [INFO] agent_smith.leader: started routine: routine="federation state pruning" +34:44.640 [INFO] agent_smith.leader: started routine: routine="intermediate cert renew watch" +34:44.640 [INFO] agent_smith.leader: started routine: routine="CA root pruning" +34:44.640 [DEBUG] agent_smith.server: Skipping self join check for node since the cluster is too small: node=Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f +34:44.640 [INFO] agent_smith.server: member joined, marking health alive: member=Node-a6ccd965-588e-b9b3-0c4a-f6c74a3ca31f +34:44.658 [INFO] agent_smith.server: federation state anti-entropy synced +34:44.844 [DEBUG] agent_smith: Skipping remote check since it is managed automatically: check=serfHealth +34:44.864 [INFO] agent_smith: Synced node info +34:44.893 [INFO] agent_smith: Synced service: service=web +34:44.906 [INFO] agent_smith: Synced service: service=web-sidecar-proxy +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 +34:44.906 [DEBUG] agent_smith: Node info in sync +34:44.906 [DEBUG] agent_smith: Service in sync: service=web +34:44.906 [DEBUG] agent_smith: Service in sync: service=web-sidecar-proxy +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 +34:44.906 [DEBUG] agent_smith: Node info in sync +34:44.906 [DEBUG] agent_smith: Service in sync: service=web +34:44.906 [DEBUG] agent_smith: Service in sync: service=web-sidecar-proxy +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 +34:44.906 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 +34:44.906 [DEBUG] agent_smith.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:47542 latency=85.891567ms +34:44.908 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy from=127.0.0.1:47542 latency=1.080533ms +34:44.964 [DEBUG] agent_smith: Node info in sync +34:44.970 [INFO] agent_smith: Synced service: service=web +34:44.978 [INFO] agent_smith: Synced service: service=web-sidecar-proxy +34:44.978 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:2 +34:44.978 [DEBUG] agent_smith: Check in sync: check=service:web-sidecar-proxy:1 +34:44.978 [DEBUG] agent_smith.http: Request finished: method=PUT url=/v1/agent/service/register from=127.0.0.1:47546 latency=40.02743ms +34:44.979 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy?hash=e79fad2be70b72e8 from=127.0.0.1:47542 latency=58.549464ms +34:44.979 [INFO] agent_smith: Requesting shutdown +34:44.979 [INFO] agent_smith.server: shutting down server +34:44.979 [DEBUG] agent_smith.leader: stopping routine: routine="federation state anti-entropy" +34:44.979 [DEBUG] agent_smith.leader: stopping routine: routine="federation state pruning" +34:44.979 [DEBUG] agent_smith.leader: stopping routine: routine="intermediate cert renew watch" +34:44.979 [DEBUG] agent_smith.leader: stopping routine: routine="CA root pruning" +34:44.979 [WARN] agent_smith.server.serf.lan: serf: Shutdown without a Leave +34:44.980 [ERROR] agent_smith.server: error performing anti-entropy sync of federation state: error="context canceled" +34:44.980 [DEBUG] agent_smith.leader: stopped routine: routine="federation state anti-entropy" +34:44.980 [DEBUG] agent_smith.leader: stopped routine: routine="federation state pruning" +34:44.980 [DEBUG] agent_smith.leader: stopped routine: routine="intermediate cert renew watch" +34:44.980 [DEBUG] agent_smith.leader: stopped routine: routine="CA root pruning" +34:44.987 [WARN] agent_smith.server.serf.wan: serf: Shutdown without a Leave +2021-10-05T16:34:44.987Z [INFO] agent.router.manager: shutting down +2021-10-05T16:34:44.987Z [INFO] agent.router.manager: shutting down +34:44.987 [INFO] agent_smith: consul server down +34:44.987 [INFO] agent_smith: shutdown complete +34:44.987 [INFO] agent_smith: Stopping server: protocol=DNS address=127.0.0.1:22253 network=tcp +34:44.987 [INFO] agent_smith: Stopping server: protocol=DNS address=127.0.0.1:22253 network=udp +34:44.987 [INFO] agent_smith: Stopping server: protocol=HTTP address=127.0.0.1:22254 network=tcp +34:44.989 [DEBUG] agent_smith.http: Request finished: method=GET url=/v1/agent/service/web-sidecar-proxy?hash=790f39b13effbdf2 from=127.0.0.1:47542 latency=9.520377ms +34:45.488 [INFO] agent_smith: Waiting for endpoints to shut down +34:45.488 [INFO] agent_smith: Endpoints down +--- PASS: TestAgentConfigWatcherSidecarProxy (1.27s) PASS -ok github.com/hashicorp/consul/connect/proxy 0.867s +ok github.com/hashicorp/consul/connect/proxy 1.536s === RUN TestConfigValidate -=== RUN TestConfigValidate/type=inferred_oidc/all_required -=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientSecret -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_BoundIssuer -=== RUN TestConfigValidate/type=inferred_oidc/missing_required_AllowedRedirectURIs -=== RUN TestConfigValidate/type=jwt/all_required_for_jwks -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSCACert -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_ClockSkewLeeway -=== RUN TestConfigValidate/type=inferred_oidc/bad_discovery_cert -=== RUN TestConfigValidate/type=jwt/valid_claim_mappings -=== RUN TestConfigValidate/type=jwt/all_required_for_public_keys -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_NotBeforeLeeway -=== RUN TestConfigValidate/type=oidc/incompatible_with_ClockSkewLeeway -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWTValidationPubKeys -=== RUN TestConfigValidate/type=oidc/incompatible_with_ExpirationLeeway -=== RUN TestConfigValidate/type=oidc/invalid_repeated_value_claim_mappings -=== RUN TestConfigValidate/type=oidc/bad_discovery_cert -=== RUN TestConfigValidate/type=inferred_oidc/garbage_discovery_cert -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSURL -=== RUN TestConfigValidate/type=jwt/incompatible_with_VerboseOIDCLogging +=== RUN TestConfigValidate/type=jwt/invalid_pubkey === RUN TestConfigValidate/bad_type +=== RUN TestConfigValidate/type=oidc/missing_required_OIDCDiscoveryURL +=== RUN TestConfigValidate/type=oidc/incompatible_with_JWTValidationPubKeys === RUN TestConfigValidate/type=inferred_oidc/valid_redirect_uris -=== RUN TestConfigValidate/type=oidc/invalid_algorithm -=== RUN TestConfigValidate/type=inferred_oidc/invalid_algorithm -=== RUN TestConfigValidate/type=jwt/invalid_repeated_list_claim_mappings -=== RUN TestConfigValidate/type=jwt/incompatible_with_OIDCClientSecret -=== RUN TestConfigValidate/type=inferred_oidc/invalid_redirect_uris -=== RUN TestConfigValidate/type=inferred_oidc/valid_algorithm -=== RUN TestConfigValidate/type=jwt/all_required_for_oidc_discovery +=== RUN TestConfigValidate/type=oidc/incompatible_with_BoundIssuer +=== RUN TestConfigValidate/type=oidc/invalid_repeated_value_claim_mappings +=== RUN TestConfigValidate/type=jwt/all_required_for_public_keys === RUN TestConfigValidate/type=jwt/incompatible_with_OIDCScopes -=== RUN TestConfigValidate/type=oidc/incompatible_with_JWTValidationPubKeys -=== RUN TestConfigValidate/type=jwt/incompatible_with_AllowedRedirectURIs -=== RUN TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks) +=== RUN TestConfigValidate/type=jwt/incompatible_with_JWKSCACert +=== RUN TestConfigValidate/type=jwt/jwks_bad_certs +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSURL +=== RUN TestConfigValidate/type=oidc/incompatible_with_ExpirationLeeway +=== RUN TestConfigValidate/type=inferred_oidc/invalid_redirect_uris +=== RUN TestConfigValidate/type=oidc/valid_algorithm +=== RUN TestConfigValidate/type=inferred_oidc/invalid_repeated_value_claim_mappings === RUN TestConfigValidate/type=jwt/jwks_invalid_404 -=== RUN TestConfigValidate/type=jwt/valid_algorithm -=== RUN TestConfigValidate/type=inferred_oidc/good_discovery_cert +=== RUN TestConfigValidate/type=oidc/invalid_repeated_list_claim_mappings +=== RUN TestConfigValidate/type=oidc/valid_claim_mappings === RUN TestConfigValidate/type=jwt/bad_discovery_cert -=== RUN TestConfigValidate/type=oidc/all_required -=== RUN TestConfigValidate/type=oidc/incompatible_with_BoundIssuer -=== RUN TestConfigValidate/type=inferred_oidc/valid_claim_mappings -=== RUN TestConfigValidate/type=oidc/valid_redirect_uris -=== RUN TestConfigValidate/type=oidc/missing_required_OIDCClientSecret -=== RUN TestConfigValidate/type=oidc/incompatible_with_JWKSURL +=== RUN TestConfigValidate/type=jwt/valid_claim_mappings === RUN TestConfigValidate/type=jwt/incompatible_with_OIDCDiscoveryCACert -=== RUN TestConfigValidate/type=jwt/too_many_methods_(jwks_+_pubkeys) -=== RUN TestConfigValidate/type=jwt/good_discovery_cert -=== RUN TestConfigValidate/type=jwt/jwks_bad_certs -=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientID -=== RUN TestConfigValidate/type=oidc/valid_claim_mappings -=== RUN TestConfigValidate/type=jwt/incompatible_with_OIDCClientID === RUN TestConfigValidate/type=jwt/too_many_methods_(discovery_+_pubkeys) +=== RUN TestConfigValidate/type=jwt/valid_algorithm +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSCACert +=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCDiscoveryURL +=== RUN TestConfigValidate/type=oidc/good_discovery_cert +=== RUN TestConfigValidate/type=inferred_oidc/missing_required_AllowedRedirectURIs +=== RUN TestConfigValidate/type=oidc/incompatible_with_ClockSkewLeeway +=== RUN TestConfigValidate/type=oidc/incompatible_with_JWKSURL +=== RUN TestConfigValidate/type=inferred_oidc/invalid_repeated_list_claim_mappings +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_JWTValidationPubKeys +=== RUN TestConfigValidate/type=oidc/all_required +=== RUN TestConfigValidate/type=jwt/all_required_for_oidc_discovery === RUN TestConfigValidate/type=jwt/invalid_algorithm -=== RUN TestConfigValidate/type=jwt/incompatible_with_JWKSCACert === RUN TestConfigValidate/type=oidc/incompatible_with_JWKSCACert -=== RUN TestConfigValidate/type=oidc/missing_required_OIDCClientID -=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCDiscoveryURL -=== RUN TestConfigValidate/type=jwt/invalid_repeated_value_claim_mappings -=== RUN TestConfigValidate/type=oidc/incompatible_with_NotBeforeLeeway -=== RUN TestConfigValidate/type=oidc/garbage_discovery_cert -=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_ExpirationLeeway -=== RUN TestConfigValidate/type=oidc/missing_required_OIDCDiscoveryURL +=== RUN TestConfigValidate/type=inferred_oidc/valid_claim_mappings +=== RUN TestConfigValidate/type=inferred_oidc/bad_discovery_cert === RUN TestConfigValidate/type=jwt/jwks_mismatched_certs -=== RUN TestConfigValidate/type=jwt/invalid_pubkey +=== RUN TestConfigValidate/type=jwt/incompatible_with_AllowedRedirectURIs +=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientSecret +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_ExpirationLeeway +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_ClockSkewLeeway +=== RUN TestConfigValidate/type=jwt/incompatible_with_OIDCClientSecret +=== RUN TestConfigValidate/type=jwt/good_discovery_cert === RUN TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks_+_pubkeys) +=== RUN TestConfigValidate/type=oidc/invalid_algorithm +=== RUN TestConfigValidate/type=inferred_oidc/garbage_discovery_cert === RUN TestConfigValidate/type=oidc/invalid_redirect_uris -=== RUN TestConfigValidate/type=oidc/good_discovery_cert -=== RUN TestConfigValidate/type=inferred_oidc/invalid_repeated_value_claim_mappings -=== RUN TestConfigValidate/type=oidc/invalid_repeated_list_claim_mappings -=== RUN TestConfigValidate/type=oidc/valid_algorithm -=== RUN TestConfigValidate/type=inferred_oidc/invalid_repeated_list_claim_mappings +=== RUN TestConfigValidate/type=oidc/bad_discovery_cert +=== RUN TestConfigValidate/type=jwt/invalid_repeated_list_claim_mappings +=== RUN TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks) +=== RUN TestConfigValidate/type=jwt/all_required_for_jwks +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_NotBeforeLeeway +=== RUN TestConfigValidate/type=inferred_oidc/good_discovery_cert +=== RUN TestConfigValidate/type=oidc/valid_redirect_uris +=== RUN TestConfigValidate/type=inferred_oidc/valid_algorithm +=== RUN TestConfigValidate/type=inferred_oidc/incompatible_with_BoundIssuer +=== RUN TestConfigValidate/type=jwt/invalid_repeated_value_claim_mappings +=== RUN TestConfigValidate/type=jwt/incompatible_with_VerboseOIDCLogging +=== RUN TestConfigValidate/type=inferred_oidc/all_required === RUN TestConfigValidate/type=oidc/missing_required_AllowedRedirectURIs ---- PASS: TestConfigValidate (0.16s) - --- PASS: TestConfigValidate/type=inferred_oidc/all_required (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientSecret (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_BoundIssuer (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_AllowedRedirectURIs (0.00s) - --- PASS: TestConfigValidate/type=jwt/all_required_for_jwks (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSCACert (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_ClockSkewLeeway (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/bad_discovery_cert (0.01s) - --- PASS: TestConfigValidate/type=jwt/valid_claim_mappings (0.00s) - --- PASS: TestConfigValidate/type=jwt/all_required_for_public_keys (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_NotBeforeLeeway (0.00s) - --- PASS: TestConfigValidate/type=oidc/incompatible_with_ClockSkewLeeway (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWTValidationPubKeys (0.00s) - --- PASS: TestConfigValidate/type=oidc/incompatible_with_ExpirationLeeway (0.00s) - --- PASS: TestConfigValidate/type=oidc/invalid_repeated_value_claim_mappings (0.01s) - --- PASS: TestConfigValidate/type=oidc/bad_discovery_cert (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/garbage_discovery_cert (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSURL (0.00s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_VerboseOIDCLogging (0.00s) +=== RUN TestConfigValidate/type=jwt/incompatible_with_OIDCClientID +=== RUN TestConfigValidate/type=oidc/missing_required_OIDCClientSecret +=== RUN TestConfigValidate/type=oidc/incompatible_with_NotBeforeLeeway +=== RUN TestConfigValidate/type=oidc/garbage_discovery_cert +=== RUN TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientID +=== RUN TestConfigValidate/type=inferred_oidc/invalid_algorithm +=== RUN TestConfigValidate/type=oidc/missing_required_OIDCClientID +=== RUN TestConfigValidate/type=jwt/too_many_methods_(jwks_+_pubkeys) +--- PASS: TestConfigValidate (0.51s) + --- PASS: TestConfigValidate/type=jwt/invalid_pubkey (0.00s) --- PASS: TestConfigValidate/bad_type (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/valid_redirect_uris (0.01s) - --- PASS: TestConfigValidate/type=oidc/invalid_algorithm (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/invalid_algorithm (0.01s) - --- PASS: TestConfigValidate/type=jwt/invalid_repeated_list_claim_mappings (0.00s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCClientSecret (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/invalid_redirect_uris (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/valid_algorithm (0.01s) - --- PASS: TestConfigValidate/type=jwt/all_required_for_oidc_discovery (0.01s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCScopes (0.00s) + --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCDiscoveryURL (0.00s) --- PASS: TestConfigValidate/type=oidc/incompatible_with_JWTValidationPubKeys (0.00s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_AllowedRedirectURIs (0.00s) - --- PASS: TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks) (0.00s) - --- PASS: TestConfigValidate/type=jwt/jwks_invalid_404 (0.01s) - --- PASS: TestConfigValidate/type=jwt/valid_algorithm (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/good_discovery_cert (0.01s) - --- PASS: TestConfigValidate/type=jwt/bad_discovery_cert (0.01s) - --- PASS: TestConfigValidate/type=oidc/all_required (0.01s) + --- PASS: TestConfigValidate/type=inferred_oidc/valid_redirect_uris (0.02s) --- PASS: TestConfigValidate/type=oidc/incompatible_with_BoundIssuer (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/valid_claim_mappings (0.01s) - --- PASS: TestConfigValidate/type=oidc/valid_redirect_uris (0.01s) - --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCClientSecret (0.00s) - --- PASS: TestConfigValidate/type=oidc/incompatible_with_JWKSURL (0.00s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCDiscoveryCACert (0.00s) - --- PASS: TestConfigValidate/type=jwt/too_many_methods_(jwks_+_pubkeys) (0.00s) - --- PASS: TestConfigValidate/type=jwt/good_discovery_cert (0.01s) + --- PASS: TestConfigValidate/type=oidc/invalid_repeated_value_claim_mappings (0.02s) + --- PASS: TestConfigValidate/type=jwt/all_required_for_public_keys (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCScopes (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_JWKSCACert (0.00s) --- PASS: TestConfigValidate/type=jwt/jwks_bad_certs (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientID (0.00s) - --- PASS: TestConfigValidate/type=oidc/valid_claim_mappings (0.01s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCClientID (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSURL (0.00s) + --- PASS: TestConfigValidate/type=oidc/incompatible_with_ExpirationLeeway (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/invalid_redirect_uris (0.00s) + --- PASS: TestConfigValidate/type=oidc/valid_algorithm (0.02s) + --- PASS: TestConfigValidate/type=inferred_oidc/invalid_repeated_value_claim_mappings (0.03s) + --- PASS: TestConfigValidate/type=jwt/jwks_invalid_404 (0.02s) + --- PASS: TestConfigValidate/type=oidc/invalid_repeated_list_claim_mappings (0.02s) + --- PASS: TestConfigValidate/type=oidc/valid_claim_mappings (0.02s) + --- PASS: TestConfigValidate/type=jwt/bad_discovery_cert (0.02s) + --- PASS: TestConfigValidate/type=jwt/valid_claim_mappings (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCDiscoveryCACert (0.00s) --- PASS: TestConfigValidate/type=jwt/too_many_methods_(discovery_+_pubkeys) (0.00s) + --- PASS: TestConfigValidate/type=jwt/valid_algorithm (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWKSCACert (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCDiscoveryURL (0.00s) + --- PASS: TestConfigValidate/type=oidc/good_discovery_cert (0.02s) + --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_AllowedRedirectURIs (0.00s) + --- PASS: TestConfigValidate/type=oidc/incompatible_with_ClockSkewLeeway (0.00s) + --- PASS: TestConfigValidate/type=oidc/incompatible_with_JWKSURL (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/invalid_repeated_list_claim_mappings (0.01s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_JWTValidationPubKeys (0.00s) + --- PASS: TestConfigValidate/type=oidc/all_required (0.02s) + --- PASS: TestConfigValidate/type=jwt/all_required_for_oidc_discovery (0.02s) --- PASS: TestConfigValidate/type=jwt/invalid_algorithm (0.00s) - --- PASS: TestConfigValidate/type=jwt/incompatible_with_JWKSCACert (0.00s) --- PASS: TestConfigValidate/type=oidc/incompatible_with_JWKSCACert (0.00s) - --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCClientID (0.00s) - --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCDiscoveryURL (0.00s) - --- PASS: TestConfigValidate/type=jwt/invalid_repeated_value_claim_mappings (0.00s) - --- PASS: TestConfigValidate/type=oidc/incompatible_with_NotBeforeLeeway (0.00s) - --- PASS: TestConfigValidate/type=oidc/garbage_discovery_cert (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/valid_claim_mappings (0.02s) + --- PASS: TestConfigValidate/type=inferred_oidc/bad_discovery_cert (0.06s) + --- PASS: TestConfigValidate/type=jwt/jwks_mismatched_certs (0.02s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_AllowedRedirectURIs (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientSecret (0.00s) --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_ExpirationLeeway (0.00s) - --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCDiscoveryURL (0.00s) - --- PASS: TestConfigValidate/type=jwt/jwks_mismatched_certs (0.01s) - --- PASS: TestConfigValidate/type=jwt/invalid_pubkey (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_ClockSkewLeeway (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCClientSecret (0.00s) + --- PASS: TestConfigValidate/type=jwt/good_discovery_cert (0.01s) --- PASS: TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks_+_pubkeys) (0.00s) + --- PASS: TestConfigValidate/type=oidc/invalid_algorithm (0.01s) + --- PASS: TestConfigValidate/type=inferred_oidc/garbage_discovery_cert (0.00s) --- PASS: TestConfigValidate/type=oidc/invalid_redirect_uris (0.00s) - --- PASS: TestConfigValidate/type=oidc/good_discovery_cert (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/invalid_repeated_value_claim_mappings (0.01s) - --- PASS: TestConfigValidate/type=oidc/invalid_repeated_list_claim_mappings (0.01s) - --- PASS: TestConfigValidate/type=oidc/valid_algorithm (0.01s) - --- PASS: TestConfigValidate/type=inferred_oidc/invalid_repeated_list_claim_mappings (0.01s) + --- PASS: TestConfigValidate/type=oidc/bad_discovery_cert (0.02s) + --- PASS: TestConfigValidate/type=jwt/invalid_repeated_list_claim_mappings (0.00s) + --- PASS: TestConfigValidate/type=jwt/too_many_methods_(discovery_+_jwks) (0.00s) + --- PASS: TestConfigValidate/type=jwt/all_required_for_jwks (0.02s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_NotBeforeLeeway (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/good_discovery_cert (0.04s) + --- PASS: TestConfigValidate/type=oidc/valid_redirect_uris (0.01s) + --- PASS: TestConfigValidate/type=inferred_oidc/valid_algorithm (0.02s) + --- PASS: TestConfigValidate/type=inferred_oidc/incompatible_with_BoundIssuer (0.00s) + --- PASS: TestConfigValidate/type=jwt/invalid_repeated_value_claim_mappings (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_VerboseOIDCLogging (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/all_required (0.01s) --- PASS: TestConfigValidate/type=oidc/missing_required_AllowedRedirectURIs (0.00s) + --- PASS: TestConfigValidate/type=jwt/incompatible_with_OIDCClientID (0.00s) + --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCClientSecret (0.00s) + --- PASS: TestConfigValidate/type=oidc/incompatible_with_NotBeforeLeeway (0.00s) + --- PASS: TestConfigValidate/type=oidc/garbage_discovery_cert (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/missing_required_OIDCClientID (0.00s) + --- PASS: TestConfigValidate/type=inferred_oidc/invalid_algorithm (0.02s) + --- PASS: TestConfigValidate/type=oidc/missing_required_OIDCClientID (0.00s) + --- PASS: TestConfigValidate/type=jwt/too_many_methods_(jwks_+_pubkeys) (0.00s) === RUN TestJWT_OIDC_Functions_Fail === RUN TestJWT_OIDC_Functions_Fail/static === RUN TestJWT_OIDC_Functions_Fail/static/GetAuthCodeURL @@ -92119,16 +92243,16 @@ === RUN TestJWT_OIDC_Functions_Fail/oidc_discovery === RUN TestJWT_OIDC_Functions_Fail/oidc_discovery/GetAuthCodeURL === RUN TestJWT_OIDC_Functions_Fail/oidc_discovery/ClaimsFromAuthCode ---- PASS: TestJWT_OIDC_Functions_Fail (0.06s) +--- PASS: TestJWT_OIDC_Functions_Fail (0.20s) --- PASS: TestJWT_OIDC_Functions_Fail/static (0.00s) --- PASS: TestJWT_OIDC_Functions_Fail/static/GetAuthCodeURL (0.00s) --- PASS: TestJWT_OIDC_Functions_Fail/static/ClaimsFromAuthCode (0.00s) - --- PASS: TestJWT_OIDC_Functions_Fail/JWKS (0.02s) - --- PASS: TestJWT_OIDC_Functions_Fail/JWKS/GetAuthCodeURL (0.01s) - --- PASS: TestJWT_OIDC_Functions_Fail/JWKS/ClaimsFromAuthCode (0.01s) - --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery (0.03s) - --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery/GetAuthCodeURL (0.02s) - --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery/ClaimsFromAuthCode (0.02s) + --- PASS: TestJWT_OIDC_Functions_Fail/JWKS (0.07s) + --- PASS: TestJWT_OIDC_Functions_Fail/JWKS/GetAuthCodeURL (0.04s) + --- PASS: TestJWT_OIDC_Functions_Fail/JWKS/ClaimsFromAuthCode (0.03s) + --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery (0.14s) + --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery/GetAuthCodeURL (0.05s) + --- PASS: TestJWT_OIDC_Functions_Fail/oidc_discovery/ClaimsFromAuthCode (0.09s) === RUN TestJWT_ClaimsFromJWT === RUN TestJWT_ClaimsFromJWT/static === RUN TestJWT_ClaimsFromJWT/static/missing_audience @@ -92152,422 +92276,422 @@ === RUN TestJWT_ClaimsFromJWT/oidc_discovery/bad_signature === RUN TestJWT_ClaimsFromJWT/oidc_discovery/bad_issuer === RUN TestJWT_ClaimsFromJWT/oidc_discovery/bad_audience ---- PASS: TestJWT_ClaimsFromJWT (0.25s) - --- PASS: TestJWT_ClaimsFromJWT/static (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/static/missing_audience (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/static/valid_inputs (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/static/unusable_claims (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/static/bad_signature (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/static/bad_issuer (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/static/bad_audience (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS (0.12s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/missing_audience (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/valid_inputs (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/unusable_claims (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_signature (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_issuer (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_audience (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery (0.11s) +--- PASS: TestJWT_ClaimsFromJWT (0.82s) + --- PASS: TestJWT_ClaimsFromJWT/static (0.04s) + --- PASS: TestJWT_ClaimsFromJWT/static/missing_audience (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/static/valid_inputs (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/static/unusable_claims (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/static/bad_signature (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/static/bad_issuer (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/static/bad_audience (0.01s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS (0.43s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/missing_audience (0.07s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/valid_inputs (0.06s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/unusable_claims (0.07s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_signature (0.07s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_issuer (0.08s) + --- PASS: TestJWT_ClaimsFromJWT/JWKS/bad_audience (0.09s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery (0.35s) --- SKIP: TestJWT_ClaimsFromJWT/oidc_discovery/missing_audience (0.00s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/valid_inputs (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/unusable_claims (0.03s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_signature (0.03s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_issuer (0.02s) - --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_audience (0.02s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/valid_inputs (0.09s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/unusable_claims (0.07s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_signature (0.07s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_issuer (0.06s) + --- PASS: TestJWT_ClaimsFromJWT/oidc_discovery/bad_audience (0.07s) === RUN TestJWT_ClaimsFromJWT_ExpiryClaims === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims (0.00s) --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway (0.03s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.03s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.03s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.05s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.07s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.04s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.02s) --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.07s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.08s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway (0.05s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.01s) --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_iat_with_clock_skew_leeway (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.01s) --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/no_expire_leeway_using_iat_with_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway (0.10s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.10s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.11s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.10s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.11s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway (0.11s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.11s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.13s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.13s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.12s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway (0.13s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.13s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.05s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.06s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.05s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.06s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.06s) - --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway (0.06s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.05s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.04s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.09s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/static/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway (0.06s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS (0.00s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.14s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.15s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.20s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_iat_with_clock_skew_leeway (0.20s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.22s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway (0.24s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.26s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.22s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_no_expire_leeway_using_nbf_with_clock_skew_leeway (0.26s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.17s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.30s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.23s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway (0.17s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_iat_with_auto_clock_skew_leeway (0.24s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_no_clock_skew_leeway (0.19s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway (0.19s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_iat_with_clock_skew_leeway_with_default_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway_and_nbf (0.24s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_no_clock_skew_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_iat_with_custom_clock_skew_leeway (0.35s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/auto_expire_leeway_using_nbf_with_auto_clock_skew_leeway (0.31s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.19s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_auto_expire_leeway_using_nbf_with_custom_clock_skew_leeway (0.20s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/no_expire_leeway_using_iat_with_clock_skew_leeway (0.14s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/expired_custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.15s) + --- PASS: TestJWT_ClaimsFromJWT_ExpiryClaims/JWKS/custom_expire_leeway_using_nbf_with_clock_skew_leeway_with_default_leeway (0.17s) === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway === RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway === PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== RUN TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== PAUSE TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway -=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway === CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway +=== CONT TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static (0.00s) --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.00s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway (0.00s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.01s) --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/nbf_leeway_using_iat_with_no_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.00s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway (0.03s) --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.03s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.03s) --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.01s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.01s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.03s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.05s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.02s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/static/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.02s) --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS (0.00s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.06s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.07s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.08s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.08s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.08s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.08s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.08s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.09s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway (0.09s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.09s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway (0.09s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.10s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.10s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.04s) - --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.05s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.15s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.20s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.20s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_nbf_leeway_using_iat_with_no_clock_skew_leeway (0.18s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.26s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.25s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.34s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/nbf_leeway_using_iat_with_no_clock_skew_leeway (0.16s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_auto_clock_skew_leeway (0.17s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.19s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_no_clock_skew_leeway_with_default_leeway (0.16s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.25s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/not_yet_valid_custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.15s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_custom_clock_skew_leeway (0.22s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/no_nbf_leeway_using_iat_with_auto_clock_skew_leeway (0.23s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/custom_nbf_leeway_using_exp_with_custom_clock_skew_leeway (0.19s) + --- PASS: TestJWT_ClaimsFromJWT_NotBeforeClaims/JWKS/auto_nbf_leeway_using_exp_with_no_clock_skew_leeway (0.16s) === RUN TestParsePublicKeyPEM === RUN TestParsePublicKeyPEM/rsa === RUN TestParsePublicKeyPEM/ecdsa === RUN TestParsePublicKeyPEM/ed25519 ---- PASS: TestParsePublicKeyPEM (1.62s) - --- PASS: TestParsePublicKeyPEM/rsa (1.62s) +--- PASS: TestParsePublicKeyPEM (2.98s) + --- PASS: TestParsePublicKeyPEM/rsa (2.98s) --- PASS: TestParsePublicKeyPEM/ecdsa (0.00s) --- PASS: TestParsePublicKeyPEM/ed25519 (0.00s) === RUN TestOIDC_AuthURL @@ -92581,11 +92705,11 @@ === CONT TestOIDC_AuthURL/missing_RedirectURI === CONT TestOIDC_AuthURL/invalid_RedirectURI --- PASS: TestOIDC_AuthURL (0.00s) - --- PASS: TestOIDC_AuthURL/missing_RedirectURI (0.02s) - --- PASS: TestOIDC_AuthURL/normal_case (0.02s) - --- PASS: TestOIDC_AuthURL/invalid_RedirectURI (0.02s) + --- PASS: TestOIDC_AuthURL/missing_RedirectURI (0.08s) + --- PASS: TestOIDC_AuthURL/invalid_RedirectURI (0.10s) + --- PASS: TestOIDC_AuthURL/normal_case (0.11s) === RUN TestOIDC_JWT_Functions_Fail ---- PASS: TestOIDC_JWT_Functions_Fail (0.02s) +--- PASS: TestOIDC_JWT_Functions_Fail (0.06s) === RUN TestOIDC_ClaimsFromAuthCode === RUN TestOIDC_ClaimsFromAuthCode/successful_login === RUN TestOIDC_ClaimsFromAuthCode/failed_login_unusable_claims @@ -92600,66 +92724,66 @@ === RUN TestOIDC_ClaimsFromAuthCode/no_response_from_provider === RUN TestOIDC_ClaimsFromAuthCode/invalid_bound_audience === CONT TestOIDC_ClaimsFromAuthCode/failed_login_-_bad_nonce ---- PASS: TestOIDC_ClaimsFromAuthCode (0.21s) - --- PASS: TestOIDC_ClaimsFromAuthCode/successful_login (0.03s) - --- PASS: TestOIDC_ClaimsFromAuthCode/failed_login_unusable_claims (0.03s) - --- PASS: TestOIDC_ClaimsFromAuthCode/successful_login_-_no_userinfo (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/missing_state (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/unknown_state (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/valid_state,_missing_code (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/failed_code_exchange (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/no_id_token_returned (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/no_response_from_provider (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/invalid_bound_audience (0.02s) - --- PASS: TestOIDC_ClaimsFromAuthCode/failed_login_-_bad_nonce (0.03s) +--- PASS: TestOIDC_ClaimsFromAuthCode (0.59s) + --- PASS: TestOIDC_ClaimsFromAuthCode/successful_login (0.08s) + --- PASS: TestOIDC_ClaimsFromAuthCode/failed_login_unusable_claims (0.07s) + --- PASS: TestOIDC_ClaimsFromAuthCode/successful_login_-_no_userinfo (0.08s) + --- PASS: TestOIDC_ClaimsFromAuthCode/missing_state (0.05s) + --- PASS: TestOIDC_ClaimsFromAuthCode/unknown_state (0.07s) + --- PASS: TestOIDC_ClaimsFromAuthCode/valid_state,_missing_code (0.05s) + --- PASS: TestOIDC_ClaimsFromAuthCode/failed_code_exchange (0.06s) + --- PASS: TestOIDC_ClaimsFromAuthCode/no_id_token_returned (0.04s) + --- PASS: TestOIDC_ClaimsFromAuthCode/no_response_from_provider (0.05s) + --- PASS: TestOIDC_ClaimsFromAuthCode/invalid_bound_audience (0.04s) + --- PASS: TestOIDC_ClaimsFromAuthCode/failed_login_-_bad_nonce (0.05s) === RUN TestExtractStringMetadata -=== RUN TestExtractStringMetadata/nested_data -=== RUN TestExtractStringMetadata/error:_a_struct_isn't_stringifiable -=== RUN TestExtractStringMetadata/error:_a_slice_isn't_stringifiable === RUN TestExtractStringMetadata/empty === RUN TestExtractStringMetadata/all === RUN TestExtractStringMetadata/some === RUN TestExtractStringMetadata/none +=== RUN TestExtractStringMetadata/nested_data +=== RUN TestExtractStringMetadata/error:_a_struct_isn't_stringifiable +=== RUN TestExtractStringMetadata/error:_a_slice_isn't_stringifiable --- PASS: TestExtractStringMetadata (0.00s) - --- PASS: TestExtractStringMetadata/nested_data (0.00s) - --- PASS: TestExtractStringMetadata/error:_a_struct_isn't_stringifiable (0.00s) - --- PASS: TestExtractStringMetadata/error:_a_slice_isn't_stringifiable (0.00s) --- PASS: TestExtractStringMetadata/empty (0.00s) --- PASS: TestExtractStringMetadata/all (0.00s) --- PASS: TestExtractStringMetadata/some (0.00s) --- PASS: TestExtractStringMetadata/none (0.00s) + --- PASS: TestExtractStringMetadata/nested_data (0.00s) + --- PASS: TestExtractStringMetadata/error:_a_struct_isn't_stringifiable (0.00s) + --- PASS: TestExtractStringMetadata/error:_a_slice_isn't_stringifiable (0.00s) === RUN TestExtractListMetadata -=== RUN TestExtractListMetadata/some_-_singular -=== RUN TestExtractListMetadata/none_-_singular -=== RUN TestExtractListMetadata/nested_data_-_singular +=== RUN TestExtractListMetadata/list_omits_empty_strings +=== RUN TestExtractListMetadata/JSONPointer === RUN TestExtractListMetadata/error:_a_struct_isn't_stringifiable_(singular) +=== RUN TestExtractListMetadata/all_-_list === RUN TestExtractListMetadata/some_-_list === RUN TestExtractListMetadata/none_-_list -=== RUN TestExtractListMetadata/nested_data_-_list -=== RUN TestExtractListMetadata/empty -=== RUN TestExtractListMetadata/list_omits_empty_strings -=== RUN TestExtractListMetadata/non-string-slice_data_(string) -=== RUN TestExtractListMetadata/all_-_list === RUN TestExtractListMetadata/error:_a_slice_isn't_stringifiable_(singular) -=== RUN TestExtractListMetadata/JSONPointer -=== RUN TestExtractListMetadata/JSONPointer_not_found +=== RUN TestExtractListMetadata/nested_data_-_singular +=== RUN TestExtractListMetadata/non-string-slice_data_(string) +=== RUN TestExtractListMetadata/some_-_singular === RUN TestExtractListMetadata/all_-_singular +=== RUN TestExtractListMetadata/none_-_singular +=== RUN TestExtractListMetadata/nested_data_-_list +=== RUN TestExtractListMetadata/JSONPointer_not_found +=== RUN TestExtractListMetadata/empty --- PASS: TestExtractListMetadata (0.00s) - --- PASS: TestExtractListMetadata/some_-_singular (0.00s) - --- PASS: TestExtractListMetadata/none_-_singular (0.00s) - --- PASS: TestExtractListMetadata/nested_data_-_singular (0.00s) + --- PASS: TestExtractListMetadata/list_omits_empty_strings (0.00s) + --- PASS: TestExtractListMetadata/JSONPointer (0.00s) --- PASS: TestExtractListMetadata/error:_a_struct_isn't_stringifiable_(singular) (0.00s) + --- PASS: TestExtractListMetadata/all_-_list (0.00s) --- PASS: TestExtractListMetadata/some_-_list (0.00s) --- PASS: TestExtractListMetadata/none_-_list (0.00s) - --- PASS: TestExtractListMetadata/nested_data_-_list (0.00s) - --- PASS: TestExtractListMetadata/empty (0.00s) - --- PASS: TestExtractListMetadata/list_omits_empty_strings (0.00s) - --- PASS: TestExtractListMetadata/non-string-slice_data_(string) (0.00s) - --- PASS: TestExtractListMetadata/all_-_list (0.00s) --- PASS: TestExtractListMetadata/error:_a_slice_isn't_stringifiable_(singular) (0.00s) - --- PASS: TestExtractListMetadata/JSONPointer (0.00s) - --- PASS: TestExtractListMetadata/JSONPointer_not_found (0.00s) + --- PASS: TestExtractListMetadata/nested_data_-_singular (0.00s) + --- PASS: TestExtractListMetadata/non-string-slice_data_(string) (0.00s) + --- PASS: TestExtractListMetadata/some_-_singular (0.00s) --- PASS: TestExtractListMetadata/all_-_singular (0.00s) + --- PASS: TestExtractListMetadata/none_-_singular (0.00s) + --- PASS: TestExtractListMetadata/nested_data_-_list (0.00s) + --- PASS: TestExtractListMetadata/JSONPointer_not_found (0.00s) + --- PASS: TestExtractListMetadata/empty (0.00s) === RUN TestGetClaim === RUN TestGetClaim/a === RUN TestGetClaim//a @@ -92709,51 +92833,51 @@ --- PASS: TestNormalizeList/struct_{_A_int_}{A:5} (0.00s) --- PASS: TestNormalizeList/ (0.00s) === RUN TestStringifyMetadataValue -=== RUN TestStringifyMetadataValue/struct === RUN TestStringifyMetadataValue/int32 -=== RUN TestStringifyMetadataValue/int64 === RUN TestStringifyMetadataValue/uint32 === RUN TestStringifyMetadataValue/uint -=== RUN TestStringifyMetadataValue/float64 -=== RUN TestStringifyMetadataValue/float32 +=== RUN TestStringifyMetadataValue/int_slice +=== RUN TestStringifyMetadataValue/nil +=== RUN TestStringifyMetadataValue/empty_string === RUN TestStringifyMetadataValue/int8 +=== RUN TestStringifyMetadataValue/int64 === RUN TestStringifyMetadataValue/uint8 -=== RUN TestStringifyMetadataValue/nil +=== RUN TestStringifyMetadataValue/map === RUN TestStringifyMetadataValue/string -=== RUN TestStringifyMetadataValue/true === RUN TestStringifyMetadataValue/false +=== RUN TestStringifyMetadataValue/float64 +=== RUN TestStringifyMetadataValue/float32 +=== RUN TestStringifyMetadataValue/int16 === RUN TestStringifyMetadataValue/uint64 === RUN TestStringifyMetadataValue/string_slice -=== RUN TestStringifyMetadataValue/empty_string -=== RUN TestStringifyMetadataValue/uint16 -=== RUN TestStringifyMetadataValue/map +=== RUN TestStringifyMetadataValue/true === RUN TestStringifyMetadataValue/json_number -=== RUN TestStringifyMetadataValue/int16 === RUN TestStringifyMetadataValue/int -=== RUN TestStringifyMetadataValue/int_slice +=== RUN TestStringifyMetadataValue/uint16 +=== RUN TestStringifyMetadataValue/struct --- PASS: TestStringifyMetadataValue (0.00s) - --- PASS: TestStringifyMetadataValue/struct (0.00s) --- PASS: TestStringifyMetadataValue/int32 (0.00s) - --- PASS: TestStringifyMetadataValue/int64 (0.00s) --- PASS: TestStringifyMetadataValue/uint32 (0.00s) --- PASS: TestStringifyMetadataValue/uint (0.00s) - --- PASS: TestStringifyMetadataValue/float64 (0.00s) - --- PASS: TestStringifyMetadataValue/float32 (0.00s) + --- PASS: TestStringifyMetadataValue/int_slice (0.00s) + --- PASS: TestStringifyMetadataValue/nil (0.00s) + --- PASS: TestStringifyMetadataValue/empty_string (0.00s) --- PASS: TestStringifyMetadataValue/int8 (0.00s) + --- PASS: TestStringifyMetadataValue/int64 (0.00s) --- PASS: TestStringifyMetadataValue/uint8 (0.00s) - --- PASS: TestStringifyMetadataValue/nil (0.00s) + --- PASS: TestStringifyMetadataValue/map (0.00s) --- PASS: TestStringifyMetadataValue/string (0.00s) - --- PASS: TestStringifyMetadataValue/true (0.00s) --- PASS: TestStringifyMetadataValue/false (0.00s) + --- PASS: TestStringifyMetadataValue/float64 (0.00s) + --- PASS: TestStringifyMetadataValue/float32 (0.00s) + --- PASS: TestStringifyMetadataValue/int16 (0.00s) --- PASS: TestStringifyMetadataValue/uint64 (0.00s) --- PASS: TestStringifyMetadataValue/string_slice (0.00s) - --- PASS: TestStringifyMetadataValue/empty_string (0.00s) - --- PASS: TestStringifyMetadataValue/uint16 (0.00s) - --- PASS: TestStringifyMetadataValue/map (0.00s) + --- PASS: TestStringifyMetadataValue/true (0.00s) --- PASS: TestStringifyMetadataValue/json_number (0.00s) - --- PASS: TestStringifyMetadataValue/int16 (0.00s) --- PASS: TestStringifyMetadataValue/int (0.00s) - --- PASS: TestStringifyMetadataValue/int_slice (0.00s) + --- PASS: TestStringifyMetadataValue/uint16 (0.00s) + --- PASS: TestStringifyMetadataValue/struct (0.00s) === RUN TestValidateAudience === RUN TestValidateAudience/boundAudiences=[]string{"a"}_audience=[]string{"a"}_strict=false === RUN TestValidateAudience/boundAudiences=[]string{"a"}_audience=[]string{"a"}_strict=true @@ -92823,11 +92947,11 @@ --- PASS: TestValidRedirect/uri="https://[::1]:5000/x/y?r=42"_allowed=[]string{"a",_"b",_"https://[::1]:5000/x/y?r=43"} (0.00s) --- PASS: TestValidRedirect/uri="%%%%%%%%%%%"_allowed=[]string{} (0.00s) PASS -ok github.com/hashicorp/consul/internal/go-sso/oidcauth 2.689s +ok github.com/hashicorp/consul/internal/go-sso/oidcauth 6.608s === RUN TestStrListContains --- PASS: TestStrListContains (0.00s) PASS -ok github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil 0.005s +ok github.com/hashicorp/consul/internal/go-sso/oidcauth/internal/strutil 0.052s ? github.com/hashicorp/consul/internal/go-sso/oidcauth/oidcauthtest [no test files] === RUN TestIsPrivateIP === RUN TestIsPrivateIP/10.0.0.1 @@ -92900,7 +93024,7 @@ --- PASS: TestIsIPv6/localhost (0.00s) --- PASS: TestIsIPv6/1.257.0.1 (0.00s) PASS -ok github.com/hashicorp/consul/ipaddr 0.006s +ok github.com/hashicorp/consul/ipaddr 0.043s === RUN TestDurationMinusBuffer --- PASS: TestDurationMinusBuffer (0.00s) === RUN TestDurationMinusBufferDomain @@ -92995,61 +93119,61 @@ === PAUSE TestJitterRandomStagger/0_percent === RUN TestJitterRandomStagger/10_percent === PAUSE TestJitterRandomStagger/10_percent -=== CONT TestRetryWaiter_calculateWait -=== RUN TestRetryWaiter_calculateWait/Defaults -=== PAUSE TestRetryWaiter_calculateWait/Defaults -=== RUN TestRetryWaiter_calculateWait/Minimum_Wait -=== PAUSE TestRetryWaiter_calculateWait/Minimum_Wait -=== RUN TestRetryWaiter_calculateWait/Minimum_Failures -=== PAUSE TestRetryWaiter_calculateWait/Minimum_Failures -=== RUN TestRetryWaiter_calculateWait/Maximum_Wait -=== PAUSE TestRetryWaiter_calculateWait/Maximum_Wait -=== CONT TestRetryWaiter_calculateWait/Defaults -=== CONT TestRetryWaiter_calculateWait/Maximum_Wait -=== CONT TestRetryWaiter_calculateWait/Minimum_Wait -=== CONT TestRetryWaiter_calculateWait/Minimum_Failures -=== CONT TestRetryWaiter_WaitChans ---- PASS: TestRetryWaiter_calculateWait (0.00s) - --- PASS: TestRetryWaiter_calculateWait/Defaults (0.00s) - --- PASS: TestRetryWaiter_calculateWait/Maximum_Wait (0.00s) - --- PASS: TestRetryWaiter_calculateWait/Minimum_Wait (0.00s) - --- PASS: TestRetryWaiter_calculateWait/Minimum_Failures (0.00s) === RUN TestJitterRandomStagger/100_percent === PAUSE TestJitterRandomStagger/100_percent === CONT TestJitterRandomStagger/0_percent +=== CONT TestRetryWaiter_WaitChans === RUN TestRetryWaiter_WaitChans/Minimum_Wait_-_Success === PAUSE TestRetryWaiter_WaitChans/Minimum_Wait_-_Success === RUN TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIf === PAUSE TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIf === RUN TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIfErr -=== CONT TestJitterRandomStagger/10_percent === PAUSE TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIfErr === RUN TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed -=== CONT TestJitterRandomStagger/100_percent ---- PASS: TestJitterRandomStagger (0.00s) - --- PASS: TestJitterRandomStagger/0_percent (0.00s) - --- PASS: TestJitterRandomStagger/10_percent (0.00s) - --- PASS: TestJitterRandomStagger/100_percent (0.00s) === PAUSE TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed === RUN TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIf === PAUSE TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIf === RUN TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIfErr === PAUSE TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIfErr === CONT TestRetryWaiter_WaitChans/Minimum_Wait_-_Success -=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIfErr -=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed +=== CONT TestRetryWaiter_calculateWait +=== RUN TestRetryWaiter_calculateWait/Defaults +=== PAUSE TestRetryWaiter_calculateWait/Defaults +=== RUN TestRetryWaiter_calculateWait/Minimum_Wait +=== PAUSE TestRetryWaiter_calculateWait/Minimum_Wait +=== RUN TestRetryWaiter_calculateWait/Minimum_Failures +=== PAUSE TestRetryWaiter_calculateWait/Minimum_Failures +=== RUN TestRetryWaiter_calculateWait/Maximum_Wait +=== PAUSE TestRetryWaiter_calculateWait/Maximum_Wait +=== CONT TestRetryWaiter_calculateWait/Defaults +=== CONT TestJitterRandomStagger/100_percent === CONT TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIfErr -=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIf === CONT TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIf +=== CONT TestJitterRandomStagger/10_percent +=== CONT TestRetryWaiter_calculateWait/Minimum_Failures +=== CONT TestRetryWaiter_calculateWait/Minimum_Wait +=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIf +--- PASS: TestJitterRandomStagger (0.00s) + --- PASS: TestJitterRandomStagger/0_percent (0.00s) + --- PASS: TestJitterRandomStagger/100_percent (0.00s) + --- PASS: TestJitterRandomStagger/10_percent (0.00s) +=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIfErr +=== CONT TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed +=== CONT TestRetryWaiter_calculateWait/Maximum_Wait +--- PASS: TestRetryWaiter_calculateWait (0.00s) + --- PASS: TestRetryWaiter_calculateWait/Defaults (0.00s) + --- PASS: TestRetryWaiter_calculateWait/Minimum_Failures (0.00s) + --- PASS: TestRetryWaiter_calculateWait/Minimum_Wait (0.00s) + --- PASS: TestRetryWaiter_calculateWait/Maximum_Wait (0.00s) --- PASS: TestRetryWaiter_WaitChans (0.00s) - --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIf (0.20s) - --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIfErr (0.20s) - --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_Success (0.20s) - --- PASS: TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed (0.25s) + --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIf (0.21s) + --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_Success (0.21s) + --- PASS: TestRetryWaiter_WaitChans/Minimum_Wait_-_WaitIfErr (0.21s) --- PASS: TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIfErr (0.25s) --- PASS: TestRetryWaiter_WaitChans/Maximum_Wait_-_WaitIf (0.25s) + --- PASS: TestRetryWaiter_WaitChans/Maximum_Wait_-_Failed (0.25s) PASS -ok github.com/hashicorp/consul/lib 0.260s +ok github.com/hashicorp/consul/lib 0.283s === RUN TestHookTranslateKeys === RUN TestHookTranslateKeys/target_of_type_struct,_with_struct_receiver === RUN TestHookTranslateKeys/target_of_type_ptr,_with_struct_receiver @@ -93081,11 +93205,11 @@ === RUN TestHookWeakDecodeFromSlice_NestedOpaqueConfig --- PASS: TestHookWeakDecodeFromSlice_NestedOpaqueConfig (0.00s) PASS -ok github.com/hashicorp/consul/lib/decode 0.006s +ok github.com/hashicorp/consul/lib/decode 0.038s === RUN TestWriteAtomic ---- PASS: TestWriteAtomic (0.00s) +--- PASS: TestWriteAtomic (0.01s) PASS -ok github.com/hashicorp/consul/lib/file 0.008s +ok github.com/hashicorp/consul/lib/file 0.050s === RUN TestDynamic === PAUSE TestDynamic === RUN TestDynamicPanic @@ -93096,10 +93220,10 @@ === CONT TestDynamicAcquire === CONT TestDynamicPanic --- PASS: TestDynamicPanic (0.00s) ---- PASS: TestDynamicAcquire (0.05s) ---- PASS: TestDynamic (0.32s) +--- PASS: TestDynamicAcquire (0.06s) +--- PASS: TestDynamic (1.45s) PASS -ok github.com/hashicorp/consul/lib/semaphore 0.325s +ok github.com/hashicorp/consul/lib/semaphore 1.478s === RUN TestContains --- PASS: TestContains (0.00s) === RUN TestEqual @@ -93139,79 +93263,79 @@ --- PASS: TestMergeSorted/one_and_none (0.00s) --- PASS: TestMergeSorted/one_and_one_dupe (0.00s) PASS -ok github.com/hashicorp/consul/lib/stringslice 0.007s +ok github.com/hashicorp/consul/lib/stringslice 0.018s === RUN TestInterpolateHIL -=== RUN TestInterpolateHIL/lowercase_var_in_middle_lower=false -=== RUN TestInterpolateHIL/lowercase_var_in_middle_lower=true -=== RUN TestInterpolateHIL/two_vars_missing_second_start_lower=false -=== RUN TestInterpolateHIL/two_vars_missing_second_start_lower=true -=== RUN TestInterpolateHIL/var_without_end_lower=false -=== RUN TestInterpolateHIL/var_without_end_lower=true -=== RUN TestInterpolateHIL/no_vars_lower=false -=== RUN TestInterpolateHIL/no_vars_lower=true +=== RUN TestInterpolateHIL/empty_lower=false +=== RUN TestInterpolateHIL/empty_lower=true +=== RUN TestInterpolateHIL/just_lowercase_var_lower=false +=== RUN TestInterpolateHIL/just_lowercase_var_lower=true === RUN TestInterpolateHIL/just_uppercase_var_lower=false === RUN TestInterpolateHIL/just_uppercase_var_lower=true -=== RUN TestInterpolateHIL/missing_map_val_lower=false -=== RUN TestInterpolateHIL/missing_map_val_lower=true -=== RUN TestInterpolateHIL/just_end_lower=false -=== RUN TestInterpolateHIL/just_end_lower=true === RUN TestInterpolateHIL/var_without_start_lower=false === RUN TestInterpolateHIL/var_without_start_lower=true -=== RUN TestInterpolateHIL/just_start_lower=false -=== RUN TestInterpolateHIL/just_start_lower=true -=== RUN TestInterpolateHIL/empty_lower=false -=== RUN TestInterpolateHIL/empty_lower=true +=== RUN TestInterpolateHIL/two_vars_missing_second_start_lower=false +=== RUN TestInterpolateHIL/two_vars_missing_second_start_lower=true +=== RUN TestInterpolateHIL/two_vars_missing_first_end_lower=false +=== RUN TestInterpolateHIL/two_vars_missing_first_end_lower=true === RUN TestInterpolateHIL/uppercase_var_in_middle_lower=false === RUN TestInterpolateHIL/uppercase_var_in_middle_lower=true -=== RUN TestInterpolateHIL/backwards_lower=false -=== RUN TestInterpolateHIL/backwards_lower=true -=== RUN TestInterpolateHIL/just_lowercase_var_lower=false -=== RUN TestInterpolateHIL/just_lowercase_var_lower=true === RUN TestInterpolateHIL/two_vars_lower=false === RUN TestInterpolateHIL/two_vars_lower=true +=== RUN TestInterpolateHIL/missing_map_val_lower=false +=== RUN TestInterpolateHIL/missing_map_val_lower=true === RUN TestInterpolateHIL/no_varname_lower=false === RUN TestInterpolateHIL/no_varname_lower=true === RUN TestInterpolateHIL/missing_map_key_lower=false === RUN TestInterpolateHIL/missing_map_key_lower=true -=== RUN TestInterpolateHIL/two_vars_missing_first_end_lower=false -=== RUN TestInterpolateHIL/two_vars_missing_first_end_lower=true +=== RUN TestInterpolateHIL/just_start_lower=false +=== RUN TestInterpolateHIL/just_start_lower=true +=== RUN TestInterpolateHIL/no_vars_lower=false +=== RUN TestInterpolateHIL/no_vars_lower=true +=== RUN TestInterpolateHIL/lowercase_var_in_middle_lower=false +=== RUN TestInterpolateHIL/lowercase_var_in_middle_lower=true +=== RUN TestInterpolateHIL/just_end_lower=false +=== RUN TestInterpolateHIL/just_end_lower=true +=== RUN TestInterpolateHIL/backwards_lower=false +=== RUN TestInterpolateHIL/backwards_lower=true +=== RUN TestInterpolateHIL/var_without_end_lower=false +=== RUN TestInterpolateHIL/var_without_end_lower=true --- PASS: TestInterpolateHIL (0.00s) - --- PASS: TestInterpolateHIL/lowercase_var_in_middle_lower=false (0.00s) - --- PASS: TestInterpolateHIL/lowercase_var_in_middle_lower=true (0.00s) - --- PASS: TestInterpolateHIL/two_vars_missing_second_start_lower=false (0.00s) - --- PASS: TestInterpolateHIL/two_vars_missing_second_start_lower=true (0.00s) - --- PASS: TestInterpolateHIL/var_without_end_lower=false (0.00s) - --- PASS: TestInterpolateHIL/var_without_end_lower=true (0.00s) - --- PASS: TestInterpolateHIL/no_vars_lower=false (0.00s) - --- PASS: TestInterpolateHIL/no_vars_lower=true (0.00s) + --- PASS: TestInterpolateHIL/empty_lower=false (0.00s) + --- PASS: TestInterpolateHIL/empty_lower=true (0.00s) + --- PASS: TestInterpolateHIL/just_lowercase_var_lower=false (0.00s) + --- PASS: TestInterpolateHIL/just_lowercase_var_lower=true (0.00s) --- PASS: TestInterpolateHIL/just_uppercase_var_lower=false (0.00s) --- PASS: TestInterpolateHIL/just_uppercase_var_lower=true (0.00s) - --- PASS: TestInterpolateHIL/missing_map_val_lower=false (0.00s) - --- PASS: TestInterpolateHIL/missing_map_val_lower=true (0.00s) - --- PASS: TestInterpolateHIL/just_end_lower=false (0.00s) - --- PASS: TestInterpolateHIL/just_end_lower=true (0.00s) --- PASS: TestInterpolateHIL/var_without_start_lower=false (0.00s) --- PASS: TestInterpolateHIL/var_without_start_lower=true (0.00s) - --- PASS: TestInterpolateHIL/just_start_lower=false (0.00s) - --- PASS: TestInterpolateHIL/just_start_lower=true (0.00s) - --- PASS: TestInterpolateHIL/empty_lower=false (0.00s) - --- PASS: TestInterpolateHIL/empty_lower=true (0.00s) + --- PASS: TestInterpolateHIL/two_vars_missing_second_start_lower=false (0.00s) + --- PASS: TestInterpolateHIL/two_vars_missing_second_start_lower=true (0.00s) + --- PASS: TestInterpolateHIL/two_vars_missing_first_end_lower=false (0.00s) + --- PASS: TestInterpolateHIL/two_vars_missing_first_end_lower=true (0.00s) --- PASS: TestInterpolateHIL/uppercase_var_in_middle_lower=false (0.00s) --- PASS: TestInterpolateHIL/uppercase_var_in_middle_lower=true (0.00s) - --- PASS: TestInterpolateHIL/backwards_lower=false (0.00s) - --- PASS: TestInterpolateHIL/backwards_lower=true (0.00s) - --- PASS: TestInterpolateHIL/just_lowercase_var_lower=false (0.00s) - --- PASS: TestInterpolateHIL/just_lowercase_var_lower=true (0.00s) --- PASS: TestInterpolateHIL/two_vars_lower=false (0.00s) --- PASS: TestInterpolateHIL/two_vars_lower=true (0.00s) + --- PASS: TestInterpolateHIL/missing_map_val_lower=false (0.00s) + --- PASS: TestInterpolateHIL/missing_map_val_lower=true (0.00s) --- PASS: TestInterpolateHIL/no_varname_lower=false (0.00s) --- PASS: TestInterpolateHIL/no_varname_lower=true (0.00s) --- PASS: TestInterpolateHIL/missing_map_key_lower=false (0.00s) --- PASS: TestInterpolateHIL/missing_map_key_lower=true (0.00s) - --- PASS: TestInterpolateHIL/two_vars_missing_first_end_lower=false (0.00s) - --- PASS: TestInterpolateHIL/two_vars_missing_first_end_lower=true (0.00s) + --- PASS: TestInterpolateHIL/just_start_lower=false (0.00s) + --- PASS: TestInterpolateHIL/just_start_lower=true (0.00s) + --- PASS: TestInterpolateHIL/no_vars_lower=false (0.00s) + --- PASS: TestInterpolateHIL/no_vars_lower=true (0.00s) + --- PASS: TestInterpolateHIL/lowercase_var_in_middle_lower=false (0.00s) + --- PASS: TestInterpolateHIL/lowercase_var_in_middle_lower=true (0.00s) + --- PASS: TestInterpolateHIL/just_end_lower=false (0.00s) + --- PASS: TestInterpolateHIL/just_end_lower=true (0.00s) + --- PASS: TestInterpolateHIL/backwards_lower=false (0.00s) + --- PASS: TestInterpolateHIL/backwards_lower=true (0.00s) + --- PASS: TestInterpolateHIL/var_without_end_lower=false (0.00s) + --- PASS: TestInterpolateHIL/var_without_end_lower=true (0.00s) PASS -ok github.com/hashicorp/consul/lib/template 0.008s +ok github.com/hashicorp/consul/lib/template 0.038s === RUN TestGatedWriter_impl --- PASS: TestGatedWriter_impl (0.00s) === RUN TestGatedWriter @@ -93301,41 +93425,41 @@ === RUN TestLogger_SetupLoggerWithInValidLogPathPermission === PAUSE TestLogger_SetupLoggerWithInValidLogPathPermission === CONT TestLogFile_timeRotation +=== CONT TestLogger_SetupLoggerWithInValidLogPathPermission +--- PASS: TestLogger_SetupLoggerWithInValidLogPathPermission (0.00s) +=== CONT TestLogger_SetupLoggerWithInValidLogPath +--- PASS: TestLogger_SetupLoggerWithInValidLogPath (0.00s) +=== CONT TestLogger_SetupLoggerWithValidLogPath +--- PASS: TestLogger_SetupLoggerWithValidLogPath (0.00s) +=== CONT TestLogger_SetupLoggerWithJSON +--- PASS: TestLogger_SetupLoggerWithJSON (0.00s) +=== CONT TestLogger_SetupLoggerWithName +--- PASS: TestLogger_SetupLoggerWithName (0.00s) +=== CONT TestLogger_SetupLoggerDebugLevel +--- PASS: TestLogger_SetupLoggerDebugLevel (0.00s) === CONT TestLogger_SetupLoggerErrorLevel === RUN TestLogger_SetupLoggerErrorLevel/ERR_log_level -=== CONT TestLogger_SetupInvalidLogLevel === RUN TestLogger_SetupLoggerErrorLevel/ERROR_log_level -=== CONT TestLogger_SetupLoggerWithValidLogPath ---- PASS: TestLogger_SetupInvalidLogLevel (0.00s) --- PASS: TestLogger_SetupLoggerErrorLevel (0.00s) --- PASS: TestLogger_SetupLoggerErrorLevel/ERR_log_level (0.00s) --- PASS: TestLogger_SetupLoggerErrorLevel/ERROR_log_level (0.00s) -=== CONT TestLogFile_deleteArchives ---- PASS: TestLogger_SetupLoggerWithValidLogPath (0.00s) -=== CONT TestLogFile_byteRotation ---- PASS: TestLogFile_deleteArchives (0.00s) -=== CONT TestLogFile_openNew -=== CONT TestLogFile_deleteArchivesDisabled ---- PASS: TestLogFile_byteRotation (0.00s) -=== CONT TestLogger_SetupLoggerWithInValidLogPath ---- PASS: TestLogFile_openNew (0.00s) -=== CONT TestLogger_SetupLoggerWithJSON ---- PASS: TestLogger_SetupLoggerWithInValidLogPath (0.00s) ---- PASS: TestLogFile_deleteArchivesDisabled (0.00s) +=== CONT TestLogger_SetupInvalidLogLevel +--- PASS: TestLogger_SetupInvalidLogLevel (0.00s) === CONT TestLogger_SetupBasic ---- PASS: TestLogger_SetupLoggerWithJSON (0.00s) --- PASS: TestLogger_SetupBasic (0.00s) -=== CONT TestLogger_SetupLoggerWithInValidLogPathPermission === CONT TestLogFile_rotationDisabled -=== CONT TestLogger_SetupLoggerDebugLevel ---- PASS: TestLogger_SetupLoggerWithInValidLogPathPermission (0.00s) -=== CONT TestLogger_SetupLoggerWithName ---- PASS: TestLogger_SetupLoggerDebugLevel (0.00s) ---- PASS: TestLogger_SetupLoggerWithName (0.00s) --- PASS: TestLogFile_rotationDisabled (0.00s) ---- PASS: TestLogFile_timeRotation (2.00s) +=== CONT TestLogFile_deleteArchivesDisabled +--- PASS: TestLogFile_deleteArchivesDisabled (0.00s) +=== CONT TestLogFile_deleteArchives +--- PASS: TestLogFile_deleteArchives (0.00s) +=== CONT TestLogFile_byteRotation +--- PASS: TestLogFile_byteRotation (0.00s) +=== CONT TestLogFile_openNew +--- PASS: TestLogFile_openNew (0.00s) +--- PASS: TestLogFile_timeRotation (2.01s) PASS -ok github.com/hashicorp/consul/logging 2.008s +ok github.com/hashicorp/consul/logging 2.070s === RUN TestMonitor_Start === PAUSE TestMonitor_Start === RUN TestMonitor_Stop @@ -93347,17 +93471,17 @@ === RUN TestMonitor_WriteStopped === PAUSE TestMonitor_WriteStopped === CONT TestMonitor_Start -=== CONT TestMonitor_ZeroBufSizeDefault --- PASS: TestMonitor_Start (0.00s) -=== CONT TestMonitor_DroppedMessages -=== CONT TestMonitor_Stop === CONT TestMonitor_WriteStopped --- PASS: TestMonitor_WriteStopped (0.00s) +=== CONT TestMonitor_ZeroBufSizeDefault +=== CONT TestMonitor_DroppedMessages +=== CONT TestMonitor_Stop +--- PASS: TestMonitor_Stop (0.10s) --- PASS: TestMonitor_ZeroBufSizeDefault (0.10s) --- PASS: TestMonitor_DroppedMessages (0.10s) ---- PASS: TestMonitor_Stop (0.10s) PASS -ok github.com/hashicorp/consul/logging/monitor 0.106s +ok github.com/hashicorp/consul/logging/monitor 0.165s === RUN TestHookPBTimestampToTime --- PASS: TestHookPBTimestampToTime (0.00s) === RUN TestHookTimeToPBTimestamp @@ -93365,7 +93489,7 @@ === RUN TestHookTimeToPBTimestamp_ZeroTime --- PASS: TestHookTimeToPBTimestamp_ZeroTime (0.00s) PASS -ok github.com/hashicorp/consul/proto 0.005s +ok github.com/hashicorp/consul/proto 0.038s ? github.com/hashicorp/consul/proto/pbacl [no test files] ? github.com/hashicorp/consul/proto/pbautoconf [no test files] ? github.com/hashicorp/consul/proto/pbcommon [no test files] @@ -93392,13 +93516,13 @@ --- PASS: TestIntervalOverlap/1:4_vs_3:6 (0.00s) --- PASS: TestIntervalOverlap/1:6_vs_3:4 (0.00s) PASS -ok github.com/hashicorp/consul/sdk/freeport 0.007s +ok github.com/hashicorp/consul/sdk/freeport 0.015s ? github.com/hashicorp/consul/sdk/testutil [no test files] === RUN TestRetryer retry_test.go:12: DM-skipped --- SKIP: TestRetryer (0.00s) PASS -ok github.com/hashicorp/consul/sdk/testutil/retry 0.004s +ok github.com/hashicorp/consul/sdk/testutil/retry 0.044s ? github.com/hashicorp/consul/sentinel [no test files] ? github.com/hashicorp/consul/service_os [no test files] === RUN TestArchive @@ -93406,110 +93530,110 @@ === RUN TestArchive_GoodData --- PASS: TestArchive_GoodData (0.00s) === RUN TestArchive_BadData ---- PASS: TestArchive_BadData (0.00s) +--- PASS: TestArchive_BadData (0.01s) === RUN TestArchive_hashList --- PASS: TestArchive_hashList (0.00s) === RUN TestSnapshot -2022-11-07T22:47:06.786Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4 Address:64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4}]" -2022-11-07T22:47:06.786Z [INFO] raft: entering follower state: follower="Node at 64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4 [Follower]" leader= -2022-11-07T22:47:07.989Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= -2022-11-07T22:47:07.989Z [INFO] raft: entering candidate state: node="Node at 64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4 [Candidate]" term=2 -2022-11-07T22:47:07.990Z [DEBUG] raft: votes: needed=1 -2022-11-07T22:47:07.990Z [DEBUG] raft: vote granted: from=server-64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4 term=2 tally=1 -2022-11-07T22:47:07.990Z [INFO] raft: election won: tally=1 -2022-11-07T22:47:07.990Z [INFO] raft: entering leader state: leader="Node at 64f3e3a1-9ccf-7d8f-673a-1ae46c8c8cb4 [Leader]" -2022-11-07T22:47:08.902Z [INFO] raft: starting snapshot up to: index=65538 -2022-11-07T22:47:08.902Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot-snapshot535313534/before/snapshots/2-65538-1667861228902.tmp -2022-11-07T22:47:08.998Z [INFO] raft: compacting logs: from=1 to=55298 -2022-11-07T22:47:09.004Z [INFO] raft: snapshot complete up to: index=65538 -2022-11-07T22:47:09.844Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9880b164-e401-0d1b-2f25-717b4f88f289 Address:9880b164-e401-0d1b-2f25-717b4f88f289}]" -2022-11-07T22:47:09.844Z [INFO] raft: entering follower state: follower="Node at 9880b164-e401-0d1b-2f25-717b4f88f289 [Follower]" leader= -2022-11-07T22:47:11.175Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= -2022-11-07T22:47:11.175Z [INFO] raft: entering candidate state: node="Node at 9880b164-e401-0d1b-2f25-717b4f88f289 [Candidate]" term=2 -2022-11-07T22:47:11.175Z [DEBUG] raft: votes: needed=1 -2022-11-07T22:47:11.175Z [DEBUG] raft: vote granted: from=server-9880b164-e401-0d1b-2f25-717b4f88f289 term=2 tally=1 -2022-11-07T22:47:11.175Z [INFO] raft: election won: tally=1 -2022-11-07T22:47:11.175Z [INFO] raft: entering leader state: leader="Node at 9880b164-e401-0d1b-2f25-717b4f88f289 [Leader]" -2022-11-07T22:47:11.339Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot-snapshot535313534/after/snapshots/2-65539-1667861231339.tmp -2022-11-07T22:47:11.411Z [INFO] raft: copied to local snapshot: bytes=16973829 -2022-11-07T22:47:11.471Z [INFO] raft: restored user snapshot: index=1 ---- PASS: TestSnapshot (4.70s) +2021-10-05T16:34:51.138Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-202a10d1-926a-cb64-bb95-be0ca00d1c72 Address:202a10d1-926a-cb64-bb95-be0ca00d1c72}]" +2021-10-05T16:34:51.140Z [INFO] raft: entering follower state: follower="Node at 202a10d1-926a-cb64-bb95-be0ca00d1c72 [Follower]" leader= +2021-10-05T16:34:52.548Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= +2021-10-05T16:34:52.548Z [INFO] raft: entering candidate state: node="Node at 202a10d1-926a-cb64-bb95-be0ca00d1c72 [Candidate]" term=2 +2021-10-05T16:34:52.548Z [DEBUG] raft: votes: needed=1 +2021-10-05T16:34:52.548Z [DEBUG] raft: vote granted: from=server-202a10d1-926a-cb64-bb95-be0ca00d1c72 term=2 tally=1 +2021-10-05T16:34:52.548Z [INFO] raft: election won: tally=1 +2021-10-05T16:34:52.548Z [INFO] raft: entering leader state: leader="Node at 202a10d1-926a-cb64-bb95-be0ca00d1c72 [Leader]" +2021-10-05T16:34:55.180Z [INFO] raft: starting snapshot up to: index=65538 +2021-10-05T16:34:55.180Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot-snapshot290043556/before/snapshots/2-65538-1633451695180.tmp +2021-10-05T16:34:55.364Z [INFO] raft: compacting logs: from=1 to=55298 +2021-10-05T16:34:55.370Z [INFO] raft: snapshot complete up to: index=65538 +2021-10-05T16:34:57.055Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-795d65c7-0739-c41b-9bb9-45c1b748ca8d Address:795d65c7-0739-c41b-9bb9-45c1b748ca8d}]" +2021-10-05T16:34:57.055Z [INFO] raft: entering follower state: follower="Node at 795d65c7-0739-c41b-9bb9-45c1b748ca8d [Follower]" leader= +2021-10-05T16:34:58.908Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= +2021-10-05T16:34:58.908Z [INFO] raft: entering candidate state: node="Node at 795d65c7-0739-c41b-9bb9-45c1b748ca8d [Candidate]" term=2 +2021-10-05T16:34:58.908Z [DEBUG] raft: votes: needed=1 +2021-10-05T16:34:58.908Z [DEBUG] raft: vote granted: from=server-795d65c7-0739-c41b-9bb9-45c1b748ca8d term=2 tally=1 +2021-10-05T16:34:58.908Z [INFO] raft: election won: tally=1 +2021-10-05T16:34:58.908Z [INFO] raft: entering leader state: leader="Node at 795d65c7-0739-c41b-9bb9-45c1b748ca8d [Leader]" +2021-10-05T16:34:59.222Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot-snapshot290043556/after/snapshots/2-65539-1633451699222.tmp +2021-10-05T16:34:59.343Z [INFO] raft: copied to local snapshot: bytes=16973829 +2021-10-05T16:34:59.461Z [INFO] raft: restored user snapshot: index=1 +--- PASS: TestSnapshot (8.36s) === RUN TestSnapshot_Nil --- PASS: TestSnapshot_Nil (0.00s) === RUN TestSnapshot_BadVerify --- PASS: TestSnapshot_BadVerify (0.00s) === RUN TestSnapshot_TruncatedVerify -2022-11-07T22:47:11.486Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ba8703f-1739-91a7-aab6-0444ef6cdc8d Address:2ba8703f-1739-91a7-aab6-0444ef6cdc8d}]" -2022-11-07T22:47:11.486Z [INFO] raft: entering follower state: follower="Node at 2ba8703f-1739-91a7-aab6-0444ef6cdc8d [Follower]" leader= -2022-11-07T22:47:12.538Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= -2022-11-07T22:47:12.538Z [INFO] raft: entering candidate state: node="Node at 2ba8703f-1739-91a7-aab6-0444ef6cdc8d [Candidate]" term=2 -2022-11-07T22:47:12.538Z [DEBUG] raft: votes: needed=1 -2022-11-07T22:47:12.538Z [DEBUG] raft: vote granted: from=server-2ba8703f-1739-91a7-aab6-0444ef6cdc8d term=2 tally=1 -2022-11-07T22:47:12.538Z [INFO] raft: election won: tally=1 -2022-11-07T22:47:12.538Z [INFO] raft: entering leader state: leader="Node at 2ba8703f-1739-91a7-aab6-0444ef6cdc8d [Leader]" -2022-11-07T22:47:13.623Z [INFO] raft: starting snapshot up to: index=65538 -2022-11-07T22:47:13.623Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_TruncatedVerify-snapshot711290943/before/snapshots/2-65538-1667861233623.tmp -2022-11-07T22:47:13.709Z [INFO] raft: compacting logs: from=1 to=55298 -2022-11-07T22:47:13.714Z [INFO] raft: snapshot complete up to: index=65538 +2021-10-05T16:34:59.494Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e8f0dc-0e38-d273-0218-406de4708b5a Address:e3e8f0dc-0e38-d273-0218-406de4708b5a}]" +2021-10-05T16:34:59.494Z [INFO] raft: entering follower state: follower="Node at e3e8f0dc-0e38-d273-0218-406de4708b5a [Follower]" leader= +2021-10-05T16:35:00.884Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= +2021-10-05T16:35:00.884Z [INFO] raft: entering candidate state: node="Node at e3e8f0dc-0e38-d273-0218-406de4708b5a [Candidate]" term=2 +2021-10-05T16:35:00.884Z [DEBUG] raft: votes: needed=1 +2021-10-05T16:35:00.884Z [DEBUG] raft: vote granted: from=server-e3e8f0dc-0e38-d273-0218-406de4708b5a term=2 tally=1 +2021-10-05T16:35:00.884Z [INFO] raft: election won: tally=1 +2021-10-05T16:35:00.884Z [INFO] raft: entering leader state: leader="Node at e3e8f0dc-0e38-d273-0218-406de4708b5a [Leader]" +2021-10-05T16:35:03.950Z [INFO] raft: starting snapshot up to: index=65538 +2021-10-05T16:35:03.950Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_TruncatedVerify-snapshot334375517/before/snapshots/2-65538-1633451703950.tmp +2021-10-05T16:35:04.099Z [INFO] raft: compacting logs: from=1 to=55298 +2021-10-05T16:35:04.104Z [INFO] raft: snapshot complete up to: index=65538 === RUN TestSnapshot_TruncatedVerify/truncate_200_bytes_from_end === RUN TestSnapshot_TruncatedVerify/truncate_16_bytes_from_end === RUN TestSnapshot_TruncatedVerify/truncate_8_bytes_from_end === RUN TestSnapshot_TruncatedVerify/truncate_4_bytes_from_end === RUN TestSnapshot_TruncatedVerify/truncate_2_bytes_from_end === RUN TestSnapshot_TruncatedVerify/truncate_1_bytes_from_end ---- PASS: TestSnapshot_TruncatedVerify (3.69s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_200_bytes_from_end (0.12s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_16_bytes_from_end (0.11s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_8_bytes_from_end (0.11s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_4_bytes_from_end (0.12s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_2_bytes_from_end (0.12s) - --- PASS: TestSnapshot_TruncatedVerify/truncate_1_bytes_from_end (0.11s) +--- PASS: TestSnapshot_TruncatedVerify (7.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_200_bytes_from_end (0.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_16_bytes_from_end (0.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_8_bytes_from_end (0.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_4_bytes_from_end (0.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_2_bytes_from_end (0.20s) + --- PASS: TestSnapshot_TruncatedVerify/truncate_1_bytes_from_end (0.20s) === RUN TestSnapshot_BadRestore -2022-11-07T22:47:15.177Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d49fb473-bbe8-7daa-e3c8-ef1598ba972f Address:d49fb473-bbe8-7daa-e3c8-ef1598ba972f}]" -2022-11-07T22:47:15.177Z [INFO] raft: entering follower state: follower="Node at d49fb473-bbe8-7daa-e3c8-ef1598ba972f [Follower]" leader= -2022-11-07T22:47:16.448Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= -2022-11-07T22:47:16.448Z [INFO] raft: entering candidate state: node="Node at d49fb473-bbe8-7daa-e3c8-ef1598ba972f [Candidate]" term=2 -2022-11-07T22:47:16.448Z [DEBUG] raft: votes: needed=1 -2022-11-07T22:47:16.448Z [DEBUG] raft: vote granted: from=server-d49fb473-bbe8-7daa-e3c8-ef1598ba972f term=2 tally=1 -2022-11-07T22:47:16.448Z [INFO] raft: election won: tally=1 -2022-11-07T22:47:16.448Z [INFO] raft: entering leader state: leader="Node at d49fb473-bbe8-7daa-e3c8-ef1598ba972f [Leader]" -2022-11-07T22:47:16.679Z [INFO] raft: starting snapshot up to: index=16386 -2022-11-07T22:47:16.679Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_BadRestore-snapshot824257225/before/snapshots/2-16386-1667861236679.tmp -2022-11-07T22:47:16.716Z [INFO] raft: compacting logs: from=1 to=6146 -2022-11-07T22:47:16.716Z [INFO] raft: snapshot complete up to: index=16386 -2022-11-07T22:47:16.890Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b Address:a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b}]" -2022-11-07T22:47:16.890Z [INFO] raft: entering follower state: follower="Node at a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b [Follower]" leader= -2022-11-07T22:47:18.197Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= -2022-11-07T22:47:18.197Z [INFO] raft: entering candidate state: node="Node at a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b [Candidate]" term=2 -2022-11-07T22:47:18.197Z [DEBUG] raft: votes: needed=1 -2022-11-07T22:47:18.197Z [DEBUG] raft: vote granted: from=server-a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b term=2 tally=1 -2022-11-07T22:47:18.197Z [INFO] raft: election won: tally=1 -2022-11-07T22:47:18.197Z [INFO] raft: entering leader state: leader="Node at a2de5cc7-8d2a-b6f7-4054-f3b0c204e89b [Leader]" -2022-11-07T22:47:18.198Z [ERROR] TestSnapshot_BadRestore: Failed to close snapshot decompressor: error="unexpected EOF" ---- PASS: TestSnapshot_BadRestore (3.02s) +2021-10-05T16:35:06.696Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6153b2a5-8642-cd42-a490-a550ba4dc7ec Address:6153b2a5-8642-cd42-a490-a550ba4dc7ec}]" +2021-10-05T16:35:06.696Z [INFO] raft: entering follower state: follower="Node at 6153b2a5-8642-cd42-a490-a550ba4dc7ec [Follower]" leader= +2021-10-05T16:35:08.584Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= +2021-10-05T16:35:08.584Z [INFO] raft: entering candidate state: node="Node at 6153b2a5-8642-cd42-a490-a550ba4dc7ec [Candidate]" term=2 +2021-10-05T16:35:08.584Z [DEBUG] raft: votes: needed=1 +2021-10-05T16:35:08.584Z [DEBUG] raft: vote granted: from=server-6153b2a5-8642-cd42-a490-a550ba4dc7ec term=2 tally=1 +2021-10-05T16:35:08.584Z [INFO] raft: election won: tally=1 +2021-10-05T16:35:08.584Z [INFO] raft: entering leader state: leader="Node at 6153b2a5-8642-cd42-a490-a550ba4dc7ec [Leader]" +2021-10-05T16:35:09.149Z [INFO] raft: starting snapshot up to: index=16386 +2021-10-05T16:35:09.149Z [INFO] snapshot: creating new snapshot: path=/tmp/consul-test/TestSnapshot_BadRestore-snapshot487753879/before/snapshots/2-16386-1633451709149.tmp +2021-10-05T16:35:09.219Z [INFO] raft: compacting logs: from=1 to=6146 +2021-10-05T16:35:09.220Z [INFO] raft: snapshot complete up to: index=16386 +2021-10-05T16:35:09.656Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a14fff90-b72b-ee04-ed13-4c35565c3e02 Address:a14fff90-b72b-ee04-ed13-4c35565c3e02}]" +2021-10-05T16:35:09.656Z [INFO] raft: entering follower state: follower="Node at a14fff90-b72b-ee04-ed13-4c35565c3e02 [Follower]" leader= +2021-10-05T16:35:11.300Z [WARN] raft: heartbeat timeout reached, starting election: last-leader= +2021-10-05T16:35:11.300Z [INFO] raft: entering candidate state: node="Node at a14fff90-b72b-ee04-ed13-4c35565c3e02 [Candidate]" term=2 +2021-10-05T16:35:11.300Z [DEBUG] raft: votes: needed=1 +2021-10-05T16:35:11.300Z [DEBUG] raft: vote granted: from=server-a14fff90-b72b-ee04-ed13-4c35565c3e02 term=2 tally=1 +2021-10-05T16:35:11.300Z [INFO] raft: election won: tally=1 +2021-10-05T16:35:11.300Z [INFO] raft: entering leader state: leader="Node at a14fff90-b72b-ee04-ed13-4c35565c3e02 [Leader]" +2021-10-05T16:35:11.302Z [ERROR] TestSnapshot_BadRestore: Failed to close snapshot decompressor: error="unexpected EOF" +--- PASS: TestSnapshot_BadRestore (4.61s) PASS -ok github.com/hashicorp/consul/snapshot 11.430s +ok github.com/hashicorp/consul/snapshot 20.212s ? github.com/hashicorp/consul/testrpc [no test files] === RUN TestConfigurator_outgoingWrapper_OK ---- PASS: TestConfigurator_outgoingWrapper_OK (0.01s) +--- PASS: TestConfigurator_outgoingWrapper_OK (0.03s) === RUN TestConfigurator_outgoingWrapper_noverify_OK ---- PASS: TestConfigurator_outgoingWrapper_noverify_OK (0.01s) +--- PASS: TestConfigurator_outgoingWrapper_noverify_OK (0.02s) === RUN TestConfigurator_outgoingWrapper_BadDC ---- PASS: TestConfigurator_outgoingWrapper_BadDC (0.01s) +--- PASS: TestConfigurator_outgoingWrapper_BadDC (0.02s) === RUN TestConfigurator_outgoingWrapper_BadCert ---- PASS: TestConfigurator_outgoingWrapper_BadCert (0.01s) +--- PASS: TestConfigurator_outgoingWrapper_BadCert (0.02s) === RUN TestConfigurator_outgoingWrapperALPN_OK ---- PASS: TestConfigurator_outgoingWrapperALPN_OK (0.02s) +--- PASS: TestConfigurator_outgoingWrapperALPN_OK (0.04s) === RUN TestConfigurator_outgoingWrapperALPN_serverHasNoNodeNameInSAN ---- PASS: TestConfigurator_outgoingWrapperALPN_serverHasNoNodeNameInSAN (0.01s) +--- PASS: TestConfigurator_outgoingWrapperALPN_serverHasNoNodeNameInSAN (0.02s) === RUN TestConfigurator_outgoingWrapperALPN_BadDC ---- PASS: TestConfigurator_outgoingWrapperALPN_BadDC (0.01s) +--- PASS: TestConfigurator_outgoingWrapperALPN_BadDC (0.02s) === RUN TestConfigurator_outgoingWrapperALPN_BadCert ---- PASS: TestConfigurator_outgoingWrapperALPN_BadCert (0.01s) +--- PASS: TestConfigurator_outgoingWrapperALPN_BadCert (0.02s) === RUN TestConfigurator_wrapTLS_OK ---- PASS: TestConfigurator_wrapTLS_OK (0.01s) +--- PASS: TestConfigurator_wrapTLS_OK (0.02s) === RUN TestConfigurator_wrapTLS_BadCert ---- PASS: TestConfigurator_wrapTLS_BadCert (0.01s) +--- PASS: TestConfigurator_wrapTLS_BadCert (0.03s) === RUN TestConfig_ParseCiphers --- PASS: TestConfig_ParseCiphers (0.00s) === RUN TestConfigurator_loadKeyPair @@ -93517,10 +93641,10 @@ === RUN TestConfig_SpecifyDC --- PASS: TestConfig_SpecifyDC (0.00s) === RUN TestConfigurator_NewConfigurator -2022-11-07T22:47:06.955Z [TRACE] TestConfigurator_NewConfigurator.tlsutil: Update: version=1 +2021-10-05T16:34:51.347Z [TRACE] TestConfigurator_NewConfigurator.tlsutil: Update: version=1 --- PASS: TestConfigurator_NewConfigurator (0.00s) === RUN TestConfigurator_ErrorPropagation ---- PASS: TestConfigurator_ErrorPropagation (0.00s) +--- PASS: TestConfigurator_ErrorPropagation (0.01s) === RUN TestConfigurator_CommonTLSConfigServerNameNodeName --- PASS: TestConfigurator_CommonTLSConfigServerNameNodeName (0.00s) === RUN TestConfigurator_LoadCAs @@ -93607,7 +93731,7 @@ --- PASS: TestGeneratePrivateKey (0.00s) --- PASS: TestGenerateCA (0.00s) PASS -ok github.com/hashicorp/consul/tlsutil 0.160s +ok github.com/hashicorp/consul/tlsutil 0.280s ? github.com/hashicorp/consul/types [no test files] ? github.com/hashicorp/consul/version [no test files] make[1]: Leaving directory '/build/consul-1.8.7+dfsg1' @@ -93665,12 +93789,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: not including original source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/12155/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/11550 and its subdirectories -I: Current time: Mon Nov 7 10:48:01 -12 2022 -I: pbuilder-time-stamp: 1667861281 +I: removing directory /srv/workspace/pbuilder/12155 and its subdirectories +I: Current time: Wed Oct 6 06:36:56 +14 2021 +I: pbuilder-time-stamp: 1633451816